Source: Original.
From Jayavel Sounderpandian:
I shall first show that the largest pentagon needs to have one vertex (call it A) on the diagonal of the square and the other four on the four sides of the square. In this orientation, the normals at the four points of contact indicate that the pentagon cannot be translated or rotated. Thus in the neighborhood of this orientation, the pentagon is largest. Since the pentagon.s two diagonals that touch the square.s sides are at 9 degrees with the normal to the sides, the orientation must be changed by at least (2 x 9 = ) 18 degrees for another competitive solution. If it is rotated by 18 degrees a vertex adjacent to A will fall on the square.s diagonal giving us the same orientation as the first.
Given this result, the calculation of the side of the largest pentagon (call it x) is mechanical. We note that the pentagon.s side opposite vertex A makes a 45 degree angle with the square.s side, and the side adjacent to it makes a 27 degree angle. Thus
x Cos (45 degrees) + x Cos (27 degrees) = 1.
This yields x = 0.625738.
Thus the largest pentagon will have a side of length 0.625738.
Let ABCDE the regular pentagon of side a and PQRS the square of side 1. There are two ways to fit a regular pentagon inside the square: 1) B on QR, C and D on RS, E on SP such as A remains inside the square. We have the relation: a + 2*a/cos(2*pi/5) = 1 . Therefore a = (sqrt(5) - 1)/2 =0.618035... 2) B on PQ, C on QR, D on RS, E on SP such as CD is perpendicular to the diagonal PR of the square and A is on PR. We have the relation: a/sqrt(2) + a*cos(3*pi/20) = 1. Therefore a = 1/(sqrt(2)/2 + cos(3*pi/20) with cos(3*pi/20) = sqrt(1/2+sqrt(10-2*sqrt(5))/8) So a = 0.62573786... This the largest regular pentalon which can fit within the square. Indeed, if we take any C on QR and D on RS such as CD=a and angle(CDR)=alpha, we can easily demonstrate that the largest pentagon is obtained with alpha=pi/4, that is to say the second configuration above described.