Source: Original. Similar to a problem at the U. of Miss' Geometry Gambit Contest.
1. The diagonal is actually the hypotenuse of a right triangle with legs A+B and B. The part that is in square A is a similar right triangle, with legs A and AB/(A+B), so the area we want is (A^2)B/(2(A+B)). 2. A=3, B=6 and A=6, B=3. The restriction is the A and B must be such that A+B's prime factors are all common factors of A and B. For this area to be an integer, we need (A^2)B to be divisible by 2(A+B). The only prime factors which B is going to have in common with A+B are ones that are also factors of A. So we need to choose A and B such that A+B's only prime factors are common factors of A and B, and since A+B is bigger than either A or B, that means A+B must have at least one repeated prime factor. So 4, 8, 9, 12, 16, 18, 20, 24, 25, 27, 28, 32, 36, ... are candidates for A+B. For each candidate, we look for partitions into two integers which both contain all A+B's prime factors. Since we want A < B or A > B, we ignore the case of A=B as with 4=2+2. So 8=2+6 is the first real possibility. If A=2, B=6, we have area 24/16, and if A=6, B=2, we have area 72/16, neither an integer. Next is 9=3+6. A=3, B=6 gives area 54/18 = 3, and A=6, B=3 gives area 108/18 = 6. These are the answers we want for this part. 3. [Upon comment from one reader, Joseph asked me to remove his answer due to errors. This is still in need of a solution. - KD]