FFT IP - correct magnitude calculation

It may be a stupid suggestion, but did you have a look at the FFT input in signaltap too, and check that the signal was correct, and did change it's amplitude as it should?

AFAIK there is no automatic gain adjustment in the FFT IP, but you could have one in your ADC chain, or there could be a problem with the signal acquisition (saturation, or bad handling of the sign bit) and as a result the amplitude of the analog input signal could have little influence on the FFT result.

Did you also have a look at the FFT output before your module that calculates the magnitude?