The easiest fix for you is to first make sure your fft scaling is ok. Not just the use of exponent but your truncation afterwards. If this is ok then your fft is ok and I wouldn't worry about the test.
If you want your test to do better then increase fft resolution.
If your audio is received upsampled then you may decimate it down to minimum (for test purposes only). Thats what I labelled speed by mistake. Remember fft resolution doesn't have to be equal to number of points per sinusoid cycle, that ruins the test unless you get the phase exactly same per each block. In short make your fft 2048 and repeat the test. Try also pass the audio to both real and imaginary so that the phase change is on both channels(again for test purpose only).