if your single clean tone is not having zero error then there should be something to explain. Either phase of Tx tone is not same at the Rx nco or your branch filters are doing something not very nice such as not cutting off well or cutting off your own wanted signal. You can check spectrum at each stage using sptool or any fft based approach.
if your external nco has phase control input that controls phase only and doesn't change its running frequency (apart from transient effect of phase change) then you complete the loop in the same way using the filtered error and scale it to match the phase register range but without the need to revert back to control frequency.
In any case the error should be scaled correctly. The principle of the Costas phase error (real * imag) is that error = -1/4*sin(2*phase diff).
in this design we are not looking at phase diff directly but its -sine which goes down then up then down depending as angle diff goes up... in other words the error could change sense if it overshoots by 1/4 cycle. The alternative is to get phase diff itself instead of sin(e.g. using cordic) plus phase unwrapping. This way you make sure the error sense is fixed either up or down. This could make control easier?
notice also that sin function for 1/4 cycle is nonlinear and horribly so at 90 degree section, only the first half of it i.e 45 degrees is linear enough.
so in short, keep using the error as it is (sin) and modify further if necessary. Good luck.
Honored Contributor
