--- Quote Start ---
I can notice that I component has a DC offset... how can I get rid of it inside for loop? And do I have to scale to unity as you did?
--- Quote End ---
You will get dc on both I and Q when it works. So if set rf freq and nco to target dc you get it. Depending on rf phase I or Q could be any pair of constants.
you don't have dc in rf signal because it is man made clean tone. In real rf you should remove dc (or else it moves to nco frequency where you can filter it.
if you get error changing sense you got it.
next how to be gentle with nco:
the error is basically mad. you need to filter it through IIR and add it proportional term (error x 8 or x 100...) and even a second accum if it helps.
finally you scale the error (division by 2^n i.e. later shift in hardware). The scaling is also critical since in your model your nco expects +/- 0 ~ 0.5 range and very gently.
run your model in feedback loop, set addition of error to zero to kill the loop effect. Then observe the error, the filtered error and final error and make sure it is gentle and with above nco range.
In hardware the tuning word will have to be scaled and things will need proper scaling.