Costas Loop oscillates in ModelSim while locks in MATLAB!

as phase gets corrected then the error tends towards zero and this depends on how you have scaled it so the nudge effect gets less and less. Every nudge should reduce the amplitude of oscillation.

If it overshoots then the sense of error reverses and so on you have a loop error that pushes one way and pulls the other way from time to time (this is acceptable withing limits of specified jitter). In real world the loop keeps track of tx/rx oscillators freq and phase drift.

If phase is correct to begin with then error starts as zero and should stay stay zero. I notice that with single tone bu haven't tried more.