Costas Loop oscillates in ModelSim while locks in MATLAB!

Data aided lock is common practice and you can add slicer result to the loop to check that +/-1 are at equal distance from zero indicating good phase. I believe you focus on modelsim as it has proper loop and delays. If you want to revert to Matlab you better model nco as per rtl exactly so you have control over phase.

I have posted example nco model on DSPrelated.com long time ago but I now know it got bug in negative frequency when phase accum goes backwards. This cn be repaired or use it for positive freq only and if you need to negate it do that after nco e.g. by inverting sine.