Costas Loop oscillates in ModelSim while locks in MATLAB!

That is good path since modelling in matlab for feedback loops needs to be dead accurate match of rtl. Yes your analysis is right if you mean your matlab model which I haven't seen its final format. The nco statement has control over phase just like rtl and this phase is used to control frequency by different jumps over sin/cos functions (luts) so it will have very high floating resolution. but you need to write as y = exp(j*2*pi*linspace(0+offset:N-1+offset,N)*.1); in order to step phase in fractions.