Costas Loop oscillates in ModelSim while locks in MATLAB!

Yes I used same error as yours (real * imag)

Any filter for high sideband will do as long as it does the job but I kept loop at Rs*2 rate (not at symbol rate).

Updating nco word conditionaly is like this: use sample counter e.g. 0~9 (not clock counter)

if counter = 9 update with error else assign initial value (do not latch error update as it will lose frequency lock)