You will need an indexed feedback loop that acts on one sample at a time fully modelling from input to output. The filtered error together with any proportional element have to be taken care of.
I have attached some old work. This was my initial attempt for qpsk/16QAM (I have lost everything else afterwards). It may not be that correct but should give some idea.
It starts with modelling TX side to generate a signal (you may not need that) but focus on the loop.
At this point you can try Modelsim before Matlab as it easier there. If you get in difficulty then you better model it in matlab.