How to simulate serdes in modelsim? Which files must I compile?

1 - Look at the "EDA" tab of your alt2gxb megawizard. It will tell you exactly what you need for simulation.

2 - I believe you need to have the megawizard generate a simulation model for you (a .vo file).

So, are you using Arria GX or Stratix II GX?

Jake