Hello,
I had a similar problem and was able to solve it.
(I needed to initialize a dual port ram block with a hex file.)
What worked was to put my IP instantiation and my .HEX file both in the main quartus project directory and then add the .vhd and the .hex (both as a reference) to my modelsim project. If the files were in different directories, quartus did ok but modelsim was unable to compile the .hex.
If I use modelsim I usually make a new project in modelsim by hand and add the files by hand. Doing it step by step can be more reliable than the "big bang approach".
Best Regards,
Johi.