Where to start and what to use

Might be.

You are on the right track.

While DSP Builder simulation is quite slow, the resulting FPGA code may be quite fast. (Assuming you are using the DSP Builder Altera Blockset to create your design in Simulink.)

The Convolution may be a bit tough, or memory intensive, and may be better suited to NIOS II code (at this time). Hrd to say.

Press on, and keep us all posted.