Programming FPGA chip

First I wouldn't advise to make a custom PCB for your project at this stage. Designing an FPGA board can take some time, and you're not even sure yet that it will solve your problem.

Then maybe some signal processing experts could give more advice on this, but in my opinion the algorithms that you plan to implement will require more CPU power than a soft core in an FPGA can provide. It doesn't look easy to implement directly in hardware either.

I would rather suggest to use more specialized hardware, such as a DSP or a faster CPU. You can have a look at Texas Instruments' DSP as an example. They provide tools to convert Matlab code and run it directly on the DSP.

You can keep the FPGA board to handle the OFDM transceiving, as you already have the VHDL code, or also use the DSP for that.

Another alternative is to keep the Matlab and C/C++ functions on a PC, and only use the FPGA board for the OFDM and video interfaces.