Trying to test Filter using NCO Megafunction and Signal Tap Analyzer?

You basically need to learn how to do a testbench. For your filter, the testbench itself is very simple if you know how to read text files.

You need to start from Matlab. Generate your random data and convert it to integers then into text file. pass your input to "filter" function using your fixed point coeffs and round up result into integers as per design then convert to text file (thus you have input file and output file).

Go to ModelSim, run your testench to read random data file and pass it to filter by assignment, read output file into your design and compare.

There will be some delay depending on reset but you should get bit accurate result. If there is rounding error or scale issue then we can discuss that.

Note that you don't need to verify filter response in design as long as matlab can be used to study it. All you need is verify that your implementation matches that of matlab.

Normally you should use ModelSim before actual hardware testing.

In hardware stage, you can design a synthesisable testbench e.g. putting your above input /output ref data in RAM then testing through a flag passed to led or signaltap. Or if you have software control you may read files from PC and capture to PC then compare in Matlab... lots of ways...