Questions about DSPBuilder

I'm not using DSPBuilder and don't know, what "can not convert to integer" means in this regard. I assume, that some kind of coefficient scaling is performed by any FIR design tool. If I use the Quartus FIR compiler, it would also have issues to realize a certain filter characteristics with insufficient coefficient length. You didn't actually tell the used coefficient length or internal data length, just I/O data length of 8 bit.

design a bigger filter could be also understood as increasing data and coefficient length, which seems to be necessary in this case. But you have to know your specification, how far 8-bit data length is meaningful for your application and about the intended filter characteristics.