I believe custom instructions have its own interface: a, b, c, read_a, read_b, write_c, ... etc... I haven't tried this out myself personally, but if you want to use your DSP Builder component as custom instructions in Nios II, then I believe you would have to:
1. remove the Avalon MM interface blocks
2. add in the correct CI interfaces (a, b, c, ... etc.)
3. somehow import that into Nios II... this is the part that I am unsure about.... theoretically, you simply need to import the VHDL or Verilog file to component editor and make sure all ports are set to CI interfaces... however, not sure how this would work with DSPB generated components (since the component is separated into multiple VHDL files....) there is a chance that you might need to file a service request with Altera to see whether it would work or not...
Best of luck!