From Custom Instruction to Custom Peripheral

If your hardware takes longer to process the data than Nios II can input the data then perhaps what you could do is put a FIFO in your custom instruction so that your code just keeps calling the custom instruction shoving operands into it without reading the results. Then you can start reading the results back out. At this point though I would have switched it over to be a memory mapped component.

In your HDL at the top one suggestion I have is break out your registers into separate always blocks. Anything that doesn't need to be a register I recommend coding as a wire using an assign statement. The way your HDL is coded currently it looks like it'll take 5 clock cycles to complete one result. With some buffering you can get it to perform a calculation every clock cycle (keep stuffing the result into the FIFO to be read later). Also assuming your logic is functionally correct you are practically 90% of the way to having a streaming component.