Accelerating a C Function with User Logic

The rule of thumb I have used is to use custom instructions if the operands and results are going to be in NIOS registers anyway (because other software is going to create or reference them), and to use Avalon-MM or Avalon-ST based components if the operands and results are going to/from memory or other components with no other software processing.

You can also use custom instructions as a prototyping tool since it's a lot easier to use the Eclipse debugger and trap inputs/outputs from your buggy logic than it is to do the same with SignalTap. The issue with staying with custom instructions long term is that although your core logic may complete execution in 1 clock cycle, the NIOS itself may take many clock cycles to issue the instruction (especially if there are any load/store operations surrounding it).

For an image processing algorithm, you can develop your kernel with custom instructions and then later migrate to a non-software based component; probably aiming for structure similar to the components in the Altera Video and Image Processing Suite (Avalon-ST for image data, Avalon-MM for control registers).