Ok if I understood correctly it sounds like you might be thinking these are the same thing (they are not).
Custom instructions have hardware with ports like clk, dataa, dataab, etc....... and they get mapped to an instruction in the assembly language of the NIOS itself (it becomes a part of the ALU reallly).
The custom hardware interface is a core that makes communication from the avalon bus to the outside world (outside of the NIOS) much easier using address/data and a few strobes.
If what you need to do is something like 1/2 inputs and 1 output back to the software then custom instructions are for you. However if you have some processing to do outside the NIOS on more then 2 inputs then I recommend the custom hardware interface (and use a done signal to find out when it is complete)