Custom Instruction writing to FIFO

You don't really need to "hack" anything. Just use Quartus II to parameterize the SCFIFO logic in whatever language you want, create a custom instruction wrapper that instantiates SCFIFO, map your ports any way you like and off you go (to component editor). I have never used custom instruction mapped FIFOs before but this is how I'd do it (if I had to):

1) Instantiate SCFIFO in my custom instruction RTL

2) Connect each side of the FIFO to a custom instruction interface (remember you need multiple interfaces since you are connecting multiple Nios II cores)

3) Map a secondary set of signals like "empty" and "full" to do some handshaking between CPUs

The reason for# 3 is if Nios A and Nios B are talking to each other through FIFOs, and Nios B goes to read from an empty FIFO then you will have to take care of that. You would either need Nios B to read from a secondary instruction that states there is data in the FIFO or have that FIFO instruction hold Nios B in a wait state by not asserting the "done" bit (potential for deadlock if you are not careful). With all this control I don't think you are buying much using this FIFO and custom instruction method.

This begs the question of why you want to implement this sharing method using custom instruction mapped FIFOs? There is a mailbox component in SOPC Builder for the purpose of passing messages back and forth between processors. If you want to use low latency memory to store messages then I recommend using tightly coupled memories.