NIOS II Multi-Processor System's On-chip Memory

You can have a fixed location in memory that you share between all your processors for message passing. You will also need a hardware mutex to ensure that two processors won't try to access it simultaneously. I think that Altera has a hardware mutex core designed for multiprocessor communication.