How to read and write value in two different systems?

Of course, shared memory, accompanied by a hardware Mutex, is a perfectly legal design. It can be done as (1) a real dual ported RAM, accessible simultaneously by both CPUs or (2) simply by using the Avalon bus to handle mutual accesses. (2) might slow down both systems, as any access to the RAM needs to wait until an access from the other CPU is over. This is especially bad if that RAM is not dedicated to the inter-CPU communication but holds other data or even code, as well.

Regarding designing a FiFo by means of an FPAG, I suppose there is lots of literature in the Internet.

Here supposedly a pair of FiFos, one in either direction, is the most useful design.

Each CPU sees it's own read port (reading the incoming FiFo), write port (writing the outgoing FiFo) and flag port. The flags being (e.g.) "outgoing FiFo full" and "incoming FiFo empty". A "clear outgoing FiFo" command port might be useful, too. (See the datasheet of some hardware FiFo.)

Additionally, for speed, it might be useful to have the FiFos create interrupts towards the appropriate CPU, if the incoming FiFo gets "not empty" and/or if the outgoing FiFo gets "not full" (or e.g. "not half full").

The CPU interface might be similar to a hardware UART with a built-in FiFo (e.g. 16550).

-Michael