Mulit-processor and shared memory

I have designed a multi-processor design with a on board shared memory between them (32 bits wide). Initially both CPUs were are able to access the memory successfully. However after more development and investigation I found that a simple memory test failed.

I stopped all accesses to the shared memory from one CPU. Then I used the 2nd CPU to write and then read data to and from the Shared Memory. When I did 32 bit accesses there was data read at consistent addresses where the bottom bits were stuck at 0. When I did 16 bit accesses then only the 2nd to last bit was stuck at 0. When I did 8 bit accesses, there were no errors at all.

I have both CPU with the same arbitration and I am not using a MUTEX core ! Any ideas ????