Using DMA for lcd screen refresh

I have done this before, successfully, but now we are having trouble with it, and I thought I would see if I am missing something.

Our system uses SDRAM for NIOS code/data and I am also using the Altera DMAC to DMA out of that SDRAM for LCD display refresh into a FIFO. I am using Avalon flow control to get the data. I connect the FIFO almost full signal_N to the "readyfordata_n" (along with data and write). The problem is that the DMA transfers appear to be lower priority than the NIOS instruction/data transfers, so the DMA gets starved for data when the NIOS is doing very much. Is there a way to make the DMA a higher transfer priority (note, not the interrupt priority, which you can set) than the NIOS instruction fetches? I can monitor the FIFO_EMPTY signal and see it going low during the frame, which should never happen.

Thanks