mSGDMA, Qsys, and Linux Integration

--- Quote Start ---

Hi Derim. I am not sure if this example design from Rocketboards (http://rocketboards.org/foswiki/view/documentation/datamoverdesignexample) would convey you the info that you need. But worth having a look :)

--- Quote End ---

Thanks-- I think I'm going to debug this a bit myself a bit more as I'm thinking that there's either an endianess or timing issue with the HPS driving the mSGDMA directly when mapping /dev/mem. It would be great to know if anyone else has tried it as the examples all rely on a Nios core, which should be redundant for an HPS-based design. Depending, of course, on one's application.

I do control mSGDMA with lightweight AXI

In QSys:

CSR, Descriptor, Response connected to lw_axi_master

mm_write connected to f2h_axi_slave

In Linux:

open /dev/mem

mmap to 0xFF200000 with offset of registers addresses

# pragma pack(push,1)
/**
 * @brief mSGDMA control and status register
 */
typedef struct {
	uint32_t	status;
	uint32_t	control;
	uint16_t	rd_fill_level;
	uint16_t	wr_fill_level;
	uint16_t	resp_fill_level;
	uint16_t	reserved_0;
	uint16_t	rd_sequence_number;
	uint16_t	wr_sequence_number;
	uint32_t	reserved_1;
	uint32_t	reserved_2;
	uint32_t	reserved_3;
} t_mSGDMA_CSR;
# pragma pack(pop)
t_mSGDMA_CSR * regs_CSR;
fd_FPGA_ctrl_regs = open("/dev/mem", O_RDWR|O_SYNC);
mem_FPGA_ctrl_regs = (unsigned int*)mmap(NULL, MAP_PIO_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED, fd_FPGA_ctrl_regs, 0xFF200000);
regs_CSR= (t_mSGDMA_CSR*)(p_mmap_mSGDMA_control + mSGDMA_CSR_offset		/ sizeof(uint32_t));
/* descriptor, response is the same */
/* and registers are ready to use */

--- Quote Start ---

I do control mSGDMA with lightweight AXI

In QSys:

CSR, Descriptor, Response connected to lw_axi_master

mm_write connected to f2h_axi_slave

In Linux:

open /dev/mem

mmap to 0xFF200000 with offset of registers addresses

--- Quote End ---

Great. See, that makes sense. f2h_axi_slave, or f2h_sdram? Where are you pushing data?

--- Quote Start ---

Great. See, that makes sense. f2h_axi_slave, or f2h_sdram? Where are you pushing data?

--- Quote End ---

I use f2h_axi_slave to put data from mSGDMA to SDRAM

--- Quote Start ---

I use f2h_axi_slave to put data from mSGDMA to SDRAM

--- Quote End ---

See, that's fascinating. So instead of using the SDRAM AXI you're using the f2h_axi_slave. I'll give it a try.

So the mSGDMA is working great, but now it almost looks like I'm seeing endianess issues when reading back the data in to the HPS. Has anyone seen this? In SignalTap the signals are definitely correct, but when reading the values back in the HPS, it looks like the endianess is flipped.

As the Nios and HPS are little-endian, does the mSGDMA do any sort of internal conversion to handle this?

--- Quote Start ---

So the mSGDMA is working great, but now it almost looks like I'm seeing endianess issues when reading back the data in to the HPS. Has anyone seen this? In SignalTap the signals are definitely correct, but when reading the values back in the HPS, it looks like the endianess is flipped.

As the Nios and HPS are little-endian, does the mSGDMA do any sort of internal conversion to handle this?

--- Quote End ---

And on a related note, how does the prefetcher handle fetching descriptors when the descriptor interface width is less than the full descriptor width? Does it fetch from LSB to MSB, or something else? If it fetches LSB to MSB, this will cause problems as the "GO" bit is in the MSB, which will then trigger a start without setting any of the interrupt enables.

--- Quote Start ---

And on a related note, how does the prefetcher handle fetching descriptors when the descriptor interface width is less than the full descriptor width? Does it fetch from LSB to MSB, or something else? If it fetches LSB to MSB, this will cause problems as the "GO" bit is in the MSB, which will then trigger a start without setting any of the interrupt enables.

--- Quote End ---

Up to the point now where the SDR ports are out of reset, and I am seeing data being streamed from the DMA in to RAM. The data ordering is odd, but I'm trying to do 128-bit transactions. I don't see why this would be an issue, but looking in to it.

EDIT: Spoke too soon. I'm able to get it to work once in awhile, but not consistently as expected. My procedure:

> Write updated descriptors in to descriptor memory

> Check for prefetcher to be stopped

> Trigger "run" and "global_int" bits of prefetcher

> Wait for run to end

> Wait for interrupt

It seems to sometimes work, sometimes not. Quite frustrating. The first run in a set seems to be missed, and unless I do things in a very specific way, no luck. It also seems to not be clearing the interrupt on the last run or clearing the interrupt takes longer than it should.

--- Quote Start ---

Up to the point now where the SDR ports are out of reset, and I am seeing data being streamed from the DMA in to RAM. The data ordering is odd, but I'm trying to do 128-bit transactions. I don't see why this would be an issue, but looking in to it.

--- Quote End ---

For any one else having issues, I was running in to an issue where the Qsys interconnect (or something on the Descriptor-write side) was causing problems with triggering the DMA, leading the DMA to act very odd.

Even so, I need to add a <100 ms delay after initiating and completing the first mSGDMA run before I can start using the DMA correctly. I'm wondering if there is some sort of initialization procedure that isn't covered by a reset that is not being accounted for?