Hello, Currently I have a system with DMA transfers and DDR2 SDRAM (and a lot of other stuff). I'm using the normal SOPC builder DMA controller. I run the system on 100 MHz, the DDR2 SDRAM runs on 125 MHz via a clock crossing bridge. If I transfer 32 megabytes at once I only get a transfer speed of 41.5670 megabyte/second. i was wondering if these kinds of low speeds are normal for the dma controllers in the in the sopc builder? The main issue for me is that I use the DMA controllers to send data to my Serial RapidIO core. But if I send data to the core(from either on-chip memory or the DDR2 memory, the latter being slower ofcourse), I get a throughput of around 10 megabytes/second, but I want this to be more like 1000 megabytes/second. I know I could try the SG-DMA, which should be faster, but I don't see why it would be THAT much faster for only one transfer. Since the SG-DMA should be able to maintain speeds of up to 10 Gbps. tx_data=(void*)ALTMEMDDR_1_BASE; rx_data=(void*)ALTMEMDDR_1_BASE+0x1FFFFF8; length = 0x1FFFFF8; txchan = alt_dma_txchan_open(DMA_TESTER_NAME); rxchan = alt_dma_rxchan_open(DMA_TESTER_NAME); PERF_BEGIN (PERFORMANCE_COUNTER_BASE, SECTION_TO_MONITOR_3); //Start timing section txrx_done=0; alt_dma_txchan_send (txchan, tx_data, length, null, null); alt_dma_rxchan_prepare (rxchan, rx_data, length, txrxDone, null); while (!txrx_done); PERF_END (PERFORMANCE_COUNTER_BASE, SECTION_TO_MONITOR_3); //End timing section alt_dma_txchan_close(txchan); alt_dma_rxchan_close(rxchan); --Performance Counter Report-- Total Time: 20.8909 seconds (2089086210 clock-cycles) +---------------+-----+-----------+---------------+-----------+ | Section | % | Time (sec)| Time (clocks)|Occurrences| +---------------+-----+-----------+---------------+-----------+ |DDR2 to DDR2 | 3.69| 0.76984| 76983631| 1| +---------------+-----+-----------+---------------+-----------+ Transfer speed DDR2 to DDR2 = 31.9999885559082/0.76984=41.5670 megabyte/second Thanks in advance.

You could put some signaltap probes on the DMA masters to see what's going on. The slow speed could be explained by the fact that the DMA controller isn't using bursts on the transfers. In that case I think that if both the read and write masters fight to get access to the DDR SDRAM, they will each get mostly single cycle operations, and you loose a lot of time due to the memory latency. You should try to enable bursts, but from a quick read of the documentation it seems that in that case the DMA transfer length mustn't be higher that the burst count, so you would have to split your test in multiple DMA transfers. If you have enough on-chip memory you could try to do transfers between the DDR SDRAM and the on-chip memory. You should have less latency problems in that case and it should give you a better idea of the DMA's performance transferring data from main RAM to a peripheral.

I don't use bursting at the moment no, because I haven't enabled that. When I tried to use bursting a while back it didn't seem to work, but I can try that again. Since it was most likely due a software fault. The max amount I can burst at one time is 1024 words (if the DDR2 can handle that) . So that indeed means I would have to set up 8192 DMA transfers if I wanted to transfer 32 megabyte. I did try using two seperate on-chip memories though for communicating with the Serial RapidIO core (again non bursting): [On-chip TX_data] -> Serial RapidIO Core loopback -> [On-chip RX_data] and I get a whopping speed of 8.2203 megabyte/second. With the following set-up: [DDR memory_1] -> Serial RapidIO Core loopback -> [DDR memory_1] I got a speed of 4,9936 megabyte/second Maybe I should use bursting for the entire system then (since the Serial RapidIO Core supports it). Since you are saying these kinds of speed aren't normal.

You definitley need to try to avoid the clock crossing and bus width adapters, they will both slow things down significantly. This is all made more difficult because the sopc builder won't tell you where it has inserted them.

There shouldn't be any bus width adapters between the 32 bit NIOS processor, the 32 bit data path RapidIO core and the 32 bit data width on-chip memory (and the DMA controllers) though? I can't really help the ones for the DDR2 though, but that doesn't really matter anymore to me, since if I want max speed I need to use on-chip memory anyway.

The on-chip memory will give about the same performance, with or without bursts. But I think that the SRIO core will slow down dramatically without bursts, as it will probably turn each read/write transaction into a SRIO packet. With bursts it would create less packets and I think that it would go faster. The SGDMA is able to do larger transfers with bursts, up to 65536 bytes IIRC. But if you do a descriptor chain and regularly update it on the interrupt generated by the SGDMA, the software won't slow down the SGDMA controller, and it will run at almost 100% capacity even for big transfers. The old DMA core seems very limited with bursts, it is a shame it isn't able to do a transfer bigger than the burst length...

Max throughput of SOPC DMA? | Altera Community

21 Replies

Altera_Forum
Honored Contributor
15 years ago
I had a look a the modular SG-DMA and it seems to be (a lot) easier to use than the normal SG-DMA: http://www.altera.com/support/examples/nios2/exm-modular-scatter-gather-dma.html

Is there any real disadvantage using that one over the normal SG-DMA? Other than the max burst is 1024 (words I think?)?
Altera_Forum
Honored Contributor
15 years ago
That's correct, if you match the burst count (and data width) of the DMA to the SDRAM then you should be able to avoid burst adapters between the DMA and SDRAM. You would however end up with a burst adapter between the DMA and on-chip RAM since that memory is not burst capable.

The modular SGDMA was designed with this in mind:

- ease of use
- support large transfer sizes and data widths
- support customization to the control plane (i.e. you can replace the dispatcher for your own controller)
- capable of pre-fetching descriptors from memory by simply adding a pre-fetch block in front of the dispatcher (one of these days I'll build one...)
- use a smaller logic and memory footprint
- achieve higher throughput and fmax
- easily support video, Ethernet, PCIe, etc... types of applications

I would say the disadvantage of the modular SGDMA over the regular SGDMA are:

- lack for descriptor pre-fetching from memory
- lack of a full blown HAL driver

There might be more but nothing comes to mind.

Altera_Forum

Honored Contributor

15 years ago

Hi BadOmen (and others),

Do you have any experience with the modular SGDMA? Because I now have the following code. (Pretty much directly taken from the example.)

And I have a question as seen in the next post.

// flag used to determine when all the transfers have completed


volatileint sgdma_interrupt_fired = 0;
 
staticvoid sgdma_complete_isr (void *context, alt_u32 id)
{
sgdma_interrupt_fired = 1;
clear_irq (MODULAR_SGDMA_ONCHIP_CSR_BASE);
}
 
void modular_sgdma_transfer()
{
sgdma_standard_descriptor a_descriptor;
sgdma_standard_descriptor * a_descriptor_ptr = &a_descriptor;  // using this instead of 'a_descriptor' throughout the code
unsignedlong length;
unsignedlong read_address;
unsignedlong write_address;
unsignedlong control_bits;
unsigned long transfer_time, test_throughput, I, number_of_transfers;
 
alt_irq_register (MODULAR_SGDMA_ONCHIP_CSR_IRQ, NULL, sgdma_complete_isr);  // register the ISR
enable_global_interrupt_mask(MODULAR_SGDMA_ONCHIP_CSR_BASE);                // turn on the global interrupt mask in the SGDMA
 
memset((void*)SRIO_ONCHIP_TX_DATA_BASE,0x00, SRIO_ONCHIP_TX_DATA_SPAN); 
memset((void*)SRIO_ONCHIP_RX_DATA_BASE,0x00, SRIO_ONCHIP_RX_DATA_SPAN);
printf("Content of TX before DMA operation\n");
read_address = SRIO_ONCHIP_TX_DATA_BASE;
write_address = SRIO_ONCHIP_RX_DATA_BASE;
length = MAXIMUM_BUFFER_SIZE; // 16384
number_of_transfers=100;
alt_timestamp_start();
control_bits = 0;  // go bit is handled 
construct_standard_mm_to_mm_descriptor (a_descriptor_ptr, (alt_u32 *)read_address, (alt_u32 *)write_address, length, control_bits);
 
for(i = 0; i < number_of_transfers; i++)
{ 
//See next post for code
}
//Write the last descriptor (is last transfer)
control_bits =  DESCRIPTOR_CONTROL_TRANSFER_COMPLETE_IRQ_MASK;  // go bit is handled 'construct_standard_mm_to_mm_descriptor'
construct_standard_mm_to_mm_descriptor (a_descriptor_ptr, (alt_u32 *)read_address, (alt_u32 *)write_address, length, control_bits);
while ((read_csr_status(MODULAR_SGDMA_ONCHIP_CSR_BASE) & CSR_DESCRIPTOR_BUFFER_FULL_MASK) != 0) {}  // spin until there is room for another descriptor to be written to the SGDMA
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
 
while (sgdma_interrupt_fired == 0)  {}  // keep spinning until the interrupt fires when the last word is written to the destination location by the SGDMA
transfer_time = alt_timestamp();  // number of clock ticks from the time that descriptors where formed and sent to the SGDMA to the time of the last memory write occuring
 
mem_compare( (void*) SRIO_ONCHIP_TX_DATA_BASE, (void*) SRIO_ONCHIP_RX_DATA_BASE, MAXIMUM_BUFFER_SIZE);
sgdma_interrupt_fired = 0;  // set back to 0 to perform another test
// /  = total test time in seconds
//throughput =  /  = ( * ) /   <--- this would be bytes per second so divide by 1024*1024 to get MB/s
length = length*128;
test_throughput = (unsignedlong)((((double)(number_of_transfers*length)) * ((double)alt_timestamp_freq())) / ((double)transfer_time * 1024 * 1024) );
 
printf("Test completed with a throughput of %ldMB/s.\n",  test_throughput);
}

Altera_Forum

Honored Contributor

15 years ago

First thing in the for loop is:

while ((read_csr_status(MODULAR_SGDMA_ONCHIP_CSR_BASE) &  CSR_DESCRIPTOR_BUFFER_FULL_MASK) != 0) {}  // spin until there is room  for another descriptor to be written to the SGDMA

Then the code in the for loop is ~13 times the code below (nasty I know :)):


write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr)  ;

So I have 128 (descriptors which is the max FIFO at the moment times) of these in a row: write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr) ;

The length for each descriptor is 16384 bytes and I do the whole thing 100 times.

i was wondering if this way i do in fact transfer 128*100*16384 bytes? Since I'm getting a maximum transfer rate of 3380MB/s between the 2 on-chip memories and that's pretty neat. (To a maximum of throughput of 3770MB/s if I increase the number of transfers.)

I know it works if I add only one:

write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr) ; and I can make the length whatever I want (and see that happening). but does adding 128 of these mean a transfer size of 128*length? I think it does, but I might be overlooking something.

What I now have are 2 on-chip memories of each 32kb and I continually write 16k between them (so I keep overwriting the RX data).

Edit: I guess I could just clear the RX memories between writes and see if it it filled after a next descriptor write.

Altera_Forum
Honored Contributor
15 years ago
If I only add one (since the buffer is able to store 128 descriptors):
while ((read_csr_status(MODULAR_SGDMA_ONCHIP_CSR_BASE) & CSR_DESCRIPTOR_BUFFER_FULL_MASK) != 0) {}

and then 128 times
write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr) ;

in the for loop.

I get a maximum throughput of 3760MB/s (did a miscalculation earlier).

The data does seem to get transfered though, if in the middle of those 128 'write_standard_descriptors' I clear the RX memory and after one 'write_standard_descriptor' I read it back out again.

Edit: Something still goes wrong with calculation, since when I double the transfer_size, the number of clock_ticks double but the transfer size decreases.
Edit2: I'm running towards the maximum I can store in an unsigned long I think >,<.
Edit3: Yup, was going over 2^32.

Edit4: Calculating gives:
Test completed with a transfer size of 40000MB
Transfer time (clocks ticks): 83374115
Transfer time (seconds): 0.833741
This gives a throughput of 47962MB/s

which means with a clock of 100 mhz a throughput of 47962 mb/s , now that can't be right or can it?

Edit5: Ah if I wrap an IF around it
if(write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr) != 0) { printf("Failed to write descriptor 0x%lx to the descriptor SGDMA port.%c", i+1, TERMINAL_KILL_CHARACTER); }
I get Failed to write descriptor 0x42 to the descriptor SGD.
I have to add
while ((RD_CSR_STATUS(MODULAR_SGDMA_ONCHIP_CSR_BASE) & CSR_DESCRIPTOR_BUFFER_FULL_MASK) != 0) {} // spin until there is room for another descriptor to be written to the SGDMA
between each write_standard_descriptor .

Altera_Forum

Honored Contributor

15 years ago

If I do it like the example (silly me doing it slightly different..). I only get a throughput of 379 MB/s. Isn't there a way to make it go faster?

 
for 
(i = 0; i < number_of_transfers; i++) 
{
while ((RD_CSR_STATUS(MODULAR_SGDMA_ONCHIP_CSR_BASE) & CSR_DESCRIPTOR_BUFFER_FULL_MASK) != 0) {} // spin until there is room for another descriptor to be written to the SGDMA
control_bits = (i == (number_of_transfers-1))? DESCRIPTOR_CONTROL_TRANSFER_COMPLETE_IRQ_MASK : 0; 
construct_standard_mm_to_mm_descriptor (a_descriptor_ptr, (alt_u32 *)read_address, (alt_u32 *)write_address, length, control_bits);
if(write_standard_descriptor (MODULAR_SGDMA_ONCHIP_CSR_BASE, MODULAR_SGDMA_ONCHIP_DESCRIPTOR_SLAVE_BASE, a_descriptor_ptr) != 0)
{
printf("Failed to write descriptor 0x%lx to the descriptor SGDMA port.%c", i+1, TERMINAL_KILL_CHARACTER);
}

Altera_Forum
Honored Contributor
15 years ago
What are the source and destination memories? What are the source and destination data widths? And what are the max burst count setup for those memories and the SGDMA?

The design example shows very limited performance because of the following:

a) The source and destination memory are the same (i.e. throughput cut in half)

b) The arbiter is letting the read and write masters access the memory with burst of 2 transactions back and forth which will thrash SDRAM (SDRAM performs best with a bunch of back to back sequential accesses).

So really the way the design example is setup for SDRAM it's giving probably the worst case performance possible.
Altera_Forum
Honored Contributor
15 years ago
My test set-up is:
32 bit data-width TX on-chip memory -> Modular SG-DMA controller (1024 word bursts (max)) -> 32 bit data-width RX on-chip memory.
I get that on-chip memory doesn't support burst transfers on it's own, but how does it work then?

So I have 2 different on-chip memories. But I can only transfer 32k at a time, so for each 32k I have to set-up a new transfer.
The reason I can only transfer up to 32k is because the RapidIO core, that is going to be the actual target after the tests, has a maximum TX buffer of 32k. So I have to transfer from address 0x0 through 0x8000 and then back to 0x0 through 0x8000 etcetera.

So the value of the length register is 32768 and the number of transfer was for example 100000. Meaning I have a lot of overhead for the amount I transfer. And the modular SG-DMA doesn't have pre-fetching, so I guess this causes for a big loss in speed due to overhead?

So I think the normal SG-DMA might be more suitable for this cause? Since it has pre-fetching?

(If they are actually are gonna use the RapidIO core for production purposes, they will probably write their own DMA controller, so it's mostly to test the speed of the RapidIO core/ see how much speed is lost compared to memory to memory copy etc.)
Altera_Forum
Honored Contributor
15 years ago
The modular SGDMA design assumes that something (host or the DMA itself) will shovel multiple descriptors into it. If you send one descriptor at a time into it you are using it like a standard DMA. There shouldn't be much overhead difference since you are either stuffing descriptors into the FIFO inside the modular SGDMA dispatcher or you are placing them into memory and letting the SGDMA go fetch them (that's actually more overhead since you have to maintain a linked-list in memory... and adding to the list while the SGDMA is operating isn't trivial)

Neither the SGDMA or modular SGDMA are capable of posting reads for a descriptor while the previous descriptor transfer reads are still trickling in. The modular SGDMA will have this added for "Full word access only" mode and I doubt the regular SGDMA will ever have this feature. This will allow the DMA to hide the latency in between transfers for sequential descriptors. This feature is handy for high latency links like PCI, PCIe, SRIO, etc....

I suspect the reason why you are seeing inefficiencies is due to the high burst count you have selected. If you chose a FIFO depth that is only 2x the max burst count I could see this being very inefficient (simulate to find out why). The only difference between a burst transfer and a non-burst transfer is that the arbiter gets locked down for the entire burst causing other masters to have to wait. Bursting is meant for interfaces like SDRAM, PCI, PCIe, SRIO, etc... Since on-chip memories don't support bursting you are having a burst adapter inserted automatically for you which will chop up the bursts of 1024 into bursts of 1 (i.e. non bursting). RapidIO if I remember correctly uses a max burst count of 32 so for your testing I recommend using 32 and a master FIFO depth of 4x or greater.
Altera_Forum
Honored Contributor
15 years ago
Hi, thanks for all the help so far.

Aren’t I already sending multiple descriptors into it then? Since I fill the FIFO with descriptors (waiting on the FIFO to have room for another descriptor) and after that I wait for the last word to be written by the SG-DMA, which causes an interrupt?

The actual data transfer is what takes up almost all the time, ofcourse.

I changed the burst count to 64 (the maximum for RapidIO) and the FIFO depth for the SGDMA to 512. And I enabled the "Full word access only". But that made zero difference, it even seemed to get slightly slower? With 1024 words burst transfer the FIFO depth was 2048.
For the record the ~360 MB/s is the transfer speed between 2 on-chip memories at the moment. The transfer speed from on-chip memory to RapidIO is somewhat slower.

(Maybe these DMA controllers simply aren't good at sending small amounts of data (32k) per descriptor? )
Edit: With transfers of 60k I get exactly the same transfer speed, so that doesn't (really) seem to matter.
Or there might be something I do wrong in the software (even though I think it's the same as the example)?

Please see the attached picture for the configuration. The on-chip RAMs are the standard SOPC builder memories. They are 32 bits wide.

Forum Discussion

Max throughput of SOPC DMA?

21 Replies

Recent Discussions

NIOS V/m dbg_reset_out signal (Q25.1 Std, MAX10)

NiosV and juart-terminal

licensing.altera.com never worked

JTAG_UART stuck in printf

Ashling IDE scripted project creation