Max throughput of SOPC DMA?

You could put some signaltap probes on the DMA masters to see what's going on. The slow speed could be explained by the fact that the DMA controller isn't using bursts on the transfers. In that case I think that if both the read and write masters fight to get access to the DDR SDRAM, they will each get mostly single cycle operations, and you loose a lot of time due to the memory latency.

You should try to enable bursts, but from a quick read of the documentation it seems that in that case the DMA transfer length mustn't be higher that the burst count, so you would have to split your test in multiple DMA transfers.

If you have enough on-chip memory you could try to do transfers between the DDR SDRAM and the on-chip memory. You should have less latency problems in that case and it should give you a better idea of the DMA's performance transferring data from main RAM to a peripheral.

I don't use bursting at the moment no, because I haven't enabled that. When I tried to use bursting a while back it didn't seem to work, but I can try that again. Since it was most likely due a software fault.

The max amount I can burst at one time is 1024 words (if the DDR2 can handle that) . So that indeed means I would have to set up 8192 DMA transfers if I wanted to transfer 32 megabyte.

I did try using two seperate on-chip memories though for communicating with the Serial RapidIO core (again non bursting):

[On-chip TX_data] -> Serial RapidIO Core loopback -> [On-chip RX_data]

and I get a whopping speed of 8.2203 megabyte/second.

With the following set-up:

[DDR memory_1] -> Serial RapidIO Core loopback -> [DDR memory_1]

I got a speed of 4,9936 megabyte/second

Maybe I should use bursting for the entire system then (since the Serial RapidIO Core supports it). Since you are saying these kinds of speed aren't normal.

You definitley need to try to avoid the clock crossing and bus width adapters, they will both slow things down significantly.

This is all made more difficult because the sopc builder won't tell you where it has inserted them.

There shouldn't be any bus width adapters between the 32 bit NIOS processor, the 32 bit data path RapidIO core and the 32 bit data width on-chip memory (and the DMA controllers) though?

I can't really help the ones for the DDR2 though, but that doesn't really matter anymore to me, since if I want max speed I need to use on-chip memory anyway.

The on-chip memory will give about the same performance, with or without bursts. But I think that the SRIO core will slow down dramatically without bursts, as it will probably turn each read/write transaction into a SRIO packet. With bursts it would create less packets and I think that it would go faster.

The SGDMA is able to do larger transfers with bursts, up to 65536 bytes IIRC. But if you do a descriptor chain and regularly update it on the interrupt generated by the SGDMA, the software won't slow down the SGDMA controller, and it will run at almost 100% capacity even for big transfers.

The old DMA core seems very limited with bursts, it is a shame it isn't able to do a transfer bigger than the burst length...

I enabled burst transfers now. The clock crossing bridge only supports 256 words (the FIFO can't be any bigger) and for the on-chip memory it can be a maximum of 1024 words. I got the following results, which already is a big improvement. It's about a factor 8 with the on-chip memory. Which would make sense, like you said Daixiwen, if the Serial RapidIO now makes packages of 256 bytes data (the max) in stead of 32 bytes.

I think the reason that burst transfers didn't work the last time I tried it, is that I didn't enable it for the clock crossing bridge :rolleyes:.

These times are for a write transaction and after that a read transaction. So for throughput of the DMA controller you can double the number.

I used a for loop to perform the multiple transactions.

two on chip memories

Number of words in hexadecimal format (max 0x2000)...0x400

How many transactions? In decimal format...50000

0x400 * 50000 = 51200000 words = 204800000 bytes = 195.3125 megabytes

one ddr2 memory

Number of words in hexadecimal format (max 0x2000)...0x100

How many transactions? In decimal format...200000

0x100 * 200000 = 51200000 words = 204800000 bytes = 195.3125 megabytes

+---------------+-----+-----------+---------------+-----------+

| Section | % | Time (sec)| Time (clocks)|Occurrences|

+---------------+-----+-----------+---------------+-----------+

|DDR2 DMA | 8.17| 8.27889| 827888631| 1|

+---------------+-----+-----------+---------------+-----------+

|On-chip DMA | 2.76| 2.79213| 279212507| 1|

+---------------+-----+-----------+---------------+-----------

Transfer speed DDR2: 195.3125 / 8.27889= 23.59 megabyte/second

Transfer speed on-chip DMA: 195.3125 / 2.79213= 69.95 megabyte/second

The set-up times aren't that big, by which I mean:

alt_dma_txchan_send (txchan, tx_data, length, null, null);

alt_dma_rxchan_prepare (rxchan, rx_data, length, txrxDone, null);

+---------------+-----+-----------+---------------+-----------+

| Section | % | Time (sec)| Time (clocks)|Occurrences|

+---------------+-----+-----------+---------------+-----------+

|Write set-up |0.429| 0.12740| 12740253| 20000|

+---------------+-----+-----------+---------------+-----------+

|Read set-up | 0.43| 0.12780| 12780087| 20000|

+---------------+-----+-----------+---------------+-----------+

So I can't really get a real improvement using the DMA controller registers directly in stead of using the drivers I suppose. Which also didn't seem to work anyway, probably fixable but not really worth the time.

question here :):

so i guess it's time for the scatter-gather dma controller. do you getting that getting that to work is doable in like 60 to 80 hours?

it doesn't seem to be to difficult if i look at the data sheet or i.e. this thread : http://www.alteraforum.com/forum/showthread.php?t=21462&highlight=sgdma (http://www.alteraforum.com/forum/showthread.php?t=21462&highlight=sgdma) or this example http://www.nioswiki.com/exampledesigns/sgdma (http://www.nioswiki.com/exampledesigns/sgdma), but you never know -,-.

Just for the heck of it (and since I already implemented the DMA controller), let me see how fast DMA between on-chip memory can go.

EDIT:

DMA burst transfer using one DMA controller between two on-chip memories:

In hexadecimal format (max 0x400)...0x400

How many transactions?

In decimal format...50000

--Performance Counter Report--

Total Time: 11.1866 seconds (1118663570 clock-cycles)

+---------------+-----+-----------+---------------+-----------+

| Section | % | Time (sec)| Time (clocks)|Occurrences|

+---------------+-----+-----------+---------------+-----------+

|On-chip DMA | 21.1| 2.36401| 236400523| 1|

+---------------+-----+-----------+---------------+-----------+

Transfer speed: 195.3125 / 2.36401= 82.619 megabyte/second (So per DMA transfer about 82.619*2 = 165,238 megabyte/second.)

With the following code:

/*Open DMA channels */
<....>
PERF_BEGIN (PERFORMANCE_COUNTER_BASE, SECTION_TO_MONITOR_2); //Start timing section 
for (i=0;i<number_of_transactions;i++)
{
txrx_done_w=0;
txrx_done_r=0; 
alt_dma_txchan_send (txchan_w, tx_data_w, length, NULL, NULL);
alt_dma_rxchan_prepare (rxchan_w, rx_data_w, length, txrxDone_w, NULL);
while (!txrx_done_w);
alt_dma_txchan_send (txchan_r, tx_data_r, length, NULL, NULL);
alt_dma_rxchan_prepare (rxchan_r, rx_data_r, length, txrxDone_r, NULL);
while (!txrx_done_r);
}
PERF_END (PERFORMANCE_COUNTER_BASE, SECTION_TO_MONITOR_2); //End timing section
/* Close channels */

And here's the final table for those who would like to know, before I strip the whole thing down again and add scatter-gather DMAs :).

Transferring 195.3125 megabytes.

On-chip memory uses bursts of 1024 words. DDR2 memory uses bursts of 256 words.

--Performance Counter Report--

+---------------+-----------+---------------+-----------+

| Section | Time (sec)| Time (clocks)|Occurrences|

+---------------+-----------+---------------+-----------+

|DDR2<->DDR2 | 6.40292| 640292166| 1|

+---------------+-----------+---------------+-----------+

|DDR<->SRIO | 6.75203| 675203128| 1|

+---------------+-----------+---------------+-----------+

|OC->SRIO->OC | 2.72834| 272833895| 1|

+---------------+-----------+---------------+-----------+

|OC<->OC | 2.69376| 269375544| 1|

+---------------+-----------+---------------+-----------+

1. Transfer speed: 195.3125 / 6.40292 = 30.5037 megabyte/second

2. Transfer speed: 195.3125 / 6.75203 = 28.9265 megabyte/second

3. Transfer speed: 195.3125 / 2.72834 = 71.5866 megabyte/second

4. Transfer speed: 195.3125 / 2.69376 = 72.5055 megabyte/second

Reasons the DDR2 is much slower are: only 256 word bursts, clock crossing bridge and probably bus width adapters.

You should be able to hit around 97% (max) efficiency using a SGDMA depending on the access pattern. I recommend using the HP2 or Uniphy controllers since they don't need burst accesses to be efficient. If you access them sequentially then internally they'll form optimal offchip bursts for you.

If you want to continue using bursting I highly recommend that you match the burst counts of the DMAs to the memory to avoid burst adaptation. You might also run into burst wrapping efficiency problems as well since the DMAs don't line themselves up on burst boundaries except this one: http://www.altera.com/support/examples/nios2/exm-modular-scatter-gather-dma.html?gsa_pos=1&wt.oss_r=1&wt.oss=sgdma

The SGDMA is a bit more complex, but isn't that difficult to use.

The main difference is that the DMA operation to execute is stored in memory, in a structure called a descriptor, instead of the DMA registers. If you have a look at the driver made by Altera, you will find functions to make a descriptor and another function to start a transfer. Just be sure when you design your SOPC system that the descriptor read and write ports are connected to the memory you will put the descriptors in.

Descriptors can be chained, i.e. each descriptor can point to another descriptor describing the new operation to accomplish. This is how you can have a very efficient operation without using the CPU.

As I said each operation is limited to 65536 bytes transferred. The quick-an-dirty solution to transfer 32MB would be to create a chain of 512 descriptors and launch the SGDMA on it.

A more elegant solution would be to use a circular buffer of 4-5 descriptors, place an interrupt each time the SGDMA finished processing a descriptor and write an interrupt handler that adds a new descriptor to the chain. By staying ahead of the SGDMA by 2 descriptors you'll manage to keep it busy almost 100% of the time. If you are familiar with NiosII interrupts handlers I think that such a solution would be doable in 60/80 hours.

@BadOmen, I didn't have access to the HP2 or Uniphy controllers (licenses), when I made the design, so I'm using the HP currently. By "match the burst counts of the DMAs to the memory" you mean the DDR2 memory I assume. I'm working with on-chip memory at the moment as this is faster and less complicated.

@Daixiwen. Actually I'm only able to burst transfer 8 kilobytes (2^13), since I have a Stratix IV GX ES. The reason for this is that the M144K blocks are bugged, meaning you can't use them in dual-port dual-clock mode: http://www.altera.com/literature/es/es_stratixiv_gx.pdf. So (part of) the SMGDAs are implemented in M9K blocks, which aparently (according to the Quartus II errors), only support a width of 13. Which is a shame since the RapidIO core supports up to 32 kilobytes for the RX and TX buffers, so I could have done 32 kilobytes bursts if it wasn't bugged.

I think I will try to see if I can get the quick-and-dirty solution to work, since I have no experience with interrupts handlers in the NIOS II and I'm running towards the end of my internship period. I can leave the rest as recommendations in my report.

Thanks for all the help :).