mSGDMA stride reads/writes

Yeah, I did it!

I finally have 256Bit Fifos which are read- and writeable by the mSGDMA-IP. They are really working fine and with some tricks I now can do some huge DMA-Transfers (some ten MB ) with Linux.

But the speed seems a little bit slow. Currently I'm meassuring a Transfer-Speed of 25,6MB/s (25.600.000 Byte/s).

I'm using PCIe Gen1 with one Lane. Theoretically this should give me about 250MB/s. I understand that this is really theoretically and there are latencies, overhead and so on. But my currently meassured speed is only a tenth of the maximum speed.

Is there something I can do to speed it up?

What Information do you need to help me?

Greetings

Steffen

By the way: do the DMA-Dispatcher or the Read-Masters have any strange behaviour about their reset-input?

I wrote big data-block to the Fifo until it was full. Then I resetted the Fifo, the Dispatcher and also the Read-Master.

While I did the Reset, the fill-level of the fifo was 0.

When I take back all the resets, the fifo showed me, that it was full again. I know, the Fifo-Entries are deleted by the reset. But it seems to me that either the Dispatcher or the Read-Master fills it immediately after the reset is deasserted.

Some common things I have heard of that help are:

1) Make sure you use the same burst size in the DMA as the PCIe core expects (it's either 64 or 32 depending on how you setup the PCIe core if I remember correctly)

2) Make sure the 'burst on aligned buffers' option in each master is disabled

3) Make sure you have multiple descriptors always queued up in the dispatcher to maintain a high throughput

When resetting the DMA you should monitor the reset status of the DMA since it doesn't stop on a dime because it has to wait for outstanding transactions to complete otherwise the fabric might lock up. So you should issue the reset, poll the DMA making sure the reset is complete, then take the DMA back out of reset. So what you might be seeing is a bunch of left over read data filling up the FIFO again after the reset is deasserted. Also you should be resetting the DMA dispatcher and masters by writing to the reset bit inside the dispatcher, if you are asserting the reset from outside the IP core through the reset inputs then you will most likely cause the DMA to violate the Avalon-MM spec (that reset should only be triggered if you are resetting the entire system).

Data in a FIFO should survive a reset because the memory cells cannot be cleared by a reset. The pointers and the fill levels get reset so the status from the FIFO will tell you it has been reset but valid data will be left behind. You shouldn't have to worry about that data being left behind becuase once you start filling the FIFO again they will get overwritten.

YEEEAAAAAAAH! Thats it!

Now I have the speed I wanted! Now my design has a throughput of about 100MB/s!!!!

It can't get faster because the other Hardware isn't faster than about 800MBit/s. I think it was about the wrong settings to the Burst read and writes.

Thank you so much BadOmen.

You are welcome.

By the way if you are not already doing so you might be able to get even more speed by using the "early done" feature of the DMA. A typical DMA transfer starts with the read and write masters being told what to do by the dispatcher then the reads start and after enough data has been read in and transferred over to the write master the write master starts writing to memory. When you get to the end of the transfer the read master stops because it is waiting for data to return and when that last word arrives then it tells the dispatcher it's done and is ready for the next descriptor (write master will still be writing out data). With early done enabled in the descriptor control field, the read master will signal to the dispatcher that it's done right when it issues it's last read without waiting for the read data to return.

Over PCIe a read will typically be over 100 cycles of latency so the early done bit lets you hide that latency by having the read master start issuing the reads for the next descriptor before the read data from the last descriptor arrives. So instead of seeing 100+ cycle gaps between transfers you'll have a couple of cycles between transfers instead. This of course assumes you have multiple descriptors already written into the dispatcher.

Hi BadOmen,

I have another question:

I noticed that the Write-Masters do a Read-Ahead if my Fifos have data and there is no DMA-Transfer triggered.

This is no Problem for me, but is there a way to read out the Fill-Level of the Internal Fifo in the Master?

I was kind of confused when I wrote data to the Fifos and they suddenly disappeared (Fifo Fill-Level stayed 0).

But a DMA-Transfer with the right size gave me the data.

My problem is: I often don't know exactly how many data are received by the Fifos. If the disappear in the Write-Master I can only guess how many Bytes I have to transfer.

I tried the Response Registers of the msgdma-Core with Low-Level access via "pci_debug-Tool" (Linux), but all I got was a complete System-Breakdown and no "Actual Bytes Transferred" or "Error".

Can you help me?

That's correct, the write master will accept data even if it's not transfering. I meant to make this parameterizable so that you could prevent this from happening but I never got around to it. This feature is included for speed reasons since the DMA write data FIFO can be readied with data before the master beings transfering. One way to prevent this from happening would be to hack the RTL so that the Avalon-ST sink ready signal is gated by the !done (i.e. while the DMA is operating only then will data flow into the FIFO). If you wanted to see the fill level of the master you would need to add a conduit interface to the write master component and wire up the fill level to that conduit.

In order to read out the response you need to enable packet support in the write master. When that's enabled and the correct SOP and EOP bits are enabled in the descriptor the write master will keep track of how much data has been written and will report it to the dispatcher when the transfer is complete. This means that the IP providing the data will need to issue SOP on the first beat of streaming data and EOP on the last one.

YES, I DID IT!!

Although I had no clue of verilog and tcl and how to use them in QSYS, I made a conduit from the fifo_usedw wire to the QSYS-System and exported it. Now I can finally see, how many entries in the internal fifo are.

Thats nice. With an external Adder-Core I added the Fill-Levels of the original Fifo and the internal fifo of the write-master. Now I have the real amount of stored Data in the FPGA and I can generate DMA-Writes with the exact number of bytes.

Thank you

Hi.. BadOmen..

Salute for your modular SGDMA. It's very great. It's very easy to use.

From me, may be it just a little suggestion to enhance your mSGDMA. Especially MM-MM transfer

1. It will be useful if adding "support package", so we can add some synchronous component between write master core and read master core.

2. Same as case number 1. Is it possible to transfer different length? In example, if we add some module like FFT. As I know, FFT has different wide bit. Output has added exponential bus.

Maybe it useful..

Thank you... Greetings from Indonesia

You are welcome. I'll follow up to them in the same order:

# 1 Actually if the read and write master have to transfer the same amount of data you can put a streaming transform block between the read and write master and use a regular MM to MM transfer. If the amount of input data does not match the amount of output data then the easiest way to handle this is to use independent mSGDMAs set for MM to ST and the other for ST to MM and place the transform block between the two mSGDMAs. That workaround is a little wasteful but luckily the only logic duplicated is the dispatcher which is a fairly small block so it's probably only adding around 30 LEs and maybe one extra RAM block. The reason why it doesn't waste too much logic is because I only include one descriptor FIFO when the mSGDMA is setup for a memory to/from stream configuration. In MM to MM mode the dispatcher has two FIFOs internally so that the read and write masters can operate independently on different descriptors.

# 2 I wanted to include two length fields but I ran out of room in this descriptor to fit it (I needed to reserve some room for 64-bit addressing). It only adds 4 bytes to the descriptor but it wouldn't allow for descriptors to be nicely lined up on 16/32 byte boundaries in memory if I ever developed a descriptor prefetching unit. That said if you know that the length for the read master is a function of the write master or visa versa you could insert a gasket between the read or write master and the dispatcher that would modify the length as it's being sent to the master. For example lets say you know that the read master will read twice as much data as the write master will need to write, you could make a little gasket that sits in the write command ST path that takes the length and shifts it down by 1 bit before it arrives at the write master.

If the transfer needs to be very specialized then the other alternative is to create a block that replaces the dispatcher block and it can issue commands any way you like to each master. Compared to the master modules the dispatcher is a fairly simple block so it shouldn't be too difficult to replace.

Hi BadOmen,

I have some problem with m-SGDMA stream to memory.

I create 4 function (sgdma_init, start_listen_stream, getDataStream, and destruct_sgdma). please look up in attachment...

I call sgdma_init in the first and destruct_sgdma the end of task.

Inside the task there is many loop and calls start_listen_stream and getDataStream

It will works well in first call of task, but stuck in second call.

Am I do something wrong? Please help..

Many thanks for your helps..

Best and regards,

Doddy Patriadi