Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
13 years ago

mSGDMA stride reads/writes

Has anyone used the Stride feature (available with the extended descriptor)? I using the Qsys PCIe example design on the wiki. When I enable the extended descriptor and stride option, the data transfer gets corrupted and the PCIe bombs out. I know you have to write to a different address to set the GO bit, and have to write to a register to set the stride length. I basically just want to do large transfers to a custom component with a 256 bit interface. I have an SR open with Altera, but they are slower than molasses.

27 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    Striding across PCIe doesn't sound like a good idea to me. For starters when stride is enabled you have to disable bursting which will kill your PCIe performance. Also related to that is that for each word transfer a new PCIe packet will need to be formed because stride will cause addresses to *not* be continous. Perhaps your impression of the feature is different than what is implemented. For example if you setup a stride of 16 for a DMA that is 64-bits wide these are the addresses you should see accessed assuming a start location of 0x0:

    0x000-0x008

    0x080-0x088

    0x100-0x108

    etc....
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    I just want to do a continuous read, and want the write address to roll over after so many bytes or reset back to the base address. As an example, I want to transfer 2 MByte of data, to a 256 bit wide FIFO. I understand I could probably just chain up many small descriptors and increment the address in software, but why should I have to do that, the DMA engine should be capable of handling this. When the stride feature is enabled, the system always ends up in an irrecoverable state and the only way to fix that is to reprogram the FPGA and restart the PC.

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    What you describbed can be accomplisted with a MM-->ST mSGDMA (i.e. no write master) and if your FIFO is wider than the data path then a data format adapter will convert it over to the correct width. For example if the PCIe slave width is 64-bit with a burst length of 32 I would use the mSGDMA in MM-->ST mode with a 64-bit data path, max burst length of 32 and then use a 1:4 data format adapter to take four 64-bit beats and cram them into the 256-bit FIFO.

    Of course you can do this with a memory mapped FIFO but it would take some additional logic to funnel multiple addresses into the same word before pushing it into the FIFO. Most of the time users will create a FIFO with a really large address span and alias the addresses internally to push data in. I personally never use MM for FIFOs unless absolutely necessary and in your case I don't think that's the case at all. Avalon-ST exists for stuff like this so I would simplify things and use the most appropriate interfaces to get your job done instead of using MM for things it's not really meant for.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    12 years ago

    Hi BadOmen,

    I have the same Problem: I need two "as wide as can be"-FiFos for some outside-of-QSYS-Logic (one to write and one to read). I also took the MM-Solution, but your way seems to be the better one. I tried to implement it somehow but without success.

    Can you explain it for a dummy like me, which IP I have to implement and what has to be configured?

    Thanks

    Steffen
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    12 years ago

    I recommend using a pair of DMAs, one setup for MM-->ST and the other for ST-->MM. This means one DMA is capable of only reads and the other is only capable of writes. Since the master blocks contain a FIFO internally it would probably be sufficient to export the streaming port of each master block so that your external logic can access the FIFO directly.

    For the read DMA you would configure the dispatcher for MM to ST mode and connect the read master to it. For the write DMA you would configure the dispatcher to ST to MM mode and connect the write master to it.

    Also I recommend visiting the mSGDMA wiki page since I uploaded an update to the read master to fix a FIFO overflow bug.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    12 years ago

    OK, I managed to get a kind-of-working-design.

    I took two OnChip-FiFos and additionally I implemented four dispatcher. Two with a write- and two with a read-master.

    I connected the read-masters to the in-ports of the fifos and the write-masters to the out-port od the fifos.

    I adjusted my driver to support the four dispatchers and voilla everything worked fine. I can read out the values which i have written to the fifos.

    So far so good.

    Now I wanted to get my design in between.

    So I added two more fifos. Now two fifos are used to write data to the outside-of-qsys-system and two are used to read data in again. As a proof that this is also working, I connected the two pairs outside the qsys-system together and again did write to the write-fifos and read on the read-fifos (somehow obvious, isn't it?).

    And this worked also.

    I allowed the FiFos to backpressure the Data, because I need a ability to tell the fifos when I need new data.

    Now I have a problem with the exported connections anyway.

    The IPs I have written used the exported MM-Connections of the FiFos. This means: They have only a Data-Bus and a single readrequest. Everytime I asserted "high" on the rdreq-signal for one clock (on rising edge), I got the next Data from the MM-FiFo. I took the Data and read them out bytewise that means that I have done 8 reads on the 64bit wide data bus. I have to do that because I have to reduce them to 8bit width. (Please don't tell me there's a better way, because it worked).

    Back to the problem: now I connected the rdreq to the ready-Input of the FiFos where I want to read Data from. And I connected the 64bit Data bus to the 64bit output.

    The other two fifos are connected similarly. The Data bus leads to the 64bit Input of them and I connected a wrreq to the valid-Input.

    I compiled it and ........ SH..!!! (sorry for that!) Only corrupt Data. OK, I'm doing something terribly wrong, but what? The Fifos has 8Bit per symbol and 8Symbols per beat. Do they change the bits somehow?

    Thanks

    Steffen

    PS: See Attachment for a quick overview
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    12 years ago

    8-bit symbols means that the smallest unit of streaming data is a byte, and since the FIFO is 64-bit that means you require eight 8-bit symbols to fill up the width so that is setup correctly.

    In the case of the output FIFOs, as long as valid is high then you should be reading the correct data. For the input FIFOs you should only write valid data into the sink port when the ready signal is high otherwise you run the risk of overflowing the FIFO.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    12 years ago

    Overflowing is not my problem. My requirements accept losing data if nobody reads them out. They are not really nessesary for normal operation, just for debugging.

    My real Probölem is that the Data are corrupt. And I suppose I'm already reading them wrong.

    Hmmm, maybe I try to implement the valid Signal in my IP.

    Another Question: Does a Streaming FiFo hold the last value on its output if the ready signal is low?
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    12 years ago

    Actually I mean overflows may result in data corruption. On-chip memory based FIFOs are dual port memory with head and tail addressing so if those two addresses cross, corruption will occur if the over/under flow detection is disabled in the FIFO instantiation.

    For the FIFO sink (input) it drives ready high when the FIFO is not full. So when it drives ready high and your logic connected to it drive valid high the data is pushed into the FIFO.

    For the FIFO source (output) it drives the valid signal high when there is data buffered in the FIFO (i.e. not empty). So when the FIFO drives valid high and your logic drives the ready signal high the data is popped from the FIFO and it's expected to be captured by your logic, if your logic isn't ready for the data then it should drive the ready signal low.

    Data being valid is not determined by the ready signal, the valid signal represents when the data provided by the source port is valid. Ready is used as an indicator from the sink port to let the source port know when it's safe to move data. So you can have a source asserting valid and the sink deasserts ready and the data will remain presented until the sink is ready again. Likewise if the sink is ready but the source is not the sink will know this because the valid signal is driven low by the source port.

    In your diagram you are not using the valid signal for the output FIFO. So there is no way for your custom logic to know which cycles contain valid data and which do not. That data is coming from memory so it takes a while to propogate to the source port of that FIFO so you will need to use that valid signal to know when the data is valid otherwise you'll be reading in old/garbage data sometimes.

    Likewise in your diagram you show that your logic is not monitoring the ready signal of the input FIFO so in that case if the FIFO is full and data is written in anyway it will either be lost or cause corruption of the FIFO depending on whether that IP uses over/under flow protection. I don't know if the overflow protection is enabled in that IP but either way driving data into a sink port when ready is low is *not* allowed in the Avalon-ST spec and if you similated this design I suspect you'll run into an assertion due to not following the spec.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    12 years ago

    You're so totally right, yes. And I know, I have to implement the valid and ready signals in my logic. But I'm currently switching from MM to ST. I can't change my logic at the moment because it is tested and reviewed and stuff. It's company policy stuff (or something like that).

    Besides, I have already managed to write and read the correct data. It was a software failure. In my case I had to change the order of the bytes to write correct data to the fifos. And then everythin worked fine.

    You have also to know that I work with a kind of multiplexer/demultiplexer on the fifos. This means I take the 64bit data from the fifo, and make a 8bit wide stream from it. The 8bit stream has a lower clock as the fifo. So when the fifo-output is valid I now and then generate a one-clock-cycle ready signal to reqeust new data. After the request, I even wait for some clocks to get a valid output from the fifo.

    And on the input-fifo I handle things similary. I collect the data, write them to the output, generate a single valid signal, and don't change the output for a while until I have new Data.

    So my design works although I'm not following the spec, because I'm slow as hell!!! ;-)

    When I finally can implement the ready and valid signals, this would also mean that I can communicate better with the fifos and speed up the reading of data. (The speed of writing the data depends on other factors.)