Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
13 years ago

Dual port memory corruption

We have system that is using an M9K memory block for IPC between two nios2 cpus.

CPU A accesses it as tightly coupled data memory, CPU B via the data port and Avalon bus (no data cache on either cpu). The memory block is set to return OLD_DATA during concurrent read and write.

One of the memory locations contains the 'write pointer' into a ring buffer, this is written by CPU A (actually updated every 125us) and polled by CPU B (along with a couple of other locations) in its idle loop.

On one card the 'OLD_DATA' option doesn't seem to be working properly, CPU B reads 0x00015460 (lots of times), then 0x00015420 (invalid) followed by the valid 0x00015480 (I've added a double read & compare to detect the error).

So it looks as though the read has latched a part-changed value.

Does any one know if 'OLD_DATA' actually works correctly, or do we have a more general timing error.

23 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    That's supposed to work with M9K blocks. Do you have the 2nd port running on a difference clock frequency or have the latency set two 2 cycles by any chance? I wouldn't expect either to be a problem but it might be something to check out if that's the case.

    The behavior you are seeing sounds more like a timing problem to me, especially if you have the read latency set to 1 on the 2nd port. With a read latency of 1, the output of the RAM is not registered and perhaps CPU 2 reading the data is not seeing a settled copy of the data. If this is happening then the timing report should be flagging this as a violation (assuming the constraints are setup properly).

    Speaking of proper constraints.... I forget which version of the tools had this problem, but there was a bug that caused data corruption in the on-chip memories due to an incorrect timing constraint being made. I can't remember if it was 11.0 or 11.1 that had that issue but if you are using Quartus II without service packs you might want to grab a SP since that issue could result in what you are seeing.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    Everything is running off the same 100MHz clock.

    I don't remember there being options in the sopc builder for read latency - but it is a while since I've used it - any values are likely to be the defaults.

    This part of the fpga (it is a large Arria part) is a dual nios with 2 tightly coupled code and 3 data blocks booted from the PCIe slave -> Avalon bridge. I had intended to dual port the 'shared data' directly between two tightly coupled data ports, but CPU B doesn't make that many accesses (when not idle) and isn't that time/clock critical so accessing via the Avalon bus lets me read it out over PCIe for debug/diagnostics.

    The device also contains a big TDM switch, an FFT block for tone detect etc.

    I'm also not sure which version or Quatus they hw guys use. I think they are frightened to change because something might break!
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    Ok, if the memory was left at the defaults then it would probably be 1 cycle read latency. So the commands and write data will be pipelined into the memory but the readdata out of the memory will not pipelined.

    I dug around my email and found the memory corruption issue. Turns out it only affected DCFIFO at 11.1 (no service pack) so you shouldn't be running into this with the on-chip RAM component.

    I would have the hardware team take a look at the timing analysis reports to make sure they don't have failing paths from the on-chip memory over to CPU 2 (or from CPU 2 into the memory). If there are failing paths you could bump the read latency of that memory port to 2 to get some extra pipelining in the readdata path. The TCM connection would need to remain set to a read latency of 1 since that's the only latency Nios II supports with the tightly coupled connections. If the failing paths are into the memory then additional pipelining in the fabric will be needed, I would probably do this with a pipeline bridge so that you don't get pipeline stages all over the place.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    Apparantly the total design has quite a lot of timing errors (and I mean 1000s) for signals that cross between clock domains (not related to these nios cpu and their memory). Some of these signals are trully 'don't care' (eg led outputs), some are probably synchronised - but the tools haven't been told, and others will be bugs waiting to strike!

    Marking about 7 clock pairs as 'false paths' (all related to a specific 96MHz clock generated from an external 8MHz input) lets the rest of the fpga be synthesised without any timing errors - and the 'failing' board passes the test it was failing on.

    The suspicion is that in attempting to meet these impossible timing constraints so much fpga real estate was used that other paths also failed.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    What might be happening is without constraints the fitter is putting most of it's effort into paths that are failing due to a lack of constraints and causing other valid paths to fail in the process. The fitter optimizes based on slack times so if you had some paths that should be false paths and they have the worst slack times due to a lack of constraints the fitter will work extra hard on those paths. So it could be the fitter was working hard on improving the slack time to LEDs instead of focusing on on-chip paths like between Nios II and the dual-port RAM.

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    After removing a few clock domain crossing issues and removing all the 'red' timing paths, the rebuilt fpga image worked on the boards that were failing.

    However we've recently has some board fail the production tests (which now perform a test which is likely to show up this issue). Program the older fpga image and these boards work.

    The whole thing is very strange, I've added a memory test in the idle loop of both cpu (using a 32bit value whose low 16 bits are the bit-reversed 1's complement of the high bits - which increment). While this test gives occaisonal errors, the ring index fails 3-4 times as often even though the read/write collision is much more likely.

    The error has to be associated with the memory write, otherwise the following read would return the 'old' data - and that just dosn't happen.

    I've tried many thing to increase the error rate, but nothing seems to make a significant difference. The only other avalon slave (to that memory block) is the PCIe interface, but it won't be accessing that memory at all. Loading the PCIe slave makes no difference at all.

    The whole nios block (and quite a lot of other stuff) is running off the same 100MHz clock, there will be a bus width adapter and clock crossing bridge between the PCIe and any avalon slaves.

    We do have to put all the Avalon bus signals from the PCIe block through a lump of vhdl in order to get the correct addres lines (otherwise the BAR becomes massive, not just 32MB ), but that doesn't contain any logic - it just renames signals.

    About the only thing I haven't tried is accesses through the other 3 BARs. They go into different logic that is in the same SOPC build, but has no shared avalon master/slave parts.

    IIRC Fmax is a lot higher than 100MHz.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    I just looked at this doc and the section "Mixed-Port RDW" suggests that you can't use a dual clocking scheme with mixed ports and the old data option all at the same time: http://www.altera.com/literature/ug/ug_ram_rom.pdf

    If that's really true then that's a bug with the on-chip memory component. If this is true I'm not sure why Quartus allowed the compile to go through unless the memory component does something to the instantation that makes it work.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    Both ports to the memory should be on the same clock. The same same 100MHz clock is fed to the nios (using it as a tightly coupled data memory), the other nios (accessing via the avalon bus) and the memory interface itself.

    I've just had a thought, the only other avalon master (to this particular memory block) is the PCIe slave. That will require a clock crossing bridge and a bus-width adapter. Will those be generated separately for each slave? Or will the interface be converted once?

    Is there an easy way to tell??

    If the former, maybe we should add the explicit bridges?

    Plausibly having two separate sets of bridges might actually work best.

    One to map the top 16MB of the BAR to the SDRAM (optimised for burst transfers).

    The other to map the low 16MB to various on-chip resources (optimised for single word transfers, this could alias addresses at 256kB ).

    I'm not sure we've really looked at any of the avalon bridges.

    If relevant we are 'still' using Q9.1sp2, moving forwards is causing grief.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    13 years ago

    With SOPC Builder you will get clock crossing adapters and width adapters per master:slave pairing when they are needed. Clock crossing adapters can be spotted during generation time since you'll see messages like "clock_0" flying by the screen for each one. Finding the width adaptation logic is not so easy.

    So from the way you described the system assuming you only have the second Nios II data master and PCIe master hooked up to the second port of the on-chip RAM, I would only expect one clock crossing and one width adapter to be placed between the PCIe and memory cores.

    Not related to the problem but one thing you probably want to avoid is the additional bursting from the PCIe through the clock crossing adapter into the memory. Those automatic clock crossing adapters only let one beat through at a time and each takes a few clock cycles to make their way through. The clock crossing bridge will do a better job because the data crosses domains using FIFOs so you can fill the FIFO with a rate of one beat/clock cycle assuming there is room in the FIFO.

    All of this is starting to jog my memory, I can't believe I managed to find this but you could be running into this: http://www.altera.com/support/kdb/solutions/rd11022009_41.html

    I think the PCIe core performs bursts of 32 beats. So what you could do is setup a clock crossing bridge to be the width of the PCIe master and non-bursting. Place the clock crossing bridge between the PCIe core and the memory. What this will do is force the burst adapter to be placed between the bridge and PCIe core, and the width adaptation between the bridge and the memory. Place the clock crossing bridge master on the same domain as your CPUs and memory, and the bridge slave should be on the same domain as the PCIe master.