mSGDMA: Incorrect operation but the behavior changes when SignalTapII is inserted.

Question

Hi all,

Please help me.

I am using two mSGDMA modules to read and write DDR3 on DE5-NET board with quartus ii ver 14.1.1. The mSGDMA is of the modules of Qsys ver 14.1.

The problem that I cannot solve is:

1) msgdma fails for some lengths of bytes, while it works for other lengths. This design is compiled without SignalTap II.

2) the behavior changes when i put signaltap ii to see registers and ports of the mSGDMA modules. Finally, the design works completely with a big signaltap ii.

The design includes:

1. PCIe-module -BAR2 is used to read and write csr, desc, and response of mSGDMAs

2. two ddr3 memories, ddr3_b and ddr3_a. ddr3_b is a master while ddr3_a shares oct, pll, and dll of ddr3_b as a slave.

3. ddr3_b and ddr3_a have 2GB each, and have memory address spaces of 0x0000_0000-0x7FFF_FFFF and 0x8000_0000 to 0xFFFF_FFFF, respectively.

4. msgdma_ur1 is of mm-to-st, and reads ddr3_b and ddr3_a.

5. msgdma_uw1 is of st-to-mm, and writes ddr3_b and ddr3_a.

6. st-src of msgdma_ur1 is connected directly to st-sink of msgdma_uw1.

The reason why I don't use a single mSGDMA of MM-to-MM, I will insert a stream-computing module between the two mSGDMAs.

The current design is for testing mSGDMAs and a control software.

7. The internal ports of ddr3_b and ddr3_a are mm-slave of a 512-bit width @ 200mhz.

8. The mm-master of msgdma_ur1 and msgdma_uw1 is of a 512-bit width @ 150mhz.

They each have configurations of

burst length is 0x20

command FIFO depth 128

descriptor FIFO depth 128

9. I put clk_crossing_bridge between the mSGDMAs and DDR3 memories.

10. The ddr3 memories works correctly when I read and write data to/from a host PC using other SGDMAs.

All the following data transfer tests satisfy 512-bit alignment and burst boundary.

When I use the design compiled without SignalTap II, I get:

+ 1 KB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : passed

+ 4 KB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : passed

+ 32 KB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : passed

+ 64 KB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : no data transferred

+ 128 KB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : no data transferred

+ 256 KB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : no data transferred

+ 512 KB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : no data transferred

+ 1024 KB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : no data transferred

+ 2048 KB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : no data transferred

+ 32 MB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : passed

+ 128 MB from 0x00000400 (ddr3_b) to 0x80000000 (ddr3_a) : passed

These results are reproduced for repeated tests with various data lengths and addresses.

The error of "no data transferred" seems to be caused by the destination addresses corruption.

For example, when I transfer data to 0x8000_0000, the actual destination can be 0x0000_0000.

When I use the design compiled with SignalTap II, tests for all the data length are passed.

So I cannot know what are happen in the mSGDMA modules.

(I observed correct descriptors and control values are written via BAR2 of PCIe.)

I doubted timing of the design, however, I made sure that sufficient constrains are given for all clocks including ddr3's 200MHz and mSGDMA's 150MHz

(generated by pll with external 50MHz). time-quest reports all the conditions are met.

So now I don't know what I should do to solve this problem.

Any hints and suggestions are helpful.

--

Mickycat

altera_forum · Answer

--- Quote Start ---   8. The mm-master of msgdma_ur1 and msgdma_uw1 is of a 512-bit width @ 150mhz.     They each have configurations of      burst length is 0x20     command FIFO depth 128     descriptor FIFO depth 128   --- Quote End ---    When I disabled the burst transfer, then the mSGDMAs work completely. However, the DDR3 memory throughput gets very low. I need burst transfer.  For max burst count = 2, 8, and 64, the mSGDMA modules have the same failure as that for max burst count = 32.  Mickycat

altera_forum · Answer

Hi,

I found that

1) mSGDMA core gets easier to operate correctly when I turn off "burst enable."

2) However, sometimes the mSGDMA with burst disabled can fail to transfer data.

With signal-tap II, I found that this failure is caused by incorrect latch to length_counter and/or

address_counter in Write_master. For example, when descriptor port of Dispatcher receives

0x0001_4000 as a length, 0x0000_4000 is written into the length_counter in Write_master

of a ST-MM mSGDMA. Since MM-ST mSGDMA sends 0x0001_4000 elements of data to the

ST-MM mSGDMA, the former mSGDMA is kept busy. And then the next DMA transfer fails.

"src_write_master_data" (a command from Dispatcher to Write_mater) has a length of

0x0000_4000, instead of 0x0001_4000. So Dispatcher wrongly works. I doubt

the_write_signal_breakout module in the_descriptor_buffers module.

The both MM-ST and ST-MM mSGDMAs are operating at 200MHz of afi_clk of DDR3 Uniphy

memory controller. The DDR3 memories operate correctly, because I can transfer data between

a host PC and the FPGA via PCI-Express. So I doubt something like timing-violation, while

TimeQuest saids no negative slack found for all clocks and conditions.

(I rechecked all the timing constraints are given sufficiently.)

Are there anyone who experienced similar phenomena?

Your any information is very helpful.

--

Mickycat

altera_forum · Answer

--- Quote Start ---