Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
7 years ago

Arria 10 PCIe Avalon-MM DMA Reference Design: Read/Write Timeout

Hi guys,

I tried searching through the forum for this and found some posts on it but no solution to the issue. I am currently in possession of a Nallatech 385a PCIe card with an Arria 10 on it. I am running this on CentOS 7 and using Quartus Prime version 17.1. What I've done with the Avalon-MM DMA reference design ( https://www.altera.com/en_us/pdfs/literature/an/an690.pdf ) is change the device and the pin assignments. The reference design archive package I am using is "hip_a10gx_g3_x8_avmm_dma256_1602_PS.qar". I am able to generate the HDL, fully compile, and program the .jic file into flash and restart the PC so the design saves and re-enumerates the PCIe bus properly. I am also able to make the test throughput code, install the driver, and run the program. However, the read/write test is timing out. I am able to see the device in lspci and output its details. Attached are screenshots of Quartus and the terminal output is below. Any and all help is greatly appreciated! 


$ cd Downloads/
$ cd avmm_dma_linux/
$ su
Password: 
# make
make -C /lib/modules/3.10.0-862.2.3.el7.x86_64/build M=/home/arclab/Downloads/avmm_dma_linux
make: Entering directory `/usr/src/kernels/3.10.0-862.2.3.el7.x86_64'
  Building modules, stage 2.
  MODPOST 1 modules
make: Leaving directory `/usr/src/kernels/3.10.0-862.2.3.el7.x86_64'
# ./install
rm -rf *.o *~ core .depend .*.cmd *.ko *.mod.c .tmp_versions *.symvers *.order
make -C /lib/modules/3.10.0-862.2.3.el7.x86_64/build M=/home/arclab/Downloads/avmm_dma_linux
make: Entering directory `/usr/src/kernels/3.10.0-862.2.3.el7.x86_64'
  LD      /home/arclab/Downloads/avmm_dma_linux/built-in.o
  CC   /home/arclab/Downloads/avmm_dma_linux/altera_dma.o
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c: In function ‘init_rp_mem’:
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:272:9: warning: unused variable ‘increment_value’ 
     u32 increment_value = 0;
         ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c: In function ‘rp_ep_compare’:
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:287:9: warning: unused variable ‘count’ 
     u32 count = 1;
         ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:286:12: warning: unused variable ‘rp_tmp’ 
     u32 j, rp_tmp, ep_tmp;
            ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c: In function ‘dma_test’:
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:362:13: warning: unused variable ‘ep_tmp’ 
 u32 rp_tmp, ep_tmp;
             ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:362:5: warning: unused variable ‘rp_tmp’ 
 u32 rp_tmp, ep_tmp;
     ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:361:12: warning: unused variable ‘j’ 
     int i, j;
            ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:360:63: warning: unused variable ‘simul_write_count’ 
     int loop_count = 0, num_loop_count = 1, simul_read_count, simul_write_count;
                                                               ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:360:45: warning: unused variable ‘simul_read_count’ 
     int loop_count = 0, num_loop_count = 1, simul_read_count, simul_write_count;
                                             ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:360:25: warning: unused variable ‘num_loop_count’ 
     int loop_count = 0, num_loop_count = 1, simul_read_count, simul_write_count;
                         ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:360:9: warning: unused variable ‘loop_count’ 
     int loop_count = 0, num_loop_count = 1, simul_read_count, simul_write_count;
         ^
In file included from /home/arclab/Downloads/avmm_dma_linux/altera_dma.c:15:0:
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c: At top level:
/home/arclab/Downloads/avmm_dma_linux/altera_dma.h:171:12: warning: ‘rp_compare’ declared ‘static’ but never defined 
 static int rp_compare(u8 *virt_addr1, u8 *virt_addr2, u32 num_dwords);
            ^
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c: In function ‘rp_ep_compare.isra.5’:
/home/arclab/Downloads/avmm_dma_linux/altera_dma.c:313:14: warning: ‘ep_tmp’ may be used uninitialized in this function 
        printk(KERN_DEBUG "ep_tmp = %08x\n", ep_tmp);
              ^
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/arclab/Downloads/avmm_dma_linux/altera_dma.mod.o
  LD   /home/arclab/Downloads/avmm_dma_linux/altera_dma.ko
make: Leaving directory `/usr/src/kernels/3.10.0-862.2.3.el7.x86_64'
#
**********************************************
** ALTERA AVMM DMA driver                   **
** version 2.02                              **
** 1) start DMA                             **
** 2) enable/disable read dma               **
** 3) enable/disable write dma              **
** 4) enable/disable simul dma              **
** 5) set num dwords (256 - 2048)           **
** 6) set num descriptors (1 - 128)         **
** 8) loop dma                              **
** 10) exit                                 **
**********************************************
Run Read                ? 1
Run Write               ? 1
Run Simultaneous        ? 1
Read Passed             ? 0
Write Passed            ? 0
Simultaneous Passed     ? 0
Read EPLast timeout     ? 1
Write EPLast timeout    ? 1
Number of Dwords/Desc   : 512
Number of Descriptors   : 128
Length of transfer      : 256 KB
Rootport address offset : 0
Read Time               : 0 s and 249767 us
Read Throughput         : 0.000978 GB/S
Write Time              : 0 s and 216415 us
Write Throughput        : 0.001128 GB/S
Simultaneous Time       : 0 s and 270368 us
Simultaneous Throughput : 0.001807 GB/S#  
$ sudo lspci -s 01:00.0 -vvv
 password for arclab: 
01:00.0 Non-VGA unclassified device: Altera Corporation Device e003 (rev 01)
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 33
    Region 0: Memory at f0120000 (64-bit, prefetchable) 
    Region 4: Memory at f0100000 (64-bit, prefetchable) 
    Capabilities:  MSI: Enable+ Count=1/4 Maskable- 64bit+
        Address: 00000000fee003d8  Data: 0000
    Capabilities:  Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities:  Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
        LnkCap:    Port# 1, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <4us, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
    Capabilities:  Virtual Channel
        Caps:    LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:    ArbSelect=Fixed
        Status:    InProgress-
        VC0:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:    Enable+ ID=0 ArbSelect=Fixed TC/VC=01
            Status:    NegoPending- InProgress-
    Capabilities:  Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
    Capabilities: # 19
    Capabilities:  Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
    Kernel driver in use: Altera DMA
$ 
$ ls /dev/ -l
total 0
crw-rw-rw-.  1 root root    243,   0 May 16 19:16 altera_dma

7 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    7 years ago

    Some additional output from dmesg: 

    
 altera_dma: loading out-of-tree module taints kernel.
 altera_dma: module verification failed: signature and/or required key missing - tainting kernel
 Altera DMA: altera_dma_init(), May 18 2018 17:29:59 
 Altera DMA 0000:01:00.0: pci_enable_device() successful
 Altera DMA 0000:01:00.0: irq 33 for MSI/MSI-X
 Altera DMA 0000:01:00.0: pci_enable_msi() successful
 Altera DMA 0000:01:00.0: using a 64-bit irq mask
 Altera DMA 0000:01:00.0: irq pin: 1
 Altera DMA 0000:01:00.0: irq line: 11
 Altera DMA 0000:01:00.0: irq: 33
 Altera DMA 0000:01:00.0: request irq: 11
 Altera DMA 0000:01:00.0: BAR 0xf0110000-0xf01101ff flags 0x0014220c, length 512
 Altera DMA 0000:01:00.0: BAR 0x00000000-0x00000000 flags 0x00000000, length 0
 Altera DMA 0000:01:00.0: BAR 0x00000000-0x00000000 flags 0x00000000, length 0
 Altera DMA 0000:01:00.0: BAR 0x00000000-0x00000000 flags 0x00000000, length 0
 Altera DMA 0000:01:00.0: BAR 0xf0100000-0xf010ffff flags 0x0014220c, length 65536
 Altera DMA 0000:01:00.0: BAR 0x00000000-0x00000000 flags 0x00000000, length 0
 Altera DMA 0000:01:00.0: BAR mapped to 0xffffabf100e46000, length 512
 Altera DMA 0000:01:00.0: BAR mapped to 0xffffabf100ee0000, length 65536
 Read DMA times out
 DWORD = 00000200
 Desc = 00000080
 Write DMA times out
 DWORD = 00000200
 Desc = 00000080
 Simultaneous DMA times out
 DWORD = 00000200
 Desc = 00000080
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    7 years ago

    Still struggling to understand what's going on here. Same error on both Windows 10 and CentOS. I have a dual boot setup on the same PC where the card is installed. Extension cable is not used here. I've also attempted to change the memory maps for RD/WR master and the device ID, still the same result. I never have an issue with seeing the device nor is the BARs disabled, I just can't read/write to the on chip RAM. I'm at a loss here as to what the issue would be.

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    7 years ago

    I have successed run PCIE reference design on my a10 board.

    I change the device and the pin assignments, then compiled.

    I think you should check your pin assignment.
  • arod412's avatar
    arod412
    Icon for New Contributor rankNew Contributor
    7 years ago

    So after months of trying to figure out what happened I finally have a solution for this. The npor and perstn signals must both be set to the reset line. Or at least that's how I got it to work. In my setup, I left the npor signal "open" and only connected perstn. I did this because I had found posts on this about which one to connect and it only mentioned perstn. The documentation also states that npor is an OR output of perstn and some other signal. After connecting both perstn and npor to the same reset the card started functioning with no problem. This makes sense since on SignalTap I only had the clocks working but the card seemed dead as in no other signals would change.

    For people searching and finding this topic, this was my environment:

    CentOS 6.9

    Nallatech N385a (Arria 10 [10AX115N3F40E2SG])

    Quartus Prime Pro 18.0.1

    I was able to verify the qsys design, which was the Arria 10 Reference Design, using Dave's program found here:

    http://www.alteraforum.com/forum/showthread.php?t=35678

  • Jose_G_Intel3's avatar
    Jose_G_Intel3
    Icon for New Contributor rankNew Contributor
    6 years ago

    Hello arod412,

    So in order to fix this issue did you connected npor and perstn signals to reset instead of configuring them as conduit?

    Raul

  • guna1's avatar
    guna1
    Icon for New Contributor rankNew Contributor
    2 years ago

    so connecting npor and persetn to which reset signal?

    did it work? timeout error is not there?

  • homeexpert12's avatar
    homeexpert12
    Icon for New Contributor rankNew Contributor
    2 years ago

    In the case of the "Arria 10 PCIe Avalon-MM DMA Reference Design: Read/Write Timeout" issue, it's essential to check for potential configuration errors or data transfer bottlenecks. This troubleshooting process resembles the attention to detail required in the UAE embassy attestation. I am sure it's right way.