Forum Discussion
Altera_Forum
Honored Contributor
Honored Contributoraltera_dma driver for PCIe
Hi everyone, i'm using an Arria V starter kit for PCIe application and i used the reference design posted in altera wiki. The application software reported success in one PC but not in other with different features. The driver code is installed in two with success but in one, user application reported timeout for read, write and simultaneous RW. Had anyone has the same problem? Have anyone some suggestion about what can i do to try solve this problem?
34 Replies
Altera_ForumHi there! I am having the same issue with timeout being reported for read and write. Have you found a solution for it? This is the output I see. I am using the driver for linux.
********************************************** ** ALTERA AVMM DMA driver ** ** version 2.02 ** ** 1) start DMA ** ** 2) enable/disable read dma ** ** 3) enable/disable write dma ** ** 4) enable/disable simul dma ** ** 5) set num dwords (256 - 2048) ** ** 6) set num descriptors (1 - 128) ** ** 8) loop dma ** ** 10) exit ** ********************************************** Run Read ? 1 Run Write ? 1 Run Simultaneous ? 0 Read Passed ? 0 Write Passed ? 0 Simultaneous Passed ? 0 Read EPLast timeout ? 1 Write EPLast timeout ? 1 Number of Dwords/Desc : 512 Number of Descriptors : 128 Length of transfer : 256 KB Rootport address offset : 0 Read Time : 1 s and 366248 us Read Throughput : 0.000179 GB/S Write Time : 1 s and 361611 us Write Throughput : 0.000179 GB/S Simultaneous Time : 1 s and 362982 us Simultaneous Throughput : 0.000358 GB/S- Altera_Forum
I am encountering the same problem, in which my DMA read and writes always time out.
When I tried the burst mode (loop dma), I got************************************************ ALTERA AVMM DMA driver ** ** version 2.02 ** ** 1) start DMA ** ** 2) enable/disable read dma ** ** 3) enable/disable write dma ** ** 4) enable/disable simul dma ** ** 5) set num dwords (256 - 2048) ** ** 6) set num descriptors (1 - 128) ** ** 8) loop dma ** ** 10) exit ** ********************************************** Run Read ? 1 Run Write ? 1 Run Simultaneous ? 1 Read Passed ? 0 Write Passed ? 0 Simultaneous Passed ? 0 Read EPLast timeout ? 1 Write EPLast timeout ? 1 Number of Dwords/Desc : 512 Number of Descriptors : 128 Length of transfer : 256 KB Rootport address offset : 0 Read Time : 1 s and 227400 us Read Throughput : 0.000199 GB/S Write Time : 1 s and 230367 us Write Throughput : 0.000198 GB/S Simultaneous Time : 1 s and 238158 us Simultaneous Throughput : 0.000394 GB/S
And by checking dmesg I got the following# DMA data error! Type in dmesg to show more details!
Does anyone have any clue why this would happen? Any suggestion or hint is much appreciated!Read DMA times out DWORD = 00000100 Desc = 00000080 Read DMA times out DWORD = 00000100 Desc = 00000080 Read DMA times out DWORD = 00000100 Desc = 00000080 Read DMA times out DWORD = 00000100 Desc = 00000080 Read DMA times out DWORD = 00000200 Desc = 00000080 Write DMA times out DWORD = 00000200 Desc = 00000080 Simultaneous DMA times out DWORD = 00000200 Desc = 00000080 command issued from user space doesn't exist 2608 command issued from user space doesn't exist 2608 - Altera_Forum
Have you checked if the FPGA has been programmed successfully and if the board has been enumerated properly in the PC? If using Linux, run the lsmod command and check if the driver is loaded. Have you checked if the device ID in the driver matches with the one in the board?
- Altera_Forum--- Quote Start --- Have you checked if the FPGA has been programmed successfully and if the board has been enumerated properly in the PC? If using Linux, run the lsmod command and check if the driver is loaded. Have you checked if the device ID in the driver matches with the one in the board? --- Quote End --- I second eapenabrm's suggestions. 1. Make sure the FPGA is programmed with the correct image. A simple way would be to connect a counter signal to an LED so it keeps blinking. 2. Check if lspci shows the Altera FPGA as a device ( On linux) or on the device manager ( on windows). Again check the device id in the PCIe QSYS component or what is returned by lspci are the same and make sure your driver is looking for the same device. 3. For the reads, I had made a mistake in connecting my QSYS components. The slave attached to the PCIe component did not return a readdatavalid, whereas the PCIe component expected it. Thus, I never "read" anything back. After I changed it, the reads started to work. 4. I would also suggest using a signal tap to monitor the BAR (register interface). You can trigger on reads or writes, while the driver is running. You can then see if the behaviour of the system is as expected on not. I used the following steps to setup the driver, and not their scripts
rmmod altera_dma.ko rf /dev/altera_dma cat /proc/devices # choose a device number that isn't already taken. In this case I have used 243 mknod /dev/altera_dma c 243 0 chmod 0644 /dev/altera_dma ls -al /dev/altera_dma insmod altera_dma.ko rm -f ./user/user gcc ./user/user.c -o ./user/user ./run - Altera_Forum
Hi eapenabrm,
Thank you for the reply! Sorry for having the following newbie questions :p:- Is there any other way to check if the FPGA has been programmed successfully besides seeing the "Programmer" in Quartus Prime shows "100% (successful)" in the progress bar?
- When you said "check if the board has been enumerated properly in the PC", I am assuming you mean I should execute
and see if the Altera device appears, right? If so, when the FPGA is not programmed with the reference design, the above command shows nothing. On the other hand, if the FPGA is programmed, I can see the following outputlscpi | grep Altera
Does this mean that it is enumerated?01:00.0 Non-VGA unclassified device: Altera Corporation Device e003 (rev ff) - May I ask how to check if the device ID matches with the one in the board? Is the "e003" in the above output the device ID the OS sees? Then how should I check whether the IP programmed on board has the same device ID?
- Using lsmod, I can see the following
or(in the cases where FPGA is programmed, lspci shows the Altera device, but /dev/altera_dma does not exist) Module Size Used by altera_dma 20480 0
some other clarification about how i got the fpga programmed I am using the "two-machine" method, as stated on page 13 in this document (https://www.altera.com/content/dam/altera-www/global/en_us/pdfs/literature/an/an456.pdf), to program the FPGA. However, there was only once I could get the "./install" and the "./run" of the ref design software to see the device /dev/altera_dma. In all the other attempts, I never be able to make /dev/altera_dma to show up again (meaning I will see "chmod: cannot access /dev/altera_dma: No such file or directory" when executing "sudo ./install" of the ref design software) dmesg shows the following relevant messages, but I am not sure what went wrong:(in the case where FPGA is programmed, lspci shows the Altera device, and /dev/altera_dma exist) Module Size Used by altera_dma 20480 2
Any suggestion is much appreciated!!!!altera_dma: module verification failed: signature and/or required key missing - tainting kernel Altera DMA: altera_dma_init(), Nov 7 2017 10:46:02 Altera DMA 0000:01:00.0: Refused to change power state, currently in D3 Altera DMA 0000:01:00.0: can't enable device: BAR 0 not claimed Altera DMA 0000:01:00.0: pci_enable_device() failed Altera DMA 0000:01:00.0: goto err_enable Altera DMA 0000:01:00.0: goto err_initchrdev Altera DMA 0000:01:00.0: goto err_bk_alloc Altera DMA: probe of 0000:01:00.0 failed with error -22 Altera DMA: altera_dma_init(), Nov 7 2017 10:46:05 Altera DMA 0000:01:00.0: Refused to change power state, currently in D3 Altera DMA 0000:01:00.0: can't enable device: BAR 0 not claimed Altera DMA 0000:01:00.0: pci_enable_device() failed Altera DMA 0000:01:00.0: goto err_enable Altera DMA 0000:01:00.0: goto err_initchrdev Altera DMA 0000:01:00.0: goto err_bk_alloc Altera DMA: probe of 0000:01:00.0 failed with error -22 - Altera_Forum
Have you tried changing the Device ID in the drivers code in "altera_dma_cmd.h" from E003 to E001 which is the default value of the Device ID for Altera dev kits. As per the documentation, this value should be E093 for the Arria 10 devices.
E - Gen3 PCIe 0 - Number of lanes (0 -1 lanes) 9 - Arria 10 Device / 7 - Arria V Devices / 0 - Stratix IV GX and so on. 3 - SGDMA mode , Desc/Data (that is, Avalon-Streaming (Avalon-ST) interface) IF E001 does not work, then try with E093. After modifying the Device ID, you may need to make and install the driver again, ie, ./make , ./install or sudo make / sudo install,etc.. - Altera_Forum
Hi eapenabrm,
Thank you for the tips! Unfortunately, none of these two Device ID's work for me . I found that
As shown, the two memory blocks are unassigned (and disabled?). Could this be related to the dmesg error message "altera_dma: module verification failed: signature and/or required key missing - tainting kernel"? Also, I looked into the system device tree, I can see the device at /sys/bus/pci/devices/0000:01:00.0. Under the folder of that device I can see the$ lspci -vv | grep Altera -A 10 01:00.0 Non-VGA unclassified device: Altera Corporation Device e003 (rev 01) Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at <unassigned> (64-bit, prefetchable) Region 4: Memory at <unassigned> (64-bit, prefetchable)
, but if I do cat /proc/devices, I can't see altera_dma in the list. Unfortunately, the altera_dma_load script tries to identify the major ID of the device from /proc/devices and mknod with that major ID, so it couldn't get pass this step. At this point I feel like to issue was because of something in the driver, but I am not sure where exactly the problem is. Could you please give me some suggestion where I can look into further? Any suggestion is much appreciated!!!$ ls /sys/bus/pci/devices/0000\:01\:00.0 broken_parity_status device irq numa_node resource subsystem class dma_mask_bits local_cpulist power resource0 subsystem_device config driver_override local_cpus remove resource0_wc subsystem_vendor consistent_dma_mask_bits enable modalias rescan resource4 uevent d3cold_allowed firmware_node msi_bus reset resource4_wc vendor - Altera_Forum
Which kit are you using the Arria V or the Arria 10 Dev Kit? When building the example design, did you notice the PCIe config space registers. They usually have these Vendor ID and DEvice ID registers in it, with the Vendor ID preset to the Altera one and Device ID being 00000 . If this is the case with your design, then you may need to set the appropriate DEvice ID in your PCIe component (E0073 for Arria V / E0093 for Arria 10/ or E001 for default) and build the system again. Program the FPGA on the board. Then in the driver code make sure you use the same Device ID that is in the HW and build the driver again. This way the Device ID will match. Once the Device is enumerated successfully, the BAR registers will get enabled and you should be able to read and write from them, perform DMA etc.
- Altera_Forum
Lets start from the beginning again.
If you're Dev it is the Arria V Starter kit then download the following files: 1. Hardware design - http://www.alterawiki.com/wiki/file:pcie_avgx_avmm_dma_on_chip_mem_1602.zip This is the Rev 3 file set for the PCIe SGDMA reference design 2. Software Driver & App : http://www.alterawiki.com/wiki/file:linux_for_avmm_dma_on_chip_mem.tar.gz This is the respective driver and App. For the Arria V GT Hardware design : http://www.alterawiki.com/wiki/file:pcie_avgt_avmm_dma_on_chip_mem_1602.zip Software files are the same. The Device ID for this design is set to E003 and need not be modified. 1. Unzip the HW & SW files. 2. Load the HW project in Quartus. If you're using the latest version 17.x an IP upgrade window may pop-up. Perform the upgrade to get the latest design files and IP. 3. Check that the modules are all properly connected in Platform Designer (Qsys) and that there are no errors during the IP upgrade process. 4. Build the project and generate the programming files. 5. Program the FPGA on the kit by following the provided instructions. Make sure the system setup is replicated as mentioned in the Reference design. 6. In the Linux system , build and install the driver. You may encounter an error when installing the drivers for the first time . This is expected. 7. Make and install the accompanying Application. 8. Run the App and follow the steps to get the DMA run. - Altera_Forum
Hi divyaganti,
Thank you for the suggestion! Sorry that I missed your message earlier. I haven't gotten a chance to try bullets 1,3 & 4 in your suggestion (I'll try them out and keep you posted), but I tried your steps for driver installation. I did pick a major id that hasn't been taken by the devices shown in /proc/devices, and I did see the following
, and insmod didn't throw any complain. Unfortunately, when I try to run the software as root, the ./run script still throwsroot# mknod /dev/altera_dma c 243 0 root# chmod 0644 /dev/altera_dma root# ls -al /dev/altera_dma crw-r--r-- 1 root root 243, 0 Nov 9 12:48 /dev/altera_dma
I had made the device ID consistent on both sides (both are e003 for now). I had planned to change the device side DID to e093 as suggested by eapenabrm, but for some reason when I tried to recompile the design after changing the device ID register in Qsys, the assembler, which was the final step, failed and complainedCouldn't open the device.
So I haven't been able to test using e093 as the device ID yet. Does anything in the above description look weird to you? Any further suggestion is much appreciated!!! --- Quote Start --- I second eapenabrm's suggestions. 1. Make sure the FPGA is programmed with the correct image. A simple way would be to connect a counter signal to an LED so it keeps blinking. 2. Check if lspci shows the Altera FPGA as a device ( On linux) or on the device manager ( on windows). Again check the device id in the PCIe QSYS component or what is returned by lspci are the same and make sure your driver is looking for the same device. 3. For the reads, I had made a mistake in connecting my QSYS components. The slave attached to the PCIe component did not return a readdatavalid, whereas the PCIe component expected it. Thus, I never "read" anything back. After I changed it, the reads started to work. 4. I would also suggest using a signal tap to monitor the BAR (register interface). You can trigger on reads or writes, while the driver is running. You can then see if the behaviour of the system is as expected on not. I used the following steps to setup the driver, and not their scriptsError (210027): Can't use configuration device EPCQL256 with selected programming mode
--- Quote End ---rmmod altera_dma.ko rf /dev/altera_dma cat /proc/devices # choose a device number that isn't already taken. In this case I have used 243 mknod /dev/altera_dma c 243 0 chmod 0644 /dev/altera_dma ls -al /dev/altera_dma insmod altera_dma.ko rm -f ./user/user gcc ./user/user.c -o ./user/user ./run
