Hi,The following reference design is based on 1SM21BH device.https://fpgacloud.intel.com/devstore/platform/19.1.0/Pro/an881-pcie-avmm-dma-gen3x16-ddr4-and-hbm2/We use Stratix 10 MX 1SM21CH device, so we cannot synthesize this reference design.I guess the difference is HBM controller interface because of the size.Could you please provide a version for 1SM21CH device?This reference is important to us, or we don't know how to access HBM2/DDR4 by PCIe driver.Thanks.

Example and reference designs are starting points. This design is specifically for the Stratix 10 MX dev kit. If you take this design, you can change the target device and regenerate the IP. Since you're probably building your own board, you would probably need to make assignment changes, such as pin locations, anyway.#iwork4intel

Yes, thanks @sstrell Hi @CHung as mentioned by sstrell, you can use the reference design as reference, eg, which component to instantiate in Qsys, and how its connect. Some modification on the top level to suit your board connection. Then make pin assignment change.

Got it.Thanks for your answers.

Hi Intel support,I've tried it. I modified the device and upgrade all IP.The sof could be programmed to the FPGA successfully.Besides, I install the Linux driver of this reference design.However, it doesn't work.Linux cannot recognize this FPGA board which has been connected to motherboard PCIe port.We followed the steps on this user guide:https://www.intel.com/content/www/us/en/programmable/documentation/sox1520633403002.html#lmm1520633289264Linux didn't find the FPGA PCIe device when we entered the following command: lspci -d 1172:000 -v | grep intel_fpga_pcie_drvWe have no idea what's wrong.Perhaps Linux driver needs to be update for the new device? Or the pin assignment of the reference design is wrong?Or sof file has some problems?Please help to check this sof and Linux driver on your board.For your reference, our project files is as follows. https://drive.google.com/file/d/1mJJYheW6pBN08tSwS4-lgF3XSJfcn-pz/view?usp=sharingThanks.

Hi do you run the $ lsmod | grep intel_fpga_pcie_drv to ensure the driver is install successful?

AN881 reference design need to be upgraded for device 1SM21CH.

54 Replies

BoonT_Intel
Frequent Contributor
6 years ago
Sorry, one dump question. After you program the SOF, I believe you did restart the host right?
Let me upgrade the design to CHU at my side and send you the SOF to retry. In case the process to upgrade the design from BHU to CHU have problem at your side.
- CHung
  Occasional Contributor
  6 years ago
  Yes, we did.
  We 'll try your sof and let you know the result. Thanks!
BoonT_Intel
Frequent Contributor
6 years ago
Hi I attached the SOF, please give it a try.
test.zip6.6 MB
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  Programmer said the device is incompatible.
  Could you please generate it again using 1SM21CHU1F53E1VG? This is full name of device.
  Thank you.
BoonT_Intel
Frequent Contributor
6 years ago
Sorry, previously I use 1SM21CHU1F53E1VGS1 and able to re-generate the IP and recompile.
However, when i change the device to 1SM21CHU1F53E1VG, I am unable to generate the HBM IP with errors as below, I will work with internal team on this errors.
Can you briefly explain how you upgrade the design and able to generate the IP without error?
At the same time, maybe you can use other non-hbm design and upgrade to 1SM21CHU1F53E1VG and retry. The goal now is want to get a non working design that able to let linux detect the device.
Error(20327): 2019.11.13.15:18:28 Error: g3x16_hbm2: hbmc_0 has port axi_0_1_araddr declared with width 28 which is declared with width 29 in file hbmc_0.ip
2019.11.13.15:18:28 Error: g3x16_hbm2: hbmc_0 has port axi_0_0_awaddr declared with width 28 which is declared with width 29 in file hbmc_0.ip
2019.11.13.15:18:28 Error: g3x16_hbm2: hbmc_0 has port axi_0_1_awaddr declared with width 28 which is declared with width 29 in file hbmc_0.ip
2019.11.13.15:18:28 Error: g3x16_hbm2: hbmc_0 has port axi_0_0_araddr declared with width 28 which is declared with width 29 in file hbmc_0.ip
2019.11.13.15:18:28 Error: The system cannot be generated when there are errors.
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  I also met these errors when I used Quartus v19.3.
  You could try v19.1, it could upgrade HBM2 smoothly.
BoonT_Intel
Frequent Contributor
6 years ago
Other non-HBM design that I mentioned can be as below:
https://fpgacloud.intel.com/devstore/platform/18.1.0/Pro/pci-express-gen3-x16-avmm-dma-with-on-chip-memory-reference-design/
- CHung
  Occasional Contributor
  6 years ago
  Thanks.
  I'll try it.
BoonT_Intel
Frequent Contributor
6 years ago
Hi I need some time to setup the server at my side.
For your side, can you give a quick try by go to bios, change the slot PCIe gen to Gen2 instead.
Also, give a try on other non-hbm design like I suggest previously.
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  We noticed a strange phenomenon when changing BIOS setting.
  
  When AN881 SOF is programmed to FPGA
  If we set PCIe slot as AUTO detect, the computer boots successfully.
  If we set PCIe slot as gen3x16, gen2x16, or gen1x16, the computer cannot boot, and the beep code is 8-short which means "display memory read/write test failure".
  When nothing is programmed to FPGA
  computer boot successfully no matter which BIOS setting is applied.
  
  So we guess, BIOS did detect something when BIOS setting is not AUTO and FPGA has SOF.
  However, we cannot enter Linux normally under this case.
BoonT_Intel
Frequent Contributor
6 years ago
Maybe there are issue from the DDR4 or HBM that caused the rd/wr fail, as mentioned before, can you give a try by using the pcie to on-chip memory design and give it a try?
- CHung
  Occasional Contributor
  6 years ago
  Hi, yes, we tried the PCIe exampled without HBM2.
  The result was the same.
CHung
Occasional Contributor
6 years ago
Hi,

We changed Quartus version from v19.1 to v19.3, and now Linux could recognize the board!
All PCIe reference designs includes AN881 could be detected by "lspci" command.

However, there is still something wrong.
We use AN881 driver, it could read configuration space, but couldn't access HBM2/DDR4.

Besides, the AN881 document doesn't describe how to set DMA clearly.
https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/an/an881.pdf
What's the flow to use above gui?

Thanks.
- CHung
  Occasional Contributor
  6 years ago
  Hi
  
  Only "4. read configuration space" works, we could read vendor ID.
  "0. Link Test" we got 100 R/W/mismatch errors.
  "5. Change BAR" we got "Could not change BAR"
  Do you have any suggestion?
BoonT_Intel
Frequent Contributor
6 years ago
Hi Sir,
Can we get the signal tap waveform for now?
From the screenshot, I can see you are unable to change to DDR4. Can you type 2 and try to change it to HBM2?
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  Yes. The signalTap shows lane_act = 0x10, and ltssmstate = 0x11.
  If you need other signals, please let me know.
  
  We have tried -1, 2, and 4, all of the options showed "Could not change BAR!".
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  As below figure, it seems BAR setting has something wrong.
  
  The path is our FPGA PCIe device:
  /sys/bus/pci/devices/0000:01:00.0
  
  And then we see resources are zero:
BoonT_Intel
Frequent Contributor
6 years ago
Hi,
When you download the original reference design, the device is 1SM21CHU1F53E1VG or 1SM21CHU1F53E2VG?
The dev kit is using 1SM21CHU1F53E1VG.
I just got the MX BHU board today. When I attempt to program the board I just aware that the design is using 1SM21CHU1F53E2VG (even the AN881 documentation also mentioned E2 instead of E1.
Are you download the design and change it from E2 to E1 then regenerate the design?
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  The original reference design we downloaded from below website is 1SM21BHU2F53E2VG.
  https://fpgacloud.intel.com/devstore/platform/19.1.0/Pro/an881-pcie-avmm-dma-gen3x16-ddr4-and-hbm2/
  
  In all previous experiments, we changed the device to 1SM21CHU1F53E1VG.
- CHung
  Occasional Contributor
  6 years ago
  Add more information:
  
  1.
  We find that in the original reference design, BAR2 and BAR4 configuration of PCIe are disable, it's different from document.
  But driver still cannot change BAR after we enable them.
  
  2.
  Although driver can read PCIe configuration space,
  but the value seems not the same with AN881 document.
  Only Vendor ID is expected 0x1172.
BoonT_Intel
Frequent Contributor
6 years ago
Hi, I am able to setup the design and I also facing the same problem as yours. I will try to contact the design author and clarify.
One thing to check with you is when i run the driver, I am unable to see HBM2=2 and DDR4=4 n preset 5. I only can see change BAR -1=None. For this I am using the unzip software folder under the reference design. Are you using other software driver?
at the same time, can give a check with command as below to see if the BAR2 and BAR4 si detect in the system.
Lspci -nn | grep 1172 to check the BDF.
Eg, if theBDF is 5:0.0 then run command as blow:
Lspci -s 05:00.0 -vvv
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  I also use the driver from reference design, but mine is called software_hbm2_ddr4 as the attached file.
  Are you using the first one of following reference designs?
  https://fpgacloud.intel.com/devstore/platform/?acds_version=any&family=stratix-10&board=104
  Mine is AN881: PCIe AVMM DMA Gen3x16 DDR4 and HBM2 .
  
  Our message is:
  
  software_hbm2_ddr4.zip72 KB
  - CHung
    Occasional Contributor
    6 years ago
    Hi,
    
    After we enabled BIOS "above 4G decoding" setting, the BAR address and memory space could be allocated.
    Besides, we modified PCIe class code from default 0xff0000 to 0x068000 (Bridge device).
    Otherwise BIOS of our motherboard doesn't allocate BAR of FPGA.
    It seems you don't have this class code issue.
    
    But capabilities are still "access denied".
    Would it be a problem? Or it's normal?
    
    And then, we can change BAR number after above modification,
    but link test and DMA functions still fail.
    All the read value of link test is 0xffffffff, which is error code replied by system.
    Is your situation the same?
    Besides, the test program shows " Stopping DMA run due to error." when we trigger DMA function.
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  By the way, please contact the design author as soon as possible to see what's wrong.
  We have to make our deadline, because there will be a demo of our design to customers in the end of December.
  
  This FPGA board has been bought for almost 2 months.
  However, no reference design works so far.
  https://fpgacloud.intel.com/devstore/platform/?acds_version=any&family=stratix-10&board=104
  We assumed that at least one of above reference designs could prove this board's function is normal.
  But the simplest "on-chip memory reference design" cannot be installed. (I've asked another question about this issue on community)
  And the other three reference designs cannot work.
  
  If there is no way to prove the functionalities of this FPGA kit,
  maybe we need to return it to Intel.