Hi,The following reference design is based on 1SM21BH device.https://fpgacloud.intel.com/devstore/platform/19.1.0/Pro/an881-pcie-avmm-dma-gen3x16-ddr4-and-hbm2/We use Stratix 10 MX 1SM21CH device, so we cannot synthesize this reference design.I guess the difference is HBM controller interface because of the size.Could you please provide a version for 1SM21CH device?This reference is important to us, or we don't know how to access HBM2/DDR4 by PCIe driver.Thanks.

Example and reference designs are starting points. This design is specifically for the Stratix 10 MX dev kit. If you take this design, you can change the target device and regenerate the IP. Since you're probably building your own board, you would probably need to make assignment changes, such as pin locations, anyway.#iwork4intel

Yes, thanks @sstrell Hi @CHung as mentioned by sstrell, you can use the reference design as reference, eg, which component to instantiate in Qsys, and how its connect. Some modification on the top level to suit your board connection. Then make pin assignment change.

Got it.Thanks for your answers.

Hi Intel support,I've tried it. I modified the device and upgrade all IP.The sof could be programmed to the FPGA successfully.Besides, I install the Linux driver of this reference design.However, it doesn't work.Linux cannot recognize this FPGA board which has been connected to motherboard PCIe port.We followed the steps on this user guide:https://www.intel.com/content/www/us/en/programmable/documentation/sox1520633403002.html#lmm1520633289264Linux didn't find the FPGA PCIe device when we entered the following command: lspci -d 1172:000 -v | grep intel_fpga_pcie_drvWe have no idea what's wrong.Perhaps Linux driver needs to be update for the new device? Or the pin assignment of the reference design is wrong?Or sof file has some problems?Please help to check this sof and Linux driver on your board.For your reference, our project files is as follows. https://drive.google.com/file/d/1mJJYheW6pBN08tSwS4-lgF3XSJfcn-pz/view?usp=sharingThanks.

Hi do you run the $ lsmod | grep intel_fpga_pcie_drv to ensure the driver is install successful?

AN881 reference design need to be upgraded for device 1SM21CH.

54 Replies

sstrell
Super Contributor
6 years ago
Example and reference designs are starting points. This design is specifically for the Stratix 10 MX dev kit. If you take this design, you can change the target device and regenerate the IP. Since you're probably building your own board, you would probably need to make assignment changes, such as pin locations, anyway.
#iwork4intel
BoonT_Intel
Frequent Contributor
6 years ago
Yes, thanks @sstrell
Hi @CHung as mentioned by sstrell, you can use the reference design as reference, eg, which component to instantiate in Qsys, and how its connect. Some modification on the top level to suit your board connection. Then make pin assignment change.
CHung
Occasional Contributor
6 years ago
Got it.
Thanks for your answers.
CHung
Occasional Contributor
6 years ago
Hi Intel support,

I've tried it.
I modified the device and upgrade all IP.
The sof could be programmed to the FPGA successfully.
Besides, I install the Linux driver of this reference design.
However, it doesn't work.
Linux cannot recognize this FPGA board which has been connected to motherboard PCIe port.

We followed the steps on this user guide:
https://www.intel.com/content/www/us/en/programmable/documentation/sox1520633403002.html#lmm1520633289264
Linux didn't find the FPGA PCIe device when we entered the following command:
lspci -d 1172:000 -v | grep intel_fpga_pcie_drv

We have no idea what's wrong.
Perhaps Linux driver needs to be update for the new device?
Or the pin assignment of the reference design is wrong?
Or sof file has some problems?

Please help to check this sof and Linux driver on your board.
For your reference, our project files is as follows.
https://drive.google.com/file/d/1mJJYheW6pBN08tSwS4-lgF3XSJfcn-pz/view?usp=sharing

Thanks.
BoonT_Intel
Frequent Contributor
6 years ago
Hi do you run the $ lsmod | grep intel_fpga_pcie_drv to ensure the driver is install successful?
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  Yes, we did, and Linux showed message as expected.
  
  By the way,
  as we know, Linux and Window OS could find PCIe devices even the driver is not installed.
  If the driver is not installed successfully, it should list an unknown device with ID information.
  However, after we entered "lspci" command, it didn't show any device new.
  So we think the hardware must have something wrong.
  (e.g. SOF file is not correct for some reason)
  
  We have asked Arrow's AE in Taiwan.
  But they and Intel's AE in asia-pacific area don't have this FPGA development kit at hand.
  If it's possible, please find someone who has this board to try the reference design.
  It seems not that straightforward to upgrade reference design from 1SMBHU device to 1SMCHU device.
  Thanks.
BoonT_Intel
Frequent Contributor
6 years ago
I also only have BHU MX board. I will suggest you download the 16G dev kit package and run the PCIE BTS test.
https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug-intel-s10-mx-devl-kit.pdf
https://www.intel.com/content/dam/altera-www/global/en_US/support/boards-kits/stratix10/mx_fpga/stratix10MX_1sm21chu1f53_fpga_revA_v18.1.2b277_v1.0.zip
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  We have tried BTS, and BTS PCIe function is OK.
  But BTS works by JTAG port, not PCIe connect.
  So we still don't know how to let Linux communicate with FPGA through PCIe connector.
BoonT_Intel
Frequent Contributor
6 years ago
Hi I agree that even driver is not install successful, it still able to detect by the host as unknown device. So now we can rule out the driver issue.
Suggestion:
is it possible to borrow the BHU variant board from field team and test it with original AN881 design using the same linux host? The reason is to rule out is issue is due to variant or the particular host actually unable to detect any FPGA gen3 card.
What is the slot PCIe capability? Have you tried go into the bios and fix the PCIe slot setting to Gen3x16? In some host, it will create some problem to detect when the setting is AUTO.
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  1. We'll try it.
  2. After confirmation, the slot PCIe capability is gen3 x16.
  We do the experiments on different computers with different CPUs, motherboards, and OS (CentOS 7.0 and Ubuntu 16).
  The result is the same.
  Besides, we have made sure the PCIe slot works well by changing PCIe cards.
  When we plugged-in other PCIe cards like nVidia graphic card,
  "lspci" command could show PCIe device.
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  Another question, is the pin assignment of 1SM21BHU the same with 1SM21CHU device?
  Currently all the user guides and documents are based on 1SM21BHU.
  We don't know whether there is difference except HBM2 size.
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  We have borrowed a BHU variant board today.
  After programming SOF of AN881 reference design (originally based on 1SM21BHU) and BTS PCIe image,
  Linux still couldn't recognize it by lspci command.
  We also tried 2 hosts, one of them is CentOS 7.0 and the other is Ubuntu 16, both got the same result.
  Besides, we tried different PCIe slots on the motherboard.
  
  If on your side, Linux could recognize the BHU board, could you provide your Linux version (include kernel version), installation files, and your sof?
  I heard that some Linux kernels don't accept PCI devices which is not certificated. Perhaps lspci command won't list the unacceptable devices. But I didn't know the detail.
  Is that possible it's Linux version related issue?
  Thanks.
BoonT_Intel
Frequent Contributor
6 years ago
Hi Yes, they should same. I check all the design from the dev kit package, all the pin location are same. As per my understanding, the different between both board is just the HBM density.
BoonT_Intel
Frequent Contributor
6 years ago
They are the same, I check the example design in edv kit package. All pin location is compatible.
This this moment, maybe you should signalap the PCIe signal and understand where the problem come from. Example of signal tap signals as below:
- CHung
  Occasional Contributor
  6 years ago
  OK. I'll do it.
  Thank you.
- CHung
  Occasional Contributor
  6 years ago
  Hi
  
  We have dumped PCIe signalTap waveforms (.jpg and .vcd), please download here:
  http://gofile.me/4v7Zs/5QrPiwKwN
  
  pcie_reset_trigger
  pcie_reset_release
  avmm_bridge_512_0_rst_n_release
  pcie_idle
  
  1, 2, 3 are the waveform around system reset.
  4 is the waveform after reset.
  Because we are not familiar with PCIe protocol and signals inside IP,
  could you please help to analyze them?
  It seems clocks and reset signals are OK. But Linux could not recognize it.
  Thanks.
BoonT_Intel
Frequent Contributor
6 years ago
Hi I am unable to get the waveform due to security reason. Can you please attach the screenshot here.
What you need to do is to capture the ltssmstate and lane_act signal to understand the Pcie issue
https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug_s10_pcie_avmm.pdf#page=80
- CHung
  Occasional Contributor
  6 years ago
  Hi,
  
  1. when reset is asserted:
  
  2. reset is de-asserted
  
  3. avmm_bridge_512_0 reset is de-asserted
  4. idle