Hello, We are having trouble bringing up the PCIe Hard IP on our custom board. The hard IP is set up for gen 1 x1. It appears that coreclkout is stuck low and not toggling. Current setup: We are using a FPGA project based on https://fpgacloud.intel.com/devstore/platform/18.0.0/Pro/cyclone-10-gx-pcie-gen1-x1-avl-st/ The only changes we made are pin assignments for the general IO to match our custom board, and using Quartus Pro 20.4.0. The custom board uses the Cyclone 10 GX and the hardware is set up for up to 2 lanes. The TX and RX lanes location have been verified to be correct using pin planner. They are located at bank 1C ch4 (lane0) and 5 (lane1). We also verified that the refclk is 100MHz coming from another custom board generated from a CDCM9102RHBT clock generator. nPERST is controlled by the root complex and located at the dedicated pin (NPERSTL0) with VCCIO at 1.8, and we can see the signal goes high using signaltap The pcie RX lane and refclk are set as CML I/O standard, and the TX lane is set at "high speed differential IO" From our root complex, we can see that the lttsmstates changes from 0x00 > 0x01 > 0x06 > 0x01 > 0x02 > 0x03 and it stays there. From the Cyclone 10 GX endpoint side using signaltap, the coreclockout/pld_clk, lttsmstates, and currentspeed are all stuck low. Any suggestions on how we can go about debugging this would be greatly appreciated, or possible reasons why coreclkout is stuck low. Thank you

The coreclkout is derived from the PCIe Refclk, below are the posibilities of this problem: How do you connect the npor signal? Please ensure this signal is asserted or you may just tie it as "1" in the design. You may try to use the PCIe refclk as the sampling clock in signal tap and check if the serdes_pll_locked signal can be asserted? If not, it could be something wrong with the refclk.Regards -SK

Thank you for your reply, SK! 1. npor is (user_input_nrst & perstn), from signal tap it shows that it goes high. I also tried tying it straight to 1'b1, but the lttsmstates shown on the root complex are still the same 2. I used the PCIe refclk as the sampling clock, and do see serdes_pll_locked goes high. I also added the coreclkout to the signaltap, but doesn't see it switching. Do you have any suggestions on what else we can look at? Thanks!

You may not see the "coreclkout_hip" switching as the sampling clock 100Mhz may be running slower. You can create a free-running counter in your design and clock by "coreclkout_hip" if the counter keeps increasing, which means the coreclkout_hip is alive. When the serdes_pll_locked is asserted, you should see the "coreclkout_hip" is running at the expected frequency.

Hi SK, We added a counter for coreclkout_hip, but the counter stayed at 0. We signaltap these signals using pcie_refclk as the sampling clock: pld_clk_inuse = 0 serdes_pll_locked = 1 pld_core_ready = 1 reset_status = 1 perstn = 1 npor = 1 (hard tied to 1) ltssmstate = 0 We also tried setting a signaltap with sampling clock coreclkout_hip, but the signaltap status is "waiting for clock". When we were watching the pcie trainings on youtube, the powerpoint said to set refclk input to CML or HCSL, but on the device datasheet, it doesn't provide the I/O standards specifications like it does for the other standards (LVDS, RSDS, LVPECL, etc). Would you know the IO standards (like Vcm, VID, etc) for CML and HCSL for Cyclone 10GX? Since we use the CDCM9102 clock generator to create the pcie refclk, we followed figure 9 (interfacing between LVPECL and HCSL) of the datasheet for the input of the refclk. Do you see any issues with setting the pcie_refclk inputs as HCSL and following that figure to shift LVPECL to HCSL? Do you have other suggestions that we can look into? Thank you so much for your help

Hi, Most of the use cases of PCIe will share the reference clock with the host via the PCIe edge finger. Do you mean your design requires using a separate clock source between the Host and Endpoint? Besides, as per the C10 pin connection guidance, the DC coupling is allowed for PCIe reference clock if the IO standard is selected is HCSL. Please ensure you are using DC coupling as per requirement. https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/dp/cyclone-10/pcg-01022.pdfDoes the pld_clk is connected with coreclkout_hip in your design? Next, you probably can create a simple design, and input the 100Mhz to an fPLL, and see if you can get the 62.5Mhz/125Mhz output.

PCIe hard IP Bringup on Cyclone 10 GX custom board

26 Replies

SengKok_L_Intel
Regular Contributor
5 years ago
The coreclkout is derived from the PCIe Refclk, below are the posibilities of this problem:

How do you connect the npor signal? Please ensure this signal is asserted or you may just tie it as "1" in the design.
You may try to use the PCIe refclk as the sampling clock in signal tap and check if the serdes_pll_locked signal can be asserted? If not, it could be something wrong with the refclk.

Regards -SK
- jpang10
  New Contributor
  5 years ago
  Thank you for your reply, SK!
  
  1. npor is (user_input_nrst & perstn), from signal tap it shows that it goes high. I also tried tying it straight to 1'b1, but the lttsmstates shown on the root complex are still the same
  
  2. I used the PCIe refclk as the sampling clock, and do see serdes_pll_locked goes high. I also added the coreclkout to the signaltap, but doesn't see it switching.
  
  Do you have any suggestions on what else we can look at?
  
  Thanks!
SengKok_L_Intel
Regular Contributor
5 years ago
You may not see the "coreclkout_hip" switching as the sampling clock 100Mhz may be running slower. You can create a free-running counter in your design and clock by "coreclkout_hip" if the counter keeps increasing, which means the coreclkout_hip is alive. When the serdes_pll_locked is asserted, you should see the "coreclkout_hip" is running at the expected frequency.
- jpang10
  New Contributor
  5 years ago
  Hi SK,
  
  We added a counter for coreclkout_hip, but the counter stayed at 0.
  
  We signaltap these signals using pcie_refclk as the sampling clock:
  
  pld_clk_inuse = 0
  
  serdes_pll_locked = 1
  
  pld_core_ready = 1
  
  reset_status = 1
  
  perstn = 1
  
  npor = 1 (hard tied to 1)
  
  ltssmstate = 0
  
  We also tried setting a signaltap with sampling clock coreclkout_hip, but the signaltap status is "waiting for clock".
  
  When we were watching the pcie trainings on youtube, the powerpoint said to set refclk input to CML or HCSL, but on the device datasheet, it doesn't provide the I/O standards specifications like it does for the other standards (LVDS, RSDS, LVPECL, etc). Would you know the IO standards (like Vcm, VID, etc) for CML and HCSL for Cyclone 10GX?
  
  Since we use the CDCM9102 clock generator to create the pcie refclk, we followed figure 9 (interfacing between LVPECL and HCSL) of the datasheet for the input of the refclk. Do you see any issues with setting the pcie_refclk inputs as HCSL and following that figure to shift LVPECL to HCSL?
  
  Do you have other suggestions that we can look into?
  
  Thank you so much for your help
SengKok_L_Intel
Regular Contributor
5 years ago
Hi,

Most of the use cases of PCIe will share the reference clock with the host via the PCIe edge finger. Do you mean your design requires using a separate clock source between the Host and Endpoint? Besides, as per the C10 pin connection guidance, the DC coupling is allowed for PCIe reference clock if the IO standard is selected is HCSL. Please ensure you are using DC coupling as per requirement.
https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/dp/cyclone-10/pcg-01022.pdf

Does the pld_clk is connected with coreclkout_hip in your design? Next, you probably can create a simple design, and input the 100Mhz to an fPLL, and see if you can get the 62.5Mhz/125Mhz output.
- jpang10
  New Contributor
  5 years ago
  Thanks SK!
  
  Sorry for the confusion. The refclk does come from the golden fingers for the Cyclone 10GX, but is generated with CDCM9102 on the host side. This clock generator output is set to LVPECL, so we were following the datasheet to convert it to HCSL. We could also set the output to LVDS.
  
  The pld_clk is connected to coreclkout_hip in the design.
  
  For the simple project suggested:
  
  I used the "fPLL Intel Arria 10/Cyclone 10 FPGA IP" selecting "CORE" for FPLL mode, and the pll isn't locking or outputting. The pll uses pcie_refclk (REFCLK_GXBL1C_CHBN) as input, and it's set to output 125MHz.
  
  Could this explain why my coreclkout_hip was low and ltssm states aren't changing?
  
  Just for some clarification, should pll_powerdown be 1 or 0 (I tried both and it still didn't work)? Does 1 mean the fpll is powered down and output = 0?
  
  The signal tap uses the pcie_refclk as sampling clock and it appears to work since I can monitor other signals.
  
  I plan to look into my input logic levels for the refclk, but if there are other suggestions of what to look into, please do let me know
SengKok_L_Intel
Regular Contributor
5 years ago
The pll_powerdown signal is high when you wanted to reset the PLL. You may do a reset after power on, for example, hold the fPLL at reset for some time, and then release the reset. Below is the example code you can use, where the clk_lock signal can connect to pll_powerdown.

reg [9:0] count_rst =0;
assign clk_lock = count_rst[9];
always @(posedge clk_100MHz) begin
if (count_rst[9]!= 1)
count_rst = count_rst +1;
else
count_rst = count_rst;
end

Besides, please try to use the IOPLL Intel FPGA IP and see if you also encounter the same problem.

Here is the Cyclone 10 GX Pin connection guideline, please review your custom board to ensure it meets the requirement:
https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/dp/cyclone-10/pcg-01022.pdf
- jpang10
  New Contributor
  5 years ago
  Hi SK,
  
  Here's the results from some of the recommendations to try:
  
  I've tried holding the PLL pins in reset for longer period of time after power up, but that didn't make any difference to the results.
  
  Using REFCLK_GXB (100MHz) as source:
  
  IOPLL works: PLL locks, counter runs, and can be measured with oscope
  
  fPLL doesn't lock, counter stays at 0
  
  Using regular clk pin (50MHz) as source:
  
  IOPLL works: PLL locks, counter runs, and can be measured with oscope
  
  fPLL: not allowed
  
  Using the output of IOPLL (100MHz) as source:
  
  fPLL doesn't lock, counter stays at 0
  
  I verified with the pin connection guide:
  
  VCCR_GXB = VCCT_GXB = 0.95V
  
  REFCLK_GXB : currently set as CML input, AC coupled with 0.1uF and 100 ohm termination
  
  unused REFCLK_GXB : are tied to ground individually
  
  CLKUSR : we have a free running 100MHz oscillator supply to that pin (in fitter report, Automatically reserve CLKUSR pin for calibration purposes is ON)
  
  RREF_[T,B] : are individually tied to a 2kΩ resistor to gnd
  
  Thank you so much for your help so far! Do you have other suggestions of what to look into or possible reasons why FPLL isn't functioning?
SengKok_L_Intel
Regular Contributor
5 years ago
In order to ensure we are aligned, I created a simple fPLL design, please find it in the attachment.

I verified this design in A10 GX setup, what you need to do is to change the device to C10 GX, pcie refclock pin, and maybe upgrade the fpll. This design is working fine on my side by checking from the signal tap.
fPLL_test_20_1_0_177.qar109 KB
- jpang10
  New Contributor
  5 years ago
  I tried using your project as is, using Quartus Pro 20.1 and 20.4, with just upgrading the IP, and assigning the input clock pin assignment.
  
  pll_cal_busy = 1
  
  pll_locked = 0
  
  counter = 0183h (not changing)
  
  In addition to trying your project, I added a iopll and a 2nd counter for the iopll output. The iopll is working, with iopll_locked = 1 and counter2 incrementing.
  
  I also uploaded 2 projects. One using the project you uploaded in the previous message with just an upgraded fpll, and pin assignment. The 2nd one is with an additional iopll in the project, which is used to generate the pictures above.
  
  Thank you!
  fpll_n_iopll_20_1_0_177.qar327 KB
  fpll_only_20_1_0_177.qar323 KB
SengKok_L_Intel
Regular Contributor
5 years ago
To identify if there is FPLL dependency, could you please change the Fpll location by using the following setting in the QSF file?

set_location_assignment FPLL_1CT -to inst|xcvr_fpll_a10_0|fpll_inst
or
set_location_assignment FPLL_1DB -to inst|xcvr_fpll_a10_0|fpll_inst

Besides, how many boards that you encounter this problem?

Regards -SK
- jpang10
  New Contributor
  5 years ago
  Hi SK,
  
  I've tried all 4 locations (1CT, 1CB, 1DT, and 1DB) of fpll, and verified that they were placed in the specified location using the chip planner after compiling. FPLL still doesn't output in any of the location, while IOPLL does.
  
  I have a total of 3 boards, and have tried on 2 of them, and they behave the same way. IOPLL works, and FPLL doesn't.
  
  I really appreciate all the help so far. Do you have other suggestions of what I can look into?
  - jpang10
    New Contributor
    5 years ago
    Hi SK,
    
    My FPLL works after I disabled calibration using
    set_global_assignment -name VERILOG_MACRO "ALTERA_XCVR_A10_DISABLE_RESET_CONNECTED_TO_CAL_BUSY=1"
    
    Would there be an issue if calibration is disabled? It appears that calibration was busy, holding my fpll in reset.
    
    What could be some possible causes that calibration wasn't able to complete?
SengKok_L_Intel
Regular Contributor
5 years ago
Do you see the coreclkout is still stuck low if using this workaround?

Regards -SK
- jpang10
  New Contributor
  5 years ago
  Yes, I still see coreclkout_hip stuck low after trying this workaround
  
  serdes_pll_locked = 1
  
  coreclkout_hip = 0
SengKok_L_Intel
Regular Contributor
5 years ago
Can you attach a simplified PCIe design that can replicate the problem here? I would like to do a sanity check on your design. Thanks.
- jpang10
  New Contributor
  5 years ago
  This project is based on https://fpgacloud.intel.com/devstore/platform/18.0.0/Pro/cyclone-10-gx-pcie-gen1-x1-avl-st/
  
  I added a separate fpll to the project and that is working (using a counter in signaltap). However, the coreclkout_hip is still stuck low (also have a counter).
  
  Debug1-4 are just LEDs on the board.
  
  Please let me know if you see any problems. Thank you!
  pcie_example_20_1_0_177.qar1.9 MB
  - jpang10
    New Contributor
    5 years ago
    We have a separate question, is there a way to set up the hard IP to do a simple loopback? Can we simply connect the tx_st and rx_st together?
    
    I see a lot of examples with the Example Application included in the platform designer.
    
    We currently want to test if we can complete lane training with our root complex, but the coreclkout_hip staying low could be one of our problems.
SengKok_L_Intel
Regular Contributor
5 years ago
Hi
This is glad to see there is good progress.
Questions:
1) After the fixed the clkusr pin, does the coreclkout work correctly with and without the following QSF setting?
set_global_assignment -name VERILOG_MACRO "ALTERA_XCVR_A10_DISABLE_RESET_CONNECTED_TO_CAL_BUSY=1

2) Does the PCIe refclk come from the gold finder? Or it is using a separate clock from your board?
3) What happens if you reboot the Host server after the program of the FPGA? Does the LTSSM behave the same?
Regards -SK
- jpang10
  New Contributor
  5 years ago
  Answers to your questions:
  
  1) After the fixed the clkusr pin, does the coreclkout work correctly with and without the following QSF setting?
  
  set_global_assignment -name VERILOG_MACRO "ALTERA_XCVR_A10_DISABLE_RESET_CONNECTED_TO_CAL_BUSY=1
  
  I no longer need this setting. I removed this from my qsf file and coreclkout_hip still functions properly (verified with counter), and I can also use coreclkout_hip as my sampling clock now.
  
  2) Does the PCIe refclk come from the gold finder? Or it is using a separate clock from your board?
  
  Yes, the PCIe refclk comes from the gold fingers.
  
  3) What happens if you reboot the Host server after the program of the FPGA? Does the LTSSM behave the same?
  
  Rebooting the root complex after programming the FPGA doesn't change the result. The LTSSM still behaves the same way. It toggles between state 0 (Detect.Quiet), 1 (Detect.Active), 2 (Polling.Active)
  
  Something I noticed on the signaltap:
  
  rx_is_lockedtodata and rx_is_lockedtoref are both toggling
  
  Does this mean that the receiver isn't seeing data?
SengKok_L_Intel
Regular Contributor
5 years ago
Hi,
Your design only has 1 lane, if this lane unable to lock, which means it can’t receive the TS1 and TS2 during the polling.active, and after 24ms timeout, it will return to the detect state, and NOT move to polling.configuration. Assuming the refclk is alright since both RP and EP are sharing the same clock source. If the rx_is_lockedtodata is toggling, it could be related to the board design where the incoming data has high BER or the board has introduced high jitter on the PCIe lane.
- jpang10
  New Contributor
  4 years ago
  Thank you for all the help you provided, and helping me through the debugging process.
  
  The problem was with my root complex PCB that I was testing with. After using a different board, everything works properly now.