I am trying to get the Cyclone V SoC Development Kit Board to behave as a PCIe End Port. I want to plug this board into a PCIe 4x slot on a PC motherboard running windows. This Dev Kit Board is really setup to be a root port and has a root port connector. So I can't plug the dev kit board directly into a slot on the motherboard. To solve this problem I purchased a male-to-male 4x adapter cable that also acts as a RX-to-TX crossover cable. A PCIe end port example design is included in the files I downloaded from Altera for the C5SoC dev kit. I had to make a few adjustments to the .qsf file, but I was able to get this end port example generated in qsys and compiled in quartus 13.1sp1. My problem is when I download the .sof into the dev kit fpga, the pc mother board doesn't seem to recognize it. I added a jtag master and a register to the design so I can read the PCIe LTSSM state bits. Using system console to read these bits, the PCIe controller appears to be going back and forth between the detect-quiet state and the polling-active state. My crossover adapter cable is fairly long (15"). I've ordered a shorter one (2") to see if this may be my problem. It will be a couple days before I get the 2" cable. In the mean time, I wanted to see if anyone else has tried to get the Cyclone V SoC Dev Kit Board to work as a PCIe End Port?

How are you getting the PCIe clock to your FPGA? The PCIe connector has a 100MHz clock output ... but you need a 100MHz clock input to have the device work as an end-point. Cheers, Dave

The Cyclone V SoC dev kit board has a PCIe clock generator device made by Silicon Labs. This clock generator has two differential 100 MHz clock outputs. One output is attached thru zero ohm resistors to the backplane connector and one is routed to the FPGA where it is used as the reference clock input to the PCIe high-speed transceivers. I removed the resistors that go from the clock generator to the backplane, since the pc motherboard also drives the PCIe REFCLK. So I don't think this is the problem and I don't think PCIe requires that the 100 MHz REFCLK come from the backplane. In fact Altera gives you the option to use a 125 MHz reference clock. -kstolp

--- Quote Start --- The Cyclone V SoC dev kit board has a PCIe clock generator device made by Silicon Labs. This clock generator has two differential 100 MHz clock outputs. One output is attached thru zero ohm resistors to the backplane connector and one is routed to the FPGA where it is used as the reference clock input to the PCIe high-speed transceivers. I removed the resistors that go from the clock generator to the backplane, since the pc motherboard also drives the PCIe REFCLK. So I don't think this is the problem and I don't think PCIe requires that the 100 MHz REFCLK come from the backplane. In fact Altera gives you the option to use a 125 MHz reference clock. --- Quote End --- PCIe in a motherboard expects a reference clock that is synchronous at all boards. That is why the SoC kit is supplying a reference clock to both the Cyclone V and the PCIe connector - because the device needs to see a synchronous reference clock. Your setup needs to figure out a way to route the reference clock being driven onto the PCIe connector over to the FPGA. I'd recommend removing both 0-ohm resistors from the SiLabs part, and jumpering the PCIe clock coming into the connector, over to the resistor that drives the FPGA REFCLK pin. Probe the pins with a scope to check that the differential clock at the FPGA looks clean. Cheers, Dave

Dave, you are correct concerning REFCLK. It was the key to getting Altera's example end port design working on the Cyclone V SoC dev kit board. For fun, here is list of things I did to get this example to work: 1) Removed R249, R251, R253 & R254. Installed R250 & R252 (zero ohm jumpers). This makes the fpga pcie_refclk_p input come from the pcie backplane connector instead of the onboard oscillator. 2) Purchased a PCIe 4x male-to-male crossover cable to connect the Cyclone V SoC board to a pc motherboard. My cable came from Adex Electronics (www.adexelec.com), part number PE-FLEX4-CX-MM-15". 3) Used the ep_gen1x4_13_0_0_balanced design example found in the c:\altera\13.0\kits\cycloneVSX_5csxfc6df31es_soc\examples\pcie\endport_example directory after installing the kit files (cycloneVSX_5csxfc6df31es_soc_v13.0.0.1.exe) downloaded from Altera's website. 4) Made the following changes to the c5SoC_EndPoint.qsf file: a) Added the following IO Bank VCCIO assignments. When I first compiled this example, VCCIO for several banks did not match the board. These lines fix that problem. Maybe these assignments are not necessary, but they match the board and make me feel better. set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 3B set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 4A set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 6A set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 6B set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 3A set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 5A set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 5B set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 8A set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7A set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7B set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7C set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7D b) I corrected the pin assignments for the user_led_fpga[3:1] pins, to be as follows: set_location_assignment PIN_Y16 -to user_led_fpga[1] set_location_assignment PIN_W15 -to user_led_fpga[2] set_location_assignment PIN_AB17 -to user_led_fpga[3] c) I changed the IO standard for the pcie_refclk input from "1.5V PCML" to "HCSL". I'm not sure this is necessary, but this is what the Cyclone V Hard IP PCIe User Guide says to do. My suspicion is that quartus ignores this assignment. d) I changed the part number from "5CSXFC6D6F31C7" to "5CSXFC6D6F31C8ES". Quartus 13.0sp1 would not generate a programming file with the original "C7" part number. A message was generated indicated specifications for this device are subject to change. no programming file will be generated. I find it interesting that the design meets timing for coreclkout (125 MHz) with the "C7" part number, but not the "8ES" part number. I hope this information helps someone else trying to use the Cyclone V SoC Dev Kit as a PCIe End Port. -kstolp

--- Quote Start --- Dave, you are correct concerning REFCLK. It was the key to getting Altera's example end port design working on the Cyclone V SoC dev kit board. --- Quote End --- Excellent! Thanks for posting the modifications you needed to do. Note that you can also create an AlteraWiki page, with a short write-up of what you did. That might be easier for the next user to locate, than this thread. Cheers, Dave

Cyclone V SoC Dev Kit PCIe End Port Example

20 Replies

Altera_Forum
Honored Contributor
12 years ago
Hi kstolp,

The example designs with the "Qsys PCIe core fails timing" thread used the PCIe Hard IP clock to clock the Qsys system primarily because that was how the first few examples I looked at implemented a simple design. In general I would not recommend doing this, but would instead use a local clock source, eg., a 100MHz Qsys clock should be fine. This allows you to implement a JTAG-to-Avalon-MM bridge into your Qsys system that will work regardless of the state of the PCIe clock.

The approach you are taking should work just fine, and would be the recommended approach.

There's a couple of things you can do;

1. Use a PCIe logic analyzer to trace the traffic that occurs when the system works and when it does not.

2. Use the SignalTap II logic analyzer to trace the traffic (if you do not have access to a PCIe analyzer)

3. Create a simulation and look to see whether you can "boot" your system, eg., enumerate the PCIe end-point and then access your Qsys system.

I'd recommend having (3) regardless of whether you do (1) or (2) as it forces you to really understand what needs to occur to get your system up-and-running. Once you have traces from (1) or (2), you can update your simulation to implement the same access sequence (eg., redundant accesses performed by the root-complex).

Cheers,
Dave
Altera_Forum
Honored Contributor
12 years ago
Dave, Thanks for the advice.
kstolp
Altera_Forum
Honored Contributor
11 years ago
Hi Dave,

My Question is a lil out of track but I would like you to ans this since You look Quite polished with PCIe related stuff.

I want to know If I can test PCIe Root Port without Connecting/attaching Endpoint with Linux Kernels.

Thanks
Angad
Altera_Forum
Honored Contributor
11 years ago
Hi Angad,

--- Quote Start ---

I want to know If I can test PCIe Root Port without Connecting/attaching Endpoint with Linux Kernels.

--- Quote End ---

You mean you just want to see if the wires on the PCIe connector are working? If the root-complex is an FPGA, then sure, you can do whatever you like.

For example, you can test a high-speed transceiver by configuring it in BIST mode and have it generate a PRBS pattern, or you can use a transceiver IP core and configure it manually and have it output a desired pattern. For example, a square wave at a low enough frequency that you can see it on a scope. You can also loop back the TX and RX differential signals using a PCIe breakout cable (Samtec sell them).

http://www.samtec.com/technical-specifications/default.aspx?seriesmaster=pcrf

I use this type of cable (the one with the peripheral style connector) as a way to access transceivers on kits that do not have HSMC or FMC connectors.

Cheers,
Dave
Altera_Forum
Honored Contributor
11 years ago
--- Quote Start ---
Hi Angad,

You mean you just want to see if the wires on the PCIe connector are working? If the root-complex is an FPGA, then sure, you can do whatever you like.

For example, you can test a high-speed transceiver by configuring it in BIST mode and have it generate a PRBS pattern, or you can use a transceiver IP core and configure it manually and have it output a desired pattern. For example, a square wave at a low enough frequency that you can see it on a scope. You can also loop back the TX and RX differential signals using a PCIe breakout cable (Samtec sell them).

http://www.samtec.com/technical-specifications/default.aspx?seriesmaster=pcrf

I use this type of cable (the one with the peripheral style connector) as a way to access transceivers on kits that do not have HSMC or FMC connectors.

Cheers,
Dave
--- Quote End ---

Hi Dave,

Thanks for the quick response.

I am testing the following design: http://rocketboards.org/foswiki/view/projects/pcierootportwithmsi
If you Look into "Open Issue" - FB 165881 – Kernel can’t boot up if don’t have end point

I have applied the patch that they are giving. It works fine on the Dev kit, But I tried the same patch for Other Cyclone V Device(5CSXFC6C6U23I7ES) The patch seems to fail.

Basically from the patch I understood that the RootPort is being forced to complete the Enumeration But I dont understand why it wont work for other device other than Dev kit.

Regards
Angad
Altera_Forum
Honored Contributor
11 years ago
Hi Angad,

--- Quote Start ---

I am testing the following design: http://rocketboards.org/foswiki/view/projects/pcierootportwithmsi
If you Look into "Open Issue" - FB 165881 – Kernel can’t boot up if don’t have end point

I have applied the patch that they are giving. It works fine on the Dev kit, But I tried the same patch for Other Cyclone V Device(5CSXFC6C6U23I7ES) The patch seems to fail.

Basically from the patch I understood that the RootPort is being forced to complete the Enumeration But I dont understand why it wont work for other device other than Dev kit.

--- Quote End ---

That is a terrible issue - its like saying "Sorry, your PC cannot boot if you do not have a card in the PCIe slot"!!!

Sorry, I've steered clear of the SoC devices for exactly this type of reason. I expected "teething" problems and didn't want to be the one to find them (been there, done that with earlier new generation devices).

I didn't read any of the code. Perhaps there is just a "bad decision" in the RTL or in the bootloader code that is the cause of this issue.

Cheers,
Dave
Altera_Forum
Honored Contributor
11 years ago
--- Quote Start ---
Hi Angad,

That is a terrible issue - its like saying "Sorry, your PC cannot boot if you do not have a card in the PCIe slot"!!!

Sorry, I've steered clear of the SoC devices for exactly this type of reason. I expected "teething" problems and didn't want to be the one to find them (been there, done that with earlier new generation devices).

I didn't read any of the code. Perhaps there is just a "bad decision" in the RTL or in the bootloader code that is the cause of this issue.

Cheers,
Dave
--- Quote End ---

Hi Dave,

Haha..that's exactly how I feel at this point of time when the Kernel hangs!!..
I have been seeking help from RocketBoards as well as Altera FAE but have not got any contextual solution to this issue. I don't know why would anyone make a device specific patch for a generic hard IP. Let me know if you get the time to go through the Patch and figure out a quick fix to the issue which I could try out.

Thanks
Angad
Altera_Forum
Honored Contributor
11 years ago
Hi Angad,

--- Quote Start ---

I have been seeking help from RocketBoards as well as Altera FAE but have not got any contextual solution to this issue. I don't know why would anyone make a device specific patch for a generic hard IP. Let me know if you get the time to go through the Patch and figure out a quick fix to the issue which I could try out.

--- Quote End ---

Sorry, I won't get a chance to look at the patch. I recommending filing a Service Request directly with Altera (use the Altera web site) and see if they can help. At least you might get a chance to chat with the person who created the patch and see why it is needed and whether there is a better solution.

Cheers,
Dave
Altera_Forum
Honored Contributor
11 years ago
Hello,

I am also facing the same issue "Kernel hangs after 'Starting kernel' ". Can you please message the link for the patch? I coudn't find rp_rtl.zip file in cv_soc_rp_full_design.tar.gz or cv_soc_rp_simple_design.tar.gz(As per solution for FB 165881 – Kernel can’t boot up if don’t have end point). Regards Kushal
Altera_Forum
Honored Contributor
11 years ago
--- Quote Start ---
Hello,

I am also facing the same issue "Kernel hangs after 'Starting kernel' ". Can you please message the link for the patch? I coudn't find rp_rtl.zip file in cv_soc_rp_full_design.tar.gz or cv_soc_rp_simple_design.tar.gz(As per solution for FB 165881 – Kernel can’t boot up if don’t have end point). Regards Kushal
--- Quote End ---

Hi Kaushal,

I am sure you would be using 13.1 or older version of Quartus. Use the latest version of Quartus 14.0 or higher. the issue has been addressed by Altera. For the higher version we do not require any patch hence Rocket boards have removed the patch from their site.

Regards
Angad

Forum Discussion

Cyclone V SoC Dev Kit PCIe End Port Example

20 Replies

Recent Discussions

System PLL of Agliex5 PCIE example design cannot be locked after configuration

JTAG Chain Broken on Agilex 7-I Dev Kit

Will serialization factor of 6 in LVDS serdes IP be supported in the future on Agilex5?

Request for Cyclone V Pinout File Information

Cyclone 10 LP's Extended Industrial parts