Cyclone V SoC Dev Kit PCIe End Port Example

How are you getting the PCIe clock to your FPGA?

The PCIe connector has a 100MHz clock output ... but you need a 100MHz clock input to have the device work as an end-point.

Cheers,

Dave

The Cyclone V SoC dev kit board has a PCIe clock generator device made by Silicon Labs. This clock generator has two differential 100 MHz clock outputs. One output is attached thru zero ohm resistors to the backplane connector and one is routed to the FPGA where it is used as the reference clock input to the PCIe high-speed transceivers. I removed the resistors that go from the clock generator to the backplane, since the pc motherboard also drives the PCIe REFCLK. So I don't think this is the problem and I don't think PCIe requires that the 100 MHz REFCLK come from the backplane. In fact Altera gives you the option to use a 125 MHz reference clock.

-kstolp

--- Quote Start ---

The Cyclone V SoC dev kit board has a PCIe clock generator device made by Silicon Labs. This clock generator has two differential 100 MHz clock outputs. One output is attached thru zero ohm resistors to the backplane connector and one is routed to the FPGA where it is used as the reference clock input to the PCIe high-speed transceivers. I removed the resistors that go from the clock generator to the backplane, since the pc motherboard also drives the PCIe REFCLK. So I don't think this is the problem and I don't think PCIe requires that the 100 MHz REFCLK come from the backplane. In fact Altera gives you the option to use a 125 MHz reference clock.

--- Quote End ---

PCIe in a motherboard expects a reference clock that is synchronous at all boards. That is why the SoC kit is supplying a reference clock to both the Cyclone V and the PCIe connector - because the device needs to see a synchronous reference clock.

Your setup needs to figure out a way to route the reference clock being driven onto the PCIe connector over to the FPGA. I'd recommend removing both 0-ohm resistors from the SiLabs part, and jumpering the PCIe clock coming into the connector, over to the resistor that drives the FPGA REFCLK pin.

Probe the pins with a scope to check that the differential clock at the FPGA looks clean.

Cheers,

Dave

Dave, you are correct concerning REFCLK. It was the key to getting Altera's example end port design working on the Cyclone V SoC dev kit board. For fun, here is list of things I did to get this example to work:

1) Removed R249, R251, R253 & R254. Installed R250 & R252 (zero ohm jumpers). This makes the fpga pcie_refclk_p input come from the pcie backplane connector instead of the onboard oscillator.

2) Purchased a PCIe 4x male-to-male crossover cable to connect the Cyclone V SoC board to a pc motherboard. My cable came from Adex Electronics (www.adexelec.com), part number PE-FLEX4-CX-MM-15".

3) Used the ep_gen1x4_13_0_0_balanced design example found in the c:\altera\13.0\kits\cycloneVSX_5csxfc6df31es_soc\examples\pcie\endport_example directory after installing the kit files (cycloneVSX_5csxfc6df31es_soc_v13.0.0.1.exe) downloaded from Altera's website.

4) Made the following changes to the c5SoC_EndPoint.qsf file:

a) Added the following IO Bank VCCIO assignments. When I first compiled this example, VCCIO for several banks did not match the board. These lines fix that problem. Maybe these assignments are not necessary, but they match the board and make me feel better.

set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 3B

set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 4A

set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 6A

set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 6B

set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 3A

set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 5A

set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 5B

set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 8A

set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7A

set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7B

set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7C

set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7D

b) I corrected the pin assignments for the user_led_fpga[3:1] pins, to be as follows:

set_location_assignment PIN_Y16 -to user_led_fpga[1]

set_location_assignment PIN_W15 -to user_led_fpga[2]

set_location_assignment PIN_AB17 -to user_led_fpga[3]

c) I changed the IO standard for the pcie_refclk input from "1.5V PCML" to "HCSL". I'm not sure this is necessary, but this is what the Cyclone V Hard IP PCIe User Guide says to do. My suspicion is that quartus ignores this assignment.

d) I changed the part number from "5CSXFC6D6F31C7" to "5CSXFC6D6F31C8ES". Quartus 13.0sp1 would not generate a programming file with the original "C7" part number. A message was generated indicated specifications for this device are subject to change. no programming file will be generated. I find it interesting that the design meets timing for coreclkout (125 MHz) with the "C7" part number, but not the "8ES" part number.

I hope this information helps someone else trying to use the Cyclone V SoC Dev Kit as a PCIe End Port.

-kstolp

--- Quote Start ---

Dave, you are correct concerning REFCLK. It was the key to getting Altera's example end port design working on the Cyclone V SoC dev kit board.

--- Quote End ---

Excellent!

Thanks for posting the modifications you needed to do.

Note that you can also create an AlteraWiki page, with a short write-up of what you did. That might be easier for the next user to locate, than this thread.

Cheers,

Dave

So here is a couple more things I've learned:

1. I was able to get the example end port design to meet timing with the "8ES" part number, by setting the routing timing optimization setting to "Maximum" in the more fitter settings window. If you make a new project from scratch, make sure to use the qsys and quartus settings found in Altera's example. These settings optimize for clock speed, which you need if you want to meet the 125 MHz requirement for coreclkout.

2. Don't allow your BAR address space to become larger than 28-bits. This caused my PC not to boot, which took me a lot of time and frustration to figure out. Dave pointed this limitation out in an old thread titled "PCIe BAR size limitations in Qsys"

-kstolp

Did you check out the PDF in this thread? It has some comments about the settings required to meet timing (for Stratix IV and Cyclone IV devices);

http://www.alteraforum.com/forum/showthread.php?t=35678

Cheers,

Dave

--- Quote Start ---

Dave, you seem to be some kind of PCIe expert.

--- Quote End ---

Nah, I'm just a guy who's had these types of problems himself :)

--- Quote Start ---

I appreciate your help. Thank you.

--- Quote End ---

You're welcome.

Cheers,

Dave

Dave,

From the thread you posted above, you did some PCIe Hard IP compilation tests and had some issues with Altera's examples meeting timing. As mentioned in one of my previous posts to this thread, I've had some difficulty with the PCIe Hard IP meeting timing when targeting the FPGA on the Cyclone V SoC dev kit board. The main issue is the application layer clock has to be 125 MHz and the avalon switch fabric generated by Qsys can't really run at this clock speed unless the interconnects are kept to a minimum. The Altera example is a good starting point, but now I want to add some other components which will increase the number of avalon interconnects that need to run at 125 MHz. In fact, when I do this, it becomes impossible to meet the 125 MHz timing requirement.

My initial fix for this issue was to add clock bridges between all of the PCIe hard IP application layer interfaces and all the other avalon-mm components. This keeps the interconnect logic running at 125 MHz to a bare minimum. I chose to run the rest of the interconnect fabric at 50 MHz, which is an easy timing requirement to meet, no matter how complex my interconnect logic becomes. This solution seemed to be working fine, but today I encountered a problem. The PC I've been using works fine with my bridged design, but the PC we are using to develop the device driver on, does not recognize the board when my bridged design is loaded in the FPGA. Looking at the device manager (Windows 7), the PCIe root node connected the C5 SoC dev kit board, is unhappy. If I reload Altera's original example, both PC's recognize the board.

Is there a problem with my clock bridged approach? Any other thoughts as to what may be my problem?

Thanks,

kstolp