Honored ContributorI wanted to know if anyone has used the Altera Stratix4GX PCIe card as
a root port. I have used it as endpoint but have not used as a root- port. As an endpoint all I have to do is plug in the card to a motherboard. But as a root-port, it needs to send out the requests to end point.
Honored ContributorIn first mode PCIe endpoint mode, I would like to to use the Avalon Streaming interface as a connection to my processor. It seems that in order to do the endpoint emulatin , the speedbridge has to have complete control over the motherboard - including PCI configuration of all devices. The PCIe side of the FPGA connects to the chipset root port on the PC (this technically may be the secondary PCIe link). My processor will send out all the PCIe config cycle over the Avalon streaming interface to configure the motherboard devices. I am not sure if this is possible using the FPGA hard IP. Can the FPGA endpoint pass all the transaction generated on Avalon Streaming TX interface without claiming them? Allso can the root-port chipset support this mode of communication?
Honored ContributorIf I just have a southbridge (not a root port) on the motherboard then it seems that I can use the FPGA in root port mode. I am not sure what happens to the root-port inside the processor that is connected to the strteaming interface
Honored ContributorReply to# 12: No, in order to emulate a PCIe end point (as layed out by me as option (a)), a speedbridge just has to behave like an end point, and it does so without being a root port. It cannot act as the root port when plugged into a PC. Regarding your final two questions: The FPGA endpoint should pass all transactions generated in your application, sent over AST TX. No, the root port chipset won’t support that kind of another root port configuring the PCIe fabric, this is the task of the processor behind the one and only root complex. In order to be the root port, you have to build a structure like documented in Section 1.3 of the PCIe spec, with your FPGA on top – not as a leaf.
Reply to# 13: If you have a southbridge chip that is connected downstream as an end point to the northbridge over PCIe, then you could use an FPGA in root port mode to drive the southbridge. Problem is: You have to connect to the southbridge over the pins that are currently connected to the northbridge/root complex. As you can’t just unsolder the northbridge and put some wires to your FPGA in its place, you have to build a completely new motherboard around your FPGA and southbridge. Otherwise it’s not possible. I have to insist: You cannot make your FPGA a root port, connect it to a COTS motherboard’s PCIe slot and expect your CPU inside/behind the FPGA to mutiny and take over the ship, act as the new root. No way. Nada. These are your options: • Find an FPGA dev board with a PICMG connector (I don’t know whether that actually exists) and make your FPGA a PCIe root port. Then you can take a PCIe backplane and connect multiple COTS PCIe cards to that backplane and let your CPU/FPGA drive them. I have not seen a backplane yet that has a southbridge on it, as most industrial CPU cards with PICMG have them around the CPU, as most southbridge chips are not connected to the northbridge over official PCIe. • Use a PCIe southbridge chip. As I don’t think that there are dev kits available with such a setup, you have to build your own board with your application and the southbridge on it. • Use a COTS northbridge chip. Again, build your own board. Currently there is no chipset that connects to the CPU over PCIe, so you will have to build a different FPGA design that speaks that CPU’s frontside bus language (HyperTransport, QPI, xGTL, etc.). • Write a PCIe end point and let the CPU on the motherboard handle PCIe configuration, drivers etc.Honored ContributorThanks for taking the time to explain possible solution to this.
For now if we make the following assumptions: (1) Custom designed motherboard with a PCIe slot and only pcie south-bridge and no Northbridge is available. (2) This PCIe slot is used to plug in the FPGA Speedbridge (3) The other side of the PCIe slot is connected to a PCIe southbridge with all the necessary peripherals to boot. Based on your explanation, it seems that I can use the FPGA speedbridge device as a Root port - provided that the the CPU behind avalon Streaming interface does not have integrated root port in it. If yes to the above case then - Can the FPGA speedbridge be completely transparent to the software? From the Avalon Streaming interface back-door when CPU is performing the PCI enumeration, then it is not going to find the FPGA speedbridge as one of the PCI device. As per my understanding, in order to access the config registers inside the FPGA hard IP, one has to use the LMI interface which is not being utilized by the CPU on avalon stream bus. Thus when CPU generates any PCIe config cycles on streaming interface then it will go out in PCIe bus and then to the Southbridge on the motherboard. I can try to run some simulations to get an answer but I thought I ask anyway. Thanks for your help.Honored ContributorI don’t know whether I precisely understand your target architecture.
At first, the architecture you specify in your bullets (1), (2) and (3) looks good, but it’s incomplete. What do you want to use the Speedbridge for? It’s an emulation tool and according to the data sheet it works only in conjunction with the proprietary Incisive Palladium simulation system. It is just thought to help you make a simulated CPU with a PCIe root port drive real PCIe hardware. It is not intended to be a PCIe extender or something similar – that’s what PCIe switches are for. That means, wherever you plug the other side of the Speedbridge into – the Palladium –, this computer cannot use the bridge ‘transparently’ at all, I suspect the simulation platform to be optimized for this simulation task and not for physical transparency. But I am not into Speedbridge/Palladium design, so I can be wrong. So, apart from Speedbridge, let’s say instead of (2) you plug a PCIe FPGA board – maybe a dev kit – into your Custom Mainboard (1). The FPGA is acting as a PCIe root. For argument’s sake, let’s assume that this works physically without mods on the dev kit. Then you put a bridge between your CPU (physical or core) and the AST of the PCIe IP block inside the FPGA. Remember that the PCIe root port is not transparent for your CPU at all, and the bridge will have to maintain all functionality specific to a root complex bridge. That includes and is not limited to CFG space access decoding, Interrupt Message handling from the PCIe leafs, etc. Again, it is impossible from a PCIe architectural standpoint to have any other PCIe root complex between the CPU and the FPGA root port. With proper OS support it might be possible to have multiple PCIe busses with multiple root complexes side-by-side, but I don’t know precisely if that’s possible at all. I think that whatever you intend should be doable with a standard PCIe architecture, so try to stick with that.Honored ContributorThanks for your feedback. Yes. I would like to use this for CPU simulation. It seems that PCIe FPGA root port hardIP can not be used. May be Cadence custom designed the pass through bridge that accepts the TLPs from Palladium side and converts into PCIe and viceversa. It may not have its own config space. Cadence speedbridge may have added other layers to meet the PCie spec.
I am not sure if there is a soft IP from FPGA vendor that lets you do similar things.Honored ContributorYou know, a PCIe root port is not a switch port. If you want to design a PCIe switch, you would have to design a (soft) L2 IP on your own, I think.
In PCIe, a Switch is an addressable and visible entity inside the fabric. It sure does most of its task on L2 which is packet forwarding and maintenance of link-specific DLLP handling like flow control, but it does quite something on L3 as well. A dual-port switch is in fact a bridge, and such a bridge could be designed to be transparent and promiscuous above L2. I think their clever part will be that they don’t emulate the physical layer of the ASIC design but they inject straight into L2 (DLL), that makes them transparent but let the user verify most of his design.Honored ContributorThanks. I will have to study what it takes to inject the traffic staright into L2 which just one lyaer below Transaction layer. Need to determine output of the Transaction layer and input into the DLL. Is there a soft IP I can use upto DLL?
Honored ContributorI don’t know of any official way to get there.
1) You have to setup the Transceivers to do PCIe L1 (Physical Layer) and supply a proper interface (PIPE). It seems one can use the instantiated pcie_serdes component from the top-level PCIe design from MegaWizard. 2) You have to build your own DLL. That’s what PCIe Spec Chapter 3 is all about. That includes: • Proper handling of L1 at line rate, • link setup, • DLLP handling including power saving stuff, TLP acknowledgement, and flow control according to your buffering capabilities, • proper TLP encapsulation, • error handling (CRC) and retry mechanism, • etc. 3) On top of the DLL you can build whatever interface is appropriate for your setup. 4) remember that while you can rate-adapt in this bridge, you have to maintain all timeouts defined by the PCIe spec, otherwise data gets lost on the lowest level, and that’s not what you want. The following Altera IP functionality should not be in this bridge IP: 1) Configuration space handling, 2) BAR decoding, 3) tag handling, 4) interrupt message generation/decoding, 5) completion timeout, 6) Filtering of unexpected TLPs, 7) etc. Happy Hacking!Honored ContributorThanks again. What if the processor has the data link layer included in it? In this case the FPGA may need to support is the physical layer supporting Link Training and Flow control.
