Point me in the right Direction - PCIe

The PCI spec probably makes slightly easier bedtime reading! Once you'e loosely understood a bit about how the bus works (logically that is), you can then think a bit about PCIe. PCIe is basically PCI using hdlc frames instead of a parallel bus.

The Altera PCIe slave isn't fast - when fed through PCIe->Avalon interface single cycles are positively lethargic (think ISA bus speeds, then slow it down some). Fortunately a PCIe transfer can carry a lot of bytes (probably 128) and the per-byte cost is small. Unfortunately you (almost always) need to start with a DMA engine in order to do multi-word cycles.

Hi Tricky,

--- Quote Start ---

Dont really know where to start with the PCIe core. Very little experience with PCI. Anyone know some design to get me going. Some nice demos of the core?

--- Quote End ---

I'm going to be looking at PCIe early next year. I'd be interested in hearing your experiences, and I'll feedback mine.

The first thing I'd suggest is getting an example design to work, without necessarily understanding the details, just to have something working to start with.

For example, I have Stratix IV GX development kits. For PCIe testing, I have a laptop with an ExpressCard that implements a PCIe-to-cabled PCIe bridge, which then connects to a motherboard containing a PCIe switch. This hardware is from OneStopSystems;

http://www.onestopsystems.com/

http://www.onestopsystems.com/pcie_over_cable_z6.php

http://www.onestopsystems.com/pcie_atx_bp.php

I prefer this to a standard motherboard, since I can test PCIe without having to deal with an OS on the PCIe motherboard, and 'unknown' hardware on the motherboard. This hardware is basically a PCIe switch.

Here's my plan of attack;

1) Test the PCIe examples that come with the kit.

* I did this, and some of the examples appeared to work, while others did not. I'll start with a working example when I move onto the next step.

2) Simulate one of the working designs using Modelsim with a PCIe BFM

* Based on I think your earlier post, it appears Altera dropped BFM support circa Quartus 11.0. I'd install that version just to get the BFMs.

3) Make sure I can access the hardware from Linux

* Accessing PCI and PCIe boards under Linux is very easy. No device driver development required until you want to use interrupts. Even when you need to write Linux drivers, they're pretty straightforward.

4) Once I can access the PCIe interface from Linux, then generate transactions to the core and use SignalTap II to capture transaction traces. Compare those traces to the simulations in (2). Adjust the BFM stimulus to match the hardware.

At this point, I'll have a working hardware/simulation setup, so then I can start on a specific design. Basically I'll just write a PCIe-to-Avalon-MM master (or test whatever Altera provides).

I'm interested in seeing how well Linux deals with PCIe hot-swap.

Cheers,

Dave

--- Quote Start ---

The PCI spec probably makes slightly easier bedtime reading!

--- Quote End ---

The PCIe spec is $3500.

http://www.pcisig.com/specifications/order_form

I doubt it is worth reading though. It'll be mostly about electrical and PHY layers that are taken care of by the Altera IP.

The specifications worth reading are the ones related to the physical form factor you are putting the card into, eg., Compact PCIe (or whatever they call it), AdvancedTCA, AdvancedMC, etc. Those specifications are about $100 to $400 each from PICMG.

https://www.picmg.org/v2internal/specorderformsec-nonmember.htm

But you can also become an affliate member for $1000 and you get everything (a stack of specifications about a foot high).

That's my recommended bedtime reading :)

Cheers,

Dave

I don’t agree. You need both the PCI spec and the PCIe spec – you won’t make it with either or none of both. The PCI spec will teach you all about the bus enumeration, config spaces and such, while the PCIe spec contains only the (added) information relevant for PCIe like the transactions, transaction ordering, etc.

Remember: You need to be PCI SIG member to get your own vendor ID assigned and reserved. And you must remain member while you are using this ID (for new products at least).

--- Quote Start ---

I don’t agree. You need both the PCI spec and the PCIe spec – you won’t make it with either or none of both. The PCI spec will teach you all about the bus enumeration, config spaces and such, while the PCIe spec contains only the (added) information relevant for PCIe like the transactions, transaction ordering, etc.

Remember: You need to be PCI SIG member to get your own vendor ID assigned and reserved. And you must remain member while you are using this ID (for new products at least).

--- Quote End ---

Ok, so help convince us why a PCI/PCIe developer would need to spend $1000 on the PCI spec and another $3500 on the PCIe specification.

Remember, we're developing with existing PCI/PCIe cores, not trying to develop our own. If I was going to develop a core, then by all means I would buy the specifications. If the specifications were both $1000 each, I'd probably just buy them too.

I have the PCI specification, and sure, while it does contain the 'official' wording, I didn't find it added much over the PLX documentation for PCI devices, or the Altera/Xilinx/Lattice/etc PCI documentation. With a PCI bus BFM and then hardware and a PCI bus analyzer, development was straight-forward. Because I was developing for CompactPCI, I definitely needed that specification, but for its mechanical details, not so much of how it use PCI (other than pin assignments).

Now, diving into PCIe, the transaction discussion should be fairly well described in Altera's PCI documentation, though, I wouldn't be surprised if it was hazy in places :), but any haze should be cleared up using a well-written BFM.

Ok, so Altera has stopped distributing their BFM. So, would I be better off spending $3500 on a PCIe BFM?

Thanks for the feedback!

Cheers,

Dave

Apparently the BFM should be back for Q12, but for now I can just use 10.1 (that has the BFM too).

The main interface is just an Avalon-ST or MM interface, so Im hoping I can just squirt a load of data in and let it get on with it!

I think the two specs can be ordered for $3100 total. Consider becoming PCI SIG member for a year ($3000), then each spec is $50.

I would not recommend visiting some darker areas of the internet that distribute prereleases of the specs. While transporting the main ideas of PCI/PCIe, those documents are different in subtle ways and might lead to more effort in finding out what’s wrong.

If you go for a MM interface then, yes, most of the work is already done by Altera. Just ‘choose’ a valid/unique vendor/device ID pair and play with the other parameters a little bit. But if you go for AST or a similar PCIe transaction-level interface, I’m pretty sure you will have a hard time understanding all about credits and transaction completion as well as handling the error interface, MSI/MSI-X, transaction priority, ordering and timeout. These things are all described in the PCIe spec, and Altera just describes the relevant parts of their IP, not the underlying techniques and ideas.

Hi Matthias,

--- Quote Start ---

I think the two specs can be ordered for $3100 total. Consider becoming PCI SIG member for a year ($3000), then each spec is $50.

I would not recommend visiting some darker areas of the internet that distribute prereleases of the specs. While transporting the main ideas of PCI/PCIe, those documents are different in subtle ways and might lead to more effort in finding out what’s wrong.

If you go for a MM interface then, yes, most of the work is already done by Altera. Just ‘choose’ a valid/unique vendor/device ID pair and play with the other parameters a little bit. But if you go for AST or a similar PCIe transaction-level interface, I’m pretty sure you will have a hard time understanding all about credits and transaction completion as well as handling the error interface, MSI/MSI-X, transaction priority, ordering and timeout. These things are all described in the PCIe spec, and Altera just describes the relevant parts of their IP, not the underlying techniques and ideas.

--- Quote End ---

Excellent advice, thanks!

Any other good references on PCIe?

How's your experience with the Altera core? Any warnings? Any suggestions for alternative BFMs?

Cheers,

Dave

Hi Dave,

I don’t know which interface you’re gonna use. If you are going for Avalon ST or another transaction-level interface with DMA, you sure will need an in-depth knowledge about inbound completion credit calculation. There is a well-written document from Xilinx which covers some possible algorithms in perfect detail, see the virtex-6 pcie user guide (http://www.alteraforum.com/forum/www.xilinx.com/support/documentation/user.../v6_pcie_ug517.pdf), Appendix E.

There are at least two pitfalls I noticed when developing with Altera on the transaction-level AST interface. The first is that there is a signal rx_st_mask<n> used to indicate that your logic is not capable of receiving any more non-posted requests, like PIO Read requests from the CPU. There are two sad things about the specific operation: You must accept up to 14 (AST 64 bit) or even 26 (AST 128 bit) more non-posted requests once this signal was asserted. Together with the requirement to not hold incoming completions and posted requests just because of a busy read completion operation, you cannot simply de-assert rx_st_ready<n> – remember, there are transaction ordering rules in PCIe. You are asking for trouble in form of deadlocks if you refuse to receive incoming transactions just because you have an outbound completion (for a read request) blocking your RX port. End result: You need a dedicated fifo on the RX port capable of holding at least 14 non-posted requests (64 bit interface assumed) – even better, make it 16 or 20 so that you don’t trigger rx_st_mask<n> right away when the first non-posted request is received. This is different from, e.g. Xilinx, where this part of the buffering and transaction reordering is done by the IP (see Table 2–13, signal trn_rnp_ok_n). De-assert rx_st_ready<n> only for those times when your internal processing (not PCIe TX related) doesn’t allow any more data, like a full received completion data buffer or a full received PIO posted write data buffer.

The second topic you have to keep in mind when designing for Altera PCIe: Any outbound transaction must be maintained at line rate and you have to be prepared to stream the whole transaction to PCIe at once. While there is tx_st_valid<n> which suggests (from the Avalon-ST spec) that you can insert wait states into the data stream at will, the signal must stay asserted between tx_st_sop<n> and tx_st_eop<n> (while the IP is ready by asserting tx_st_ready<n>), you are not allowed to de-assert it just because you cannot supply the data at full rate and have to wait for it. Again, this is different from Xilinx where you can choose to use such a streamed mode of operation (trn_tstr_n='0') or use an IP-level buffer (see Table 2–12 in the above mentioned document). Bottom line: Either design your data source to supply data at full rate, or add an explicit transaction fifo that starts to transmit transactions to the IP only when they are completely written to the fifo.

Side note: This comparison with the competitor is not meant as an advertisement or as a list of all differences between the different IP core interfaces – there are significantly more – but to point out the major pitfalls where the designer’s assumption about the IP core might not match the actual implementation, and the Altera UG for PCIe wording might be interpreted wrongly at first reading.

One thing that is still not guided by Altera correctly, is the completion timeout mechanism. PCIe requires the application to perform the completion timeout which means that any outbound posted request – i.e. DMA read request issued by the application – which does not receive any or enough completion data within 50 &#956;s to 50 ms (PCIe suggests to not timeout quicker than 10 ms), must abort or retry the operation and indicate a fatal or non-fatal completion timeout error on cpl_err[1] or cpl_err[0], respectively. If you wonder how this is done in the Chaining DMA design example – stop wondering, it is actually not implemented :(. Even more, the IP core claims that it handles unexpected completions properly, especially if “[…] The completion packet has a tag that does not match an outstanding request.” (ref: Table 12–4, page 12–4 of the current UG). I would like to ask Altera how they think the IP knows which transactions are outstanding if the application is responsible for invalidating requests based on the timeout mechanism. At the end of the day, the application has to perform completion filtering by itself rendering this IP automatism useless or even wrong.

– Matthias

Hi Matthias,

Thanks for the warnings! I'm sure we'll be having more discussions when I get started on this in a few weeks.

--- Quote Start ---

I don’t know which interface you’re gonna use.

--- Quote End ---

I'm not sure yet either. I need to transfer relatively small volumes of data between multiple FPGA-based boards and a host CPU. I'll be using DMA, but have still to investigate what I'll need internal to the FPGA.

I'll ask for your advice when I take a look at the existing infrastructure.

Thanks again!

Cheers,

Dave