PCIe packet in cyclone VI GX

Well, there are multiple ways to exchange data between PCIe endpoints, say send data from endpoint 1 to endpoint 2. The easiest one is to route the data through main memory: The device 1 writes the data with a DMA write into main memory (kernel space), next device 2 will do a DMA read from the same memory location to fetch the data. You only have to synchronize the two devices regarding the actual buffer address and order of operation via their drivers in the operating system. If you don’t care for zerocopy, you could let the data pass on to user space and by that separate the drivers for the devices even more.

Another way is to use direct memory writes from device 1 to device 2, or even memory reads from device 2 to device 1. I have never used this technique, so keep your salt at hand. Here, your device 1 has to be informed by its driver of the system memory address of device 2’s BAR you want to use. It might even be possible for device 1 to fetch the BAR settings from device 2 by issuing Configuration Read Requests based on device 2’s ID.

Now device 1 can use this address in a memory write, and the PCIe switches will route the packet to device 2, based on the switch’s knowledge of assigned BARs on its ports (address routing). Device 2 will receive this memory write request just like a memory write triggered by the CPU/driver, and the FPGA application will finish the write based on the indicated BAR hit.

If you want to pull your data by device 2 from device 1, you have to swap the roles: now the driver has to inform device 2 of the BAR address assigned to device 1, so device 2 can fetch memory from there. Device 1 will receive this memory read request and respond with one or more Completions. As the request contained the requester ID of device 1, device 2 addresses the completions to this ID and the switches will now route these response packets based on the ID (ID routing).

The third and final option is, one can use Message TLPs. Messages are typically used only by the IP blocks for Power Management events, Error propagation or Legacy Interrupt indication, but they can be used by the application to let multiple devices communicate. Again, I have never tried that, so I can’t say whether this is fully supported by your favorite (hard) IP block. Different routing mechanisms exist as there are in fact 6 (out of 8) different message TLPs with different routing options applied, identified by the Message Routing Sub-Field. One will typically choose from address routing (001b) or ID routing (010b), and you can choose to send a message with or without data and give it an application-specific Message Code (8 bits).

Messages are always posted, so the proper answer from the other side is not a Completion but another Message TLP, if needed. In such a setup, Message TLPs would be used for everything: data write, return data, signalling, acknowledgment, flow control, error handling. You would try to avoid any other system-bound communication, say, over the OS driver, because this could bring you in trouble regarding transaction ordering. How much of a problem this is in reality might depend on the application. And, again, I have zero experience with that.

Remember that any communication based on ID routing may go severely wrong if the system PCI bus is hit by a rescan and following renumbering. Similarly, if the rescan leads to BAR changes, any address routing will be wrong after that. So the devices should stop talking directly to each other in response to the system starting the rescan, which is typically precluded by a driver unload procedure.

Good luck,

– Matthias