I'm not sure if the PCI Express MegaCore hard IP can do what I want. I'll describe what I'm trying to achieve. I'd like to set up a bus-master scatter-gather DMA between on-device memory and a host memory buffer. The host memory buffer is contiguous in virtual address space, but when locked down into 4K-sized physical memory pages, the physical addresses of those pages are randomly scattered all over physical memory - a classic usage case of scatter-gather DMA. I'm not sure how this fits in with the Avalon-MM-to-PCI Express address translation table. Would I have to set up a translation table entry for each 4K page or could I set up a few translation table entries to cover the whole host memory physical address space (or at least the bottom 3 GB ? I'm more familiar with "PCI to local bus" bridges (e.g. the ones made by PLX Technologies) that have completely separate address spaces on each side of the bridge. The only example I've got to go on is the WinDriver code for the "PCI Express in Qsys Example" design, but that's nothing like what I want as it allocates a contiguous area of physical memory in the bottom 16 MiB of physical memory.

Hmmm..... I'd have thought that the DMA controller would be able to pass a full 32bit PCIe-space address, rather than an Avalon MM slave address of the PCIe controller. This partially important because you need the PCIe controller to issue single PCIe requests for large blocks (say 128 bytes min) in order to get reasonable throughput - which is likely to mean that you need to use a DMA controller embedded into the PCIe interface. (That is what we had to do on the pcc when the fpga was a slave...)

To be honest, I'm not entirely sure what the address translation tables are actually translating, as each entry only contains one 64-bit address rather than say a PCI address and a matching local address. Since the areas mapped by each table entry are all the same size, I'm guessing that consecutive table entries map to consecutive regions of local address space starting at some local base address. (EDIT: To avoid any confusion, my device is normally a PCI target except during bus-master DMA operations. I.e. it's like a PCI plug-in peripheral card for a PC.)

--- Quote Start --- I'd like to set up a bus-master scatter-gather DMA between on-device memory and a host memory buffer. The host memory buffer is contiguous in virtual address space, <...> I'm not sure how this fits in with the Avalon-MM-to-PCI Express address translation table.<...> --- Quote End --- That's perfectly possible. You do not need any (BAR) translation table on the FPGA, because the FPGA is doing SGDMA directly to/from host memory, so there is no BAR access involved. For a ready to go SGDMA solution with drivers, see http://www.lancero-pcie.com/ Best regards, Leon Woestenberg

Our design is basically a plug-in PCIe card that is normally a PCI target, but which will support PCI bus-master DMA transfers. We'll be using the PCIe hard-IP on Cyclone IV with the Qsys design flow and the Avalon-MM interface. We were thinking of using Altera's Modular SGDMA (mSGDMA) controller. As I understand it, that only works with 32-bit Avalon addresses and wouldn't know anything about PCI bus addresses, so some mapping between PCI addresses and Avalon addresses would be required. As I understand it, the Altera PCIe hard-IP provides the standard PCI BAR registers to map certain regions of the Avalon address space into the PCI bus address space. The Avalon base address and size of each BAR region is assigned in the FPGA design, and the assignment of the PCI bus base address of each BAR region is assigned by the PCI host (on a PC, firstly by the PCI-BIOS code, and later by the host operating system). Things like the mSGDMA controller only know about Avalon addresses, so in order for it to access regions in the PCI bus address space, the PCI bus addresses need to be mapped into the Avalon address space. That's what the "Avalon-MM-to-PCI Express Address Translation Table" is for. This table is either fixed or dynamic and maps one or more "pages" of PCI bus address space into the Avalon-MM address space. Up to 16 fixed table entries or up to 512 dynamic table entries can be defined and the page size can be configured to any value from 2^12 (4096 bytes) to 2^32 (4 gibibytes). There are three main possibilities I can think of: Use a single, very large fixed page to map PCI bus address 0. Using a page size of 2^30 or 2^31 would allow access to PCI bus addresses in the range 0 to 2^30 - 1 (0 - 1 GB ) or 0 to 2^31 - 1 (0 - 2 GB ). (I don't think a page size of 2^32 could be used, as that would use up all the Avalon address space.) Use a single or a few fairly large dynamic pages. This would allow access to one or a few fairly large, contiguous regions of PCI bus address space, dynamically configurable by the host PC software. Use a large number of small (4096-byte) dynamic pages. This would allow a large number of 4096-byte pages to be simultaneously mapped from PCI bus address space into Avalon address space. The page size of 4096 would match the physical memory page size of the host system (at least for most host architectures). If using mSGDMA controllers, you'd need one page table entry for each committed scatter-gather descriptor, up to a maximum of 512, so the maximum amount of PCI memory that could be mapped simultaneously would be 2 MB (4096x512 byte). (The above is just my understanding and may be partly or completely wrong. Corrections of my misunderstanding are appreciated!) I don't think we have the budget to use third-party IP in this project, but I'm interested in how the Lancero solution compares to the Altera offerings.

To get any performance at all out of PCIe, you have to use PCIe burst transfers (these are hdlc frames that carry multiple words of data). So if you are doing a PCIe read, the PCIe interface logic has to know how many words of data you are going to require before it requests any of the data. This can only really be done if the DMA controller in integrated into the PCIe master logic. Although there are pipelined Avalon cycles, I suspect they are not capable of being converted into PCIe read bursts (they might be translatable into PCIe write bursts as that is easier). But I've not used the Altera PCIe master (just the slave).

PCIe bus master random access to all host memory

14 Replies

Altera_Forum
Honored Contributor
14 years ago
Hmmm..... I'd have thought that the DMA controller would be able to pass a full 32bit PCIe-space address, rather than an Avalon MM slave address of the PCIe controller.
This partially important because you need the PCIe controller to issue single PCIe requests for large blocks (say 128 bytes min) in order to get reasonable throughput - which is likely to mean that you need to use a DMA controller embedded into the PCIe interface.
(That is what we had to do on the pcc when the fpga was a slave...)
Altera_Forum
Honored Contributor
14 years ago
To be honest, I'm not entirely sure what the address translation tables are actually translating, as each entry only contains one 64-bit address rather than say a PCI address and a matching local address. Since the areas mapped by each table entry are all the same size, I'm guessing that consecutive table entries map to consecutive regions of local address space starting at some local base address.

(EDIT: To avoid any confusion, my device is normally a PCI target except during bus-master DMA operations. I.e. it's like a PCI plug-in peripheral card for a PC.)
Altera_Forum
Honored Contributor
14 years ago
--- Quote Start ---

I'd like to set up a bus-master scatter-gather DMA between on-device memory and a host memory buffer. The host memory buffer is contiguous in virtual address space,
<...>

I'm not sure how this fits in with the Avalon-MM-to-PCI Express address translation table.<...>

--- Quote End ---

That's perfectly possible.

You do not need any (BAR) translation table on the FPGA, because the FPGA is doing SGDMA directly to/from host memory, so there is no BAR access involved.

For a ready to go SGDMA solution with drivers, see http://www.lancero-pcie.com/

Best regards,

Leon Woestenberg
Altera_Forum
Honored Contributor
14 years ago
Our design is basically a plug-in PCIe card that is normally a PCI target, but which will support PCI bus-master DMA transfers. We'll be using the PCIe hard-IP on Cyclone IV with the Qsys design flow and the Avalon-MM interface.

We were thinking of using Altera's Modular SGDMA (mSGDMA) controller. As I understand it, that only works with 32-bit Avalon addresses and wouldn't know anything about PCI bus addresses, so some mapping between PCI addresses and Avalon addresses would be required.

As I understand it, the Altera PCIe hard-IP provides the standard PCI BAR registers to map certain regions of the Avalon address space into the PCI bus address space. The Avalon base address and size of each BAR region is assigned in the FPGA design, and the assignment of the PCI bus base address of each BAR region is assigned by the PCI host (on a PC, firstly by the PCI-BIOS code, and later by the host operating system).

Things like the mSGDMA controller only know about Avalon addresses, so in order for it to access regions in the PCI bus address space, the PCI bus addresses need to be mapped into the Avalon address space. That's what the "Avalon-MM-to-PCI Express Address Translation Table" is for. This table is either fixed or dynamic and maps one or more "pages" of PCI bus address space into the Avalon-MM address space. Up to 16 fixed table entries or up to 512 dynamic table entries can be defined and the page size can be configured to any value from 2^12 (4096 bytes) to 2^32 (4 gibibytes).

There are three main possibilities I can think of:

Use a single, very large fixed page to map PCI bus address 0. Using a page size of 2^30 or 2^31 would allow access to PCI bus addresses in the range 0 to 2^30 - 1 (0 - 1 GB ) or 0 to 2^31 - 1 (0 - 2 GB ). (I don't think a page size of 2^32 could be used, as that would use up all the Avalon address space.)
Use a single or a few fairly large dynamic pages. This would allow access to one or a few fairly large, contiguous regions of PCI bus address space, dynamically configurable by the host PC software.
Use a large number of small (4096-byte) dynamic pages. This would allow a large number of 4096-byte pages to be simultaneously mapped from PCI bus address space into Avalon address space. The page size of 4096 would match the physical memory page size of the host system (at least for most host architectures). If using mSGDMA controllers, you'd need one page table entry for each committed scatter-gather descriptor, up to a maximum of 512, so the maximum amount of PCI memory that could be mapped simultaneously would be 2 MB (4096x512 byte).

(The above is just my understanding and may be partly or completely wrong. Corrections of my misunderstanding are appreciated!)

I don't think we have the budget to use third-party IP in this project, but I'm interested in how the Lancero solution compares to the Altera offerings.
Altera_Forum
Honored Contributor
14 years ago
To get any performance at all out of PCIe, you have to use PCIe burst transfers (these are hdlc frames that carry multiple words of data).

So if you are doing a PCIe read, the PCIe interface logic has to know how many words of data you are going to require before it requests any of the data. This can only really be done if the DMA controller in integrated into the PCIe master logic.

Although there are pipelined Avalon cycles, I suspect they are not capable of being converted into PCIe read bursts (they might be translatable into PCIe write bursts as that is easier).

But I've not used the Altera PCIe master (just the slave).
Altera_Forum
Honored Contributor
14 years ago
Quoting last two posts (different authors):

--- Quote Start ---
The above is just my understanding and may be partly or completely wrong.

--- Quote End ---

I think you are quite right there.

--- Quote Start ---
This can only really be done if the DMA controller in integrated into the PCIe master logic.
--- Quote End ---

There are a lot of optimizations and removal of limitations if the SGDMA controller is integrated in the PCIe master logic. Arbitrary access to 64-bit host memory is just one thing.

--- Quote Start ---
... I'm interested in how the Lancero solution compares to the Altera offerings.
--- Quote End ---

I don't think this forum is meant to discuss Altera Partner IP, so I consider to look at its web page and ask.
Altera_Forum
Honored Contributor
13 years ago
We have started a new design using Qsys targeting a Stratix V and have arrived at the same dilemma. In fact, I had come up with almost the same possibilities you listed to work with the Avalon-MM Stratix V Hard IP for PCI Express. I’ve begun considering the unpleasant option of trying to use the Avalon-ST version of the PCIe Hard IP and developing our own TLP encoders and decoders, DMA controllers, etc. I’m very curious which solution you ended up using, ijabbott. Cheers!
Altera_Forum
Honored Contributor
13 years ago
IP Compiler for PCI Express generates the Avalon slave 'tsx' port with txs_burstcount[6..0] - this suggests bursts of up to 64 cycles * 64bits data width = 512 bytes (128 DW) which is a reasonable size for a burst TLP. With this port the TLP headers can be correctly set ahead of payload transmission.

Since the Core uses a fixed hardware buffer for retransmission, with fixed 'pages' for each TLP (rather than circular buffer) there is a payload limit for packets of either 256 bytes or 128 bytes. So certain values of the txs_burstcount imply packets too big to buffer, with 256 bytes limit would be 64DW or 32 burst cycles.

Given the Altera DMA controller uses this same HW port to do it's DMA writes you should be able to get at same performance writing to that port yourself without resorting to the streaming interface. Using the streaming interface I doubt you can exceed 64DW as you need retransmission support.
Altera_Forum
Honored Contributor
13 years ago
Shuckc is correct about the bursting properties of the bridge -- if you create a burst transaction on the Avalon side of the bus, it will pack all the data it can (based on other PCIe rules) into one TLP for maximum throughput.

However, I think the question you're asking is more basic. There are two modes you can use the PCIe IP in: Completer only or requester/completer. If you're going to use a DMA in the FPGA, you need to use requester/completer mode. The DMA will write data to the "txs" (transmit slave) port on the IP (this port is in addition to one or more BARs, which are avalon master ports). Then you'll use an address translation table to convert Avalon addresses to PCIe (64-bit) addresses. For the txs port, the translation table can contain 512 entries of a given size (so you could have 512 x 1MB pages, or 512 x 2MB pages, or fewer entries 4 x 256MB pages, or smaller -- 2x 4kB, etc.). These can be static or dynamically allocated (through the CRA port). Have a look at the IP for PCI Express User Guide section called PCI Express Avalon MM Bridge, and the "address translation" section. Basically on the avalon side, the MSbits of your master address (going to the txs slave) will be translated to (a greater number of) the MSbits of the 64-bit PCI address space, based on the table.

This is separate from the address translation table for BARs, which operates a lot differently, but won't get you the throughput you're looking for.

So if your txs port was set up with translation table size of 1KB (to make the math easy) and you had 2 entries, and you set up the txs port to start at avalon location 0x0, you'd have an address range of 0x0 to 0x800 for the txs port. Now your translation table would look like:
0x0-0x3ff -> some 64-bit address range that's 1k in size
0x400-0x7ff -> another (possibly totally different) 64-bit address range that's 1k in size.
Altera_Forum
Honored Contributor
11 years ago
--- Quote Start ---
Shuckc is correct about the bursting properties of the bridge -- if you create a burst transaction on the Avalon side of the bus, it will pack all the data it can (based on other PCIe rules) into one TLP for maximum throughput.

However, I think the question you're asking is more basic. There are two modes you can use the PCIe IP in: Completer only or requester/completer. If you're going to use a DMA in the FPGA, you need to use requester/completer mode. The DMA will write data to the "txs" (transmit slave) port on the IP (this port is in addition to one or more BARs, which are avalon master ports). Then you'll use an address translation table to convert Avalon addresses to PCIe (64-bit) addresses. For the txs port, the translation table can contain 512 entries of a given size (so you could have 512 x 1MB pages, or 512 x 2MB pages, or fewer entries 4 x 256MB pages, or smaller -- 2x 4kB, etc.). These can be static or dynamically allocated (through the CRA port). Have a look at the IP for PCI Express User Guide section called PCI Express Avalon MM Bridge, and the "address translation" section. Basically on the avalon side, the MSbits of your master address (going to the txs slave) will be translated to (a greater number of) the MSbits of the 64-bit PCI address space, based on the table.

This is separate from the address translation table for BARs, which operates a lot differently, but won't get you the throughput you're looking for.

So if your txs port was set up with translation table size of 1KB (to make the math easy) and you had 2 entries, and you set up the txs port to start at avalon location 0x0, you'd have an address range of 0x0 to 0x800 for the txs port. Now your translation table would look like:
0x0-0x3ff -> some 64-bit address range that's 1k in size
0x400-0x7ff -> another (possibly totally different) 64-bit address range that's 1k in size.
--- Quote End ---

How about PCIe Qsys design flow by using avalon-mm interface? How user logic feed data and receive from PCIe core inside Qsys?

Forum Discussion

PCIe bus master random access to all host memory

14 Replies

Recent Discussions

Agilex3/5 GTS Hard Ethernet IP 10G example design pin loc and io std wanted

Agilex 7 slew rate reconfiguration

Agilex-7 AXI MCDMA for PCIe hang

Constraints not being picked for DCFIFO

Can't generate F-Tile Ethernet Hard IP Design Example