PCIe bus master random access to all host memory

Our design is basically a plug-in PCIe card that is normally a PCI target, but which will support PCI bus-master DMA transfers. We'll be using the PCIe hard-IP on Cyclone IV with the Qsys design flow and the Avalon-MM interface.

We were thinking of using Altera's Modular SGDMA (mSGDMA) controller. As I understand it, that only works with 32-bit Avalon addresses and wouldn't know anything about PCI bus addresses, so some mapping between PCI addresses and Avalon addresses would be required.

As I understand it, the Altera PCIe hard-IP provides the standard PCI BAR registers to map certain regions of the Avalon address space into the PCI bus address space. The Avalon base address and size of each BAR region is assigned in the FPGA design, and the assignment of the PCI bus base address of each BAR region is assigned by the PCI host (on a PC, firstly by the PCI-BIOS code, and later by the host operating system).

Things like the mSGDMA controller only know about Avalon addresses, so in order for it to access regions in the PCI bus address space, the PCI bus addresses need to be mapped into the Avalon address space. That's what the "Avalon-MM-to-PCI Express Address Translation Table" is for. This table is either fixed or dynamic and maps one or more "pages" of PCI bus address space into the Avalon-MM address space. Up to 16 fixed table entries or up to 512 dynamic table entries can be defined and the page size can be configured to any value from 2^12 (4096 bytes) to 2^32 (4 gibibytes).

There are three main possibilities I can think of:

1. Use a single, very large fixed page to map PCI bus address 0. Using a page size of 2^30 or 2^31 would allow access to PCI bus addresses in the range 0 to 2^30 - 1 (0 - 1 GB ) or 0 to 2^31 - 1 (0 - 2 GB ). (I don't think a page size of 2^32 could be used, as that would use up all the Avalon address space.)

2. Use a single or a few fairly large dynamic pages. This would allow access to one or a few fairly large, contiguous regions of PCI bus address space, dynamically configurable by the host PC software.

3. Use a large number of small (4096-byte) dynamic pages. This would allow a large number of 4096-byte pages to be simultaneously mapped from PCI bus address space into Avalon address space. The page size of 4096 would match the physical memory page size of the host system (at least for most host architectures). If using mSGDMA controllers, you'd need one page table entry for each committed scatter-gather descriptor, up to a maximum of 512, so the maximum amount of PCI memory that could be mapped simultaneously would be 2 MB (4096x512 byte).

(The above is just my understanding and may be partly or completely wrong. Corrections of my misunderstanding are appreciated!)

I don't think we have the budget to use third-party IP in this project, but I'm interested in how the Lancero solution compares to the Altera offerings.