Help: Windows didn't assign Bar0 to PCie (address mapping is not correct)

--- Quote Start ---

Then how to use Qsys to connect to large memory chip (DDR II)?

If I assign large Bar size covering the whole address of DDR II, computer will not boot.

What to do to make large memory chip to work with PCIe?

--- Quote End ---

You need to use slight-of-hand and indirection :)

You need to think of the memory map of your board as being independent/separate to the memory map of your x86 host. If you want to move results between your Qsys memory map and your x86 host memory, then you use DMA. The DMA controller needs to be able to see both PCIe addresses and Qsys addresses to be efficient.

If you use the x86 host to directly read and write to your Qsys system, it will not be very efficient (low data rate). Typically you use (slow) x86 transactions to setup the DMA controller, and then let it perform efficient transfers.

Cheers,

Dave

but will it reduce BAR0 size? I just concern if my computer can boot.

Do you mean using DMA can reduce BAR0 size if the x86 host memory is still connected to Qsys ?

--- Quote Start ---

You need to use slight-of-hand and indirection :)

You need to think of the memory map of your board as being independent/separate to the memory map of your x86 host. If you want to move results between your Qsys memory map and your x86 host memory, then you use DMA. The DMA controller needs to be able to see both PCIe addresses and Qsys addresses to be efficient.

If you use the x86 host to directly read and write to your Qsys system, it will not be very efficient (low data rate). Typically you use (slow) x86 transactions to setup the DMA controller, and then let it perform efficient transfers.

Cheers,

Dave

--- Quote End ---

--- Quote Start ---

but will it reduce BAR0 size? I just concern if my computer can boot.

Do you mean using DMA can reduce BAR0 size if the x86 host memory is still connected to Qsys ?

--- Quote End ---

The BAR size will be reduced to whatever the common set of control registers requires.

Consider the following simple case. Lets say I have a DMA controller that has the following registers; 64-bit PCIe address, 32-bit Avalon (Qsys) address, direction (PCIe-to-Avalon or Avalon-to-PCIe), length in bytes, a control register and a status register. You need 7 32-bit registers to describe this. The minimum practical BAR size is about 256-bytes, but its more typical to use a page of the host memory, i.e., 4kB or 8kB.

You can program the DMA controller to move a block of data to or from any PCIe address (which includes the host memory) to an Avalon address, which includes your DDR memory.

Note how the host does not need to see the Avalon DDR memory, it is the DMA controller that needs to see those addresses.

Cheers,

Dave

Thank you! Dave,

Do you mean DDR2 is not connected to PCIe Hard IP, it is only connected to DMA? What is the differences between PCIe address and Avalon address? Isn't PCIe address also Avalon address since it often uses Avalon MM interface?

Also, do you have examples/more detailed documents about this?

Thank you very much!

--- Quote Start ---

The BAR size will be reduced to whatever the common set of control registers requires.

Consider the following simple case. Lets say I have a DMA controller that has the following registers; 64-bit PCIe address, 32-bit Avalon (Qsys) address, direction (PCIe-to-Avalon or Avalon-to-PCIe), length in bytes, a control register and a status register. You need 7 32-bit registers to describe this. The minimum practical BAR size is about 256-bytes, but its more typical to use a page of the host memory, i.e., 4kB or 8kB.

You can program the DMA controller to move a block of data to or from any PCIe address (which includes the host memory) to an Avalon address, which includes your DDR memory.

Note how the host does not need to see the Avalon DDR memory, it is the DMA controller that needs to see those addresses.

Cheers,

Dave

--- Quote End ---

--- Quote Start ---

Do you mean DDR2 is not connected to PCIe Hard IP, it is only connected to DMA?

--- Quote End ---

DDR2 and DMA and the Qsys PCIe bridge would be connected to a Qsys system. However, its the operation of a PCIe-to-Qsys bridge that you need to understand.

--- Quote Start ---

What is the differences between PCIe address and Avalon address? Isn't PCIe address also Avalon address since it often uses Avalon MM interface?

--- Quote End ---

No, PCIe addresses and Qsys addresses are two completely separate address maps.

Consider the case of 4 identical FPGA boards plugged into a motherboard. Those four boards could be configured with identical Qsys systems, right? Their Qsys address maps are therefore identical. How can the PCIe address map be identical to the Qsys address map? It cannot.

In this case, how would each of the boards DMA into x86 memory? Well, the host would access each of the board's DMA controllers via registers mapped into a BAR. Each board would be uniquely mapped into the x86 address map by the BIOS. The Qsys source address on each board could be identical, but the PCIe destination address would be unique to each board.

Draw a picture of the address maps, and you'll see that a DMA controller is really a 'bridge' that can see the Qsys address map on one side, and the PCIe address map on the other.

--- Quote Start ---

Also, do you have examples/more detailed documents about this?

--- Quote End ---

No. This is pretty standard for PCI development. However, its not exactly documented anywhere, its more like "that is just the way it is". Once you understand it, it makes sense.

Read the data sheet of the PLX PCI9054 PCI bridge, or read the PCI section of the Freescale MPC8349EA PowerQuicc II Pro processor documentation, and you'll see very similar descriptions of the operation of the PCI bridges.

Cheers,

Dave

Thank you! Dave, another question:

How to have a "Qsys PCIe bridge"? because I can not find it in Qsys component library. what is the exact name? or do we need to build one ourselves?

Thank you very much!

--- Quote Start ---

DDR2 and DMA and the Qsys PCIe bridge would be connected to a Qsys system. However, its the operation of a PCIe-to-Qsys bridge that you need to understand.

Cheers,

Dave

--- Quote End ---

--- Quote Start ---

How to have a "Qsys PCIe bridge"? because I can not find it in Qsys component library. what is the exact name? or do we need to build one ourselves?

--- Quote End ---

I'm investigating the Altera IP at the moment:

http://www.alteraforum.com/forum/showthread.php?t=35678

The existing Qsys component does not implement a DMA controller as part of the bridge, so it does not meet the requirements for what I would consider to be a decent bridge. I need to complete the MegaWizard flow review and then some of the SOPC components on the Altera wiki. If none of them have features that meet my requirements, I'll create a design. I'll then document it and post it to the Altera wiki. I'm currently working on a board design, so might not get started on the bridge design for a little while (since it does not impact the board layout, just the resources used in the FPGA).

What exactly are the requirements of your design? Perhaps I or someone else on the forum can suggest a solution that will get you working now.

Cheers,

Dave

Thank you, Dave.

We have a PCIe Hard IP, 128M 16 bit DDR2 SDRAM and 21K 64-bit on-chip memory.

Qsys assigns 256MB for the DDR2 SDRAM due to Avalon MM interface, plus on-chip memory, the total size of Bar0 exceeds 256 MB, and we can not make it work with WinDriver and our computer (Windows XP).

It seems the maximum size of Bar0 for our computer is 256 MB.

So we want to make both DDR2 SDRAM and on-chip memory working, we want to reduce the size of Bar0.

--- Quote Start ---

I'm investigating the Altera IP at the moment:

http://www.alteraforum.com/forum/showthread.php?t=35678

The existing Qsys component does not implement a DMA controller as part of the bridge, so it does not meet the requirements for what I would consider to be a decent bridge. I need to complete the MegaWizard flow review and then some of the SOPC components on the Altera wiki. If none of them have features that meet my requirements, I'll create a design. I'll then document it and post it to the Altera wiki. I'm currently working on a board design, so might not get started on the bridge design for a little while (since it does not impact the board layout, just the resources used in the FPGA).

What exactly are the requirements of your design? Perhaps I or someone else on the forum can suggest a solution that will get you working now.

Cheers,

Dave

--- Quote End ---

--- Quote Start ---

We have a PCIe Hard IP, 128M 16 bit DDR2 SDRAM and 21K 64-bit on-chip memory.

Qsys assigns 256MB for the DDR2 SDRAM due to Avalon MM interface, plus on-chip memory, the total size of Bar0 exceeds 256 MB, and we can not make it work with WinDriver and our computer (Windows XP).

It seems the maximum size of Bar0 for our computer is 256 MB.

So we want to make both DDR2 SDRAM and on-chip memory working, we want to reduce the size of Bar0.

--- Quote End ---

That tells me what the hardware is, but it does not tell me anything about why. Here's a few questions;

1) What FPGA do you have on the board?

2) Why does the FPGA have memory? What is it to be used for?

3) Why does the host CPU need to see that memory?

4) What is your data flow?

For example, in my application I have high-speed ADCs and signal processing logic in the FPGA. The FPGA results will be written to DDR3 RAM, and then those results DMAed to the host CPU. There is a few MB of data that needs to be transferred every ~20ms. However, those results can be queued, and transferred as a larger DMA transfer every 500ms.

The host CPU will run Linux, and a device driver will setup the DMA controllers on multiple boards to DMA their results into main memory. The main memory will contain scatter-gather DMA buffers. The data will be moved out of the buffers once the host has processed the results.

You need to come up with a similar data flow description for your system.

Cheers,

Dave

1) It is Cyclone IV GX speed grade 7, I modified the design in

http://www.alterawiki.com/wiki/pci_express_in_qsys_example_designs

changed the DDR2 to our 128 M 16 bit DDR2 SDRAM.

2) The memory is to be used for storing some parameters/commands for signal processing

3) There is a lot of RF data from ADCs for the FPGA to process, CPU needs to send commands/parameters, also obtains processed data storage in DDR2 SDRAM from PCIe. I don't know if CPU needs to see that memory or not, if it does not need to see that memory, how to realize ? to get the data from DDR2 SDRAM?

4) FPGA processes the ADC data,CPU sends commands/parameters, access data in on-chip memory or in DDR2 SDRAM. displays processed data for real-time (30frams/s) images on screen.

--- Quote Start ---

That tells me what the hardware is, but it does not tell me anything about why. Here's a few questions;

1) What FPGA do you have on the board?

2) Why does the FPGA have memory? What is it to be used for?

3) Why does the host CPU need to see that memory?

4) What is your data flow?

For example, in my application I have high-speed ADCs and signal processing logic in the FPGA. The FPGA results will be written to DDR3 RAM, and then those results DMAed to the host CPU. There is a few MB of data that needs to be transferred every ~20ms. However, those results can be queued, and transferred as a larger DMA transfer every 500ms.

The host CPU will run Linux, and a device driver will setup the DMA controllers on multiple boards to DMA their results into main memory. The main memory will contain scatter-gather DMA buffers. The data will be moved out of the buffers once the host has processed the results.

You need to come up with a similar data flow description for your system.

Cheers,

Dave

--- Quote End ---