Resource Collisions when installing PCIe DD

I am now adding printk's to the kernel and re-building it ... I do know where it fails but need to track back as to why the dev-> resource's are start = 0, end = 0xFFFFF, for the first resource ... where I believe each resource is a block of memory map allocated to the BAR register.

To make it simple I first reduced the BAR size to be 20 bits .. from 28 bits since that amount of memory was not available in the map.

I then tried making BAR 0 32 bits and not pre-fetchable. BAR 1 disabled and BAR 2 32 bits.

I have now reduced to just BAR 0, non-prefetch ...

I have to believe the system supports non-prefetch memory which is necessary for any control registers.

The particular test that is failing is for the first dev->resource.start != 0 and dev->resource.end = 0xFFFFF. the first test fails and the printk reports a

"resource collision". I need to work back to see why the start value isn't a meaningful number ...

The dmesg outpus doesn't seem to have any particular errors.

Any ideas on this ?

Thanks BCD.

BAR registers are typically for access to small regions, eg., typically no more than 1MB regions (20-bits). Any larger than this, and a lot of PCs will not even boot, eg., your 32-bit BAR is asking for a 4GB region in the host memory map. If that host is 32-bits ... well ... that ain't going to happen :)

Try smaller BAR regions and see if that helps.

Ignore any arguments in favor of large BARs, that discussion can come after you resolve this issue :)

Cheers,

Dave

Thanks Dave,

I did reduce the BAR sizes to 1MB without changing anything and there was a dmesg line that indicated the same but at the end the resource collision message was still the result. The system is an ARM 32 bit system ... I traced the kernel source back to a resource allocation section which I believe must be responsible for setting the resource->start and resource->end values. ... later on in mach-msm the pcie.c code run through a 0 .. 6 for loop examining the start and end values for each resource. The fail is where the start value is 0 which I am able to see via a printk. The indx vale in the for loop was 0 ... and I interpreted that at the BAR0 resource but may be mistaken.

Currently I have reduced to a single BAR0 (32bit address ) and I did notice there are some requirements checked before the resource allocation code does it's job ... like checks for if pre-fetching is allowed or not ... I need to check on ARM where this decision is made. in /proc/mem I do see space allocated for the memory beyond the PCIe bridge ( Synopsys IP ) The original reference design has BAR0/1 as pre-fetchable and BAR2 not ... I guess this is since BAR2 handles NIOSII IP where pre-fetching could have side-effects that we can't handle. So I will run with and without pre-fetching on BAR0 and see what happens .

BTW, 1BM isn't mach menory per BAR. The system memory map allocates 128MB of space for PCI ... That seems a fair bit for the BAR's and possibly ROM ... but I ca see that 4GB is too large and the dmesg output indicate that.

Best Regards, Bob

--- Quote Start ---

I did reduce the BAR sizes to 1MB without changing anything and there was a dmesg line that indicated the same but at the end the resource collision message was still the result. The system is an ARM 32 bit system ...

--- Quote End ---

As a debug "tool", I'd try booting the board in an x86 host and see what the Linux messages are on that machine. The "problem" may not be due to your PCIe setup, but due to the PCIe configuration logic in the ARM bootloader.

--- Quote Start ---

I traced the kernel source back to a resource allocation section which I believe must be responsible for setting the resource->start and resource->end values. ... later on in mach-msm the pcie.c code run through a 0 .. 6 for loop examining the start and end values for each resource. The fail is where the start value is 0 which I am able to see via a printk. The index vale in the for loop was 0 ... and I interpreted that at the BAR0 resource but may be mistaken.

--- Quote End ---

Sorry, I haven't dug around in that code. The place to look might actually be in the bootloader. If you boot to U-Boot, then you should be able to browse the PCIe devices.

--- Quote Start ---

I will run with and without pre-fetching on BAR0 and see what happens .

--- Quote End ---

I have not seen any hosts have problems with PCI/PCIe BARs marked as pre-fetchable versus not. I typically make the regions pre-fetchable and then make sure there are no read "side effects".

--- Quote Start ---

BTW, 1BM isn't mach menory per BAR. The system memory map allocates 128MB of space for PCI ... That seems a fair bit for the BAR's and possibly ROM ... but I ca see that 4GB is too large and the dmesg output indicate that.

--- Quote End ---

Type lspci on your host PC and look at the BAR sizes. I doubt you will find many with large BAR sizes.

The BAR regions allow host CPU access, and that access is typically terribly low performance. All of your "real work" should be done using DMA, and the DMA controller usually exists on the peripheral board, and the peripheral board needs to have bus master capability. The DMA controller can access/generate 32-bit and 64-bit PCIe addresses, so it can "see" the complete memory map of the PCIe segment it resides on.

Another use for BARs is as a window, eg., you can have a BAR with control registers and another that is the "window", and via the control registers you can set a base address that defines where the "window" looks. I have PowerPC processor boards where I can use a 1MB BAR to copy FPGA images of many MB to DDR memory and then run PowerPC DMA controller tests. The key is that a large BAR is not needed to access all of the memory on your board if you just want to "poke" around.

Cheers,

Dave

Hi Dave and team, I owe you an answer here ... I wasn't really getting anywhere adding printk messages to the PCIe kernel code so elected a different approach.

I tried a $30 dual USB PCIe endpoint from Fry's and it had resources allocated just fine. I recorded the dmesg output to a text file then did the same with the Arria V starter kit board with my

Gen1 X1 PCie design and again recorded the dmesg output to a text file. I then diffed the two outputs and evaluated the differences. Oh, I should say that I modified the FPAG design to match the BAR's and sizes that were showing up on the dyal USB endpoint ... it still failed with a "resource conflict" Anyhow the first difference was the Class code returned from the endpoint config space. The FPGA was returning 0x00000000, since I was not setting it to anything and the USB PCIe endpoint card returned a non-zero value indicating some serial bus endpoint. So ... I copied that class code to the FPGA design and the resources were allocated as expected. I have not traced through the PCIe kernel code to determine why a Class of 0x00000000 causes problems but the PCIe reference indicates a value of 0x00000000 indicates some early adapter so I assume the kernel code is not supporting a 0x00000000 code.

I have made progress on the Linux device driver based on the 2 samples provided from Altera. I also found that the Linux I have doesn't support pre-fetchable memory attribute on the BARS.

And I set the BAR0 to 32 bit address space and defined the BAR1 as 32 bit address to access the DMA controller and mail boxes for NIOS <-> ARM system ( rc ) communications .

I now have IOCTL functions supporting the ARM system as a "producer" and have NIOS code to implement the "consumer" side ...

I plan to add dedicated PCI order checking state machines at the output of the hard PCIe IP at the Avaon MM BAR0 output and where read completion data is mastered ... to monitor dedicated "data" coming down the BAR0 master and a dedicated "flag" pattern which will be a NIOS read completion. As a debug tool I plan to have the order tracking state machine inputs ( patterns) drive FPGA outputs to drive user LED's and be tracked by logic analyzer.

Dave, on a new subject, I want to run CVP to manage the FPGA configuration from the system side ... but I also want to load the NIOS .elf or .flash code from the system side as well. Where

should I go for advice on say configuring the NIOS Flash from the system side .... I believe maintaining both from the system side is the only reasonable way to manage "configuration" and "code" levels since as a manual operation it would be difficult to manage. Would it be as easy as adding a FLASH controller and having the system update the FLASH diirectly wiith a new .flash file ?

Thanks, Bob.

--- Quote Start ---

I owe you an answer here ...

--- Quote End ---

Thanks! :)

--- Quote Start ---

I wasn't really getting anywhere adding printk messages to the PCIe kernel code so elected a different approach.

I tried a $30 dual USB PCIe endpoint from Fry's and it had resources allocated just fine. I recorded the dmesg output to a text file then did the same with the Arria V starter kit board with my

Gen1 X1 PCie design and again recorded the dmesg output to a text file. I then diffed the two outputs and evaluated the differences. Oh, I should say that I modified the FPAG design to match the BAR's and sizes that were showing up on the dyal USB endpoint ... it still failed with a "resource conflict" Anyhow the first difference was the Class code returned from the endpoint config space. The FPGA was returning 0x00000000, since I was not setting it to anything and the USB PCIe endpoint card returned a non-zero value indicating some serial bus endpoint. So ... I copied that class code to the FPGA design and the resources were allocated as expected. I have not traced through the PCIe kernel code to determine why a Class of 0x00000000 causes problems but the PCIe reference indicates a value of 0x00000000 indicates some early adapter so I assume the kernel code is not supporting a 0x00000000 code.

--- Quote End ---

I haven't tried a class code of 0, so hadn't had that one trip me yet :)

--- Quote Start ---

I have made progress on the Linux device driver based on the 2 samples provided from Altera. I also found that the Linux I have doesn't support pre-fetchable memory attribute on the BARS.

--- Quote End ---

In what way?

The tests I did in this thread (see the PDF)

http://www.alteraforum.com/forum/showthread.php?t=35678

have pre-fetchable regions.

--- Quote Start ---

And I set the BAR0 to 32 bit address space and defined the BAR1 as 32 bit address to access the DMA controller and mail boxes for NIOS <-> ARM system ( rc ) communications .

I now have IOCTL functions supporting the ARM system as a "producer" and have NIOS code to implement the "consumer" side ...

--- Quote End ---

Why IOCTL codes? You can use the mailboxes and interrupts to create an inter-locked handshake. Here's an old PCI example ...

http://www.ovro.caltech.edu/~dwh/correlator/pdf/cobra_driver.pdf

--- Quote Start ---

I plan to add dedicated PCI order checking state machines at the output of the hard PCIe IP at the Avaon MM BAR0 output and where read completion data is mastered ... to monitor dedicated "data" coming down the BAR0 master and a dedicated "flag" pattern which will be a NIOS read completion. As a debug tool I plan to have the order tracking state machine inputs ( patterns) drive FPGA outputs to drive user LED's and be tracked by logic analyzer.

--- Quote End ---

Why do you need this? If you are DMAing data, then use an interrupt to indicate the end of the transfer.

--- Quote Start ---

Dave, on a new subject, I want to run CVP to manage the FPGA configuration from the system side ... but I also want to load the NIOS .elf or .flash code from the system side as well. Where

should I go for advice on say configuring the NIOS Flash from the system side .... I believe maintaining both from the system side is the only reasonable way to manage "configuration" and "code" levels since as a manual operation it would be difficult to manage. Would it be as easy as adding a FLASH controller and having the system update the FLASH diirectly wiith a new .flash file ?

--- Quote End ---

Here's what I would do (assuming you have SDRAM on the board for the NIOS II processor);

1. Instantiate a NIOS II core that boots from SDRAM.

This core powers-on with its reset asserted. The reset register would be something the host can toggle, eg., an Avalon-MM register.

2. Use the host to copy the NIOS II images into SDRAM.

3. Use the host to release (deassert) the reset line.

The NIOS II processor would then boot. The host can wait for a message from the NIOS II in the mailbox.

Here I assume that you have a "mailbox" for the host-to-device and another for device-to-host transfers, and that these two mailboxes generate interrupts at their respective device.

Cheers,

Dave

Dave Thanks for the reply.

"have pre-fetchable regions."

I believe pre-fetchable space is generally supported but I believe the particular Linux I am working with which is a development version may not support pre-fetchable memory on the endpoint somewhere down in the kernel resource related code ... I can investigate this but not right now.

"Why IOCTL codes? You can use the mailboxes and interrupts to create an inter-locked handshake. Here's an old PCI example ..."

I am just using IOCTL at this tome to provide some simple system side facility to test code running at the application level.

"Why do you need this? If you are DMAing data, then use an interrupt to indicate the end of the transfer."

I can understand why this may not be obvious ... the short answer is that I am taking a verification test suite which includes a "produce consumer" stress test and

supporting it in an Emulation and Post-Silicon environment . The produce / consumer test could use an interrupt to signal that the transfer is complete but the implement producer / consumer

test transfers data , and has a flag and status value to synchronize between the producer and consumer. The flag and status can randomly mapped to endpoint memory or system memory in all possible permutations. So the ARM processor in the system produces data and the NIOS core consumes data. The reason for this is to stress bridges between the producer and consumer which need to obey PCI ordering rules.

So ... this isn't a simple DMA data transfer but a stress test to run verification tests in a post-silicon validation environment.

In general MSI interrupts will stay in order across bridges since the MSI is just another write and will stay ordered behind the write data. In the producer / consumer test the write data may target endpoint memory while the endpoint NIOS polls the flag in system memory to determine the data write has completed. If the bridges honor PCI ordering rules , the flag read completion will flush posted data writes in bridges. I am stressing that activity in this test. I hope that makes sense as a "normal" data transfer may involve DMA followed by an interrupt but we can't constrain all activity to be "normal".

Best Regards, Bob

Hi Bob,

Thanks for the clarifications. It sounds like you've got things under control :)

Cheers,

Dave

--- Quote Start ---

Hi Bob,

Thanks for the clarifications. It sounds like you've got things under control :)

Cheers,

Dave

--- Quote End ---

Thanks Dave, I very much appreciate your support.

The local supplier rep. wasn't sure about loading the NIOS executable over the PCIe link and your suggestion makes sense.

Right now the NIOS is executing from internal memory since the code is small and it was an easy starting place.

If I can figure out how to load the .elf into that memory, I believe you are saying the NIOS core can be released from reset or

controlled via a register output to start executing the reset vector and the system ( rc ) side could do the load and reset register operation.

I will investigate that approach.

Thanks, Bob.

Dave ,

on re-reading your suggestion to load the NIOS code via the PCIe link ... I now understand you are saying to load up the SRAM data and the SRAM controller will be in reset ... upon releasing that reset the NIOS core will start ... does that mean that the NIOS core will be stalled with a fetch to the reset vector memory location, until the SRAM controller is active and can honor the first Avalon MM bus read request ?

Thanks, Bob