Problems with SGDMA through PCIe in several motherboards

I have no practical experience with the SGDMA example, except for looking at the generated code and … deciding to write my own application code connecting to the PCIe IP on the Avalon ST layer.

I cannot be of big help here, but such issues typically arise if one (=the old) chipsets or bridge chips are forgiving about some irregularities in the PCIe transactions while another (=the new) chipsets or bridge chips are more strict.

For example, PCIe requires the application to issue requests with 4 DW headers only for addresses where the upper 32 bits are nonzero. If the upper address bits are zero, a 3 DW header request must be used. Now, some systems work without an issue with just 4 DW header requests, even for addresses in the lowest 4 GiB range. You change the motherboard and – bang! – nothing works because the illformed requests are dropped.

Other sources of trouble, when changing the system, are changes in the timing that uncover race conditions that were hidden due to limitations of the old system.

For example, consider an MSI that should push out all written data so that the interrupt service routine (or bottom halve) can read this value and act on the new value accordingly. Now, if the application is wrongly designed so that the MSI could overtake the final write request, the old system might act slow enough on the MSI that the data is already fresh when read by the ISR. But the new system could be faster in activating the ISR or slower in updating the written data, resulting in the ISR reading the old value and probably finishing without having done anything. Of course, in such a case it’s not the fault of either the old or the new chipset, but the application was buggy.

Similar race conditions can be present in the driver as well, reading/writing not strictly in correct order, and at the moment the system takes the chance of changing order within spec, things start to mysterically fail. Memory barriers are something most developers don’t care about much and instead think they can rely on volatile memory locations.

Another source of trouble when changing the system architecture might be the setting of the memory BARs, e.g. difference between prefetchable and non-prefetchable, which might be handled differently by each system, e.g. by prefetching data for read, or by collecting multiple write requests into a single request.

As I said, not much concrete help, but maybe something to consider.

– Matthias