Bug! Quartus Pro 20.1.1, Cyclone V, utilizing PCIe example from 16.1.

I got it working.

The last hurdle was the MSI interface. I'm not sure why it wasn't working, but restarting the design from scratch with my recently acquired knowledge got everything working.

I've attached my qsys file and socfpga.dtsi. Hopefully it will help others get a jump on things so they don't have to learn everything the hard way as I did.

This design is not optimized for speed, nor is it optimized for space.

NVMe read speed is around 80 MB/s.

NVMe write speed is around 50 MB/s.

Faster drives will do a little better, but even the over a gig per second on PC 4 lane part I have doesn't do much better than 110 MB/s read. Bandwidth is limited by the ARM memory interface and the fact that bursting logic at 125MHz causes timing violations. A burst length of one on the Txs interface has got to slow things down.

Performance is quoted with the 5.11 kernel, the 5.4 kernel is not quite as fast. Still fast enough for an embedded system though.

Don't forget to enable the fpga, pcie and msi modules in your top level board dts file.

And don't forget to reserve the first 64K of DRAM as stated above. If you don't you will get read errors.

There are probably better configurations, and eventually I'll probably try to optimize for size since my logic will need to get bigger on the next project. But for now. It finally works.

Good luck to you all.