PCIE example issues

Dear KhaiChein_Y, thanks for reaching out.

I assume the name of the zipfiles speak for its content.

Main changes:

1. splitted altpcietb_bv_rp_gen2_x8 into two files: altpcie_bfm_rpvar and altpcietb_bv_rp_gen2_x8_ex_rpvar, to make the editing of the last more easy
2. added many $fdisplay statements to more easily track bitstreams (i always get lost in the wave screen)
3. changed a lot in tlp_parser to generate headers for 8 byte CplD. The result of that can e.g. be seen in outparsout.txt
4. added 8 byte wide write to and read from memory (interconnect_low and ~high, and mem_low and ~high) in pcie_ed
5. changed a bit in downstream_driver to generate 8 bytes.
6. in pcie_ed, line 546 and 566, 1 is added to each read byte
7. In the downstream_drive file, line 465, Length = 4, as written by INTEL. If you run the testbench, it fails of course because of the adding of 1 to each byte. But downstream_drive is modifies such, that it does not stop, it just reports a difference.
8. Change this line 465 to 8, and the tb sends 8 byte, the apps writes it to MEM in pcie_ed, read it back, adapts the TLP headerfiles, of which you can see the result in pcie_ed_tx.txt, and eventually sends seemingly correct data to the HardIP.
9. At receipt in TB, in rpvar, rx_data remains zero, but rx_desc0 has some recognizable values. Also, there are 'x' in the serial wires tx0..3 from the DUT, to the rx0...3 of rpvar. This suggests that the HardIP of PCIE is not controlled correct by TLP headers? configuration setting?
10. I hope this clarifies, please do not hesitate to ask clarification if needed.

Regards ,Pieter

Hi KhaiY,

Glad to hear you are still on it....

Of course, i understand, it is a F**ng difficult testbench / apps / bfm. But i was triggered by INTEL if the answer provided was sufficient to close the call. (i thought with all those smart AI of today, the bot would have seen there is just a question (yours) and an answer (mine).....)

Regards, Pieter

Dear KhaiY. Yes, of course i tried without modification It runs fine. Take care: i modified the data being sent, and being send back, by 1/ making them different per loop, and 2/ adding 1 to each byte. Of course the comparator does not like that, so i changed it that it reports a difference, but does not stop as a result of that (other than the design example, where it would stop).

For more input/comments, check the attached word file, and check the transcript line 3640 -- end for the output of the comparator when Byte = 4.
I also attached a transscript_08.txt to show for Bytes = 8

regards, Pieter

Now i see: in your folder customer/ip/pcie_id/ there is only a small subset of files. Even more strange, is that in my design those files do not exist at all!!!???

There should be the set of files as shown in the pci_202_mod/pcie_a10_hip_blabla/ip/pcie_ed/directory. My assumption was you would load the design example from INTEL website, and then replace the changed files with my files. (quartus 20.2) Apparantly the design example is not properly loaded into your machine, hence it will definitly not run.

The problem is: the fully compressed design is 0.7 GB (with 762 files), which does not fit in the drag & drop field below, so i cannot send this design to you, unless via a direct link to your email.

So, my suggestion is:

So, please install the full design from the INTEL website, and make sure it runs. I am afraid you loaded a different design, given those clock_in and reset_in files that do not exist in my design.

Then replace the files from INTEL example, with the files in the attachements, that i sent you earlier.

But maybe another appraoch example is also possible:

What i really want, is connect my ethernet PHY, via the INTEL Low Latency MAC, to the PCIE Hard IP (gen2 x 4) and map it onto the cyclone10GX development board.

(I have in the INTEL design example for Low Latency MAC, replaced the INTEL PHY iwth my own PHY, and it runs in simulation, and i still have difficulties in getting the tranceiver properly mapped onto hte cyclone10GX)

If you do have an example with this, (but then of course with the INTEL PHY), that would be most welcome!

Regards, Pieter

Hi KhaiY,

Thanks for your response. Something in our communication seems not working.

the design example you suggest, is the one i am working with:

Cyclone 10 GX PCIe Gen2 x4 Avl-ST

The design example i started with, and for which i ask your advice, is the one described in UG-01145_avst | 2020.06.02 (newer versions may exist), and which you can download from the IP Catalog in Quartus PCI Express / Intel Arria 10 / Cyclone 10 Hard IP for PCI Express => Platform Designer => Parameter setting: System Settings: Standalone, Avalon-ST, Gen2x4, 64 bit @ 250 MHz, Native Endpoint, Balanced, RX Buffer: Header: 112 Data: 440. All other settings: default. Example Design: PIO, Development Kit: Arria 10 GX Development Kit (I have the Cyclone 10GX development kit, and i understand this should not be a problem)

I asked you: is it possible to change the testbench and application, such that transfers with Payloads of many bytes is possible, the design example only allows for 4 Bytes. I changed the testbench, the TLP Parser such, that in my opinion it should transfer 8 Bytes, but so far no success. My assumption is: if I manage 8 Bytes, i can also change to many bytes, as long as within the boundaries of the maximum payload.
What i really want: connect INTEL's Low Latency MAC 10-GBASER design example to the hard IP PCIE of the PCIE example design as described above. Instead of finding out the wheel myself, an example design doing exactly this will be most appreciated. The list of design examples you send me, do not contain (unless i missed something, you can never exclude that) such an example.
So, I really hope somebody did this is exercise (connect LL MAC to PCIE) and is willing to share this code with me. Otherwise, I have to do it myself, and a step to achieve this, is to make the PCIE design example suitable for transfers >4 Bytes, for which i need your help.

I hope this clarifies my requests...

Regards, Pieter

Hi KhaiY, thanks for updating me. It is really not an easy part of code.

What i could not figure out, or not understand is: if you load the board with the example code, and connect it to the other PC, as described in document an-855 paragraph 1.7 Hardware Installation, does this also use the very same configuration? Because it says: ($1.8) Set the Transfer length to 100,000 bytes and the Sequence to Write only,....transfer data from the FPGA to the system memory in chunks of 100,000 bytes. So, you would expect the very same application (APPS and the others) really do accept and send back packets of length 512 / 1024 / 2048 packet to larger chunks... If so, then why not in the testbench?

(we have this software test working)

Looking forward to your progress / feedback

Regards, Pieter

Hi KhaiY,

thanks for response, but i don't understand it.

I thought i explained my goals (see earlier posts):
What i really want, is connect my ethernet PHY, via the INTEL Low Latency MAC, to the PCIE Hard IP (gen2 x 4) and map it onto the cyclone10GX development board.
(I have in the INTEL design example for Low Latency MAC, replaced the INTEL PHY iwth my own PHY, and it runs in simulation, and i still have some difficulties in getting the tranceiver properly mapped onto the cyclone10GX).
If you do have an example with this, (but then of course with the INTEL PHY), that would be most welcome!

For PCIE testbench I am using the example of UG-01145_avst. The difficulty is not in the testbench, but in the APPS component, in fact, i believe it is in the TLP parser that is not built to send anything back but ONE DWORD of 4 byte. I tried to modify the TLP parser to make it suitable for 2 DWORD of data, but somehow the Hard_IP PCIE block makes a mess out of it.

The UG-01145 document says (page 157): "It can only handle received read requests that are less than or equal to the currently set Maximum payload size option specified under PCI Express/PCI
Capabilities heading under the Device tab using the parameter editor. Many systems are capable of handling larger read requests that are then returned in multiple completions.

That would be good enough for me, but i dont get it working! It only seems to work for 4 Bytes!
And if you look at the code, (driver_downstream.v), line 270 - 289, the length is cut off at 4 Bytes, and there is a remark:
line 271: //TODO extend to more than 1 DW.

So, it seems as there is some work to be done ??????!!!!!

So, you can help me in several ways:

1/ provide me with a TLP parser that is able to reply correctly with packets of many bytes. I think i can manage to integrate that into the APPS entity and the MEM entity, and modify the testbench such, that it maybe does not compare, but at least i can verify manually that received = send

OR:

2/ provide me with an example where your LL_MAC_10GBASE_R example is already integrated into a PCIE gen2x4 example

OR

3/ explain me how, without using the testbench, but by directly building the above (PCIE + LLMAC_10GBASER) and synthesise/place/route it onto the Cyclone10GX development board, and run a software test similar to the one described in AN 855: PCI Express* High Performance Reference Design for Intel® Cyclone® 10 GX.

Thanks for your feedback!

regards, Pieter

Dear KhaiY,

As i told you, we already worked with hte example in UG-20162. I understand you don't have an example design with the two connected.

What i don't understand, is that you write in the manual for the PCIE example:

The UG-01145 document says (page 157): "It can only handle received read requests that are less than or equal to the currently set Maximum payload size option specified under PCI Express/PCI
Capabilities heading under the Device tab using the parameter editor. Many systems are capable of handling larger read requests that are then returned in multiple completions.

meanwhile, i managed to send 8 bytes, but is a hard struggle. In fact, i am disappointed that this testbench / example cannot handle anything else but those 4 bytes. I reluctantly seem to have to accept that INTEL is not intented to repair this shortcoming, although the manual says it does accept TLP until max payload. Regards Pieter