How to obtain bandwidth information of CXL memory on Intel Agilex 7 FPGA I-Series Development Kit

Hi there,

1. Does my host and CXL device currently follow CXL 1.1 have the ability to monitor CXL memory access bandwidth?

[Altera] you can try the “MLC” tool from intel, please refer to the Design example guide

1.1 In https://www.intel.com/content/www/us/en/developer/articles/technical/fourth-generation-xeon-scalable-family-overview.html , it say 4th Gen Intel® Xeon® Scalable Processors support CXL 1.1,

but on lspci, my CXL device show "CXL: Intel Corporation Device 0ddb (rev 01) (prog-if 10 [CXL Memory Device (CXL 2.x)])"

I am very confused whether my device is CXL 1.1 or CXL2.0 ?

[Altera] For both CXL1.1 and CXL2.0 , the link speed is 32GT/s, but there is new added filed in the Training Sequence(TS) to indicate the CXL2.0 link capability (R-Tile doesn’t support this filed as it the silicon limitation) thought the finial link speed is still 32GT/s , but from the Transaction layer perspective, R-Tile supports most of the CXL 2.0 feature, that is the reason we claim R-Tile as the CXL1.1 link but CXL2.0 device, regarding the "CXL: Intel Corporation Device 0ddb (rev 01) (prog-if 10 [CXL Memory Device (CXL 2.x)])" during the kernel info display, this depends on the Kernel behavior, so it might discover CXL2.0 feature on the R-Tile (32GT/s with CXL1.1 link)

2. If have, how should I monitor it?

My attempt, but it didn't work：

1. I can see 16GB CXL memory work as numa node4, and I can see /dev/dax0.0 work in "system-ram" mode.

But i can not see /dev/cxl/mem0 device, and demsg report : "cxl_pci 0000:bd:00.0: Device DVSEC not present, skip CXL.mem init"

So I turn for CXL driver code, follow https://lore.kernel.org/lkml/ZaGVm5tHXoTZd4Ad@debian/T/ modified my cxl driver from linux-6.5.0, then I get /dev/cxl/mem0 at address 0000:bd:00.0.

However, dmesg reports an error :CXL port topology not found, and /dev/cxl/mem0 is in a disabled state and cannot be activated.

[Altera] this also depends on the Kernel version , because there is some update from the Kernel to identify the CXL device, regarding the “/dev/cxl/mem0” , might this is the new feature of the Kernel OS, in our 5.16 Kernel version we only monitor the “/dev/dax0.0 work in "system-ram" mode” , we don’t have the platform to support up to 6.5 kernel , and actually you can access “/dev/dax0.0 as "system-ram" mode, regarding the CXL driver you mentioned, not sure if this is the mature driver since we never use it

1. 
   
2. 2. I found some CXL related counters in perf:
3. CXLCM_TxC_PACK_BUF_INSERTS.MEM_DATA, CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA from https://github.com/intel/pcm/blob/master/src/cpucounters.cpp#L10426 UNC_CHA_TOR_INSERTS.IA_MISS_RFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFRFO_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_DRD_PREF_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFDATA_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_LLCPREFCODE_CXL_ACC,UNC_CHA_TOR_INSERTS.IA_MISS_RFO_PREF_CXL_ACC from https://github.com/intel/pcm/blob/master/src/pcm-memory.cpp#L1257
4. but when i monitor them, i only get 0 value, such as
5. UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC: pcm-raw -e cha/config=0x10c8178200000135,name=UNC_CHA_TOR_INSERTS.IA_MISS_DRD_CXL_ACC
6. or
7. CXLCM_RxC_PACK_BUF_INSERTS.MEM_DATA : pcm-raw -e cxlcm/config=4161
8. I see that in the pcm-memory code, it first check if the CXL unit exists before monitor the CXL memory bandwidth. Is it because I do not have an active /dev/cxl/mem0 device that causes the results of monitoring the above counters to be 0, or there is an error when i use pcm-raw?
9. Looking forward to your help.

4th Gen Intel Xeon Processor Scalable Family, sapphire rapids

A technical overview of the 4th Gen Intel® Xeon® Processor Scalable Family based on the formerly codenamed Sapphire Rapids architecture.

[Altera] we don’t know what is the code you mentioned “https://github.com/intel/pcm/blob/master/src/cpucounters.cpp#L10426” , might this is for some specific usage, and seems it for the CPU only because CHA is mentioned, but no such Agent in the FPGA

So , most of the questions are about whatever Platform/Kernel/CPU related code, and could you share us your final purpose regarding FPGA CXL solution ? might we can share something we once verified from the FPGA CXL example design perspective, thanks

Best regards,

WZ

Hi there,

If there's no further questions, would you mind me set this case to close pending first? The close-pending status will last ~20 days before completely closed, so let me know if further assistance is needed, I'll re-open it, thank you for your understanding.

Have a nice day!

Best regards,

WZ