Hello,I'm using Intel FPGA AI 2023.2 on ubuntu 20.04 host computer and trying to infer a custom CNN in a Intel Arria 10 SoC FPGA.I have followed Intel FPGA AI Suite SoC Design Example Guide and I'm able to copile the Intel FPGA AI suite IP and run the M2M and S2M examples.I have also compiled the grpah for my custom NN and I'm trying to run it with the Inter FPGA AI suite IP but I have not clear how to do it. I'm trying to use the dla_benchmark app provided but for example, the input data of my NN (it is trained and graph was compiled in this way) must be float whereas the input data of the IP must be int8 if I'm not wrong.Another problem I have is regarding the ground truth file. I have a ground truth file for each imput file because each groud truth is a 225 array.Is there any additional information or guide to run custom model with Intel FPGA AI Suite?Thank you in advance

Hi,You may refer to AN993 https://cdrdv2-public.intel.com/777191/an993-777190-777191.pdf.Thanks.

Hi @JohnT_Intel ,That application note only describes the process of compiling a graph from a custom NN and address potential issues during compilation. I have already successfully compiled the graph, and now the question is how to perform inference with the custom graph using the Intel FPGA IP. Is this information addressed in another AN or User Guide?

I read in "Intel® FPGA AI Suite: PCIe-based Design Example User Guide", which is not aimed for the device that I am using, that there are two different tools: OpenVINO™ FPGA Runtime Plugin and Intel FPGA AI Suite Runtime. The document explain the classes bot tools have but I do not find any information about how to modify them according to custom NN needs, compile or depoly the NN with them. Is there any guideline?

Hi,Once you have completed the flow in https://www.intel.com/content/www/us/en/docs/programmable/777190/current/example-1-customized-resnet-18-model.html then you can use the generate IR file on OpenVINO. It will be performing as how you run the example design.

Hello,IR and compiled graph were succesfully obtained. The question is how to run the model. In the example S2M image_streaming_app and streaming_inference_app are executed. The problems are:The input date must be in 224x224x3 BGR .bmp format, while my NN input data is 3 channel float data array. The NN was trained in this way and IR and graph was get accordingly. I can convert the data to bmp by my input size is 3360x360x3. I can change width and height with image_streaming_app flags but I don't know if it works because the input buffer.The output is comparted directly with "/app/categories.txt" file and the top 5 scores are given in result.txt. My output is completely different. I have 225 outputs. How can I get the gross output? The other example, M2M example uses dla_benchmark app. And the problem is the same, how to change the input or how to use it with different input size and how to read the output

Intel FPGA AI Suite Inference Engine | Altera Community

31 Replies

JohnT_Altera
Regular Contributor
2 years ago
Hi,

How do you run the OpenVINO application?
- RubenPadial
  Contributor
  2 years ago
  Hello @JohnT_Intel,
  I modified the dla_benchmark application. So load the architecture file and the compiled graph with the dl_benchmark application. I read rawOutputData and write it in a txt file. Then I compute the accuracy with this raw output data.
JohnT_Altera
Regular Contributor
2 years ago
Hi,

How about the model optimization? Do you generate it based on the new architecture?
RubenPadial
Contributor
2 years ago
Hello @JohnT_Intel ,
What changes are needed?
JohnT_Altera
Regular Contributor
2 years ago
Hi,

May I know if you regenerate the model using the TensorFlow 2 models optimization?
https://docs.openvino.ai/2023.2/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html
- RubenPadial
  Contributor
  2 years ago
  Hello @JohnT_Intel,
  According to the link you sent only mo --saved_model_dir <SAVED_MODEL_DIRECTORY> for saved model files. That exacly what I'm doing.
  - RubenPadial
    Contributor
    2 years ago
    Please, find in the following link the complete TF model, IR model and Compiled Graph https://consigna.ugr.es/download.php?files_ids=63731 or https://consigna.ugr.es/?s=download&token=779d3377-e785-43aa-ab82-6e14c50072fe
JohnT_Altera
Regular Contributor
2 years ago
Hi,

I observed that the file provided does not support certain layer on FPGA side. This will cause the you to pass between FPGA and CPU.

Hi,

I have check on the file you provided, it looks there is multiple layer that is not supported by FPGA.

Layer (Name: StatefulPartitionedCall/model/dense_1/BiasAdd, Type: Eltwise) is not supported: FPGA plugin: all parent nodes are not supported by FPGA.
Layer (Name: Constant_2560, Type: Constant) is not supported:
Layer (Name: StatefulPartitionedCall/model/dense_1/Tensordot, Type: Reshape) is not supported:
FPGA plugin: layer is not executed on FPGA because it does not have preceding layer on FPGA.
Layer (Name: StatefulPartitionedCall/model/dense_1/Tensordot/MatMul, Type: FullyConnected) is not supported:
FPGA plugin: this Fully Connected layer does not have preceding convolutional layer / sequence.
Layer (Name: StatefulPartitionedCall/model/re_lu_7/Relu, Type: Relu) is not supported:
FPGA plugin: layer is not executed on FPGA because it does not have preceding layer on FPGA.
Layer (Name: StatefulPartitionedCall/model/dense/BiasAdd, Type: Eltwise) is not supported:
FPGA plugin: 'Constant' nodes are not supported as input of nodes Eltwise
Layer (Name: Constant_2559, Type: Constant) is not supported:
RubenPadial
Contributor
2 years ago
Hello @JohnT_Intel ,
According to Intel FPGA AI Suite. IP Reference Manual section 2.3. some of that layers should be compatible like the ReLU layer. Could you confirm? In addition, the model is compiled for HETERO plugin (--fplugin option), if the layer cannot be implemented in FPGA, it should be implemented in the CPU. What solution do you suggest?
Plase note that someof that layers are not in the intioan TensorFlow model and they are included by the OpenVINO model optimizer (mo) application.
Add tags
JohnT_Altera
Regular Contributor
2 years ago
You will need to check on A10_Performance.arch on what type of confiuguration setting you are using on the FPGA design.
JohnT_Altera
Regular Contributor
2 years ago
Hi,

Can we consolidate all the AI Suite disucssion into single forum discussion?
RubenPadial
Contributor
2 years ago
Hello @JohnT_Intel ,
I prefer to manage the issues or questions in different threads as they seem to have different root causes.
RubenPadial
Contributor
1 year ago
Hello @JohnT_Intel,
Is there any official documentation on the DLA runtime or inference engine for managing the DLA from the ARM side? I need to develop a custom application for running inference, but so far, I’ve only found the dla_benchmark (main.cpp) and streaming_inference_app.cpp example files. There should be some documentation covering the SDK.
From what I understand, the general inference workflow involves the following steps:
Identify the hardware architecture
Deploy the model
Prepare the input data
Send inference requests to the DLA
Retrieve the output data

Forum Discussion

Intel FPGA AI Suite Inference Engine

31 Replies

Recent Discussions

AI Suite - Is layout transformation useful in more scenarios?

AI Suite - How to interpret these graphs in Model Analyzer GUI

AI Suite - Spatial IP outputs wrong value

Agilex 7 I-Series "aocl diagnose acl0" error following OFS

AI Suite - core_hw.tcl error