Intel FPGA AI Sutie Inference Engine

Hi,

Yes, your understanding is correct where you should be able to continuously feed new data for inference. Your understanding of accelerator is correct.

Hi Ruben,

Unfortunately I do not have the setup to this up and I am working with engineering to see how we can implement the flow that you requested.

I am sorry if I am not able expediate this support and I am trying my best to help you resolved the issue.

Hello @JohnT_Intel ,

I only commented out the // Flip Vertically and // Group the data into 9 slaves sections to avoid excessively extending the log, as they only modify the already retrieved output. You can run the full code if you’d like. Do you have any questions about that?

The raw output is printed in the previous section, as you can see in the log file.

Again, do you have an example or pseudocode for properly handling the inference requests?

Hi Ruben,

From the output log file, I observed that it is different from your code where certain printout is missing.

I do not observed the print out below.

// Flip Vertically

flipHorizontally(processed_output);

std::cout << "Flipped Output Array:" << std::endl;

for (const auto& row : processed_output) {

std::cout << "[ ";

for (const auto& value : row) {

std::cout << value << " ";

}

std::cout << "]" << std::endl;

}

// Group the data into 9 slaves

std::vector<std::vector<int>> grouped_output = groupData(processed_output);

std::cout << "\nGrouped Output Array:" << std::endl;

for (size_t i = 0; i < grouped_output.size(); ++i) {

std::cout << "Group " << i + 1 << ": [ ";

for (const auto& value : grouped_output[i]) {

std::cout << value << " ";

}

std::cout << "]" << std::endl;

}

std::cout << "RIS Resolution: " << output_shape[3] << "-bit." << std::endl;

std::vector<int> flattened_output;

for (const auto& group : grouped_output) {

flattened_output.insert(flattened_output.end(), group.begin(), group.end());

}

// std::vector<uint64_t> groups = prepareData(flattened_output, output_shape[3]);

std::vector<uint64_t> groups = prepareData(grouped_output, output_shape[3]);

std::cout << "Prepared Data for SPI:" << std::endl;

for (size_t i = 0; i < groups.size(); ++i) {

std::cout << "Group " << i << ": 0x"

<< std::hex << groups[i]

<< std::dec << std::endl;

}

const std::string throughput_file_name = "throughput_report.txt";

std::ofstream throughput_file;

throughput_file.open(throughput_file_name);

throughput_file << "Throughput : " << totalFps << " fps" << std::endl;

throughput_file << "Batch Size : " << batchSize << std::endl;

throughput_file << "Graph number : " << exeNetworks.size() << std::endl;

throughput_file << "Num Batches : " << num_batches << std::endl;

throughput_file.close();

// Output Debug Network Info if COREDLA_TEST_DEBUG_NETWORK is set

ReadDebugNetworkInfo(ie);

if (return_code) return return_code;

Thanks.

John Tio

Hi,

it seems like you are creating a new inference request for every new input, and it failed at the 6th. Instead of creating a new inference request for every new input, you should keep using the same set of inference requests, wait for one to become available, and supply input data to it.

Hi Ruben,

If that is the case then I suspect that the FPGA AI suite might not be able to run as it is already pre-occupied with the previous inferencing. It has already not able to run further inferencing unless the previous task is already fully completed and it can move towards a new inferencing.

Hi Ruben,

Can I confirm that you are running below command multiple time where the 6 times, you are facing the error?

./ris_app \

-arch_file=$arch \

-cm=$model \

-plugins_xml_file=$plugins \

-nireq=1 \

-niter=1 \

-d=HETERO:FPGA,CPU

Hi Ruben,

I check on the log provided but it does not provide the full information on how you run it. Is it running the same graph? Or can you provide the step you use to run the application?

Hi Ruban,

Sorry that I forget to check which FPGA AI Suite version are you running? As the latest FPGA AI Suite (2024.3) runtime application code is different to yours.

Hi,

Can you also the full log when you are running it multiple time until you observed the error?