Intel FPGA AI Sutie Inference Engine

Hi,

it seems like you are creating a new inference request for every new input, and it failed at the 6th. Instead of creating a new inference request for every new input, you should keep using the same set of inference requests, wait for one to become available, and supply input data to it.

Hi Ruben,

From the output log file, I observed that it is different from your code where certain printout is missing.

I do not observed the print out below.

// Flip Vertically

flipHorizontally(processed_output);

std::cout << "Flipped Output Array:" << std::endl;

for (const auto& row : processed_output) {

std::cout << "[ ";

for (const auto& value : row) {

std::cout << value << " ";

}

std::cout << "]" << std::endl;

}

// Group the data into 9 slaves

std::vector<std::vector<int>> grouped_output = groupData(processed_output);

std::cout << "\nGrouped Output Array:" << std::endl;

for (size_t i = 0; i < grouped_output.size(); ++i) {

std::cout << "Group " << i + 1 << ": [ ";

for (const auto& value : grouped_output[i]) {

std::cout << value << " ";

}

std::cout << "]" << std::endl;

}

std::cout << "RIS Resolution: " << output_shape[3] << "-bit." << std::endl;

std::vector<int> flattened_output;

for (const auto& group : grouped_output) {

flattened_output.insert(flattened_output.end(), group.begin(), group.end());

}

// std::vector<uint64_t> groups = prepareData(flattened_output, output_shape[3]);

std::vector<uint64_t> groups = prepareData(grouped_output, output_shape[3]);

std::cout << "Prepared Data for SPI:" << std::endl;

for (size_t i = 0; i < groups.size(); ++i) {

std::cout << "Group " << i << ": 0x"

<< std::hex << groups[i]

<< std::dec << std::endl;

}

const std::string throughput_file_name = "throughput_report.txt";

std::ofstream throughput_file;

throughput_file.open(throughput_file_name);

throughput_file << "Throughput : " << totalFps << " fps" << std::endl;

throughput_file << "Batch Size : " << batchSize << std::endl;

throughput_file << "Graph number : " << exeNetworks.size() << std::endl;

throughput_file << "Num Batches : " << num_batches << std::endl;

throughput_file.close();

// Output Debug Network Info if COREDLA_TEST_DEBUG_NETWORK is set

ReadDebugNetworkInfo(ie);

if (return_code) return return_code;

Thanks.

John Tio

Hello @JohnT_Intel ,

I only commented out the // Flip Vertically and // Group the data into 9 slaves sections to avoid excessively extending the log, as they only modify the already retrieved output. You can run the full code if you’d like. Do you have any questions about that?

The raw output is printed in the previous section, as you can see in the log file.

Again, do you have an example or pseudocode for properly handling the inference requests?

Hi Ruben,

Unfortunately I do not have the setup to this up and I am working with engineering to see how we can implement the flow that you requested.

I am sorry if I am not able expediate this support and I am trying my best to help you resolved the issue.

Hi,

Yes, your understanding is correct where you should be able to continuously feed new data for inference. Your understanding of accelerator is correct.

Hi Ruben,

Sorry for the delay. If you are using the benchmark source code then you will need to include “wait_all” so the inference is completed before you proceed with new input.

You might want to refer to OpenVINO’s classes instead: https://docs.openvino.ai/2024/openvino-workflow/running-inference/integrate-openvino-with-your-application/inference-request.html

Hi,

I think it should be different. You may refer to openvino.AsyncInferQueue — OpenVINO™ documentationCopy to clipboardBack ButtonFilter Button — Version(2024)

You may also refer to for the example that contain the wait_all. Throughput Benchmark Sample — OpenVINO™ documentationCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardBack ButtonFilter Button — Version(2024)

Hi Ruben,

You may also refer to 2023.3 version of document from OpenVINO. The sample design can be use to be run with FPGA.Throughput Benchmark Sample — OpenVINO™ documentationCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboardCopy to clipboard — Version(2023.3)

It has both Python and C++ sample code.

Hi Ruben,

I think in C++ it is using below code which is wait()

for (ov::InferRequest& ireq : ireqs) {

ireq.wait();

}

Hello @JohnT_Intel ,

As I said, it also included in dla_bechmark as well as the application I shared with you. It doesn't work. Find below the code extracted:

 for (size_t iireq = 0; iireq < nireq; iireq++) {
                            auto inferRequest = inferRequestsQueues.at(net_id)->getIdleRequest();
                            if (!inferRequest) {
                                THROW_IE_EXCEPTION << "No idle Infer Requests!";
                            }
                            
                            if(niter != 0LL){
                                std::cout << "#Debug: 10. Set output blob.\n";
                                for (auto & item : outputInfos.at(net_id)) {
                                    std::string currOutputName = item.first;
                                    auto currOutputBlob = ioBlobs.at(net_id).second[iterations.at(net_id)][currOutputName];
                                    inferRequest->SetBlob(currOutputName, currOutputBlob);
                                }
                                std::cout << "#Debug: 10. Set input blob.\n";

                                for (auto & item: inputInfos.at(net_id)){
                                    std::string currInputName = item.first;
                                    auto currInputBlob = ioBlobs.at(net_id).first[iterations.at(net_id)][currInputName];
                                    inferRequest->SetBlob(currInputName, currInputBlob);
                                }
                            }

                            // Execute one request/batch
                            if (FLAGS_api == "sync") {
                                inferRequest->infer();
                            } else {
                                // As the inference request is currently idle, the wait() adds no additional overhead (and should return immediately).
                                // The primary reason for calling the method is exception checking/re-throwing.
                                // Callback, that governs the actual execution can handle errors as well,
                                // but as it uses just error codes it has no details like ‘what()’ method of `std::exception`
                                // So, rechecking for any exceptions here.
                                inferRequest->wait();
                                inferRequest->startAsync();
                            }

                            iterations.at(net_id) ++;
                            if (net_id == exeNetworks.size() - 1) {
                                execTime = std::chrono::duration_cast<ns>(Time::now() - startTime).count();
                                if (niter > 0) {
                                    progressBar.addProgress(1);
                                } else {
                                    // calculate how many progress intervals are covered by current iteration.
                                    // depends on the current iteration time and time of each progress interval.
                                    // Previously covered progress intervals must be skipped.
                                    auto progressIntervalTime = duration_nanoseconds / progressBarTotalCount;
                                    size_t newProgress = execTime / progressIntervalTime - progressCnt;
                                    progressBar.addProgress(newProgress);
                                    progressCnt += newProgress;
                                }
                            }
                        }