Forum Discussion
New ContributorStuck during execution on Mustang-F100-A10, Intel® Vision Accelerator Design with Intel® Arria® 10 FPGA
Hello,
we have Mustang F100-A10 and TANK-870 AIoT Dev. Kit.
We have followed the official guide for the latest openvino to setup the 2019R1 openvino and our FPGA card.
We were able to run "aocl diagnose" and we see there is the card as you can see below.
ieisw@ieisw-SER0:~$ aocl diagnose
--------------------------------------------------------------------
Device Name:
acl0
BSP Install Location:
/opt/altera/aocl-pro-rte/aclrte-linux64/board/a10_1150_sg1
Vendor: Intel(R) Corporation
Phys Dev Name Status Information
acla10_1150_sg10Passed Intel Vision Accelerator Design with Intel Arria 10 FPGA (acla10_1150_sg10)
PCIe dev_id = 2494, bus:slot.func = 01:00.00, Gen3 x8
FPGA temperature = 61.6406 degrees C.
DIAGNOSTIC_PASSED
--------------------------------------------------------------------
The
./demo_squeezenet_download_convert_run.sh -d HETERO:FPGA,CPUsucceeded as well as demo_security_barrier_camera.sh with device HETERO:FPGA,CPU
However when we try to run classification_sample demo as below, it sometimes stuck forever. Not after certain amount of inference calls but randomly. Sometimes it finished with -ni set to 3000 but sometimes it stuck when we set -ni to 10. Below is the command we used.
/<path>/inference_engine_samples_build/intel64/Release/classification_sample -i /opt/intel/openvino/deployment_tools/demo/car.png -m /<path>/squeezenet1.1_FP16/squeezenet1.1.xml -d HETERO:FPGA,CPU -ni 10The same thing happens with our custom code, which runs without issue on CPU.
While this execution is running we are not able to call "aocl diagnose" at all.
We are really out of ideas and we are starting to suspect that the FPGA card is broken. So any help or suggestion is highly appreciated.
My second question is if it's possible to call "aocl diagnose" from two different terminals simultaneously.
PS: We have skipped the part in documentation which is about USB Blaster (we dont have this thing).
37 Replies
JohnT_AlteraRegular Contributor
Hi,
The 2019R2 for FPGA is schedule to be released in 2 weeks time.
- JohnT_Altera
- list all log files where FPGA plugin and HETERO:FPGA,CPU plugin and fpga itself store any logs
No, log file is available. You will need to modify your host code to see where it is stuck.
- Any way how to get any informations regarding utilization of FPGA while there is something running on the board?
No. The bitstream and driver is different as we update it and optimize to have a better performance.
VDemc2we’ve found another way how to reproduce the freezing issue which is happening with intel’s human_pose_estimation demo as we.
From some reason the human_pose demo freezes when we open Google Chrome and Chromium browser. it is not happening for Firefox browser.
It seems is not just about docker but a virtualization in general.
How to reproduce:
1. Build and run intel’s human pose demo with an input from a video file on HETERO:FPGA,CPU device.
~/inference_engine_samples_build/intel64/Release/human_pose_estimation_demo -i ~/Downloads/yt-openvino-interactive-face-detection-demo.mp4 -m /opt/intel/openvino/deployment_tools/tools/mod_downloader/Transportation/human_pose_estimation/mobilenet-v1/dldt/human-pose-estimation-0001-fp16.xml -d HETERO:FPGA,CPU2. Open Google Chrome browser
3. Demo should freeze
- JohnT_Altera
Hi,
There is some issue with the 2019R1 released for this board. Please upgrade your OpenVINO to 2019R1.1 released.
You may call "aocl diagnose" from two different terminals simultaneously as long as you are able to confirm that there is no application is running on the FPGA.
- VDemc2
Hi,
thanks for your answer.
we downloaded "l_openvino_toolkit_fpga_p_2019.1.144" so we do have 2019R1.1.
- JohnT_Altera
Hi,
Is that mean that you are also facing the same issue on 2019R1.1?
- VDemc2
Yes we are facing the issue on 2019R1.1. We use the latest openvino.
Hovewer we haven't tried on 2019R1 (I thought R1.1 and R1 is almost the same so I incorrectly wrote 2019R1 in my original post). But I assume it wouldn't help.
We tried 2018R5, because our second FPGA, which we received and installed like 6months ago, worked well (we have 2 FPGAs, the new with the problem on 2019R1.1 and the second one - works well but not on the latest version of openvino)
Unfortunately with the new FPGA we were not able to call "aocl diagnose" with success message on 2018R1 version.
- JohnT_Altera
Hi,
May I know how frequent are you observing the hang issue?
- VDemc2
Hi,
it's like 95% for classification_sample demo from openvino toolkit when we set number of iterations to 10. When we run just classification_sample without number of iteration (without -ni parameter, so there is only 1 itteration) it works well without freezing problem.
But when we set number of iterations to 100+ it always freezes.
From stream input e.g. from camera it stuck every time after ~0-3 inferences so it's useless in cases like that.
- JohnT_Altera
Hi,
We are in the progress of looking into the freeze issue and will be fixed in 2019R2 released.
- MČule
Hi John, please do you know, when the 2019R2 for FPGA will be released? For now, there is just R1.1 available.
thank you!
Martin
- VDemc2
Hi JohnT,
now the 2019 for FPGA has been released. Could you please confirm if the issue is resolved? I cannot find it in Release notes.
We will definitely try it but would be good to know any details what was wrong etc.
In the near future we want to create FPGA docker for FPGA as it is described in the documentation. There is a similar freezing issue (marked as #31 in issue list) and it's still in the "Known Issues" (in Release notes) and it looks like it's still not resolved.
31Inference on FPGA freezes when run in Docker*Inference EngineN/A
Thanks
- JohnT_AlteraHi, If you look into the release note below, the bitstream for Mustang F100-A10 is not included. We are still in the progress of debugging the hang issue. "Bitstreams for Intel® Vision Accelerator Design with an Intel® Arria 10 FPGA (Mustang-F100-A10) Speed Grade 1 and Speed Grade 2 are not included in the OpenVINO R2 release. These bitstreams will be added at a future date."
- VDemc2
Hi,
we have prepared some steps how to reproduce our issue. I might help you in the process of debugging
- JohnT_AlteraHi, The link provided is issue observed when using docker container. If it is running without docker container then no issue is observed. May I know what is the reason that you need to run in docker container? The reason is that the demo application is not validated in docker container environment.
- VDemc2
The link provided is issue observed when a FPGA program is running locally and there is another docker containers running on the same host - it this example kafka and zookeeper.
We would like to run FPGA in docker in the future, but for now we run FPGA locally however we have another components which run in docker.
- JohnT_Altera
Hi,
From the link provided, I observed it mention that it is successfull process 3 time and 0 time failed when performed using "HETERO:FPGA,CPU without docker kafka"
Could you confirm if my understanding is correct?
- VDemc2
Yes exactly ... but when we run the same application on the same host and there is running a docker container (eg kafka with zookeeper) it freezes.
