Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
7 years ago

OpenCL kernel takes more than 36 hours to compile on Arria 10

Dear community,

Before the weekend I started the compilation of my OpenCL kernel using the Intel SDK for OpenCL for the Arria 10 FPGA board. I had compiled an unoptimized version of this kernel before, and that took about 4 hours of compiling and synthesis. When I checked the status of the compilation yesterday (more than 36 hours after starting), I saw it was still busy, and the quartus_sh_compile.log (see attachment) seemed to be stuck in the middle of a line of output. I read on this forum that for the Arria 10 it should take at most around 12 hours to synthesize a large design, so this made me think my compilation was just hanging.

The compilation used about 28GB of the 64GB available RAM of my build machine, and when it was stuck there was only 1 core being used by the `quartus_syn top -c top_synth` process. If it is of any help, the emulation and verification of the kernel was fine, and gave the same results as the (compilable) unoptimized version of the kernel. The main optimizations are the use of local memory and loop unrolling, and I also changed from a task kernel to ND-Range kernel type. The resource usage is significantly more than the unoptimized version, but still estimated to be within the available resources on the Arria 10 (as you can see in the Optimization Report attached as "my_kernel.txt").

I don't know what I can do to further debug this as there is no output, so I hope there is someone on this forum that maybe experienced this before, and could help me out.

Cheers!

6 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    7 years ago

    Well, place and routing large designs on Arria 10 could take 20 hours or maybe even more, but 36 seems to be very extreme, especially since it seems the process has not even gone past synthesis in your case. This could be transient issue or a bug in the synthesis engine. I cannot think of any specific recommendations other than cancelling and retrying the compilation.

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    7 years ago

    Thanks for your reply!

    --- Quote Start ---

    ... cancelling and retrying the compilation.

    --- Quote End ---

    That is what I did, but now, after 24 hours, it got stuck again on the same line in the quartus_sh_compile.log... On another machine I started another compilation, but unrolling a certain loop N times instead of fully. That compiled and synthesized within 4 hours... So apparently it is a resource issue? Which is weird, because the Optimization Report shows the following resource utilization:

    +--------------------------------------------------------------------+

    ; Estimated Resource Usage Summary ;

    +----------------------------------------+---------------------------+

    ; Resource + Usage ;

    +----------------------------------------+---------------------------+

    ; Logic utilization ; 55% ;

    ; ALUTs ; 24% ;

    ; Dedicated logic registers ; 32% ;

    ; Memory blocks ; 82% ;

    ; DSP blocks ; 25% ;

    +----------------------------------------+---------------------------;

    I read somewhere that the estimated resource usage could be off significantly, so maybe in reality it is allocating more than available. But I would expect an error message in that case :)
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    7 years ago

    Yes, the resource estimation can be significantly off. However, if a resource is overutilized, the process will fail during fitting with a clear error message. The only cases I have seen where placement and routing continues forever is if the design narrowly fits on the FPGA, but the routing never succeeds and keeps retrying to no avail. Can you sort the files generated by the compiler based on time and tell me what is the last file that follows the pattern of top.X.summary (X: map, merge, fit, flow, sta)?

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    7 years ago

    ls -tr path/to/generated/files | grep top 

    top.v
top.sdc
top_post.sdc
top_synth.qsf
top.qsf
top.qpf

    There don't seem to be any files generated that match the pattern you described.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    7 years ago

    Well, then it is really getting stuck during synthesis, which is VERY strange and uncommon. You can probably try using one of the newer or older versions of the compiler if you have a compatible BSP and see what happens. You can probably also complain to Altera, but they will likely not pay much attention unless you are using the latest version of the compiler.

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    7 years ago

    Ok, thanks a lot for your help. I will try to ask the system administrator if he can install the latest version of the SDK. If I get more news on that I will certainly post it here.