OpenCL error on code compilation

3 hours is indeed excessive. The first stage of compilation should only take a few minutes for typical kernels. Based on the line numbers in the log, you seem to have a relatively large kernel. Furthermore, the compiler is auto-unrolling a lot of loops, which might not necessarily be what you want do (especially from a resource usage point of view) while it is also removing some out-of-bound accesses to some of your buffers which shows you have logical issues in your code. I think your kernel is probably too large and complex for the compiler to handle and it is probably running into a memory leak somewhere and filling your memory and finally crashing when it runs out of memory.

My recommendation is to first make sure to modify your code to remove all the warnings in the log and then try to simplify your kernel. As it is, even if your kernel passes the first stage of the compilation, it will probably be too big to fit on the device.

It seems your code has been initially written to run on a standard CPU, hence not every construct used in the code is suitable for FPGA acceleration. There are lots of opportunities to improve your code:

• Starting from the top function I can see that you are processing 1024 of data while only writing 512 points back. This will result in a significant waste of computing cycles and also FPGA area. You should modify the code to only compute what you are going to write back to external memory and later read in the host.
• You are unrolling the read and write loops in the top function, which is the correct thing to do to achieve compile-time access coalescing. However, the unroll factor is far too large (512). Supporting such large accesses results in significant waste of FPGA resources, especially Block RAMs. The external memory bandwidth of the FPGA will be saturated with one 512-bit read and one 512-bit write per loop iteration (in case of two DDR memory banks and an II of one). This effectively translates to an unroll factor of 16 for the "int" datatype. What you should consider doing is to reconstruct your code so that you are reading, processing and writing back 16 points per loop iteration. Assuming that the FPGA is overutilized, you can then reduce the number of parallel points to fit the design.
• There is excessive use of function calls in your code. Every function call will be implemented individually as a circuit on the FPGA, resulting in excessive use of FPGA resources. This is similar to the case of a fully-unrolled loop. Furthermore, such calls prevent the compiler from correcting reporting the area usage per kernel line in the HTML report (as is evident in your report where "No Source Line" is occupying half the area), which in turn makes performance debugging very difficult. You should avoid function calls as much as possible and try to use loops over the functions instead and partially or fully unroll the loops based on the available area.
• The way the "ibfly4_16" is currently written is very inefficient on FPGAs (loop inside of a branch). Since the loops inside of both sides of the branch over "type" are the same, you should instead use one loop and move the branch inside of the loop. Furthermore, using the "out = (condition) ? in_1 : in_2;" construct rather than if/else could lead to area savings in some cases.
• The main problem in your code seems to stem from the cpack_16_64 function which cannot achieve an II of one due to dependency on "x", resulting in the depth of all the buffers in the loop being increased by the II. Since the function is instantiated multiple times, it leads to huge area waste. I think the dependency exists since you are reading from the x[i+1] point and then overwriting it. If you can split "x" into two buffers and write to x1[i] and x2[i] instead, you might be able to avoid this problem. Of course this will require significant code rewriting which will likely propagate all the way to the top function.

There are probably other things that can be done to improve the code but I cannot find and list them all since the code is relatively large. You can try converting each function to a separate kernel manually and then compile them one by one and optimize each separately based on the information you get from the report and then put them back in the original kernel.

Those dependencies look like false write-after-write dependencies. The compiler seems to be assuming that the store addresses might overlap and cause undefined behavior in the pipelined loop but since the loop bound is fixed and the addresses do not seem to overlap, it is probably safe to add #pragma ivdep to the loop to avoid the false dependency.

Different output on FPGA compared to emulation can have two reasons:

• A bug in the compiler that results in the generation of an incorrect hardware circuit (less likely)
• Race condition in global memory accesses or incorrect usage of ivdep pragama (more likely)

It is possible that I missed some important detail in your kernel and my suggestion of adding ivdep to avoid the dependencies was incorrect. You can try removing them to see if you will get correct output (at the cost of lower performance).

I wouldn't rely too much on the numbers reported by the profiler; in my experience, these numbers are not very accurate. The peak external memory bandwidth of your board is 25.6 GB/s (23.8 GiB/s); however, you should not expect to get close to that number unless in extremely ideal situations. You can find the math behind calculation of the external memory bandwidth and my recommendations on how to improve external memory performance in this thread (check the reply before the last, usernames have been lost after migration from Altera's forum):

https://forums.intel.com/s/question/0D50P00003yyTK3SAM/global-memory-access-512-bit-width-constrain

Regarding operating frequency, it largely depends on loop-carried dependencies and area usage. OpenCL users have very little control over the kernel operating frequency and it is difficult to give recommendations as to how it can be improved. You can try changing the default target operating frequency from 240 to some higher number using the -fmax switch and force the compiler to insert more registers into the pipeline; this can potentially improve operating frequency. However, it might result in higher II for loops that are the fmax bottleneck. In that case you should focus on optimizing those loops to resolve whatever dependency that is causing the bottleneck.

Do you still get incorrect output after removing the ivdep pragmas? Also, as I mentioned before, there is really no point in fully unrolling your memory reads and writes since the memory bandwidth will be saturated with an unroll factor of 16, and you will be just wasting FPGA area with such large unroll factors.

It is unlikely that your problem is caused by a bug in the compiler; however, if it is, there is nothing any of us can do about it other than reporting it to Intel and hoping that they would fix it in a later version. It might also be possible to avoid bugs in certain cases by changing the design strategy.

I don't really see a race condition in your current code snippet and cannot think of any reason why it would generate incorrect output. One test you can do is to extract that part of the code and convert it to a separate kernel and do the rest of the computation on the host and see if the output is still incorrect. If it is, then you can go ahead and add a "lightweight" printf to your code and try to see if you can figure out where things are going wrong on the hardware.

As far as I know, each set of nested loops in a kernel will be implemented as an individual pipeline. If the compiler does not detect any dependency between two such pipelines, it might reorder or parallelize them. However, this should not happen in your case since there is a data dependency between the loop nests. In your case, the compiler must guarantee that the each loops nest is completely processed and its pipelines is flushed before starting the next one. I am not sure what could be causing the problem in your case; it could as well be a compiler bug.

Regarding run time, you should probably first calculate the total amount of data that is transferred between the FPGA and its external memory in your code and divide it by the FPGA external memory bandwidth. This will give you an upper-bound for the performance you can achieve. If this upper-bound is higher (worse) than your goal, then your goal is unachievable. If your goal is higher (worse) than the upper-bound, the further it is from the upper-band, the more likely it will be to achieve. Of course there is never any guarantee you would be able to achieve this upper-band in practice.

You can try reporting your issue to Intel, but unless you have Premier Support access, I don’t think it is possible to open tickets with Intel anymore. They have offloaded support for people who do not have Premier Support access to the forums. Maybe you can send a PM to one of the Intel-affiliated moderators in the forum and then they can open a ticket with the engineering team on your behalf.

Regarding the memory performance, the memory of the DE5-Net board operates at 1600 MHz (800 MHz double data-rate) and the memory controller operates at 200 MHz. The peak external memory bandwidth of the board is 2 x 64 bit x 1600 MHz = 25.6 GB/s = 23.8 GiB/s. Assuming that your OpenCL kernel is running at 200 MHz, you will need to read/write a minimum of 128 bytes per clock to saturate the external memory bandwidth. Since the kernel and the memory controller operate at different clocks, and the clock of the memory controller is fixed, you can saturate the memory bandwidth using less bytes per clock at a higher kernel operating frequency, or using more bytes per clock at a lower kernel operating frequency. The compiler places buffers between the kernel and the memory interface to allow this. If the total amount of data you need to transfer between the FPGA and its external memory is only 8192 bits, then you would probably be better off just running your code on the host since your bottleneck will definitely be the PCI-E transfer. The computation on the FPGA in this case will be latency-bound since you will not be saturating the pipeline, and your run time will then depend on the depth of the pipeline and kernel operating frequency - parameters which the user has very little control over in a high-level design.

Depending on the size of your data and how much "data movement" you have in each step, your bottleneck could fall on PCI-E, DDR or computation of the FPGA itself. If you transfer one byte of data through PCI-E to DDR, read it once from DDR, and process it once on the FPGA, your bottleneck is going to be the PCI-E transfer since it has the least bandwidth. If you transfer once through PCI-E but read multiple times from DDR but process only once each time, then the bottleneck will fall on the DDR transfer. Finally, if you transfer once through PCI-E, read once from DDR but do a lot of compute on the FPGA for every byte you read, then you might finally be able to actually saturate the compute capabilities of the FPGA. In your case, since your data size is very small, unless you are processing the same data a couple million times, your bottleneck is going to be the PCI-E transfer.

Please attach the quartus_sh_compile.log file from your compilation folder. How much memory does your compilation machine have? Also note that Terasic's latest BSP for the DE5-Net board targets Quartus v18.0; it is not guaranteed to work with 18.1.