Dear all,I would like to use OpenCL SDK for a Terasic DE5-Net to deploy my algorithm (I am using Quartus 18.1 and OpenCL SDK 18.1). I successfully run the examples provided by Intel FPGA for OpenCL and they work. Thus, I am trying to compile my code, which is an integer iFFT based on the Cooley-Tukey algorithm. The OpenCL emulator gives me correct results when I run the code on the x86 machine. But, when I try to generate the .aocx for the DE5-Net, the compiler returns with the following warnings and one error (after 5/6 hours of compilation):..several warnings about auto unrolled loop..Compiler Warning: removing out-of-bounds accesses to xtmp1.i.i.iCompiler Warning: removing out-of-bounds accesses to xtmp2.i.i.iCompiler Warning: removing out-of-bounds accesses to xtmp3.i.i.iintelFPGA/18.1/hld/board/terasic/de5net/tests/fft1Dint/device/fft1dint.cl:1424: Compiler Warning: Aggressive compiler optimization: removing unnecessary storage to local memoryError: exception "std::bad_alloc" in PostBuildSteps [ifft_512_function ifft_512_function ]Error: Verilog generator FAILED.Could you please provide me more information about this error? How can I understand which error in my code is linked to this output (buffer allocation or something else)?Thank you for your support.Best Regards,Federico

3 hours is indeed excessive. The first stage of compilation should only take a few minutes for typical kernels. Based on the line numbers in the log, you seem to have a relatively large kernel. Furthermore, the compiler is auto-unrolling a lot of loops, which might not necessarily be what you want do (especially from a resource usage point of view) while it is also removing some out-of-bound accesses to some of your buffers which shows you have logical issues in your code. I think your kernel is probably too large and complex for the compiler to handle and it is probably running into a memory leak somewhere and filling your memory and finally crashing when it runs out of memory.My recommendation is to first make sure to modify your code to remove all the warnings in the log and then try to simplify your kernel. As it is, even if your kernel passes the first stage of the compilation, it will probably be too big to fit on the device.

It seems your code has been initially written to run on a standard CPU, hence not every construct used in the code is suitable for FPGA acceleration. There are lots of opportunities to improve your code:Starting from the top function I can see that you are processing 1024 of data while only writing 512 points back. This will result in a significant waste of computing cycles and also FPGA area. You should modify the code to only compute what you are going to write back to external memory and later read in the host.You are unrolling the read and write loops in the top function, which is the correct thing to do to achieve compile-time access coalescing. However, the unroll factor is far too large (512). Supporting such large accesses results in significant waste of FPGA resources, especially Block RAMs. The external memory bandwidth of the FPGA will be saturated with one 512-bit read and one 512-bit write per loop iteration (in case of two DDR memory banks and an II of one). This effectively translates to an unroll factor of 16 for the "int" datatype. What you should consider doing is to reconstruct your code so that you are reading, processing and writing back 16 points per loop iteration. Assuming that the FPGA is overutilized, you can then reduce the number of parallel points to fit the design.There is excessive use of function calls in your code. Every function call will be implemented individually as a circuit on the FPGA, resulting in excessive use of FPGA resources. This is similar to the case of a fully-unrolled loop. Furthermore, such calls prevent the compiler from correcting reporting the area usage per kernel line in the HTML report (as is evident in your report where "No Source Line" is occupying half the area), which in turn makes performance debugging very difficult. You should avoid function calls as much as possible and try to use loops over the functions instead and partially or fully unroll the loops based on the available area.The way the "ibfly4_16" is currently written is very inefficient on FPGAs (loop inside of a branch). Since the loops inside of both sides of the branch over "type" are the same, you should instead use one loop and move the branch inside of the loop. Furthermore, using the "out = (condition) ? in_1 : in_2;" construct rather than if/else could lead to area savings in some cases.The main problem in your code seems to stem from the cpack_16_64 function which cannot achieve an II of one due to dependency on "x", resulting in the depth of all the buffers in the loop being increased by the II. Since the function is instantiated multiple times, it leads to huge area waste. I think the dependency exists since you are reading from the x[i+1] point and then overwriting it. If you can split "x" into two buffers and write to x1[i] and x2[i] instead, you might be able to avoid this problem. Of course this will require significant code rewriting which will likely propagate all the way to the top function.There are probably other things that can be done to improve the code but I cannot find and list them all since the code is relatively large. You can try converting each function to a separate kernel manually and then compile them one by one and optimize each separately based on the information you get from the report and then put them back in the original kernel.

Those dependencies look like false write-after-write dependencies. The compiler seems to be assuming that the store addresses might overlap and cause undefined behavior in the pipelined loop but since the loop bound is fixed and the addresses do not seem to overlap, it is probably safe to add #pragma ivdep to the loop to avoid the false dependency.

Different output on FPGA compared to emulation can have two reasons:A bug in the compiler that results in the generation of an incorrect hardware circuit (less likely)Race condition in global memory accesses or incorrect usage of ivdep pragama (more likely)It is possible that I missed some important detail in your kernel and my suggestion of adding ivdep to avoid the dependencies was incorrect. You can try removing them to see if you will get correct output (at the cost of lower performance).I wouldn't rely too much on the numbers reported by the profiler; in my experience, these numbers are not very accurate. The peak external memory bandwidth of your board is 25.6 GB/s (23.8 GiB/s); however, you should not expect to get close to that number unless in extremely ideal situations. You can find the math behind calculation of the external memory bandwidth and my recommendations on how to improve external memory performance in this thread (check the reply before the last, usernames have been lost after migration from Altera's forum):https://forums.intel.com/s/question/0D50P00003yyTK3SAM/global-memory-access-512-bit-width-constrainRegarding operating frequency, it largely depends on loop-carried dependencies and area usage. OpenCL users have very little control over the kernel operating frequency and it is difficult to give recommendations as to how it can be improved. You can try changing the default target operating frequency from 240 to some higher number using the -fmax switch and force the compiler to insert more registers into the pipeline; this can potentially improve operating frequency. However, it might result in higher II for loops that are the fmax bottleneck. In that case you should focus on optimizing those loops to resolve whatever dependency that is causing the bottleneck.

Do you still get incorrect output after removing the ivdep pragmas? Also, as I mentioned before, there is really no point in fully unrolling your memory reads and writes since the memory bandwidth will be saturated with an unroll factor of 16, and you will be just wasting FPGA area with such large unroll factors.It is unlikely that your problem is caused by a bug in the compiler; however, if it is, there is nothing any of us can do about it other than reporting it to Intel and hoping that they would fix it in a later version. It might also be possible to avoid bugs in certain cases by changing the design strategy.

OpenCL error on code compilation | Altera Community

24 Replies

FCive
Occasional Contributor
7 years ago
Thank you for your reply.
I can't attach the quartus_sh_compile.log since there is no such file in my folder. The compilation fails before generating the report.html. The command below for the intermediate compilation also fails with the same output. I noticed that this command takes around 3 hours to be completed and the code does not exceed 1500 lines with comment included ( the Intel guide says that this command takes some minutes to complete). I can attach the "fft1dint.log" file.
aoc -c fft1dint.cl -report
About the Terasic support, it also works with 18.1. There were some issues with the BSP, such as a kernel module bug that I fixed, but finally it worked. I tested the BSP with both examples provided by Intel (hello_world and fft1d_offchip) and another opencl algorithm that I had. All the tests were ok.
Best Regards,
Federico
fft1dint.log23 KB
FCive
Occasional Contributor
7 years ago
Sorry, I forgot to write that my PC has 32 GB of RAM
FCive
Occasional Contributor
7 years ago
Thank you HRZ.
I tried to reduce the number of the loops and also the number of variables. Finally, I was able to compile without the previous error. Unfortunately my design is still too big to fit into the FPGA. In particular, there are some variables that requires too much logic utilization. For example, I have the variable 'xtmp1', which is an array of 16 elements of 16 bits each, and in the report I can see the following comment:
Private Variable: - 'xtmp1' (fft1dint.cl:393):
Type: Register
512 registers of width 16 and depth 1
256 registers of width 16 and depth 7 (depth was increased by a factor of 7 due to a loop initiation interval of 7.)
Why the compiler uses such a big number of registers for 'xtmp1' variable? How can I reduce the resources for this variable?
Federico
HRZ
Frequent Contributor
7 years ago
It is difficult to determine the reason for the high resource usage without seeing the source code; however, one thing I can say based on the snippet of the report you have posted is that due to the high initiation interval of your loop, all buffers inside of the loop need to be enlarged to accommodate for the high initiation interval, resulting in even higher resource usage. One thing you should consider doing is to optimize your loop and reduce the initiation interval. If you are comfortable with posting your kernel code, or a simplified version of it that exhibits the same issue, I might be able to give you a more concrete answer.
FCive
Occasional Contributor
7 years ago
Thank you again for your support.
I do not have many loops with an high initiation interval but some variables that require a lot of resources are inside an high II. Thus, I am trying to reduce the II of loops and the ALUs and FFs utilization. No problem for me to post the code and I really appreciate your help. I also attached the report which can give you more details.
Moreover, I was able to reach only 50% of RAM but the ALUs and FFs were always well beyond the 100%. Could you please provide me some suggestions on how to reduce ALUs, FFs and also RAM? If possible, I would like to achieve the maximum performance in terms of execution time.
Federico
code_reports.zip1.5 MB
FCive
Occasional Contributor
7 years ago
Thank you, HRZ. I really appreciate your help.
I tried to follow you suggestions and I made the kernel smaller (now it works with 256 samples instead of 512). Now It is much better but I have some problems with loops. Indeed, I still have memory dependency and I am not able to understand how I can fix this. Moreover, the variables involved in this loops require a lot of resources to be instantiated.
Please find the attached new report: could you please tell me how I can resolve these dependencies?
Federico
reports.zip1.2 MB
FCive
Occasional Contributor
7 years ago
Thank you again for your reply.
Finally I was able to put the kernel into the HW. At the moment the resource utilization is around 50%. Your suggestions were really helpful.
Unfortunately the results of the opencl kernel in HW are different from the emulator. indeed, the kernel returns a buffer that has some correct elements and some other not. Do you know what could be the reasons for this behavior?
Looking the dynamic profiler, I checked that the transmission throughput from global memory to FPGA is 574 MB/s and from FPGA to global memory is 147 MB/s but I got around 2500 MB/s with the "aocl diagnose" command. Also the kernel clock frequency is not the optimal one since I got 155 MHz but I was able to reach more than 200 MHz with other kernels.
Could you please let me know what I can do to improve the performances in terms of execution time and global memory reading/writing troughput?
Thank you.
FCive
Occasional Contributor
7 years ago
Thank you for the information and calculation about accessing the global memory. In this way I can estimate the latency communication, considering the PCIe interface between host and device.
I will try to force the maximux frequency with -fmax option and I will see if the performance are better. I asked you how I can improve performances since my kernel lasts 600 us (according to the profiler) and I would like to reach hundreds of nanoseconds of processing, if possible.
About the different output, I followed your suggestion about the pragma ivdep but the area usage still was too much. So I tried to remove dependencies and I reached 50% of area only optimizing the loops, without using the ivdep pragma. The accesses to the global memory are only for reading and writing (plese see the code below).
Is there anyway a race condition in the global memory access? In case of bug compiler, could you please tell me how I can fix this?
reading: #pragma unroll for (ushort i = 0; i < 512; i++) data[i] = x[i]; writing: #pragma unroll for (ushort i = 0; i < 512; i++) y[i] = data[i];
Thank you.

FCive

Occasional Contributor

7 years ago

Yes. I still have incorrect output and I am not using the pragma ivdep. After a debug session, I have found out that the problem is in the following code.

for (uchar j=0; j<8; j++) {
	k = j*16;
	p = j*16+128;
	h = j*16+256;
	t = j*16+384;
	
	for (uchar i=0; i<16; i+=2) {
		result2[k+i] = (((result[p+i+1]*tw256[k+i+1] + result[p+i]*tw256[k+i] + result[h+i+1]*tw256[p+i+1] + result[h+i]*tw256[p+i] + result[t+i+1]*tw256[h+i+1] + result[t+i]*tw256[h+i]) >> 15) + result[k+i])>>1;
		result2[k+i+1] = (((result[p+i+1]*tw256[k+i] - result[p+i]*tw256[k+i+1] + result[h+i+1]*tw256[p+i] - result[h+i]*tw256[p+i+1] + result[t+i+1]*tw256[h+i] - result[t+i]*tw256[h+i+1]) >> 15) + result[k+i+1])>>1;
		result2[p+i] = (((result[t+i+1]*tw256[h+i] - result[t+i]*tw256[h+i+1] - result[h+i+1]*tw256[p+i+1] - result[h+i]*tw256[p+i] - result[p+i+1]*tw256[k+i] + result[p+i]*tw256[k+i+1]) >> 15) + result[k+i])>>1;
		result2[p+i+1] = (((result[p+i+1]*tw256[k+i+1] + result[p+i]*tw256[k+i] - result[h+i+1]*tw256[p+i] + result[h+i]*tw256[p+i+1] - result[t+i+1]*tw256[h+i+1] - result[t+i]*tw256[h+i]) >> 15) + result[k+i+1])>>1;
		result2[h+i] = (((result[h+i+1]*tw256[p+i+1] + result[h+i]*tw256[p+i] - result[t+i+1]*tw256[h+i+1] - result[t+i]*tw256[h+i] - result[p+i+1]*tw256[k+i+1] - result[p+i]*tw256[k+i]) >> 15) + result[k+i])>>1;
		result2[h+i+1] = (((result[h+i+1]*tw256[p+i] - result[h+i]*tw256[p+i+1] - result[t+i+1]*tw256[h+i] + result[t+i]*tw256[h+i+1] - result[p+i+1]*tw256[k+i] + result[p+i]*tw256[k+i+1]) >> 15) + result[k+i+1])>>1;
		result2[t+i] = (((result[p+i+1]*tw256[k+i] - result[p+i]*tw256[k+i+1] - result[h+i+1]*tw256[p+i+1] - result[h+i]*tw256[p+i] - result[t+i+1]*tw256[h+i] + result[t+i]*tw256[h+i+1]) >> 15) + result[k+i])>>1;
		result2[t+i+1] = (((result[t+i+1]*tw256[h+i+1] + result[t+i]*tw256[h+i] - result[h+i+1]*tw256[p+i] + result[h+i]*tw256[p+i+1] - result[p+i+1]*tw256[k+i+1] - result[p+i]*tw256[k+i]) >> 15) + result[k+i+1])>>1;
		}
}

The buffer "result" is correct and it is calculated previously in the code. The buffer "tw256" is constant and read-only. The variables k, p, h, t are initialized. The buffer "result2" is incorrect. Both "result" and "result2" are declared as registers (512 registers of width 16 and depth 1). I also tried to divide the "result2" writings using several unrolled loops. The output has less errors but still wrong anyway.

Could you please let me know if something is wrong? I can share the complete code if it helps.

Thank you for your support.

FCive
Occasional Contributor
7 years ago
Finally I found the error. In my kernel code, I have 3 different stages to complete the FFT. The first and third ones are composed by an unrolled nested loop while the second one consists on a pipelined nested loop. Converting the pipelined nested loop to unrolled nested loop in the second stage resolved the problem. About this issue, I would like to ask you a question to confirm my idea: is it possible that the third stage with unrolled loop is run before that the data from second stage are ready? This can explain the issue that I had. If not, what could be the reason of this error?
I noticed that the execution time is around 700 us: what are possible good practices to accelerate the execution time? My goal is to reach 5/600 ns or 1 us as maximum. Do you think it is feasible?
Thank you very much.

Forum Discussion

OpenCL error on code compilation

24 Replies

Recent Discussions

Quartus® II Web Edition Design Software Version 13.0sp1 for Windows Still Free to Use?

USB-Blaster II malfunction

TU Darmstadt: Base CXL License

Bug Report: project_open Tcl Command Resets Current Revision to Base Revision in Quartus

How to organize a shared cache for qsys systems?