Forum Discussion
Occasional ContributorThe weird aggressive aocl optimization "removing unnecessary storage to local memory"
With respect to functional verification, what I do is that I construct my host code in a way that both run-time and offline compilation are supported, the latter for FPGAs and the former for other devices, and I use AMD's OpenCL SDK for other devices. In this case, as long as the run-time OpenCL driver is installed, the same host code can then be used to execute the same kernel on any type of CPU, GPU or FPGA. You can take a look at the host code/makefiles of the optimized benchmarks in the following repository as example of achieving this:
https://github.com/fpga-opencl-benchmarks/rodinia_fpga
I emulated all of those kernels on CPUs/GPUs using the same host and kernel codes. What I would tell you is that if an NDRange kernel with sufficiently large local and global size performs correctly on a GPU, it should also perform correctly on an FPGA (unless there is a bug in the FPGA compiler). A CPU should also work fine even if the whole kernel runs on one core, since there will still be multiple threads (work-items) running on that core that could be issued out of order and this is usually enough to show concurrency issues but a GPU would likely be more trustworthy in this case.
With respect to, let's say HDL vs. OpenCL, many old-school HDL programmers tend to think that OpenCL or HLS tools in general are insufficient and it is possible to achieve better results using HDL. This is indeed true in some cases like latency-sensitive or low-power applications where clock-by-clock control over the code is required, or applications that are limited by logic resources, but I would not say this is the case for high-throughput applications where limitation is Memory/PCI-E bandwidth or DSP count since these limitations are independent of the programming language. With respect to the particular case of unpipelinable nested loops, HDL or OpenCL would not make a difference. If you have a regular outer loop with an irregular inner loop, the outer loop cannot be pipelined; it doesn't matter how you "describe" the code. There are two ways to approach such loops on FPGAs:
1- Use NDRange and let the run-time work-item scheduler do its best in maximizing pipeline efficiency and minimizing the average loop II.
2- Collapse the nested loop as long as it is not too irregular and get an II of one at the cost of a noticeable Fmax hit. Though by "collapse" I mean manual collapse and not the compiler's "coalesce" pragma. Take a look at Section 3.2.4.3 in this document:
https://arxiv.org/abs/1810.09773
Even though the provided example involves collapsing a regular nested loop, this optimization also sometimes applies to irregular nested loops. I such case, the condition inside the collapsed loop that is used to increment the variable of the original outer loop will have more than one statement (which complicates the critical path and reduces the Fmax). Indeed the possibility also exists to implement parts of your application in HDL and use it as an HDL library in an OpenCL kernel but you are going to run into complications if your HDL library does not have a fixed latency and I highly doubt you would be able to achieve much better results in the end.
Finally, with respect to NDRange vs. Single Work-item, I recommend reading Section 3.1 (and particularly 3.1.4) of the document I posted above.
Frequent Contributor@hiratz This is the best way I can simplify the problem of local memory and barriers:
1- Is there any instance in your code where work-item "i" writes to point X in a local buffer and work-item "j" reads from that point? (based on your last reply it seems the answer is no)
Yes: You need a local memory barrier after every such write operation. --> END
No: You do not need to use local memory. --> GOTO 2
2- Is there any instance in your code where work-item "i" reads point Y from global memory multiple times? (based on your last reply it seems the answer is yes)
Yes: Then create a private variable (rather than local) to store the point and reuse it. The size of this variable will depend on how many such points you need to store on-chip per work-item. You do not need to account for the work-group size or SIMD factor in this case. The compiler will automatically create one such buffer for as many work-items as your SIMD factor. --> END
No: There is no point in optimization using local/private memory in this case. --> END
Getting correct results with one work-item but incorrect results with more than one sounds very much like a concurrency issue to me. If there is no overlapping between the regions that the work-items access in global memory, then the problem must come from local memory or incorrect synchronization. Of course there are also cases where the compiler generates incorrect logic due to some compiler bug, but such cases are quite rare. Considering the fact that you are not using channels (or any other FPGA-specific constructs), have you tried running your modified code on a GPU? I personally would not use Intel's emulator unless I am using such constructs. Running on a GPU is not only much faster, but also allows debugging concurrency issues.
Finally, I wouldn't mind academic collaborations but right now I am occupied by my own projects and since I have little knowledge of the code you are trying to port and, to be honest, it looks quite big, I am not sure if I could allocate enough time for such collaboration. One advice I have for you is to first make sure porting this code for FPGAs is worth the time you are spending on it. If it ends up being too slow since the algorithm is not suitable for FPGA acceleration, it would be difficult to justify the time and effort spent on it. If the code is memory-intensive/memory-bound, if it largely involves random or indirect memory accesses, or if it cannot be properly pipelined due to loop-carried dependencies or variable loop exit conditions, I wouldn't say the code is a good candidate for FPGA acceleration.
hiratz@HRZ Thank you for so specific code guidance and suggestions ! I'll look through my code again with your method 1 and 2.
Speaking of concurrency, actually my project has three different implementation versions for now: "CPU Serial", "OpenCL CPU" and the one I showed here "OpenCL FPGA". The "CPU Serial" is used to generate the correct results for comparison with the other two versions; the "OpenCL CPU" is used to detect potential concurrency bugs or problems.
I forgot to mention in last reply that another important reason why I remove all barriers in my code is: only doing so makes all workitems in the "OpenCL CPU" run on DIFFERENT CPU cores. On the one hand, Intel's OpenCL runtime driver for CPU views a core as a "Compute Unit (CU)" and puts work items in the same workgroup into one core. On the other hand, as we all know, OpenCL barriers are only applied within one workgroup. As a result, my earlier "OpenCL CPU" versions (one single big kernel, one group) with many barriers only can run on a single core, and no concurrent running happened! But if I put workitems into different groups with each group containing only one item, the results became incorrect because the barrier cannot play role. Therefore, I decided to break that big single kernel into multiple ones and meanwhile remove all barriers. Then it worked stable and well and always gives the correct results in my own computers. (Note that Intel's OpenCL CPU runtime uses Intel TBB as its underlying mechanism to create work items (threads)). (I also think putting a group into one core is a limitation of Intel OpenCL CPU runtime itself, not the OpenCL itself. I have not tried other CPU type like AMD's).
So this is why I did not pay much attention to the concurrency issue. I have not implemented an OpenCL GPU version yet. Considering both CPU and GPU are "instruction-decoding-based" execution style, I feel that there should be no concurrency issue too on GPU if my CPU version can work correctly. I may be wrong, of course. If you think this (my thinking) is not correct, I'll consider also implementing it on GPU and it should be not difficult. Unlike CPU and GPU, FPGA implements its concurrency using direct "circuit-based" execution style. So maybe there are still chance for concurrency issues to happen on FPGA. In other words, there may be no "concurrency-test portability" among CPU, GPU and FPGA.
I totally agree with you about how to apply applications on FPGA. Not all applications are suitable for FPGA acceleration. But here let me first clarify some concepts before further discussion: "OpenCL-based FPGA" vs "FPGA itself". One faculty in our group thinks it is the problem of the OpenCL compiler that cannot generate a high-efficient circuit, not the problem of the FPGA itself; using a low level RTL implementation may solve this problem.
Take my case for example, the "(de)compression" kernel actually is a big top-level loop which contains many small loops, many of which are IRREGULAR. By "irregular", I mean these loops do not have fixed number of iterations. As you said, they have "variable loop exit conditions" that are determined by the runtime data. So the top-level loop will be never pipelined by the current compiler. (It seems also difficult to break this big top loop into multiple parts according to the code's semantics) Do you think that it is possible to handle such irregular cases including other cases you mentioned with a RTL implementation without considering an OpenCL compiler? (In my opinion, the issues caused by program logic or semantics are intrinsic and hard to solve by changing the implementation)
Actually now I'm considering two solutions: 1) implement these irregular loops with RTL language and then call it by OpenCL Library (Intel compiler supports this). But I'm not very sure if it is feasible (especially in the statement-level, like call a API which is a RTL implementation of an irregular loop in somewhere inside a kernel). I'm concerned that the top level loop might still not be pipelined even it is feasible; 2) Implement the whole (de)compression function with RTL. Undoubtedly this will take much more time.
Though Intel's compiler supports both task and NDRange type, it seems that Intel focuses more on the former and implicitly suggests people to use the former by which more aggressive optimizations can be conducted (according to the manuals "Programming Guide" and "The Best Practices"). In my case, a SIMD-style optimization instead of a pipelined one should be more appropriate because multiple work items access non-overlapping data. But even for NDRange one, Intel still uses a pipeline to implement it across work items. Only their "kernel Vectorization" is real SIMD-style, but it only targets the kernel level (not the statement level.). Another optimization "Multiple Compute Units" is obviously not feasible for my case. A pair of "decomp" and "compress" kernels already occupy ~50% resource. From this perspective, the "instruction-decoding based" execution style is not limited by your code's logic complexity and size. That's why thousands of simple cores can be put into a GPU to do massive SIMD-style processing. So I think pipeline is the main advantage of FPGA, which is what both CPU and GPU lack (Note that the microarchitecture pipeline inside the CPU is specifically for instruction level parallelism, not the function-level pipeline we are talking about in the FPGA). It seems impractical to achieve a comparable performance to GPU by replicating function units (especially for those complex ones).
PS: Quartus 18.1 and 19.1 provide many example OpenCL designs (in 18.1/hld/examples_aoc, and 19.1/hld/examples_aoc, respectively). I looked through them quickly and noticed that most loops are regular.
Finally, I totally understand that you are busy with your projects and don't have much time. That's fine and I still appreciate your kindness and your advice!
