Memory usage during compilation

The memory recommendations in Altera's OpenCL docs are mostly for placement and routing and not OpenCL conversion/optimization, the latter doesn't need more than a few GBs of memory in most cases. Still, I have seen cases where the OpenCL optimization process runs into some kind of a deadlock or memory leak and even fills 256 GB of memory until it runs out of memory and crashes; your case is very likely similar. I am afraid in such cases there is no choice but to follow another approach or use some other algorithm for your kernel. This happens if the algorithm is not suitable for FPGA synthesis or the kernel has been written in a way that the compiler cannot understand and optimize it properly.