out of memory in 17.0 compiler

Too be honest, you probably are encountering a very rare problem. I have been using the toolchain for three years now on both Stratix V and Arria 10, starting from v15.0, and despite the numerous problems I have had, I have not had this particular issue. I have encountered "out of memory" crashes, but that is because I run multiple compilations in parallel and I am using a cluster that is shared between everyone in my lab and sometimes the memory fills due to multiple people using the machine at the same time. Your cases seems to be running out of memory without the memory actually being full, which does not seem right. I have one final suggestion: try increasing your swap size and see what happens. You can also try to find an automated way to periodically log the memory usage of the machine (or do it manually using top/htop) and see how the memory usage changes during compilation and if it fails again, figure out what situation it fails in.