May I know what are the device that you were using? Some device require larger ram.
What I suggest is to try different machine to see if you still face this problem. Or you can do the below:
1) Can you try on different project to see if is project specific issue?
2) install currprocess https://www.nirsoft.net/utils/cprocess.html and disable all the software running behind there
3) disable the anti virus
4) close all the Quartus software
go to c:/user/<username>/ rename the quartus2.ini and quartus2.req for windows
cd to home/<user>/.altera.quartus rename the quartus2.ini and quartus2.req for linux
and reopen the quartus. this will reset the initial setting of the quartus prime.
5) try your project in different computer to see if the problem exist? Or you can send it to me to test it out
6) did you install all the devices?
7) check if you have spaces in your directory names? If you had, try to replace the spaces with underscores.
8) delete .altera.quartus