How to determined the physical memory used

The peak virtual memory reported in Quartus is a total of all the memory allocated by that executable. In general, disk swapping will slow down your compilation significantly, so having a machine that has enough physical RAM is important. Altera publishes recommendations on how much memory you need depending on the device you are using (see the release notes for the Quartus version you are using).

When you run out at ~2 GB, it is usually a sign that you are using a 32-bit Windows machine, where Quartus typically has access to only 2 GB of RAM. There is an option to change XP to give you access to 3 GB, but it often makes your machine unstable. 32-bit Linux gives you access to 3 GB.

For designs that need more than 2 or 3 GB, the solution is to upgrade to a 64-bit capable OS, use 64-bit Quartus, and have enough physical RAM to match the Altera recommendations.