quartus_fit out of memory

Once in the fitter, the memory usage is pretty independent of your coding style. (Naturally if you could code to use less physical resources, that would help, but I assume that's a general goal already.) Timing analysis is run during the fitter, and that can have an affect on memory footprint. There were some improvements with various releases of Quartus, so if you're on an old version, upgrading might help.

The /3GB option is tricky. I have heard of a number of people successfully using it, but some people do have issues like the blue screen. It's really a MS issue at that point, but being limited to 2GB, I can certainly see a Stratix III going over that. My reference was the Stratix II 180, whereby that always seemed to be on the 2GB threshold, where some designs worked all the time and others went over the limit.

I would not try to design around it, as there are very few tricks to reduce memory footprint. (If you have extremely complex timing constraints in TimeQuest, it may help to re-organize them, but that's difficult to do since you don't know the internal data structures). Upgrading Quartus is one thought, but really it seems like a new machine is in order.