1) RAMs as logic take up lots of memory. Not small RAMs, but if you have something like a 16Kx8 RAM, which easily fits into a few M9K blocks, but it needs to get mapped to registers, it will make the internal memory usage explode, as it's one small hierarchy using 128K registers with huge muxes across them all. If this is the case, the problem isn't how much memory you have(most designs should work with 2GB of memory, especially through synthesis), and is a problem of these RAMs not going through.
2) I don't see the rest of the messages. Possibly it's due to too many RAMs inferred in the design and eventually running out and having to use logic. If you know how much RAM is in your design, you can tell.
3) Or a RAM isn't getting inferred due to some logical reason(like it has aclr, or is coded in a way the tools can't recognize). If so, do you have a hierarchy report(the top-left window). Right-click -> Customize it to show FFs, and then look for a hierarchy with a huge FF count. There's where your RAM is.
- If you don't get a report due to the out of memory, make a copy of your project, and go to Assignments -> Settings -> General and set the top-level entity to something lower in your project. Compile that and look for a huge FF count. If it's not in there, go to the next sub-hierarchy and compile that.
In general, if you've built up your code through sub-modules, you probably have a good idea where the problem is already. If this is just a huge blob of code handed to you, it's going to take some investigating.