In HDL, you can code a hierarchy for every primitive in a schematic and hook them up, thereby creating the same thing. So just switching to schematic will not buy you anything. Now, when you take your state-machine and convert it to a schematic, it won't be a 1:1 translation, and it could be smaller. (Although it could be larger, and even if it does become smaller there's a good chance the same thing could be done in HDL).
I find it very difficult to reduce area without removing functionality. The places to look are:
- Seeing if something can be done a different way. As an example, if you had a wide and long shift-register, say 16 bits wide and 20 bits deep, that takes up a ton of registers. You can build a shift-register out of a memory block that is much smaller. (Quartus does this automatically assuming there isn't an asynchronous reset, and based on the size of your design I don't think you have anything like this, it's just an example)
- The other case is when you write more abstract code like a case statement, there might be some behavior that is inferred which you don't want. It's just worthwhile to study it and see if you might be able to reduce it through some options.
But again, these are usually small changes that are almost in the noise of the design.