State-machine encoding settings?

We don't know the function to be implemented, so detail suggestions are hardly possible.

Generally, I would set up a simple test design, compile it with different encoding settings (or possibly testing other design alternatives) and analyse the synthesized gate level logic to understand how the resource usages comes up.