Omitting dcache AND icache in f-core

If your algorithm has tight loops then you will take a performance hit removing the instruction cache (not that you can with the f core anyway though). You can't turn off the icache in the f-core due to the design of the core (without an instruction cache you might as well use the e-core). The main purpose of an icache is so that if you have repetative structure in your code (a loop for example), then you don't need to be constantly going out on the avalon bus to read the next instruction if you can keep that information cached. The only case (well one of the only cases) where cache does not help you is if you have a lot of code to be executed within a repetative structure (the code inside the loop is larger then the size of your cache). That's a simplistic view of it (there's a bit more to it than that).

Since you are worried about onchip resources, I recommend picking your memory wisely then (if you have MRAM available use it when you can). Also don't forget to use compiler optimizations in the Nios II IDE to reduce the memory footprint.