Jacob,
If you can budget for 8Mb of RAM I think you'll be a lot happier.
A kernel with Network and USB will push 2Mb of code space alone (remember the kerenl is copied from flash to RAM for execution). Busybox, depending on how it is configured, will range from 300-700K (a bit of an estimate on the upper bound -- have never turned on all the options). If you run parallel commands that are handled by busybox each instance requires its own memory (we have not implemented shared library support).
Thus I don't think you'd have enough RAM left over for the kernel to adequately run (it needs run-time memory for page tables, data structures etc in addition to the code space).
One of the items on our to-do list is further exploration of XIP (execute in place) which would allow the kernel to run out of flash directly. Superficially it sounds like a "good thing" but there is certainly a performance hit due to the slower access times of flash.
In summary, if you jump really hard on the elephant you might squeeze him into the bottle but he won't be able to move and really won't be very happy.
mike