first, about the kernel size.
elf is big, because it contains a lot of debug symbols.
you can strip off the debug info, and the striped size will be around 2MB in your case.
the srec is ascii hex, so it will be more than two times of the binary.
the binary size is about 1.8MB in your case.
if you compress the kernel with zImage or (gzip + u-boot) , it will be around 1MB.
it looks like problem of kernel size?
I had kernel of 3.6MB binary, but did not have such problem.
check usr/initramfs_list
my suggestions,
1. always start with minimal hardware, and minimal kernel config .
2. then add more stuff step by step .
I know this takes a long time. but this is what I do when I faced problem like this.
did you enable large memory >1MB allocation in kernel config?
try add some debug messages in init/initramfs.c .
other stuff that might (or might not) be helpful,
1. zImage
2. stand alone init, sash