Hi Ken,
Sorry, we didn't mean to leave you out in the cold. I see you have a number of things going on. Stephen gives one suggestion for boot-loading (where to put the reset vectors). Really the answer is "where ever the code is"....so if you have a boot loader in EPCS flash at a known offset, then put the reset vector there. Likewise, if you're using an on-chip ram that is configured with the boot-loader, you'd reset there.
About resetting/booting from EPCS:
When you choose to put your prog/data memory into, say, SDRAM for a software project (in the IDE), and your reset vector is pointing to some other memory (flash, EPCS chip), the IDE, during software build, will assume that you want to boot out of the non-volatile memory. It will create flash programming files that place a small boot loader before your code + initialized data, and when you use the flash programmer it will program this data into flash starting at the reset address you specified in SOPC Builder. This boot-loader is Altera-provided and gets placed in without any user code. In the case of a conventional flash chip the boot loader prepends the data going into flash, and for the EPCS flash, the boot loader will live in a small onchip memory (more detail on this below).
About resetting/booting from on-chip memory:
If you take this approach you're on your own to write a boot loader. You'd do so by setting up a small software project in the IDE and specifying that program/data memory be housed in that on-chip memory. As part of the software build process, the IDE will generate memory initialization files that Quartus will pick up and use to initialize the memory with during place & route. The "small" example designs and hello_led software app demonstrate this. So... this path isn't nearly as automated as our flash boot-loaders. However, it can provide you with additional flexibility. For example, you can setup an onchip memory that Nios boots from, and then once that boot procedure is done, use that same memory for some other useful purpose like your vector table, scratch pad RAM, or the mailbox between CPUs.
On multiple CPUs:
You are correct that you can run multiple CPUs out of the same physical memory, provided that their program/data spaces won't clash with each other. The caveat here is that with the first release of Nios II, we don't provide a clean way of doing this (for example, in the IDE you can choose a memory device for program/data space in a given application -- not a memory *range* within a memory device). Really what this sets up is a linker script which tells the linker where it can and cannot use memory. So, for multiple CPUs sharing a single memory you'll need to modify a linker script to segregate a single memory into two areas, and then use this linker script for each processor's software build. Writing linker scripts is not what I'd call exciting, so if you really do need to share a common memory device between two Nios', I'd suggest creating a software project using the auto-generated linker script (which will make a file called 'generated.x' in your syslib), and then tweak this linker script to dice up the memory between the two CPUs, and finally, use the new scripts for building software for each CPU. Just so you know, supporting multiple CPUs in a clean manner (from the IDE GUI) is definitely on the road map as far as things to come!
On the no-fits due to memory:
It sounds like the error about not placing the 254 ram blocks is a separate issue -- is this still a problem? Instantiating the EPCS controller will use some memory - 1Kbyte of memory in the form of a M4K blocks if I remember correctly, which is used for the boot-loader code that I mentioned above. If I remember correctly a RAM block is (at smallest) 512 bits of memory (M512)… so the message that 254 could not be placed is a bit weird. I've seen this problem when, for example, I'd done something silly like make a 2K ram and tell SOPC Builder to use an M-RAM block that my device didn't have. If the problem is beyond these basics, please let me know and perhaps we can talk about it over email.
edit: Sorry, I just realized you want to boot both CPUs out of the same EPCS chip. I'm not sure if this will fly -- time for an experiment. If Stephen doesn't know off hand one of us will probably try an experiment out. I suppose whether this can be done depends on how flexible the boot loader is that I described above.