using open() C command in the Nios II

For open and write/printf to work you need a file system. The NIOS II does not have a file system unless you include some storage device and a device driver for it. You can either log the data on the serial port as suggested. Another alternative is to use the JTAG serial port if your real serial port have issues as you described (or try to lower the serial port speed to see if it helps). Another method would be to include some serial file transfer protocol which includes error check and retry, e.g. kermit or x/y/zmodem.