About a custom UART and the JTAG UART

The JTAG UART doesn't have a baud rate, it runs as fast as necessary to transfer the data.

I think your problem is likely to be related to flow control. The standard UART doesn't have flow control enabled by default so if its buffer fills up it will lose data. There is an ioctl which you can use to turn on flow control (I can't remember exactly what, but I think it was something like tcsetattr).