I have solved the problem.
It was as expected a problem with the memory latency.
On my SDRAM controller are now:
- the NIOS II data and instruction master
- the Frame Reader master
However, I had to change the arbitration for the Frame Reader to give it more priority.
I found this in
this (https://www.altera.com/en_us/pdfs/literature/tt/tt_nios2_system_architect.pdf) document.
Right click in the Systems Contents view in Qsys.
Check Show Arbitration Shares.
The priority is on the point where you connect different interfaces as a number. Default is 1.
I changed the frame reader priority to 50.
There are still some artifacts, but that should be easy to fix.
I hope it can also help someone else.