The problem is mostly if your two clocks aren't at the same exact frequency. If they come from the same oscillator, but are shifted, then you won't miss any data. But if there is a slight frequency difference, then from time to time either one word won't get read in time (it will fill the fifo with one additional word), or the read side will attempt to read the fifo but no data will be available. If you have a 100ppm difference this will happen approximately every 0.2ms so it's not negligible.
Edit: oups, didn't see you didn't have a continuous stream... then as long as you use the fifo's full and empty signals to control the flow, you should be ok