From one Ethernet to the other

I don't think your delay blocks are enough. You are sending data from one clock domain to the other, with no special synchronization between the two clocks. The RX clock from each PHY is recovered from the incoming packet on the cable, while the TX clock is generated locally with the crystal. There is no guarantee that the two clocks are synchronized, and not even at the same frequency. When transferring data you need a proper clock domain crossing circuitry, for example with dual clock fifos.