Custom-PHY Error in Qsys

Every interface in a component, custom or otherwise, must be synchronous to a clock interface. I see tx_clkout0 in your screenshot, not tx_clkout1, so perhaps you did not create that clock interface.

If you think it should not be synchronous to that clock (it's just asynchronous control signals), you could set it as a conduit interface instead of a streaming sink.