The PLL has a loopback path that it compensates for(generally this is one of the PLL clocks going through a global, and as one of its outputs, feeding back into the PLL). This is what it can compensate for. When you use a dedicated clk input pin, then the input to the PLL is a really quick path and has little affect on the overall clock delay(and can almost be accounted for with a phase-shift). So the net affect is your entire clock tree is compensated for, and it can be nicely correlated to the clock coming into the FPGA.
When coming from a transceiver, there is some delay through the GXB and to the PLL that is not part of the loopback path and can't be compensated for. It can be calculated as a raw delay, but the PLL just won't be compensating for it over PVT. This is perfectly fine, as it's not like the recovered clock has some known reference edge that you need to relate to. In essence, it's physically impossible to compensate for the delay before the PLL, and that's perfectly fine in your design.
(To think of where it would be useful, let's say you had two FPGA's recovering data. You couldn't synchronously pass data between the two recovered clocks, because they wouldn't be edge aligned. That's expected when recovering a clock and using it.) Hope that helps.