How to use Shared pll in LVDS transmiter and resceiver functions

It seems you misunderstood the serdes functionality. Why do you connect the two clocks you mentioned?

I assume your 100MHz is the ref clock for serdes PLL(it has nothing to do with data). The PLL should then produce two clocks:

At Rx: the PLL should generate 40MHz for your 20bit data processing.

At tx:The PLL should generate 800MHz internally to kick serial bit stream out at 800 Mbps. This fast clk is not accessible to you, it is for serdes use.