We present a polarization diverse receiver architecture for continuous variables quantum key distribution. We demonstrate experimentally that our system is capable of reliably achieving theoretical security against collective attacks, even under very adverse random polarization drift scenarios, and is capable of functioning indefinitely without any user input. We show that, in a similar scenario, a standard CV-QKD system without polarization diversity cannot distribute keys securely. Our receiver architecture forgoes the need for any polarization matching calibrations or feedback loops and is usable in any locally generated local oscillator key distribution system. A theoretical study on the security impact of polarization drift is presented, based on which we show that, when using our polarization diverse receiver, it is possible to achieve the performance of a channel with zero polarization drift.