Continuous Variables (CV) Quantum Key Distribution (QKD) enables secure communications using long-established optical fiber infrastructures. CV-QKD can be implemented with Gaussian modulation using coherent states, which is theoretically optimal. Recently, Polarization-Division Multiplexing (PDM) CV-QKD systems have been proposed to improve the spectral efficiency, increasing significantly the secret key rate and improving infrastructure protection against attackers. In this work we consider a PDM-CV-QKD system with a polarization diversity receiver. Moreover, in the receiver we use double homodyne detection to measure both quadratures of the two orthogonal polarization components of the quantum signal. Nevertheless, when implemented over optical fibers, CV-QKD systems face a random polarization drift, particularly impactful for PDM setups. Depending on the fiber installation, the variations of the state of polarization (SOP) of the quantum signal over the fiber can significantly impact the performance of the system. Monitoring SOP variations requires additional equipment and increases the complexity of the system. Adding extra hardware to reverse the polarization drift can increase the excess noise of the CV-QKD system. In order to address this issue we design, test and implement a dual-polarization Constant Modulus Algorithm (CMA) in the digital signal processing step of the CV-QKD system, using a pilot signal as a reference to compensate for the polarization fluctuations. We focus on studying the impact of the polarization drift over the quantum channel using a post-compensation algorithm, i.e. CMA, on a PDM-CV-QKD system. To achieve this, we simulate both the system and the polarization drift in the channel, and analyze how the polarization drift affects the correlation between the data exchanged between Alice and Bob, before and after post-compensation. Furthermore, we investigate the scenarios for which the CMA can compensate for the SOP fluctuations without introducing noise and measure the impact of the polarization drift on the performance of the system.