Deep learning (DL)-based coding has recently become very popular for multimedia data, notably images and point clouds (PCs). Training a DL coding model using the backpropagation algorithm requires a differentiable loss function. Thus, for PC joint geometry and color coding, both the PC geometry and color distortion metrics must be differentiable. Since the distortion/quality metrics commonly used for the final PC quality assessment do not meet this criterion, new PC distortion metrics have to be designed for DL-based training purposes. Moreover, for PC joint geometry and color coding, it is critical to define the balance between the geometry and color distortions in a meaningful way, ideally driven by the human perception and subjective quality assessment. In this context, this paper proposes a perceptually-driven design for a differentiable PC joint geometry and color distortion metric to be used for training purposes in DL-based coding, notably to define the relative weights for the geometry and color distortions. The obtained perceptually-driven weights achieve a rate reduction of around 3% regarding the default balanced weights at no complexity cost. This is the first proposal in the literature with this purpose and this perceptual approach.