In this work, the error vector magnitude (EVM) is
used as a tool to study the impact of in-band crosstalk on the
performance of a coherent receiver with polarization division
multiplexing at a bit rate of 100 Gbps. The accuracy of the
EVM method on the estimation of the optical signal-to-noise
ratio (OSNR) penalty due to in-band crosstalk is studied by
comparison with the results obtained from direct error counting
using Monte-Carlo simulation. We show that the EVM method
is sufficiently accurate to estimate the crosstalk level that leads
to a 1 dB OSNR penalty. However, it becomes inaccurate when
estimating the OSNR penalty due to higher crosstalk levels and
lower duty-cycles on the selected and interfering signals.