The optical manipulation of small particles has many important applications in nanotechnologies. In particular, fluctuation-induced optical Casimir-Polder forces are relevant in a variety of situations. Typically, when a particle stands nearby a perfectly smooth material surface the lateral force vanishes in equilibrium situations due to the translational invariance of the system. Recently, it was shown that graphene biased with a drift-current can be a very promising platform for nonreciprocal plasmonics in the infrared range. The drift current enables the propagation of unidirectional surface plasmons as the wave can be dragged by the moving electrons. Here, we investigate how the drift-current bias affects the light-matter interactions and demonstrate that it gives rise to an anomalous lateral drag force