This paper deals with the challenging problem of visual anomaly detection in a cluttered environment using videos acquired with a moving camera. The anomalies considered are abandoned objects. A new method is proposed for comparing two videos (an anomaly-free reference video and a target one possibly with anomalies) by using convolutional neural networks as feature extractors for a subsequent anomaly-detection stage using a classifier. Two classifier strategies are considered, namely a fully-connected neural network and a random forest algorithm. Results for a comprehensive abandoned object database acquired with a moving camera in a cluttered environment indicate that the proposed architecture can match even the state-of-the-art algorithms in terms of object-detection performance, with a reduction in processing time of 80%.