Automatic classification of lesions for gastroenterology imaging scenarios poses novel challenges to computer-assisted decision systems, which are mostly attributed to the dynamics of the image acquisition conditions. Such challenges demand that automatic systems are able to give robust characterizations of tissues irrespective of camera rotation, zoom, and illumination gradients when viewing the inner surface of the gastrointestinal tract. In this paper, we study the invariance properties of Gabor filters and propose a novel descriptor, the autocorrelation Gabor features (AGF). We show that our proposed AGF is invariant to scale, rotation, and illumination changes in the images. We integrate these new features in a texton framework (Texton-AGF) to classify images from two complementary gastroenterology imaging scenarios (chromoendoscopy and narrow-band imaging) broadly into three different groups: normal, precancerous, and cancerous. Results show that they compare favorably to using state-of-the-art texture descriptors for both imaging modalities.