An accurate and early diagnosis of the Alzheimer’s Disease (AD) is of fundamental importance for the development of effective treatments to palliate the effects of the disease. Computer Aided Diagnosis (CAD) allows physicians to detect early stages of the disease, and functional brain images have been proved to be very useful in this task. This paper presents a new CAD system that consists of three stages: voxel selection, feature extraction and classification. Voxels are selected in terms of their significance, by using Mann–Whitney–Wilcoxon U-Test. Then, Factor Analysis is proposed to carry out the feature reduction step, by extracting common factors and factor loadings from the selected voxels. Finally, a Linear Support Vector Machine (SVM) classifier is trained to perform clustering of the input images. Two different databases are considered for testing the proposed methods: the first one, consists of 96 Single Photon Emission Computed Tomography (SPECT) images from the “Virgen de las Nieves” Hospital in Granada, Spain, and a 196 Positron Emission Tomography (PET) database from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). The proposed method achieves accuracy results of up to 93.7% and 92.9% for SPECT and PET images respectively, and reports benefits over recently reported methods.