One recently developed model for pattern recognition is the structural hidden Markov models (SHMMs) [15,16]. To avoid the complexity problem inherent to the determination of the higher level states, the SHMMs provide a way to explicitly control them via an unsupervised clustering process. This capability is offered through an equivalence relation built in the visible observation sequence space. This approach also allows the user to weight substantially the local structures within a pattern that are difficult to disguise. This provides a SHMM recognizer with a higher degree of robustness. Therefore, the SHMM is well suited to simultaneously model the within and between structures information of any sequential pattern (such as a face or a fingerprint). Indeed, the concept of SHMMs have been shown to outperform HMMs in a number of applications including handwriting recognition [15], but have yet to be applied to face recognition.