Since the past several years, face recognition from video has received significant attention due to wide range of commercial and law enforcement applications, such as surveillance systems, closed circuit TV (CCTV) monitoring, etc. Human face detection is the first and important task in a dynamic environment, such as video, where noise conditions, illuminations, locations of subjects and pose can vary significantly from one frame to another frame. In this paper, a novel elastic window, which does not make any assumption about the pose, expression or prior localization of a face in a video frame is presented for finding boundary of face region. The window locates the possible face boundaries by elastically expanding its size using local image gradients. Prior to this, a video-frame undergoes in several pre-processing tasks in order to remove noise, background, etc. And producing thin binary image representing only possible face boundaries and scattered noises. After detecting faces from video frames, we extract discriminant facial features from these cropped face images. A multi-class SVM is used as a classifier for face recognition based on these facial features. The proposed method was evaluated on Honda/UCSD video database and the experimental results show that the proposed method outperforms several existing video-based face recognition methods in terms of face recognition.