Video surveillance system is a process of monitoring and analyzing video sequences for the purpose of checking the behavior, activities and other certain information in a video sequence. It is really a very upcoming area in the real time system. It takes lot of data to storage in computer system. Video surveillances can be performed using three different activities; it can be manual, semi-autonomous or fully autonomous [1]. In Manual video surveillance the video contents are analyzed by human being and this type of Manual systems are mostly used in real time system. But Semiautomated video surveillance system consists of both human intervention and up to some extent of video processing. Most useful examples of this type of system are found in simple moving object detection in video sequence [2]. A fully autonomous system is used for extra ordinary motion in captured video and then sent to the human expert for further analysis. In case of fully-autonomous system [1], video sequence is the input which is taken from the source where we had performed the surveillance system. In such type of system, the system perform all kind of tasks, such as motion detection and tracking, and it also perform the high level of decision making activity such as unusual event detection. Video surveillance system must be able to supports automatically moving object detection and then tracking. Because analyzing of video sequence for a long time by human or by surveillance system is no longer practical and not feasible. But automatic detection of moving object generates batter human intervention.