This paper describes an efficient approach for irregular moving object detecting and tracking in real-time system based on color and shape information of the target object from realistic environment. Firstly, the data is gotten from a real-time camera system at a stable frame rate. And then, each frame is processed by using proposed method to detect and track the target object immediately in consecutive frames. Finally, the target position based modifying controlling signal is used to control pan-tilt-zoom camera (PTZ camera) in order to automatically follow the target object. Our experiment results were obtained by using pan-tilt-zoom camera Sony EVI D70 under variety environments in real-time and our algorithm is verified that it can be implemented effectively and accurately at high frame speed, even 29.97 fps.