Object tracking in image and design of controller for fast operation, as well as techniques and algorithms are proposed to response the requirements of real applications. Meanwhile, estimating position and trajectory of object is utilized as a part of object tracking since object position can be fed back for camera motion control. Generally, using vision of a single camera for object pose estimation requires camera calibration in order to find camera parameter. However, even vision provides accepted accuracy; Drift of estimated object trajectory is found. This work presents a new technique for target tracking on a horizontal plane using both visions from a camera and distance information from an infrared sensor for estimating object pose and tracking by keeping object at the center of frame all the time. The 2 DOF camera system is constructed using DC motors and web camera for testing the proposed method. The distinct of the proposed technique is that there is no need for camera calibration and estimation of object pose is simple using Euler angle theory. A color ball, no light absorbed, is used in the experiment. Even though the infrared data contains noise, the Kalman Filter is applied for signal fusion and accurate object trajectory on a plane is obtained.