Estimating the pose of a camera (virtual or real) in which some augmentation takes place is one of the most important parts of an augmented reality (AR) system. Availability of powerful processors and fast frame grabbers have made vision-based trackers commonly used due to their accuracy as well as flexibility and ease of use. Current vision-based trackers are based on tracking of markers. The use of markers in- creases robustness and reduces computational re- quirements. However, their use can be very com- plicated, as they require certain maintenance. Di- rect use of scene features for tracking, therefore, is desirable. To this end, we describe a general sys- tem that tracks the position and orientation of a camera observing a scene without any visual mark- ers. Our method is based on a two-stage process. In the first stage, a set of features is learned with the help of an external tracking system while in action. The second stage uses these learned fea- tures for camera tracking when the system in the first stage decides that it is possible to do so. The system is very general so that it can employ any available feature tracking and pose estimation sys- tem for learning and tracking. We experimentally demonstrate the viability of the method in real-life examples.