One of the leading augmented reality researches using color-based tracking is SixthSense [1]. This system was initiatively proposed by Mistry and his group from MIT Media Laboratory. As shown in Figure 1, SixthSense is a gestural system that allows natural hand gestures interaction with digital information. The system projects the digital information onto the surfaces and real objects around us, making any surface into a digital interface. The camera recognizes the user’s colored fingertip markers and physical objects using the vision-based tracking algorithm [2]. However, using colored fingertip markers sometimes seems to be unnatura