This paper firstly proposes a new algorithm for finger touch detection based on front-projected vision-based table system. The algorithm, taking advantage of shadow information, can locate possible finger touch with low complexity. An adaptive structured light method is introduced to improve the precision of touch detection effectively. Secondly, we present a new prototype system architecture and corresponding hardware design for portable bare-hand human computer interaction (HCI). Experimental results show that the accuracy of touch detection of single finger and multi-finger is 98.3% and 92.6% respectively