8 Conclusion We have presented a basic concept for a vision based interactive robot control as a part of multi-modal man-machine interaction system. Implementing a straight forward technique for tracking a human hand and extracting a poiting gesture we could demonstrate how man and machine become a coupled system in a very natural way. The object recognition system can use those pointing gestures to minimize the searching area or, in the common case, can search the whole scene for a special object. Thereby the object’s position and orientation is extracted very robustly. By using multiple channels of information, the whole system is able to overcome ambiguities and the robot can react in an adequate matter. For the future we plan to extend the multi-modal manmachine interaction system by integrating recognition of the human’s direction of view.