The semaphore recognition system which is a humanrobot interaction is developed in this paper. The SOM algorithm is proposed to identify the semaphore signal which is captured by the Kinect sensor and is transferred to skeleton information by Kinect SDK. The contribution of this paper is that the universal semaphore posture recognition system is accomplished. This system can be applied into the application of human-robot interaction system.