For the video surveillance system nowadays, identifying the color of certain footage is paramount. Every time when it comes to a crime scene, the police will be able to extract useful information from the surveillance cameras on the scene. Among that important information, “color” plays a major role and is not affected by the size, location, time or changes in shape of an object. Facts affecting the accuracy and efficiency of realtime color identification include the material and the parameter of camera lens, the parameters of infrared for the hardware part, and efficiency of software algorithms, accuracy and practical degree for the software part, and so forth. Although many methods using static analysis of color have been proposed, they cannot effectively solve the color recognition in the real-time video system. In this paper, we provide an approach using dynamic algorithm for the real-time video system. The methodologies include the reduction of color dimension, color transformation, color classification and real-time color recognition. Lastly, we provide the methods for effectively improving both the efficiency and accuracy of a real-time surveillance system.