In this paper, a new background modeling method using Gaussian mixture model based on Bayer color filter array is proposed. It improves the efficiency of obtaining the prospect moving objects in the color video. Besides, it is proposed a method that can track the multi-moving objects using the combination of Camshift and Kalman filter based on the prospect image. Take actual moving objects detection and tracking as examples, this algorithm has advantages of quick speed and good accuracy. Meanwhile, it can realize multi-moving objects tracking and overcome occlusion problem in the process of tracking.