Abstract: A model switching based background modeling approach (MSBM) has been proposed. This approach uses entropy image as ligament to realize adaptive switching between background models with different elaborations in a spatio-temporal domain. For background regions with high complexity of pixel's value distribution, we adopt an elaborate model to guarantee the accuracy of moving object detection; otherwise, we adopt a coarse model to reduce the computational load. Combining adaptive model structure with adaptive model parameter, MSBM can improve the processing speed greatly without sacrificing the accuracy. A double-model-switching moving object detection algorithm based on Gaussian mixture model and temporal average model has been used in the experiment and the results show that it can possesses almost the same detection accuracy and much higher image processing frame rate than Gaussian mixture model.