Abstract: Gaussian mixture model (GMM) is one of the best models for modeling a background scene with gradual changes and repetitive motions. However, it fails when the scene changes suddenly, e.g., a door is opened or closed. To handle such problems, we propose a memory-based Gaussian mixture model (MGMM) inspired by the way human perceives the environment. The human memory mechanism is introduced to model the background. Each pixel of every frame is processed and transferred through three spaces: ultra-short time memory space, short time memory space, and long time memory space. The proposed memory-based model can remember what the scene has ever been, which helps the model adapt to the variation of the scene more quickly.