Abstract: In the surveillance applications, the scenes are known. So background subtraction can be used to detect the moving pixels. As a panning-tilting camera is used, several video frames are needed to represent the scene of surveillance. How to organize and index these frames to represent the background of surveillance belongs to the problem of camera tracking. We propose a simple camera tracking approach that makes it easy to determine a frame as the current background without the need of camera calibration. This approach needs estimating the initial motion parameters by the technology of image registration. We propose a fast and robust method for registering the current frame and the background frame, by combining the linear regression and the robust regression on the basis of masking the outliers. To fast estimate the motion parameters between two adjacent frames, an optimized algorithm for motion tracking with the 4-parameter model is given. A non-parametric model of background maintenance is introduced to reject the false alarms. Finally, we present experimental results in two applications demonstrating the effectiveness and efficiency of our methods.