Abstract: The traditional Mean Shift tracking algorithm partitions uniformly the whole color space, leading to a great number of void histogram bins, and is unable to represent accurately the color distribution of the object. To address the problems, we present an improved algorithm. Firstly the object color is analyzed using a clustering algorithm, and according to the clustering result the color space of the object is partitioned into subspaces by matrix factorization and orthonormal transformation. Then a new color model is defined by considering the weighted number of pixels as well as within-cluster distribution with Gaussian, and a novel measure is derived to evaluate the similarity between the reference color model and the candidate model. Finally an improved algorithm is proposed based on the color model. Experiments show that the improved algorithm has better performance than and is computationally comparable to the conventional mean shift algorithm.