Abstract: We propose a hierarchical mean shift (HMS) algorithm for object tracking. Firstly, cluster modal points are obtained by mean-shift iteratively processing all the data points in the region so that they can represent foreground object in a succinct manner. The target model and the target candidate model are described by the cluster modal points, and match processes of clustered blocks are performed. Then, on the basis of cluster blocks match, similarity measure function is set up to match between target model and target candidate at pixel level. And the pixel shift vector of target is calculated with the introduction of the neighborhood consistency concept. So, the centroid of tracking object is got layer by layer in the consecutive frames, and the HMS match iteration for object tracking is presented. Experimental comparisons with other two MS algorithms demonstrate the validity and performance of the proposed algorithm.