Abstract: In the proposed algorithm, different from other video object segmentation algorithms, the motion vector filed is mainly analyzed for segmentation and the spacial relativity is assistant information for segmentation. Firstly, the motion vectors are processed by accumulation and median filter. Secondly, the spacial relativity variable is defined in potential function and maximum a posteriori probability (MAP) is used to segment video moving object. Lastly, the edge of video moving object is made more exactly. The experimental results for different video sequences show the proposed algorithm has a better veracity of segmentation compared to other algorithms.