Abstract: Currently, in the vision-based hand gesture recognition, almost all the technologies on hand gesture segmentation are based on simple backgrounds or on gloves in special colors, which give the human-computer interaction some limitation. This paper presents a new method, which segments hand gestures with complex backgrounds by fusing skin chrominance and coarse image motion, and by using the seed algorithm twice. With the segmented hand areas, the algorithm for motion appearance parameters is accelerated greatly. By integrating temporal information, motion and shape appearances, a spatio-temporal appearance model is proposed for representing dynamic hand gestures. This paper also presents an independent distributed multistate Gaussian probability model(IDMGPM) for recognition. In this system the average recognition rate is 97.8% on the training set and 95.6% on the testing set.