Abstract: We propose a new method for real-time 3D human tracking and modeling, and implement a robust reference application system. Focusing on applications, e.g. desktop human computer interface, which require low tracking accuracy, the method makes a good compromise between the accuracy and system simplicity. It uses multiple cameras and recovers the depth maps in real-time, and then uses both color and depth information for tracking. It adopts a simple 3D human upper body model, and uses color-histogram-based particle filtering to track human head and hands, and hereby reconstructs the 3D model. By using face detection and hand color clustering algorithms, the system initialization is fully automatic. Extensive experiments demonstrate that the system can robustly track and model human motion from complex background, and can automatically recover from losing of tracking object. The system runs at 25Hz on a 2.4GHz PC.