Abstract: This paper presents a system that is able to reliably track multiple faces under varying poses (tilted and rotated) in real time. The system consists of two interactive modules. The first module performs the detection of the face that is subject to rotation. The second module carries out online learning-based face tracking. A mechanism that switches between the two modules is embedded into the system to automatically decide the best strategy for reliable tracking. The mechanism enables a smooth transit between the detection and tracking modules when one of them gives either nil or unreliable results. Extensive experiments demonstrate that the system can reliably carry out real time tracking of multiple faces in a complex background under different conditions such as out-of-plane rotation, tilting, fast nonlinear motion, partial occlusion, large scale changes, and camera motion. Moreover, it runs at a high speed of 10~12 frames per second (fps) for an image of 320~240.