Abstract: A scheme is proposed to integrate multiple cues with kernel tracking by adaptive fusion to improve the robustness of object tracking in the time-variant scenario. The tracked object is represented by a set of submodels of each cue, and then the multiple cues are combined by linear weighting to realize kernel-based tracking. According to the discriminability of each cue between target and background, measured by Fisher rule, an adaptive mechanism is presented to update the cue weight. Furthermore, a selective submodel update strategy is utilized to alleviate the model drift. In experiments, we employ color cue and local binary pattern (LBP) texture cue to implement the scheme, and the results demonstrate the effectiveness of the proposed method in several real sequences testing.