Abstract: An efficient vision system is proposed for lane detection and multiple car tracking on the highway. The main modules of the system are: lane detection, separate 2-D model-based trackers (rectangular model for passing car and U-shape model for distant car), heuristic car detection, and a process coordinator. In the system, the dynamical creation and termination of tracking processes optimizes the amount of spent computational resources. Lane detection performance is improved by robust estimation technique. And ear tracking is realized by a polygon fitting approach in three-parameter state space. The system is successfully tested with the image sequence from PETS2001 and the average processing time per frame is 12ms on a Pentium Ⅲ 450MHz PC.