Abstract: A novel multitarget tracking algorithm in unknown clutter is proposed in this paper. In the proposed algorithm, the multitarget likelihood function is described based on the finite mixture model (FMM), whose parameters are estimated according to the algorithm of expectation maximum (EM) and the technology of model merging and pruning. The estimation of clutter model, target number, and multitarget states can be derived based on the estimated parameters. Similar to the multitarget tracking algorithms based on random finite set (RFS), the association process between the targets and measurements can be avoided in the algorithm proposed. The simulation shows that the estimation results of the proposed algorithm are much better than those of the multitarget tracking algorithms without the fitting of clutter model, especially when the clutter models are complicated and unknown.