Abstract: Human activity recognition is an important issue in the fields of video content analysis and computer vision. Based on analyzing multiple scales of motion details contained in the human activities, we propose a novel human activity recognition approach named hierarchical durational-state dynamic Bayesian network (HDS-DBN). The HDS-DBN contains multiple levels of states and represents multiple scales of motion details as well. Experiments are conducted on the recognition of individual activities and two-person interacting activities. Experimental results show that the HDS-DBN recognizes human activities with high rates and has good robustness to the noise and loss of information. In addition, experimental results demonstrate that the HDS-DBN can represent multiple scales of motion details correctly.