Abstract: A novel algorithm is proposed in this paper to detect anomalous human behaviors based on motion directions. According to different motion direction rules for different events, we introduce block-based motion directions to model those events, and use support vector machine (SVM) to detect the abnormalous actions from real-time monitoring video sequences. To increase the robustness against noise and to capture the slight movement of the object, we select the foreground frames (the frames having human object) with a background edge model before the action feature extraction. Then, action features are extracted using normalized histogram analysis from the motion directions of all the foreground frames. Our experiments on public areas such as hallway show that our algorithm is able to track complex actions of single and multiple people accurately and is robust against the variation of object size, lighting, and noise during their movements. Our algorithm is of low computation complexity thus it can be used for real time monitoring.