Abstract: An algorithm is presented to analyze rain-affected videos captured by outdoor vision systems, including the detection of moving objects affected by raindrops and rain removal at background. It is indicated that the rain-affected pixel has the same color with its background by analyzing the physical imaging model of raindrops and the human visual system. But due to the degradation of video quality, the hue of color cannot be used directly to detect raindrops. Therefore, a measure function in HSV space is proposed to distinguish the rain-affected areas from moving objects. The results show that the measure function has better performance in suppressing the interference caused by raindrops, and overcomes the limitation of the existing models which are only able to identify some special kinds of raindrops by statistical methods. The measure function possesses strong robustness for video degradation and is adequate for fast detection.