Abstract: We present an algorithm for color image understanding, which segments the image after analyzing surface with color variations due to lighting condition and object colors. The work is based on a theory-the Dichromatic Reflection Model-whilch describes the color of the reflected light as a linear combination of light from surface reflection (highlight) and body reflection (object color). A peculiar cluster shape (skewed T) is formed by color pixels of an object area in 3D histogram in color space. The analysis of cluster property helps to determine the color of light and objects, but at the same time, the meaningful cluster is generated by the projection of certain object area of the image. According to the strategy of hypothesis and test, following the continuity of the image and the feature of color clusters, the algorithm completes image segmentation and gives forth the physical description of imaging process, including intrinsic images, segmented image, the color of light and objects. Reflecting respectively the properties of light condition and every object, both the matte image and highlight image compose the intrinsic images.