Abstract: Automatic image annotation, which aims at automatically identifying and then assigning semantic keywords to the meaningful objects in a digital image, is not a very difficult task for human but has been regarded as a difficult and challenging problem to machines. In this paper, we present a hierarchical annotation scheme considering that generally human's visual identification to a scenery object is a rough-to-fine hierarchical process. First, the input image is segmented into multiple regions and each segmented region is roughly labeled with a general keyword using the multi-classification support vector machine. Since the results of rough annotation affect fine annotation directly, we construct the statistical contextual relationship to revise the improper labels and improve the accuracy of rough annotation. To obtain reasonable fine annotation for those roughly classified regions, we propose an active semi-supervised expectation-maximization algorithm, which can not only find the representative pattern of each fine class but also classify the roughly labeled regions into corresponded fine classes. Finally, the contextual relationship is applied again to revise the improper fine labels. To illustrate the effectiveness of the presented approaches, a prototype image annotation system is developed, the preliminary results of which showed that the hierarchical annotation scheme is effective.