Abstract: In this paper, we propose a Markov random field (MRF) based representation for the Gestalt law, and suggest using a message passing-like scheme to infer the segmentation. Different from other grabcut models, our MRF function is specially encoded to consider orientations along the contour, thus the Gestalt law is embedded into the inference. As a basic framework of the research in figure-ground separation and Gestalt law, our system is designed in reference to neurophysiology, and the architecture is composed of three modules: primal visual cortex (V1), extra-striate cortex (V2), and the interest selected region. To validate our method, we conduct experiments in both auto and interactive segmentation algorithm. The results are better than those of grabcut and other related algorithms.