Abstract: We present a novel approach to behaviour recognition in visual surveillance under which scene events corresponding to object behaviours are modelled as groups of affiliated autonomous pixel level events automatically detected using Pixel Change Histories (PCHs). The Expectation Maximisation (EM) algorithm is employed to cluster these pixel-level events into semantically more meaningful blob-level scene events, with automatic model order selection using modified Minimum Description Length (MDL). The method is computationally efficient allowing for realtime performance. Experiments are presented to demonstrate the effectiveness of recognising these scene events without object trajectory matching.