A novel framework for pruning a category of images is proposed in this paper. We assume no prior information about the contents or semantics of the images. Our framework discovers consistencies and knowledge about the spatial relations of the categories unsupervised using iterative image segmentation and spatial grouping. A measure for deciding how well an image fits to a category is proposed and the effectiveness of this measure is investigated.