Robust semantic labeling of image regions is a basic problem in representing and retrieving image/video content. We propose an SVM-MRF framework to model features and their spatial distributions, leading towards a "semantic" representation. Eigenfeatures of Gabor wavelet features and Gaussian mixture model are used for feature clustering. Since similar feature vectors in one cluster can come from several different semantic classes, SVM is applied to represent conditioned feature vector distributions within each cluster, and a Markov random field is used to model the spatial distributions of the semantic labels. A semantic layout representation is proposed to describe the semantics of the images. Experiments show that this method can improve semantic labeling and is useful in similarity search. optimization. The sites for the MRF are blocks of pixels, each of which is described by a visual feature descriptor. The spatial distribution of semantics labels of each image can be considered as a Markov random field (MRF). Therefore, the analysis phase involves the automatic identification of the semantic classes in a given image, and is a three-step procedure. The first step is to cluster the features of the image blocks using the Gaussian mixture model (GMM) 2. The GMM is used to model the principal components of the original feature vectors. Each Gaussian represents a cluster in this model. This can improve the clustering performance if the number of clusters is not large. Since similar feature vectors in one cluster can come from several different semantic classes, the second step is the application of the "one-against-others" SVMs to classify the image blocks into candidate.