We investigate the problem of image classification within a supervised learning framework that exploits implicit mutual information in different visual features and their associated classifiers.  In our proposed two stage hierarchical processing, visual features are first clustered with the objective of maximizing diversity. Majority vote within each cluster is used to enforce diversity. Many partitioning variations are evaluated using K-nearest neighbor to obtain the highest inter-cluster entropy. In the second step, a richer measure of discrimination is obtained using a fully connected conditional random fields (CRF) over clusters. The unary and interaction potentials are defined over mutual information within each cluster and inter-dependencies across clusters respectively. Experimenting over five distinct datasets, we demonstrate an average performance gain of 30% compared with state of the art techniques.