Model Based Document Classification and Clustering

Abstract

In this paper we develop a complete methodology for document classification and clustering. We start by investigating how the choice of document features influences the performance of a document classifier and then use our findings to develop a clustering method suitable for document collections. From our study of the effect of frequency transformation, term weighting and dimensionality reduction through principal components analysis on the performance of a simple K-nearest-neighbors classifier, we conclude that: (a) applying a logarithm or square-root transformation to the term frequencies reduces error rates; (b) term weighting of the transformed frequencies does not appear to help much; and (c) increasing the feature space dimension beyond 50 does not improve performance. We use these findings in the construction of a Gaussian Mixture Document Clustering (GMDC) algorithm. This algorithm models the data as a sample from a Gaussian mixture. The model is used to build clusters based on the likelihood of the data, and to classify documents according to Bayes rule. One main advantage of our approach is the ability to automatically select the number of clusters present in the document collection. Our experiments with the Topic Detection and Tracking Corpus demonstrates the ability of GMDC to choose a sensible number of clusters and to generate meaningful partitions of the data.