Information theory offers a coherent, intuitive view of model selection. This perspective arises from thinking of a statistical model as a code, an algorithm for compressing data into a sequence of bits. The description length is the length of this code for the data plus the length of a description of the model itself. The length of the code for the data measures the fit of the model to the data, whereas the length of the code for the model measures its complexity. The minimum description length (MDL) principle picks the model with smallest description length, balancing fit versus complexity. Variations on MDL reproduce other well-known methods of model selection. Going further, information theory allows one to choose from among various types of models, permitting the comparison of tree-based models to regressions. A running example compares several models for the well-known Boston housing data.
Abramovich, Felix, Yoav Benjamini, David Donoho, and Ian Johnstone. 2000. “Adapting to Unknown Sparsity by Controlling the False Discovery Rate.” Technical Report 2000–19, Department of Statistics, Stanford University, Stanford, CA.
2.
Barron, Andrew R., Jorma Rissanen, and Bin Yu.1998. “The Minimum Description Length Principle in Coding and Modeling.”IEEE Transactions on Information Theory44:2743–2760.
3.
Belsley, David A., Edward Kuh, and Roy E. Welsch. 1980. Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. New York: John Wiley.
4.
Breiman, Leo and Jerome H. Friedman. 1985. “Estimating Optimal Transformations for Multiple Regression and Correlation.”Journal of the American Statistical Association80:580–598.
5.
Breiman, Leo, Jerome H.Friedman, Richard Olshen, and Charles J. Stone. 1984. Classification and Regression Trees. Belmont, CA: Wadsworth.
6.
Bryant, Peter G. and Olga I. Cordero-Braña. 2000. “Model Selection Using the Minimum Description Length Principle.”American Statistician54:257–268.
7.
Cover, Thomas M. and Joy A. Thomas. 1991. Elements of Information Theory. New York: John Wiley.
8.
Elias, Peter. 1975. “Universal Codeword Sets and Representations of the Integers.”IEEE Transactions on Information Theory21:194–203.
9.
Foster, Dean P. and Edward I. George. 1994. “The Risk Inflation Criterion for Multiple Regression.”Annals of Statistics22:1947–1975.
10.
Foster, Dean P. and Robert A. Stine. 1996. “Variable Selection Via Information Theory.”Technical Report Discussion Paper 1180, Center for Mathematical Studies in Economics and Management Science, Northwestern University, Chicago.
11.
Foster, Dean P. and Robert A. Stine. 1999. “Local Asymptotic Coding.”IEEE Transactions on Information Theory45:1289–1293.
12.
Foster, Dean P. and Robert A. Stine. 2002. “Variable Selection in Data Mining: Building a Predictive Model for Bankruptcy.”Manuscript submitted for publication.
13.
Foster, Dean P. and Robert A. Stine. 2003. “The Boundary Between Parameters and Data in Stochastic Complexity.” In Advances in Minimum Description Length: Theory and Applications, edited by Peter Grunwald, I. J. Myung, and M. Pitt. Cambridge, MA: MIT Press.
14.
Fox, John. 1984. Linear Statistical Models and Related Methods With Applications to Social Research. New York: John Wiley.
15.
George, E. I. and D. P. Foster. 2000. “Calibration and Empirical Bayes Variable Selection.”Biometrika87:731–747.
16.
Hansen, Mark H. and Bin Yu.2001. “Model Selection and the Principle of Minimum Description Length.”Journal of the American Statistical Association96:746–774.
17.
Harrison, David and Daniel L. Rubinfeld. 1978. “Hedonic Prices and Demand for Clean Air.”Journal of Environmental Economics and Management5:81–102.
18.
Hinkley, David V.1970. “Inference About the Change-Point in a Sequence of Random Variables.”Biometrika57:1–17.
19.
Langdon, Glen G.1984. “An Introduction to Arithmetic Coding.”IBM Journal of Research and Development28:135–149.
20.
Lanterman, Aaron D.2001. “Schwarz, Wallace, and Rissanen: Intertwining Themes in Theories of Model Selection.”International Statistical Review69:185–212.
21.
Lee, Thomas C. M.2001. “An Introduction to Coding Theory and the Two-Part Minimum Description Length Principle.”International Statistical Review69:169–183.
22.
Manish, Mehta, Jorma Rissanen, and Rakesh Agrawal. 1995. “MDL-Based Decision Tree Pruning." In Proceedings of the First International Conference on Knowledge Discovery and Data Mining (KDD’95), pp. 216–221.
23.
McCullagh, Peter. 2002. “What Is a Statistical Model?”Annals of Statistics30:1225–1310.
24.
Miller, Alan J.2002. Subset Selection in Regression. 2nd ed.London: Chapman & Hall.
25.
Rissanen, Jorma. 1978. “Modeling by Shortest Data Description.”Automatica14:465–471.
26.
Rissanen, Jorma. 1983. “A Universal Prior for Integers and Estimation by Minimum Description Length.”Annals of Statistics11:416–431.
27.
Rissanen, Jorma. 1986. “Stochastic Complexity and Modeling.”Annals of Statistics14:1080–1100.
28.
Rissanen, Jorma. 1987. “Stochastic Complexity.”Journal of the Royal Statistical Society, Series B49:223–239.
29.
Rissanen, Jorma. 1989. Stochastic Complexity in Statistical Inquiry. Singapore: World Scientific.
30.
Rissanen, Jorma and MatiWax. 1988. “Algorithm for Constructing Tree Structured Classifiers.” Technical Report No. 4,719,571, U.S. Patent Office, Washington, DC.
31.
Venables, William N. and Brian D. Ripley. 2002. Modern Applied Statistics With S-PLUS. 4th ed.New York: Springer.
32.
Wallace, C. and P. Freeman. 1987. “Estimation and Inference by Compact Coding.”Journal of the Royal Statistical Society, Series A49:241–252.
