The Application of the Cumulative Logistic Regression Model to Automated Essay Scoring

Abstract

Most automated essay scoring programs use a linear regression model to predict an essay score from several essay features. This article applied a cumulative logit model instead of the linear regression model to automated essay scoring. Comparison of the performances of the linear regression model and the cumulative logit model was performed on a large variety of data sets. It appears that the cumulative logit model performed somewhat better than did the linear regression model.

Keywords

deleted residual cross-validation PRESS regression

References

Attali Y.

Burstein J.

.Automated essay scoring with e-rater v.2.Journal of Technology, Learning and Assessment. 2006;4:1-29

Attali Y.

Burstein J.

Andreyev S.

E-rater version 2.0: Combining writing analysis feedback with automated essay scoring. Princeton, NJ: Educational Testing Service; 2003:

Burstein J.

Chodorow M.

Leacock C.

.Automated essay evaluation: the criterion online writing service.AI Magazine. 2004;25:27-36

Cohen J.

.A coefficient of agreement for nominal scales.Educational and Psychological Measurement. 1960;20:37-46

Davey T.

.. Paper presented at the Annual meeting of the National Council of Measurement in Education, April; 2009San Diego, CA; 2009. .

Draper N. R.

Smith H.

Applied regression analysis. 3rd ed.New York, NY: Wiley; 1998:

Feng X.

Dorans N. J.

Patsula L. N.

Kaplan B.

Improving the statistical aspects of e-rater®: Exploring alternative feature reduction and combination rules (ETS Research Rep. No. RR-03-15). Princeton, NJ: ETS; 2003:

Gilula Z.

Haberman S. J.

.Models for analyzing categorical panel data.Journal of the American Statistical Association. 1994;89:645-656

Haberman S. J.

.Concavity and estimation.The Annals of Statistics. 1989;17:1631-1661

10.

Haberman S. J.

Handbook of statistics.

Rao C. R.

Sinharay S.

, ed. Amsterdam, Netherlands: North-Holland; 2007:205-233.

11.

Haberman S. J.

.When can subscores have value?.Journal of Educational and Behavioral Statistics. 2008;33:204-229

12.

Johnson V. E.

.On Bayesian analysis of multirater ordinal data: An application of automated essay scoring.Journal of the American Statistical Association. 1996;91:42-51

13.

McCullagh P.

Nelder J. A.

Generalized linear models. 2nd ed.Boca Raton, FL: Chapman and Hall; 1989:

14.

Neter J.

Kutner M. H.

Nachtsheim C. J.

Wasserman W.

Applied linear regression models. Homewood, IL: Irwin; 1996:

15.

Pratt J. W.

.Concavity of the log likelihood.Journal of the American Statistical Association. 1981;76:103-109

16.

Spitzer R. L.

Cohen J.

Fleiss J. L.

Endicott J.

.Quantification of agreement in psychiatric diagnosis.Archives of General Psychiatry. 1967;17:83-87