This study used eight methods to objectively score 6-minute writing samples from students in compensatory and specialized educational programs. Each sample was also evaluated subjectively by four trained judges, using a holistic judgment of communicative effectiveness. Direct counts were made of the number of words: (a) written, (b) written legibly, (c) spelled correctly, (d) written in correct sequence, and (e) written in continuous correct sequences (averaged over sequences). Secondary calculations produced the percentage of words: (a) written legibly, (b) spelled correctly, and (c) written in correct sequence. In both a cluster analysis and factor analysis of the eight measures, two factors were identified, with the first four measures clustering together (representing a production-dependent factor) and the latter four measures clustering together (reflecting a production-independent factor). Regression of holistic ratings on objective scores produced moderately strong results for two production-independent indices—percentage of words correctly spelled and percentage of words correctly sequenced. The production-independent factor scores were much stronger predictors of holistic ratings than were production factor scores, but were weaker than each of the two strongest individual predictors.
References
1.
Charney, D. (1984). The validity of using holistic scoring to evaluate writing: A critical overview. Research in the Teaching of English, 18, 65–81.
2.
Chase, C.I. (1966). The impact of some obvious variables on the essay test scores. Journal of Educational Measurement, 5, 315–318.
3.
Cooper, C.R., & Odell, L. (1977). Evaluating writing: Describing, measuring, judging.Urbana, IL: National Council of Teachers of English.
4.
CTB/McGraw-Hill. (1985). California achievement tests, Form E.Monterey, CA: CTB/McGraw-Hill.
5.
Deno, S., Marston, D., & Mirkin, P. (1982). Valid measurement procedures for continuous evaluation of written expression. Exceptional Children, 48, 368–371.
6.
Diederich, P.B. (1974). Measuring growth in English.Urbana, IL: National Council of Teachers of English.
Grobe, C. (1981). Syntactic maturity, mechanics, and vocabulary as predictors of quality ratings. Research in the Teaching of English, 15, 75–85.
9.
Hasbrouck, J. (1987). Objective and holistic scoring of writing: A manual for collecting and scoring writing samples using holistic and 8 objective scoring procedures (Module No. 2). Eugene: University of Oregon, Resource Consultant Training Program.
10.
Hiller, J., Marcotte, D., & Martin, T. (1969). Opinionation, vagueness, and specificity distinctions: Essay traits measured by computer. American Educational Research Journal, 13, 175–185.
11.
Hunt, K. (1970). Syntactic maturity in school children and adults. Monographs of the Society for Research in Child Development (Serial No. 134), 35, 1–44.
12.
Isaacson, S. (1985). Assessing written language. In C.S. Simon (Ed.), Communication skills and classroom success: Assessment methodologies for language-learning disabled students (pp. 403-424). San Diego: College-Hill Press.
13.
Klein, S., & Hart, F.M. (1968). Chance and systematic factors affecting essay grades. Journal of Educational Measurement, 5, 197–206.
14.
Marshall, J.C., & Powers, J.C. (1969). Writing neatness, composition errors, and essay grades. Journal of Educational Measurement, 6, 97–101.
15.
Marston, D., & Deno, S. (1981). The reliability of simple, direct measures of written expression (Research Report No. 50). Minneapolis: University of Minnesota, Institute for Research on Learning Disabilities.
16.
Marston, D., Lowry, L., Deno, S., & Mirkin, P. (1981). An analysis of learning trends in simple measures of reading, spelling, and written expression: A longitudinal study (Research Report No. 49). Minneapolis: University of Minnesota Institute for Research on Learning Disabilities.
17.
McColly, W. (1970). What does educational research say about judging of writing ability?Journal of Educational Research, 64, 147–156.
18.
McColly, W., & Remstad, R. (1965). Composition rating scales for general merit: An experimental evaluation. Journal of Educational Research, 59, 55–56.
19.
Moran, M.R. (1987). Options for written language assessment. Focus on Exceptional Children, 19, 1–10.
20.
Moss, P.A., Cole, N.S, & Khampalikit, C. (1982). A comparison of procedures to assess written language skills at grades 4, 7, and 10. Journal of Educational Measurement, 19, 37–47.
21.
Mullis, I. (1980). Using the primary trait system for evaluating writing (Research Report No. 10-W-51). Princeton, NJ: Educational Testing Service.
22.
Mullis, I.V.S. (1984). Scoring direct writing assessments: What are the alternatives. Educational Measurement: Issues and Practice, 3, 16–18.
23.
Nold, E., & Freedman, S. (1977). An analysis of readers' responses to essays. Research in the Teaching of English, 11, 164–174.
24.
Page, E. (1968). Analyzing student essays by computer. International Review of Education, 14, 210–225.
25.
Phelps-Gunn, T., & Phelps-Terasaki, D. (1982). Written language instruction: Theory and remediation.Rockville, MD: Aspen.
26.
Polloway, E.A., Patton, J.R, & Cohen, S.B. (1981). Written language for mildly handicapped students. Focus on Exceptional Children, 14, 1–16.
27.
Poteet, J. (1980). Informal assessment of written expression. Learning Disability Quarterly, 3, 88–98.
28.
Remondino, C. (1959). A factorial analysis of the evaluation of scholastic compositions in the mother tongue. British Journal of Educational Psychology, 30, 242–251.
29.
Schenck, S.J. (1981). The diagnostic/instructional link in individualized education programs. The Journal of Special Education, 14, 337–345.
30.
Slotnick, H., & Knapp, J. (1971). Essay grading by computer: A laboratory phenomena?Educational Measurement, 9, 253–263.
31.
Smith, L.S. (1980). Measures of high school students' expository writing: Direct and indirect strategies (CSE Report No. 133). Los Angeles: University of California, Center for the Study of Evaluation.
32.
Spandel, V. (1981). Classroom applications of writing assessment: A teacher's handbook.Portland, OR: Northwest Regional Educational Laboratory.
33.
Spandel, V., & Stiggins, R. (1981). Direct measures of writing skills: Issues and applications.Portland, OR: Northwest Regional Educational Laboratory, Clearinghouse for Applied Performance Testing (CAPT).
34.
Stewart, M.F., & Grobe, C.H. (1979). Syntactic maturity, mechanics of writing, and teachers' quality ratings. Research in the Teaching of English, 13, 207–215.
35.
Stiggins, R. (1982). A comparison of direct and indirect writing assessment methods. Research in the Teaching of English, 16, 101–114.
36.
Veal, L.R., & Hudson, S.A. (1983). Direct and indirect measures for large scale evaluation of writing. Research in the Teaching of English, 17, 290–296.
37.
Videen, J., Deno, S., & Marston, D. (1982). Correct word sequences: A valid indicator of proficiency in written expression (Research Report No. 84). University of Minnesota: Institute for Research on Learning Disabilities.
