Bad Tests or Bad Test Use? A Case of SAT Use to Examine Why We Need Stakeholder Conversations on Validity

Abstract

Background

Much is still unknown or unclear about how and where validity issues arise in high stakes testing situations in education, and ways by which we can rectify validity problems in practice and policy contexts.

Purpose

This paper is the Foreword to the special issue of the Teachers College Record, When Education Measures Go Public – Stakeholder Perspectives on How and Why Validity Breaks Down.

Method

The paper analyzes a recent case involving an application of the SAT to highlight tensions between validity and test score use in high stakes school accountability environments driven by the No Child Left Behind (NCLB) Act of 2001. It uses the case study as a vehicle to introduce the individual papers and authors in the section.

Conclusions

There are information and power gaps among those who set societal priorities for using tests for high stakes purposes, those who design and conduct psychometric research on tests and testing programs, and those who could eventually face consequences of assessment misuse. These gaps could be addressed through thoughtful exchanges among key assessment stakeholders, as this special issue shows.

References

American Educational Research Association, American Psychological Association, & National Council Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Braun

(2005). Value-added modeling: What does due diligence require. In Lissitz

R. W.

(Ed.), Value added models in education: Theory and applications (pp. 19–39). Maple Grove, MN: JAM Press.

Brennan

R. L.

(2006). Perspective on the evolution and future of educational measurement. In Brennan

R. L.

(Ed.), Educational measurement (4th ed., pp. 1–16). Westport, CT: Praeger Publishers.

Brookhart

S. M.

(2011). Educational assessment knowledge and skills for teachers. Educational Measurement: Issues and Practice, 30(1), 3–12.

Cronbach

L. J.

(1971). Test validation. In Thorndike

R. L.

(Ed.), Educational measurement (2nd ed., pp. 443–507). Washington, DC: American Council on Education.

Cronbach

L. J.

, & Meehl

P. E.

(1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302.

Doran

H. C.

, & Cohen

(2005). The confounding effects of linking bias on gains estimated from value-added models. In Lissitz

R. W.

(Ed.), Value added models in education: Theory and applications (pp. 80–110). Maple Grove, MN: JAM Press.

Dorans

N. J.

(2002). The recentering of SAT^® scores and its effects on score distributions and score interpretations. College Board Research Report 2002-11.

Kane

M. T.

(2006). Validation. In Brennan

R. L.

(Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: Praeger Publishers.

10.

Lawrence

, Rigol

, Van Essen

, & Jackson

C. A.

(2002). A historical perspective on the SAT® 1926-2001 (pp. 1–13). Retrieved from http://professionals.collegeboard.com/profdownload/pdf/rr20027_11439.pdf

11.

Layton

, & Brown

(2012, September 24). SAT reading scores hit a four-decade low. The Washington Post. Retrieved from: http://www.washingtonpost.com/local/education/sat-reading-scores-hit-a-four-decade-low/2012/09/24/7ec9cb1e-0643-11e2-afff-d6c7f20a83bf_story.html

12.

Mazzeo

, Lazer

, & Zieky

M. J.

(2006). Monitoring educational progress with group-score assessments. In Brennan

R. L.

(Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: Praeger Publishers.

13.

Messick

(1989). Validity. In Linn

R. L.

(Ed.), Educational measurement. New York, NY: American Council on Education.

14.

Rothstein

(2002, August 28). Better at sums than at summarizing: The SAT^® gap. The New York Times. Retrieved from http://www.epi.org/publication/webfeat_lessons20020828/

15.

Sanders

W. L.

, & Rivers

J. C.

(1996). Cumulative and residual effects of teachers on future student academic achievement. Knoxville, TN: University of Tennessee Value-Added Research and Assessment Center. Retrieved from http://beteronderwijsnederland.net/files/cumulative%20and%20residual%20effects%20of%20teachers.pdf

16.

Schatz

C. J.

, VonSecker

C. E.

, & Alban

T. R.

(2005). Balancing accountability and improvement: Introducing value-added models to a large school system. In Lissitz

R. W.

(Ed.), Value added models in education: Theory and applications (pp. 1–18). Maple Grove, MN: JAM Press.

17.

Schmidt

W. H.

, Houang

R. T.

, & McKnight

C. C.

(2005). Value-added research: Right idea but wrong solution. In Lissitz

R. W.

(Ed.), Value added models in education: Theory and applications (pp. 145–165). Maple Grove, MN: JAM Press.

18.

Shaw

E. J.

, & McKenzie

(2010). The national SAT® validity study: Sharing results from recent college success research. Presented at the Annual Conference of the Southern Association for College Admission Counseling.

19.

Shepard

L. A.

(1991). Psychometricians’ beliefs about learning. Educational Researcher, 20(7), 2–16.