Longitudinal Stability of IRT and Equivalent-Groups Linear and Equipercentile Equating

Abstract

A multiscale criterion-referenced test that featured two presumably equivalent forms (A and B), was administered to 1,667 Head Start children at each of four points over an academic year. Using a randomly equivalent groups design, three equating methods were applied: common-item IRT equating using concurrent calibration, linear transformation, and equipercentile transformation. The methods were compared by examining mean score differences, weighted mean squared difference, and Kolmogorov's D statistics for each subscale. The results indicated that over time the IRT equating method and conventional equating methods exhibited different patterns of discrepancy between the two test forms. IRT equating yielded marginally smaller form-to-form mean score differences and generated slightly f ewer distributional discrepancies between Forms A and B than both linear and equipercentile equating. However, the results were mixed indicating that more studies are needed to provide additional information on the relative merits and weaknesses of each approach.

References

Allen

M. J.

Yen

W. M.

(1979) Introduction to measurement theory. Monterey, CA: Brooks/Cole.

Carrow-Woolfolk

(1995) Oral and Written Language Scales — Listening Comprehension Scale. Circle Pines, MN: American Guidance Service, Inc.

Chen

Revicki

D. A.

Lai

Cook

K. F.

Amtmann

(2009) Linking pain items from two studies onto a common scale using item response theory. Journal of Pain Symptom Management, 38(4), 615–628.

Conover

W. J.

(1999) Practical nonparametric statistics. (3rd ed.) New York: John Wiley.

Cook

L. L.

Douglas

J. B.

(1982). Analysis of fit and vertical equating with the three-parameter model. Paper presented at the American Educational Research Association Annual Meeting, New York, March.

Cook

L. L.

Eignor

D. R.

(1991) An NCME instructional module on IRT equating methods. Educational Measurement: Issues and Practice, 10(3), 37–45.

Crocker

Algina

(1986) Introduction to classical and modern test theory. Fort Worth, TX: Harcourt.

du Toit

(Ed.) (2003) IRT from SSI, BILOG-MG, MULTILOG, PARSCALE, TESTFACT. Lincolnwood, IL: Scientific Software International.

Dunn

L. M.

Dunn

L. M.

(1997) Peabody Picture Vocabulary Test-Third Edition Form A. Circle Pines, MN: American Guidance Service, Inc.

10.

Francis

D. J.

Santi

K. L.

Barr

Fletcher

J. M.

Varisco

Foorman

B. R.

(2008) Form effects on the estimation of students' oral reading fluency using DIBELS. Journal of School Psychology, 46, 315–342.

11.

Ginsburg

H. P.

Baroody

A. J.

(2003) Test of Early Mathematics Ability-Third Edition Form A. Austin, TX: PRO-ED.

12.

Green

B. F.

(1995) Comparability of scores from performance assessments. Educational Measurement: Issues and Practice, 14(4), 13–15.

13.

Hambleton

R. K.

Swaminathan

Rogers

H. J.

(1991) Fundamentals of item response theory. Newbury Park, CA: Sage.

14.

Han

Kolen

Pohlmann

(1997) A comparison among IRT true- and observed-score equating and traditional equipercentile equating. Applied Measurement in Education, 10, 105–121.

15.

Hanson

B. A.

Béguin

A. A.

(1999) Obtaining a common scale for IRT item parameters using separate versus concurrent estimation in the common item nonequivalent groups equating design. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Montreal, April.

16.

Harris

D. J.

Crouse

J. D.

(1993) A study of criteria used in equating. Applied Measurement in Education, 6(3), 195–240.

17.

Harris

D. J.

Kolen

M. J.

(1990) A comparison of two equipercentile equating methods for common item equating. Educational and Psychological Measurement, 50, 61–71.

18.

Hendrickson

A. B.

Kolen

M. J.

(2003) IRT equating of the MCAT. Association of American Medical Colleges, Washington, DC. Retrieved from https://www.aamc.org/students/linkableblob/85396–2/data/monograph4-data.pdf.

19.

High/Scope Educational Research Foundation. (2003) Preschool Child Observation Record. (2nd ed.) Ypsilanti, MI: High/Scope Press.

20.

Jodoin

M. G.

Keller

L. A.

Swaminathan

(2003) A comparison of linear, fixed common item, and concurrent parameter equating procedures in capturing academic growth. The Journal of Experimental Education, 71(3), 229–250.

21.

Kirkman

T. W.

(1996) Statistics to use. Retrieved from http://www.physics.csbsju.edu/stats/.

22.

Kolen

M. J.

(1981) Comparison of traditional and item response theory methods for equating tests. Journal of Educational Measurement, 18, 1–18.

23.

Kolen

M. J.

Brennan

R. L.

(2004) Test equating, scaling, and linking: methods and practices. (2nd ed.) New York: Springer-Verlag.

24.

Kolen

M. J.

Harris

D.J.

(1990) Comparison of item preequating and random groups equating using IRT and equipercentile methods. Journal of Educational Measurement, 27, 27–39.

25.

Kolen

M. J.

Whitney

D. R.

(1982) Comparisons of four procedures for equating the General Tests of Educational Development. Journal of Educational Measurement, 19, 279–293.

26.

Lee

Ban

(2010) A comparison of IRT linking procedures. Applied Measurement in Education, 23, 23–48.

27.

Livingston

S. A.

(2004) Equating test scores (without IRT). Retrieved from http://tigger.uic.edu/∼georgek/HomePage/EdMeasurement/livingston.pdf.

28.

Lord

F. M.

(1977) Practical applications of item characteristic curve theory. Journal of Educational Measurement, 14, 117–138.

29.

McDermott

P. A.

Fantuzzo

J. W.

Waterman

Angelo

L. E.

Warley

H. P.

Gadsden

V. L.

Zhang

(2009) Measuring preschool cognitive growth while it's still happening: the Learning Express. Journal of School Psychology, 47, 337–366.

30.

Montague

Penfield

R. D.

Enders

Huang

(2010) Curriculum-based measurement of math problem solving: a methodology and rationale for establishing equivalence of scores. Journal of School Psychology, 48(1), 39–52. DOI: 10.1016/j.jsp.2009.08.002.

31.

Petersen

N. S.

Cook

L. L.

Stocking

M. L.

(1983) IRT versus conventional equating methods: a comparative study of scale stability. Journal of Educational Statistics, 8, 137–156.

32.

Petersen

N. S.

Marco

G. L.

Stewart

E. E.

(1982) A test of the adequacy of linear score equating models. In Holland

P. W.

Rubin

D. R.

(Eds.), Test equating. New York, NY: Academic Press, Inc. Pp. 71–135.

33.

Public Law 107–110 (No Child Left Behind Act of 2001). (2002) Most recent set of amendments to the Elementary and Secondary Education Act of 1965. Retrieved from http://www.ed.gov.

34.

Reid

D. K.

Hresko

W. P.

Hammill

D. D.

(2001) Test of Early Reading Ability-Third Edition Form A. Austin, TX: PRO-ED.

35.

Skaggs

Lissitz

R. W.

(1986) IRT test equating: relevant issues and a review of recent research. Review of Educational Research, 56, 495–529.

36.

Thissen

Wainer

(2001) Test scoring. Mahwah, NJ: Erlbaum.

37.

Tong

Kolen

M. J.

(2005) Assessing equating results on different equating criteria. Applied Psychological Measurement, 29, 418–432.

38.

U.S. Department of Health and Human Services. (2001) Head Start FACES: longitudinal findings on program performance. Third progress report. Washington, DC: Administration for Children and Families.

39.

U.S. Department of Health and uman Services. (2003) Head Start National Reporting System – Direct Child Assessment Fall and Spring. Washington, DC: Administration for Children and Families, Administration on Children, Youth, and Families, & Head Start Bureau.

40.

U.S. Department Health and Human Services. (2010) Head Start Impact Study Technical Report. Washington, DC: Administration for Children and Families.

41.

van der Linden

W. J.

(2000) A test-theoretic approach to observed score equating. Psychometrika, 65, 437–456.

42.

van der Linden

W. J.

Luecht

R. M.

(1998) Observed-score equating as a test assembly problem. Psychometrika, 63, 401–418.

43.

Wood

Wilson

Gibbons

Schilling

Muraki

Bock

(2002) TESTFACT (Ver. 4.0). Lincolnway, IL: Scientific Software International. [Computer program].

44.

Zimowski

Muraki

Mislevy

Bock

(1999) BILOG-MG (Ver. 3.0). Lincolnway, IL: Scientific Software International. [Computer program].