Few prevention studies have examined the degree to which different measures of program implementation adherence predict youth outcomes. The current study was conducted with rural middle school youth participating in a longitudinal school-based preventive intervention program. Study participants’ average age at the pretest assessment was 12.3 years. The association between program implementation ratings supplied by provider self-reports and trained independent observer reports were evaluated. In addition, the relationship between measures of implementation and youth outcomes were examined. Results indicated that although program providers tended to report higher implementation than independent observers, most ratings were correlated significantly across raters. Observer-reported implementation ratings significantly predicted several youth substance-related outcomes, while provider-reported self-ratings did not.Program provider characteristics predicted several youth outcomes. Findings suggest that there might be a social desirability bias in provider self-reported ratings of implementation and that caution must be used when interpreting self-reported ratings of implementation.
References
1.
1. Botvin GJ, Baker E, Dusenbury L, Botvin EM, Diaz T: Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. JAMA273: 1106-1112, 1995.
2.
2. Botvin GJ, Baker E, Dusenbury L, Tortu S, Botvin E: Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3-year study. J Consult Clin Psychol58: 437-446, 1990.
3.
3. Domitrovich C, Greenberg, M: The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. J Educ Psychol Consult11: 193-221, 2000.
4.
4. Institute of Medicine: Reducing Risks for Mental Disorders: Frontiers for Preventive Intervention Research. Washington, DC, National Academy Press, 1994.
5.
5. Pentz MA, Trebow EA, Hansen WB, MacKinnon DP, Dwyer JH, Johnson CA, Flay BR, Daniels S, Cormack C: Effects of program implementation on adolescent drug use behavior: The Midwestern Prevention Project (MPP). Eval Rev14: 264-289, 1990.
6.
6. Peters RD, McMahon RJ: Preventing childhood disorders, substance abuse, and delinquency, in Peters RD, McMahon RJ (eds.): Banff International Behavioral Science Series, Vol. 3. Thousand Oaks, CA, Sage, 1996.
7.
7. Rohrbach LA, Graham JW, Hansen WB: Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Prev Med22: 237-260, 1993.
8.
8. Spoth R: Challenges in defining and developing the field of rural mental disorder preventive intervention research. Am J Comm Psychol25: 425-448, 1997.
9.
9. Durlak J: Why program implementation is important. J Prev Inter in Community17: 5-18, 1998.
10.
10. Greenberg M, Domitrovich C, Graczyk P, Zins J: The Study of Implementation in School-based Preventive Interventions: Theory, Research, & Practice. Report to Center for Mental Health Services (CMHS), Substance Abuse Mental Health Services Administration, U.S. Department of Health and Human Services, in press.
11.
11. Spoth R, Guyll M, Trudeau L, Lillehoj C: Two studies of proximal outcomes and implementation quality of universal preventive interventions in a community-university collaboration context: J Comm Psy30(5): 499-518, 2002.
13. Dane AV, Schneider BH: Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clin Psychol Rev18: 23-45, 1998.
14.
14. Botvin GJ, Griffin K, Diaz T, Ifill-Williams M: Drug abuse prevention among minority adolescents: Posttest and one-year follow-up of a school-based preventive intervention. Prev Sc2: 1-13, 2001.
15.
15. Fors S, Doster M: Implication of results: Factors for success. J Sch Health55: 332-334, 1985.
16.
16. Ross J, Luepker R, Nelson G, Saavedra P, Hubbard B: Teenage health teaching modules: Impact of teacher training on implementation and student outcomes. J Sch Health61: 31-34, 1991.
17.
17. Connell D, Turner R, Mason E: Summary of findings of the school health education evaluation: Health promotion effectiveness, implementation, and costs. J Sch Health55: 316-321, 1985.
18.
18. Flay BR: Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev Med15: 451-474, 1986.
19.
19. Parcel G, Perry C, Taylor W: Beyond demonstration: Diffusion of health promotion innovations, in Bracht N (ed.): Health Promotion at the Community Level. Newbury Park, CA, Sage, 1990, pp. 229-251.
20.
20. Detrich R: Increasing treatment fidelity by matching interventions to contextual variables within the educational setting. Sch Psychol Rev28: 608-620, 1999.
21.
21. Hall S, Loucks S: A developmental model for determining whether the treatment is actually implemented. Am Educ Res J14: 263-276, 1977.
22.
22. Heck S, Steigerbauer S, Hall GE, Loucks SF: Measuring Innovation Configurations: Procedures and Applications. Austin, Research and Development Center for Teacher Education, University of Texas, 1981.
23.
23. Basch C, Sliepcevish E, Gold R, Duncan D, Kolbe L: Avoiding Type III errors in health education program evaluations: A case study. Health Educ Q21: 315-331, 1985.
24.
24. Sobol D, Rohrbach L, Dent C, Gleason L, Brannon B, Johnson CA, Flay B: The integrity of smoking prevention curriculum delivery. Health Educ Res Theory & Practice4: 59-67, 1989.
25.
25. Scheirer MA, Rezmovic EL: Measuring the degree of program implementation: A methodological review. Eval Rev7: 599-633, 1983.
26.
26. Durlak JA: Successful Prevention Programs for Children and Adolescents. New York, Plenum, 1995.
27.
27. Young R, DeMoor C, Wildey M, Gully S, Hovell M, Elder J: Correlates of health facilitator performance in a tobacco use prevention program: Implications for recruitment. J Sch Health60: 463-467, 1990.
28.
28. Botvin GJ: Life Skills Training: Promoting Heath and Personal Development. Princeton, NJ, Princeton Health Press, 1986.
29.
29. Botvin GJ: Life Skills Training: Promoting Heath and Personal Development. Princeton, NJ, Princeton Health Press, 2000.
30.
30. Bandura A: Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ, Prentice Hall, 1977.
31.
31. Jessor R, Jessor S: Problem Behavior and Psychosocial Development: A Longitudinal Study of Youth. New York, NY, Academic Press, 1977.
32.
32. Molgaard V, Kumpfer K, Fleming B: The Strengthening Families Program: For Parents and Youth 10-14. Ames, Iowa State University Extension, 1997.
33.
33. DeMarsh J, Kumpfer KL: Family-oriented interventions for the prevention of chemical dependency in children and adolescents. Prevention18: 117-151, 1986.
34.
34. Kumpfer KL, Molgaard V, Spoth R. The Strengthening Families Program for the Prevention of Delinquency and Drug Use, in Peters R, McMahon RJ (eds.): Preventing Childhood Disorders, Substance Abuse, and Delinquency. Banff International Behavioral Science Series, Thousand Oaks, CA, Sage, 1996, pp. 241-267.
35.
35. Molgaard V, Spoth R, Redmond C: Competency Training: The Strengthening Families Program for Parents and Youth 10-14. OJJDP Juvenile Justice Bulletin (NCJ 182208). Washington, DC, U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention, 2000.
36.
36. Tortu S, Botvin GJ: School-based smoking prevention: The teacher training process. Prev Med18: 280-289, 1989.
37.
37. Blakely C, Mayer J, Gottschalk R, Schmitt N, Davidson W, Roitman D, Emshoff J: The fidelity-adaptation debate: Implications for the implementation of public sector social programs. Am J Community Psychol253: 253-269, 1987.
38.
38. Adalbjarnardottir S, Selman R: “I feel I have received a new vision:” An analysis of teachers’ professional development as they work with students on interpersonal issues. Teaching Teacher Educ13: 409-428.
39.
39. Fors S, Doster M: Implication of results: Factors for success. J School Health55: 332-334, 1985.
