This study explored some practical issues for single-case researchers who rely on visual analysis of graphed data, but who also may consider supplemental use of promising statistical analysis techniques. The study sought to answer three major questions: (a) What is a typical range of effect sizes from these analytic techniques for data from “effective interventions”? (b) How closely do results from these same analytic techniques concur with visual-analysis-based judgments of effective interventions? and (c) What role does autocorrelation play in interpretation of these analytic results? To answer these questions, five analytic techniques were compared with the judgments of 45 doctoral students and faculty, who rated intervention effectiveness from visual analysis of 35 fabricated AB design graphs. Implications for researchers and practitioners using single-case designs are discussed.
Allison, D. B. (1992). When cyclicity is a concern: A caveat regarding phase change criteria in single-case designs. Comprehensive Mental Health Care, 2, 131-149.
2.
Allison, D. B., & Gorman, B. S. (1993). Calculating effect sizes for meta-analysis: The case of the single case. Behavior, Research, and Therapy, 31, 621-631.
3.
Baer, D. M. (1977). Perhaps it would be better not to know everything. Journal of AppliedBehavior Analysis, 10, 167-172.
4.
Ballard, K. D. (1983). The visual analysis of time series data: Issues affecting the assessment of behavioral interventions. New Zealand Journal of Psychology, 12, 69-73.
5.
Barlow, D. H., & Hersen, M. (Eds.). (1984). Single case experimental designs: Strategies for studying behavior change (2nd ed.). Oxford, England: Pergamon.
6.
Berry, W. D., & Lewis-Beck, M. S. (1986). Interrupted time series. In W. D. Berry& M. S. Lewis-Beck (Eds.), New tools for social scientists: Advances and applications in research methods (pp. 209-240). Beverly Hills, CA: Sage.
7.
Box, G. E. P., & Jenkins, G. M. (1970). Time-series analysis:Forecastingandcontrol. New York: Cambridge University Press.
8.
Busk, P. L., & Marascuilo, L. A. (1988). Autocorrelation in single-subject research: A counterargument to the myth of no autocorrelation. Behavioral Assessment, 10, 229-242.
9.
Busk, P. L., & Marascuilo, L. A. (1992). Statistical analysis in single-case research: Issues, procedures, and recommendations, with applications to multiple behaviors. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis: New directions for psychology and education (pp. 159-185). Hillsdale, NJ: Lawrence Erlbaum.
10.
Busk, P. L., & Serlin, R. C. (1992). Meta-analysisforsingle-caseresearch.In T. R. Kratochwill& J. R. Levin (Eds.), Single-case research design and analysis: New directions for psychology and education (pp. 187-212). Hillsdale, NJ: Lawrence Erlbaum.
11.
Carver, R. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.
12.
Center, B. A., Skiba, R. J., & Casey, A. (1985-1986). A methodology for the quantitative synthesis of intra-subject design research. Journal of Special Education, 19, 387-400.
13.
Cochrane, D., & Orcutt, G. H. (1949). Application of least-squares regression to relationships containing autocorrelated error terms. Journal of the American Statistical Association, 44, 32-61.
14.
Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426-443.
15.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
16.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
17.
Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
18.
Crosbie, J. (1987). The inability of the binomial test to control Type I error with single-subject data. Behavioral Assessment, 9, 141-150.
19.
Crosbie, J. (1993). Interrupted time-series analysis with brief single-subject data. Journal of Consulting and Clinical Psychology, 61, 966-974.
20.
Crosbie, J. (1995). Interrupted time-series analysis with short series: Why it is problematic; how it can be improved. In J. M. Gottman (Ed.), The analysis of change (pp. 361-395). Mahwah, NJ: Lawrence Erlbaum.
21.
Dar, R., Serlin, R. C., & Omer, H. (1994). Misuse of statistical tests in three decades of psychotherapy research. Journal of Consulting and Clinical Psychology, 62, 75-82.
22.
Darlington, R. B., & Carlson, P. M. (1987). Behavioral statistics: Logic & methods. New York: Free Press.
23.
DeProspero, A., & Cohen, S. (1979). Inconsistent visual analyses of intrasubject data. Journal of Applied Behavior Analysis, 12, 573-579.
24.
Faith, M. S., Allison, D. B., & Gorman, B. S. (1996). Meta-analysis of single-case research. In R. D. Franklin, D. B. Allison, & B. S. Gorman (Eds.), Design and analysis of single-case research (pp. 245-277). Mahwah, NJ: Lawrence Erlbaum.
25.
Fox, J. (1991). Regression diagnostics. Newbury Park, CA: Sage.
26.
Franklin, R. D., Allison, D. B., & Gorman, B. S. (Eds.). (1996). Design and analysis of single-case research. Mahwah, NJ: Lawrence Erlbaum.
27.
Franklin, R. D., Gorman, B. S., Beasley, T. M., & Allison, D. B. (1996). Graphical display and visual analysis. In R. D. Franklin, D. B. Allison, & B. S. Gorman (Eds.), Design and analysis of single-case research (pp. 119-158). Mahwah, NJ: Lawrence Erlbaum.
28.
Furlong, M. J., & Wampold, B. E. (1982). Intervention effects and relative variation as dimensions in experts’use of visual inference. Journal of Applied Behavior Analysis, 15, 415-421.
29.
Gibson, G., & Ottenbacker, K. (1988). Characteristics influencing the visual analysis of single-subject data: An empirical analysis. Journal of Applied Behavioral Science, 24, 298-314.
30.
Glass, G. V., McGaw, B., & Smith, W. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
31.
Glass, G. V., Willson, V. L., & Gottman, J. M. (1975). Design and analysis of time-series experiments. Boulder: Colorado Associated University Press.
32.
Gorman, B. S., & Allison, D. B. (1996). Statistical alternatives for single-case designs. In R. D. Franklin, D. B. Allison, & B. S. Gorman (Eds.), Design and analysis of single-case research (pp. 159-214). Mahwah, NJ: Lawrence Erlbaum.
33.
Gorsuch, R. L. (1983). Three methodsfor analyzingtime-series (N of 1) data. Behavioral Assessment, 5, 141-154.
34.
Gottman, J. M., & Glass, G. V. (1978). Analysis of interrupted time-series experiments. In T. R. Kratochwill (Ed.), Single subject research: Strategies for evaluating change (pp. 197-235). New York: Academic Press.
35.
Greenspan, P., & Fisch, G. S. (1992). Visual inspection of data: A statistical analysis of behavior. Paperpresentedat the annual meeting of the American Statistical Association, Boston, MA.
36.
Greenwald, A. G. (1976). Within-subjects designs: To use or not to use?Psychological Bulletin, 8, 314-320.
37.
Harbst, K. B., Ottenbacher, K. J., & Harris, S. R. (1991). Interrater reliability of therapists’judgments of graphed data. Physical Therapy, 71, 107-115.
38.
Hartmann, D. P., Gottman, J. M., Jones, R. R., Gardner, W., Kazdin, A. E., & Vaught, R. S. (1980). Interrupted time-series analysis and its application to behavioral data. Journal of Applied Behavior Analysis, 13, 543-559.
39.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
40.
Hintze, J. (2000). NCSS 2000[Computersoftware]. Kaysville, UT: NCSS Statistical Software.
41.
Hojem, M. A., & Ottenbacker, K. J. (1988). Empirical investigation of visual-inspection versus trend-line analysis of single-subject data. Journal of the American Physical Therapy Association, 68, 983-988.
42.
Holtzman, W. H. (1963). Statistical methods for the study of change in the single case. In C. W. Harris (Ed.), Problems in measuring change (pp. 199-211). Madison: University of Wisconsin Press.
43.
Horne, G. P., Yang, M. C. K., & Ware, W. B. (1982). Time series analysis for single-subject designs. Psychological Bulletin, 91, 178-189.
44.
Huitema, B. E. (1985). Autocorrelationin appliedbehavioranalysis: A myth. Behavioral Assessment, 7, 107-118.
45.
Huitema, B. E. (1986). Autocorrelation in behavioral research: Wherefore art thou? In A. Poling & R. W. Fuqua (Eds.), Research methods in applied behavior analysis: Issues and advances (pp. 187-208). New York: Plenum.
46.
Huitema, B. E., & McKean, J. W. (1991). Autocorrelation estimation and inference with small samples. Psychological Bulletin, 110, 291-304.
47.
Huitema, B. E., & McKean, J. W. (1998). Irrelevant autocorrelation in least-squares intervention models. Psychological Methods, 3, 104-116.
48.
Jaccard, J., & Wan, C. K. (1993). Statistical analysis of temporal data with many observations: Issues for behavioral medicine data. Annals of Behavioral Medicine, 15, 41-50.
49.
Jacobson, N. S., Follette, W. C., & Revenstorf, D. (1984). Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behavior Therapy, 15, 336-352.
50.
Jones, R. R., Vaught, R. S., & Weinrott, M. (1977). Time-series analysis in operant research. Journal of Applied Behavior Analysis, 10, 151-166.
51.
Jones, R. R., Weinrott, M. R., & Vaught, R. S. (1978). Effects of serial dependency on the agreement betweenvisual and statistical inference. Journalof AppliedBehaviorAnalysis, 11, 277-283.
52.
Kazdin, A. (1984). Statistical analysis for single-case experiments designs. In D. H. Barlow & M. Hersen (Eds.), Single case experimental designs (2nd ed., pp. 285-324). New York: Pergamon.
53.
Kazdin, A. E. (1982). Single-case research designs: Methods for clinical and applied settings. New York: Oxford University Press.
54.
Keppel, G. (1982). Design and analysis: A researcher’s handbook (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.
55.
Kirk, R. E. (1996). Practical significance: A conceptwhose time has come. EducationalandPsychological Measurement, 56, 746-759.
56.
Knapp, T. J. (1983). Behavior analysts’ visual appraisal of behavior change in graphic display. Behavioral Assessment, 5, 155-164.
57.
Kratochwill, T. R., & Brody, G. H. (1978). Single subject designs: A perspective on the controversy over employing statistical inference and implications for research and training in behavior modification. Behavior Modification, 2, 291-307.
58.
Kratochwill, T. R., & Levin, J. R. (Eds.). (1992). Single-case research design and analysis: New directions for psychology and education. Hillsdale, NJ: Lawrence Erlbaum.
59.
Kromrey, J. D., & Foster-Johnson, L. (1996). Determining the efficacy of intervention: The use of effect sizes for data analysis in single-subject research. The Journal of Experimental Education, 65, 73-93.
60.
Kupfersmid, J. (1988). Improving what is published: A model in search on an editor. American Psychologist, 43, 635-642.
61.
Matyas, T. A., & Greenwood, K. M. (1990). Visual analysis of single-case time series: Effects of variability, serial dependence, and magnitude of intervention effects. Journal of Applied Behavior Analysis, 23, 341-351.
62.
Matyas, T. A., & Greenwood, K. M. (1996). Serial dependency in single-case time series. In R. D. Franklin, D. B. Allison, & B. S. Gorman (Eds.), Design and analysis of single-case research (pp. 215-243). Mahwah, NJ: Lawrence Erlbaum.
63.
Maxwell, S. E., Camp, C. J., & Avery, R. D. (1981). Measures of strength of association: A comparative examination. Journal of Applied Psychology, 66, 525-534.
64.
Maxwell, S. E., & Delaney, H. D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
65.
Michael, J. L. (1974). Statistical inference for individual organism research: Mixed blessing or curse?Journal of Applied Behavior Analysis, 7, 647-653.
66.
Mitchell, C., & Hartmann, D. P. (1981). A cautionary note on the use of Omega squared to evaluate the effectiveness of behavioral treatments. Behavioral Assessment, 3, 93-100.
67.
Nourbakhsh, M. R., & Ottenbacher, K. J. (1994). The statistical analysis of single-subjectdata: A comparative examination. Physical Therapy, 74, 80-88.
68.
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
69.
Ostrom, C. W., Jr. (1978). Time series analysis:Regression techniques. BeverlyHills, CA: Sage.
70.
Ostrom, C. W., Jr. (1990). Time series analysis: Regression techniques (2nd ed.). Newbury Park, CA: Sage.
71.
Ottenbacher, K. J. (1986a). An analysis of serial dependency in occupational therapy research. Occupational Therapy Journal of Research, 6, 211-216.
72.
Ottenbacher, K. J. (1986b). Reliability and accuracy of visually analyzing graphed data from single-subject designs. American Journal of Occupational Therapy, 40, 464-469.
73.
Ottenbacher, K. J. (1990a). Visual inspection of single-subject data: An empirical analysis. Mental Retardation, 28, 283-290.
74.
Ottenbacher, K. J. (1990b). When is a picture worth a thousand p values? A comparison of visual and quantitative methods to analyze single subject data. Journal of Special Education, 23, 436-449.
75.
Park, H., Marascuilo, L., & Gaylord-Ross, R. (1990). Visual inspection and statistical analysis of single-case designs. Journal of Experimental Education, 58, 311-320.
76.
Parsonson, B. S., & Baer, D. M. (1978). The analysis and presentation of graphic data. In T. R. Kratochwill (Ed.), Single-subject research: Strategies for evaluating change (pp. 101-165). New York: Academic Press.
77.
Parsonson, B. S., & Baer, D. M. (1986). The graphic analysis of data. In A. Poling & R. W. Fuqua (Eds.), Research methods in applied behavior analysis: Issues and advances (pp. 157-186). New York: Plenum.
78.
Parsonson, B. S., & Baer, D. M. (1992). The visual analysis of data, and current research into the stimuli controlling it. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis (pp. 15-40). Hillsdale, NJ: Lawrence Erlbaum.
79.
Prentice, D. A., & Miller, D. T. (1992). When small effects are impressive. Psychological Bulletin, 112, 160-164.
80.
Rojahn, J., & Schulze, H. (1985). The linear regression line as a judgmental aid in the visual analysis of serially dependent A-B time-series data. Journal of Psychopathology and Behavioral Assessment, 7, 191-206.
81.
Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage.
82.
Rosenthal, R. (1991). Meta-analytic procedures for social research (Rev. ed., Vol. 6). Newbury Park, CA: Sage.
83.
Rosnow, R., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
84.
Scheff, H. (1959). The analysis of variance. New York: John Wiley.
85.
Sharpley, C. F., & Alavosius, M. P. (1988). Autocorrelation in behavioral data: An alternative perspective. Behavioral Assessment, 10, 243-251. (The autocorrelation debate)
86.
Shaver, J. P. (1991). Quantitative reviewing of research. In J. P. Shaver (Ed.), Handbook of research on social studies teaching and learning (pp. 83-97). New York: Macmillan.
87.
Skiba, R., Deno, S., Marston, D., & Casey, A. (1989). Influence of trend estimation and subject familiarityonpractitionersjudgementsofinterventioneffectiveness. Journalof SpecialEducation, 22, 433-446.
88.
Suen, H. K., & Ary, D. (1987). Autocorrelation in applied behavior analysis: Myth or reality?Behavioral Assessment, 9, 125-130.
89.
Suen, H. K., & Ary, D. (1989). Analyzing quantitative behavioral observation data. Hillsdale, NJ: Lawrence Erlbaum.
90.
Toothaker, L. E., Banz, M., Noble, C., Camp, J., & Davis, D. (1983). N = 1 designs: The failure of ANOVA-based tests. Journal of Educational Statistics, 8, 289-309.
91.
Wampold, B. E., & Furlong, M. J. (1981). The heuristics of visual inference. Behavioral Assessment, 3, 79-92.
92.
White, D. M., Rusch, F. R., Kazdin, A. E., & Hartmann, D. P. (1989). Applications of meta-analysis in individual subject research. Behavioral Assessment, 11, 281-296.
93.
White, O. R., & Haring, N. G. (1980). Exceptionalteaching (2nded.). Columbus, OH: Merrill.
94.
Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis (Vol. 59). Beverly Hills, CA: Sage.
