Research Pyramid: A New Evidence-Based Practice Model for Occupational Therapy

Background

Many authors have noted the limitations in the single-hierarchy EBP model and have called for revisions (Carpenter, 2004; Hammel, 2004; Hyde, 2001; Miller & Crabtree, 2000; Scheer et al., 2008; Tickle-Degnen & Bedell, 2003; White, 2009). No solution, however, has yet been forthcoming in the English language literature. The Research Pyramid of Borgetto et al. (2007), published in the German language journal Ergoscience, is the first attempt known to us to resolve the evidence tangle and provide a model that treats all evidence important for occupational therapy practice equitably.

The deconstruction of the single-hierarchy model undertaken by the Hildesheim group revealed that the multiple dimensions of rigor and applicability were oversimplified in the widely used model. Correctness of conclusion (internal validity) and applicability of findings (external validity, or generalizability) were often confounded by researchers compiling or interpreting findings from quantitative studies (Scheer et al., 2008; Tickle-Degnen & Bedell, 2003). Authors have taken the internal validity of a study as a sufficient condition for its generalizability, when it constitutes, at best, only a necessary condition.

The sentences imply that studies with higher internal validity will automatically have higher external validity. Actually, a separate evaluation of external validity is always in order. For example, a carefully controlled, double-blind experimental protocol enacted on a carefully screened group of volunteers, who are systematically provided with the most thorough application of a new intervention approach by the best-trained therapists available, in an optimal environment, with no concerns for cost or time, may indeed provide strong evidence for a cause-and-effect conclusion (i.e., the new intervention, as delivered, caused a superior outcome) with a recognized excellent internal validity. It is obvious that this study would be of uncertain generalizability to the real world (i.e., low external validity). Taking internal validity as a sufficient condition for generalizability is to assume that contextual factors of clinical settings and life circumstances do not matter—in direct contradiction to the assumptions of the classical experimental method, in which all such contextual factors must be strictly controlled because they can indeed affect the outcome of the study.

Alternatively, a large-scale outcome study of occupational therapy clients undergoing typical interventions might reveal that those receiving a certain approach usually had more favorable outcomes. Because the clients were not randomly assigned to intervention type, it would remain uncertain whether it was the difference in intervention that caused the difference in outcome. Preexisting differences among clients may have contributed, as may have different amounts of concurrent treatments received, varying cultural or environmental settings, unequal proportions of age or gender, or other factors. The conditions of the study are plausible (indicating good generalizability), and the findings may point toward the superiority of one approach over another (some internal validity), but our confidence in these findings must be tempered. Thus, there is a tendency among experimental and outcome studies for the amounts of internal and external validity to be inversely related. There also remains the inescapable uncertainty in all studies that rely on statistical analysis for their findings (Tomlin, 2008).

Qualitative research studies cannot be appraised according to the quantitative understandings of internal and external validity (Krefting, 1991). Instead, researchers speak of authenticity and transferability (roughly analogous to internal validity and generalizability, respectively), which are supported by many aspects of methodological rigor: participant observation, triangulation of data, auditable trail of analysis, member checking, and peer checking, among others (Krefting, 1991).

In part because of the nature of the questions pursued by qualitative studies, the absence of researcher manipulation in the design, and the reflexive analysis typically contained in each study, there tends to be a direct relationship between authenticity and transferability. That is, the nearer a qualitative study has come to providing a “thick description” of the lived experience at hand that illuminates (Geertz, 1973), the more that study’s findings can be transferred via theory to our general understanding of human experience and human nature (Borgetto et al., 2007).

Borgetto et al. (2007) sought to portray these two different relationships graphically. They established two orthogonal axes: quantitative–qualitative and internal validity–external validity (where these latter are understood as authenticity and transferability for qualitative studies). Basic methodology types (experimental, outcome, qualitative) occupy specific quadrants of the graph. If the authenticity and transferability of qualitative studies do indeed tend to be directly related, then the two qualitative quadrants collapse into one, forming a triangle.

Borgetto et al. (2007) then considered how different studies of the same methodology type have differing amounts of rigor and thus provide different amounts of confirmable evidence of intervention effectiveness and applicability. A hierarchy could be established on each side of the triangle, that is, a vertical dimension of rigor could be added, turning the triangle into a pyramid (Figure 1). For example, on the experimental methodology side, single subject studies (in which each participant is his or her own control) are placed at the bottom level. Controlled clinical trials would occupy the next tier, RCTs are placed on the next tier, and meta-analyses of RCTs are placed on the top tier (see “Experimental Research,” Items 1–4, in Figure 2).

OPEN IN VIEWER

Figure 1.

Research Pyramid.

OPEN IN VIEWER

Figure 2.

Research Pyramid levels of evidence.

On the outcome side of the pyramid, one-group pre–post studies occupy the lowest level, and case–control studies occupy the next. Preexisting group comparison studies with covariate analysis, where covariates are investigated to identify which secondary factors may have affected the outcome of the study, occupy the next level. Population-based studies seeking to uncover relative risk factors and their interactions (e.g., Classen, Awadzi, & Mkanta, 2008) would also occupy this level. The top tier would be occupied by meta-analyses of outcome studies (see “Outcome Research” in Figure 2).

On the qualitative pyramid side, Borgetto et al. (2007) surmised that the triangulation and reflexive analysis inherent in such studies meant that each study itself ranged from lower tier to upper tier standing. Alternatively, one could conceptualize the height of the pyramid sides as corresponding to increasing replicability. In that case, lower tier qualitative studies could be those original studies using an a posteriori approach to theme analysis. Those using an a priori approach, if that scheme of analysis were confirmed by the data, would occupy a higher tier. Another approach would be to judge each qualitative study by the combined effect of its use of techniques of rigor (whether counted or appraised holistically remains to be established). The best-fitting conceptualization for the qualitative face of the pyramid thus has yet to be determined. In any case, the top tier on the qualitative side of the pyramid would be occupied by studies that combine the results of multiple qualitative studies—called meta-syntheses (Thorne, Jensen, Kearney, Noblit, & Sandelowski, 2004; see “Qualitative Research” in Figure 2).

The current paper’s first author (Tomlin) favors assigning descriptive research (case studies, normative studies, basic descriptive reports) to the base of the pyramid, for reasons enumerated in the section on developmental methodology and epistemology. Borgetto et al. (2007) had placed descriptive research at the lowest tier of the outcome face of the pyramid. If the distinguishing feature of the outcome side of the pyramid is that its studies represent an attempt to go beyond mere description and the analysis of patterns in the data (through correlational, or associative tests) to the detection of plausible differences among groups via statistical tests, then the simpler types of descriptive research would fit well as the base of the pyramid (see “Descriptive Research” in Figure 2).

In either case, the three-sided pyramid model of research evidence for the profession represents a more comprehensive portrayal of the types of evidence (and knowledge) practitioners need to make good decisions about the services they provide to clients (Figure 2). The most powerful foundation for the profession of occupational therapy would consist of a solid platform at the top of the pyramid where the three sides converge. In terms of literature, this platform would be a meta-meta-analysis–synthesis of evidence from studies on the base and all three sides of the pyramid (a “mega-synthesis”). No such study has yet been published to our knowledge.

The theoretical advantages of the Research Pyramid over the single-hierarchy model now in common use are that (1) it acknowledges the distinctness of internal validity–authenticity and external validity–transferability; (2) it separates, and values at parity, outcome studies, many of which seek to answer real-world questions of professional practice that cannot be investigated through RCTs; and (3) it includes qualitative research at parity in EBP, a set of methodologies said to be particularly well-suited to addressing an essence of the professional concerns in occupational therapy: the lived experience of the client (Voigt-Radloff, 2007). The pyramid model also retains the hierarchical aspect of increasing rigor or dependability or replicability in studies, but only within methodology types, not across methodology types.

Some disadvantages must also be acknowledged. First, research questions and methodologies do not always align themselves neatly into the three sides of the pyramid. RCTs, with or without blinding, could be, and have been, carried out as outcome studies (Clark et al., 1997). Qualitative methodologies could be used to explore and provide evidence for causal relationships in the realm of human occupation (Hammel, 2004). Likewise, they could provide evidence for the outcomes of practice (Cohn, 2001; Spencer et al., 2002). It is even conceivable that a qualitative study could be conducted in which participants (clients) have been randomly assigned to be interacted with in a “manipulated” way by the data gatherers (practitioners) to determine whether that factor makes a difference in their “lived experience” of therapy.

Second, there does not seem to be a straightforward way to calibrate the tiers on the three sides of the pyramid to one another with respect to rigor. That is, can tiers of equal height on adjacent sides of the pyramid be said to be of equal rigor? Perhaps the question itself is not meaningful because the criteria for rigor are different on each side of the pyramid. Moreover, in the single-hierarchy model, an analogous problem of heterogeneity of criteria exists. Trombly, Tickle-Degnen, Baker, Murphy, and Ma (1999) devised a four-criteria level of evidence classification for quantitative studies that gives implicit recognition to the problem.

Finally, some studies incorporated both qualitative and quantitative methodologies (Spencer et al., 2002). They would appear on two (or more) sides of the pyramid. The single-hierarchy model is even less adept at placing such studies at an appropriate single level of evidence.

Application of the Model

Using the Research Pyramid model of evidence evaluation would lead to a different portrayal of evidence than in a traditional, single-hierarchy model critically appraised topic (CAT). Three examples follow.

First, in preparation for a graduate research project, Knackstedt and Taba investigated the published evidence for efficacy of occupational therapy interventions for clients with chronic pain. They found four studies of the experimental type and six outcome studies but no qualitative studies dealing with the lived experience of people with chronic pain. (Fisher, 2007, has since appeared in print.) Thus, Knackstedt designed and implemented a qualitative study in which she interviewed three volunteers who had a long-standing diagnosis of complex regional pain syndrome. A common point of testimony among the participants was the experience of being disbelieved and misunderstood by most of their health care providers (Knackstedt, 2007), a finding that makes an important contribution to the evidence of intervention efficacy provided by quantitative studies.

The CATs of Trombly and Ma (2002) and Ma and Trombly (2002) on treatments for people with a cerebrovascular accident purposely excluded qualitative studies. The remaining studies they analyzed were distributed across the tiers of the other two sides of the pyramid and provided evidence for the efficacy of occupational therapy interventions to address impairments and disabilities arising from the cerebrovascular accident. Absent was evidence of how clients experienced these interventions, for example, whether the timing of the services provided was optimal from their perspective in resolving life challenges as they arose.

The recent CAT of Case-Smith and Arbesman (2008) on efficacy of interventions for children with autism spectrum disorder (ASD) likewise deliberately did not include qualitative studies. They found and analyzed 49 studies, mostly of Level 2 and Level 3, that provided evidence of effectiveness for six types of interventions (sensory based, relationship based, developmental skill based, social cognitive skill training, parent-directed or -mediated approaches, and intensive behavioral intervention). They noted five limitations in the studies as a group: “lack of long-term evaluation of effects, use of focused evaluation instruments that did not measure children’s occupations or participation, lack of randomization, inadequate measures of treatment fidelity, and inappropriate data analysis” (Case-Smith & Arbesman, 2008, p. 427). They observed that individual studies often could not identify why the intervention worked for some children and their families and not for others.

Using the same search criteria over the same time period, but pointedly seeking only qualitative studies, another 49 studies were found on the experience of living with autism, or of interventions for autism, in people with ASD and their families. One example of how these findings could inform occupational therapists seeking evidence for help with intervention selection was Gray (2001). Through extensive interviews with family members of children with ASD in the vicinity of Brisbane, Queensland, Australia, Gray found three “family narratives” on the lived experience of having a child with this diagnosis, which he labeled accommodation, resistance, and transcendence. The narratives served to create “coherence out of the disordering effects of autism on family life” (Gray, 2001, p. 1247). For a therapist attempting to select an intervention approach for a given child and family that is effective in promoting meaningful participation, which of the three types of narrative the family had constructed for itself would be important.

The point of shifting from a single-hierarchy model of evidence evaluation to the pyramid model is not to claim that experimental studies are not important for occupational therapists. It is instead to assert that trustworthy evidence of different types can be discovered through disciplined inquiry, and all are important to the profession. The point is not to overthrow the “gold” standard for evaluating evidence and the notion of rigor along with it but rather to replace that quest with one for the best-balanced “alloy.” If any component of the alloy is missing, its strength will be deficient: Without experiments, the evidence may be misleading as to cause and effect in an intervention study or as to which intervention, all other conditions being equal, is likely to be the most effective in bringing about a desired outcome. Without outcome studies, the feasibility of realizing the promised treatment effects, their applicability to various groups in real, heterogeneous social contexts, and the financial sustainability of the intervention, may be unknown. Without qualitative studies, client perspective, satisfaction, the meaningfulness of participating in the treatment, and psychosocial side effects may remain unknown, thus imperiling the effectiveness of intervention.

Clinical Reasoning and the EBP Pyramid Model

An evidence hierarchy model, indeed EBP itself, should exist to guide researchers, research reviewers, and clinician consumers of research in sorting out evidence to comprehensively support the decision making needs of the profession. An evidence hierarchy for occupational therapy, therefore, should provide for rigorous means of marshalling evidence for the types of decisions occupational therapy practitioners must make on a daily basis. The Research Pyramid model more aptly aligns with the clinical reasoning demands of occupational therapy.

In the now classic model of clinical reasoning of Mattingly and Fleming (1994), three distinct types of reasoning needs were identified as arising in practice. They named them procedural, conditional, and interactive reasoning. Procedural reasoning “guides the therapist in thinking about the patient’s physical performance problems,” interactive reasoning “is used when the therapist wants to understand the patient as a person,” and conditional reasoning “is used to project an imagined future condition or situation for the person” (Fleming, 1991, p. 1007). Although other types of professional reasoning have since been identified (Schell & Schell, 2008), these three remain at the center of the therapist–client interaction. Perhaps it is not accidental that these three tracks of clinical reasoning, dealing as they do with occupation and therapy, person, and environment, respectively, align closely with the three sides of the Research Pyramid. Experimental research methodologies can indicate cause-and-effect relationships in the processes of evaluation and intervention, pointing to superior approaches to use in the procedures of therapy (procedural reasoning). Outcome studies, providing the results of therapeutic interventions as implemented in real-world situations, can inform practitioners as they project possible life futures for their clients (conditional reasoning). Interactive reasoning can be strengthened—and therapeutic use of self can be made more powerful—by the distinctive insights of qualitative research (Schwartzberg, 2002).

Evidence-Based Medicine and the Research Pyramid

In the revised EBM model of Sackett et al. (2000), evidence from the research literature is one of three sources of information for health care providers making clinical decisions. The other two are the client’s preferences and the practitioner’s experience. Fleming and Mattingly (1994) regarded conditional reasoning as largely influenced by the prior clinical experience of the therapist. Obviously, interactive reasoning by the therapist must be mindful of client preferences. Thus, the three sources of information of the Sackett et al. (2000) model align with the sides of the Research Pyramid.

Developmental Methodology and Epistemology

The transition from the conception of a single hierarchy of evidence to a pyramidal one raises the question, what does the third-dimensional height of the pyramid represent? One interpretation is afforded by the designation of descriptive research as the base of the pyramid. The pyramid would thus serve as a model of the evolution of research methodologies or of the increasing sophistication of professional knowledge itself and the criteria by which it becomes established.

Every practitioner can portray and convey professional experiences through description. What has been observed and what has been experienced in the care of clients can be documented and retold (e.g., case studies). Suppositions about trends, patterns, client perspectives, and cause-and-effect relationships between intervention and outcome can be made. Descriptive statistics can be enlisted to convey the information contained in the many clinical measurements and treatment encounters of a practitioner or group of practitioners. Individual inferences can be misleading, however, because fortuitous patterns can be taken as underlying rules in the wider world. For more accurate, reliable, generalizable, and transferable—and less misleading—conclusions to be drawn, the three categories of research methodology, as labeled in the Research Pyramid model, have evolved over the past few hundred years.

Qualitative research methodologies allow us to escape, albeit partially, from our limited individual perspectives through careful observation, interview, and reflexive analysis. Outcome research attempts to detect stable patterns statistically and to make fair comparisons using careful consideration of covariates between groups that were not created by random assignment. Experimental research, seeking to escape trial and error as a means of discovering cause and effect, has evolved the techniques of blinding, control of extraneous factors, application of the independent (intervention) variable and, of course, randomization. Each methodology type has evolved under the challenge of needing to answer the three fundamental types of questions in professional practice: the underlying cause-and-effect mechanisms of occupational dysfunctions and of our interventions; their outcome in a complex, ever-changing real world; and—what should be the ultimate point of practice—the actual lived experience of our clients.

Conclusions

The Research Pyramid model incorporates more essential types of evidence for practice than the single-hierarchy model—and at parity. It recognizes that certain research questions of crucial interest to the profession can only be answered by certain methods of research. There is no gold standard of research design for answering all questions of importance. Rather, an alloy of different types of evidence, melded together, would serve the decision-making needs of the profession best. The pyramid model aligns closely with the clinical reasoning model of the three-track mind in occupational therapy (Fleming, 1991) and with the revised formulation of evidence-based medicine of Sackett et al. (2000). It can better guide research in the field by revealing more comprehensively where there are evidence gaps. The model may spur “mega-synthesis CATs” in the future to help clinicians in their intervention decision making even more than current CATs do. The model suggests a coherent developmental evolution for research methodologies. The Research Pyramid model may also offer a coherent framework for teaching EBP, clinical reasoning, and research in occupational therapy education programs.