Abstract
Although many researchers have discussed replication as a means to facilitate self-correcting science, in this article, we identify meta-analyses and evaluating the validity of correlational and causal inferences as additional processes crucial to self-correction. We argue that researchers have a duty to describe sampling decisions they make; without such descriptions, self-correction becomes difficult, if not impossible. We developed the Replicability and Meta-Analytic Suitability Inventory (RAMSI) to evaluate the descriptive adequacy of a sample of studies taken from current psychological literature. Authors described only about 30% of the sampling decisions necessary for self-correcting science. We suggest that a modified RAMSI can be used by authors to guide their written reports and by reviewers to inform editorial recommendations. Finally, we claim that when researchers do not describe their sampling decisions, both readers and reviewers may assume that those decisions do not matter to the outcome of the study, do not affect inferences made from the research findings, do not inhibit inclusion in meta-analyses, and do not inhibit replicability of the study. If these assumptions are in error, as they often are, and the neglected decisions are relevant, then the neglect may create a good deal of mischief in the field.
Keywords
We should value replication more than we do, treasure it even. We were all routinely taught the value of replication in our first research methods course, but it seems some have forgotten the lesson. (Roediger, 2012, para. 6)
Many authors have made general recommendations for increased sharing of study details as a means of addressing problems of direct 1 (Schmidt, 2009; Sidman, 1960), systematic 2 (Sidman, 1960), and conceptual 3 (Schmidt, 2009) replication (e.g., Asendorpf et al., 2013; Eich, 2014; Grant et al., 2013; LeBel et al., 2013; LeBel & Peters, 2011). We claim these calls do not go far enough. Our proposal builds on previous suggestions in several ways. First, we outline the information necessary for adequate correlational and causal inferences, replication, and inclusion in meta-analyses from a view grounded in Mill’s (1843) Canons—constrained in terms of what (a) a researcher can realistically attend to or be in control of and (b) has been shown to influence the outcomes of several studies. Second, we place the responsibility for reporting of sampling decisions on authors and journals equally and assert these descriptions should be available with published studies. Third, we address how researcher failures to describe sampling decisions may damage several aspects of scientific self-correction beyond replication, including meta-analysis and inferential validity of a study. Finally, we document that a range of psychological journals underdescribe many sampling decisions, not just purely methodological ones, and we argue that they matter.
That science self-corrects is a truism both old and new (e.g., Bordens & Abbott, 2008; Cohen & Nagel, 1934; Goodwin & Goodwin, 2013). Scientists count this self-correcting “way of knowing” as unique because its strong empirical ties allow reproducibility and, through that, self-correction. Among the old, Cohen and Nagel (1934) claimed, “Other methods . . . are all inflexible, that is, none . . . can admit that it will lead us into error. Hence, none of them can make provision for correcting its own results” (p. 195). Among the new, Lilienfeld, Lynn, Namy, and Woolf (2009) named “lack of self-correction” as the second deadly sin of pseudoscience (pp. 45–46). Replication is a major mechanism for self-correction, and without a willingness to share complete description of our studies with others, “the scientific enterprise grinds to a screeching halt, because research progress hinges on the ability to evaluate other investigators’ findings objectively” (Lilienfeld et al., 2009, p. 29).
Recent special sections and issues of journals reflect a growing apprehension over the role of direct, systematic, and conceptual replications, as well as meta-analyses, suggesting that science may be self-correcting more in theory than in practice. 4 Ioannidis (2005) has been at the forefront of researchers identifying factors interfering with self-correction. He has claimed that journal editors selectively publish positive findings and discriminate against study replications, permitting errors in data and theory to enjoy a long half-life (see also Ferguson & Brannick, 2012; Ioannidis, 2008, 2012; Shadish, Doherty, & Montgomery, 1989; Stroebe & Strack, 2014). We contend there are other equally important, yet relatively unexplored, problems.
One neglected problem is a failure of original authors to report sampling decisions adequately. This creates a three-fold problem for self-correcting science: (a) it forces replicating researchers to guess what sampling decisions an original researcher made; (b) it leaves meta-analytic researchers unable to include all eligible studies, especially when testing potential moderators; and (c) it leaves replicating researchers, meta-analysts, editors, and readers unable to accurately assess the validity of inferences made by the researcher of a study. To understand why this is, we need to understand what exactly sampling decisions are and how they can change inferences made.
What Are Sampling Decisions?
Although researchers in the field often think that a sampling decision, taking a small sample from a larger population, refers to the number and type of participants chosen, the majority of decisions that researchers make involve sampling decisions. A researcher must choose samples (often nonrandom samples) from several distinct populations: most obviously, the populations of participants, independent (predictor) variables, and dependent (outcome) variables, as well as measures of them. Asendorpf et al. (2013) pointed out that sampling decisions extend to experimental situations and time points relevant to a study’s design. Even more broadly, researchers must also sample from populations of experimenters, assessors, physical settings, measures, and available statistical analyses. We use the phrase sampling decisions 5 to refer to all of these sampling choices.
The same psychological judgment biases that plague humans in general may taint any sampling decision (Fiedler, 2011). As important, although many studies use large N samples for the participants, the remaining samples are small N (often N = 1). It is common to see studies run by one (undescribed) experimenter, under one (partially described) set of experimental settings, using a few operationalizations of an independent/predictor or dependent/outcome variable. The small Ns involved in these samples make it increasingly likely that the study will produce an extreme outcome—an error (Wainer, 2007). Coupled with journals’ tendencies to publish “hot” new results, biases in the literature become even more probable.
The Role of Sampling Decisions and Inferential Validity
In his classic, System of Logic, John Stuart Mill (1843) gave us a set of Canons allowing us to infer that a specific event caused or is related to another event with some confidence. Mill proposed these inductive principles as a way to regulate (and regularize) scientific inquiry.
We have included Table 1, which illustrates three of these principles, to remind us that the logic of our correlational and experimental designs rests on Mill’s (1843) Canons. A failure to describe the sampling decisions used to instantiate the logic of a Canon limits or eliminates the ability to assess the inferential validity of a researcher’s interpretation of a study’s outcome, to conduct replications of any kind, and to include eligible studies in meta-analyses. Consequently, this jeopardizes both self-correction and the entire inferential enterprise of the science.
Mill’s (1843) Canons
Note: This table contains a summary of the logical syllogisms that constitute Mill’s (1843) Canons that are relevant to our discussion: the Method of Agreement, the Method of Difference, and the Method of Concomitant Variations. Researchers in the field apply these methods to make inferences regarding the causal relations among phenomena (or events).
To illustrate this problem, consider the implications of a recent article appearing in Nature Methods. The authors reported profound and differential effects of the presence of male and female humans on the performance of the most common of subjects—rats and mice.
[Exposure to] Male- [but not female-] related stimuli induced a robust physiological stress response that results in stress-induced analgesia. This effect could be replicated with T-shirts worn by men, bedding material from gonadally intact and unfamiliar male mammals, and presentation of compounds secreted from the human axilla. Experimenter sex can thus affect apparent baseline responses in behavioral testing. (Sorge et al., 2014, p. 629)
Researchers in behavioral or physiological psychology, physiology, and the neurosciences seldom describe the sex of the experimenters running their subjects or participants. Without that knowledge, we cannot know whether a basic premise of Mill’s (1843) Method of Difference (that all conditions in an experiment are, in the beginning, identical) holds, making all causal inferences based on the data suspect. This fact has led some to claim, “Decades of science are going to be—perhaps not voided but certainly called into question” (Petri, 2014, para. 7).
The controversy surrounding the well-known priming study (Bargh, Chen, & Burrows, 1996) in which young participants walked more slowly after completing a scrambled-sentence task priming an elderly stereotype than participants in a control condition provides an example from literature in which human participants were used. Despite its fame, many failed to replicate the study’s findings. In the original report, Bargh et al. (1996) did not describe all their procedures, including the fact that the experimenter who handed the priming or control task to participants was the same individual who packaged the materials (see Yong, 2012, for an extended discussion). In 2012, Doyen, Klein, Pinchon, and Cleeremans demonstrated the potential for this methodological “detail” to play a crucial role in the outcome of the original study. Bargh et al. inferred that participant priming caused the outcome, but others could not replicate the finding purportedly supporting this causal inference without this overlooked information. Researchers have spent more than a decade wondering what went awry in their replication attempts and even now continue to propose alternative inferences (e.g., Cesario, Plaks, & Higgins, 2006; Doyen et al., 2012; Hull, Slone, Metayer, & Matthews, 2002; Klatzky & Creswell, 2014; Pashler, Harris, & Coburn, 2011). 6
As outlined earlier, to implement Mill’s (1843) Canons, researchers make a large number of sampling decisions. They make judgments about which of these decisions are relevant, they control for factors that they believe are germane, and they ignore those that they intuit are not. Each choice is a sampling decision. Note that almost none of the resulting samples are taken randomly, instead they are often chosen on the basis of pilot studies or the intuition of the original researchers (see Fiedler, 2011, who has outlined the profound effects that such nonrandom samples can have on the outcome of a given study, artificially inflating effect sizes). Each of these choices potentially influences the results of a study and, when ignored, may lead to numerous problems—faulty causal and correlational inferences, initial and replication studies producing divergent results, and studies included in a meta-analysis appearing to come to widely different conclusions (see, e.g., Wainer, 2007; Yong, 2012). In an ideal world, these would not be problems at all. The original researchers would decide and know which sampling decisions are relevant to their research and would provide sufficient description of each. In the real world, however, researchers cannot always know which sampling decisions are important—and, over time, the decisions assumed unimportant may be forgotten.
We take four lessons from these facts. First and obviously, to conduct a direct, systematic, or conceptual replication properly, the replicating researcher needs an adequate description of the original researchers’ sampling decisions. Without this description, when the replicating researcher makes inferences regarding the consistency (or reliability) of a finding, he or she cannot determine whether or where his or her sampling decisions differed from the original research. Second, if meta-analysts are to examine quantitatively the robustness of a given phenomenon (e.g., Glass, 1976), they need data permitting them to derive effect sizes (e.g., number of participants and inferential statistics and, more ideally, the central tendency and variance for each measure). Moreover, to evaluate and identify possible moderators of effect sizes, which may reflect sampling decisions in the studies under analysis (e.g., Eysenck, 1994), and to make decisions about how to weight a study, 7 meta-analysts need access to adequate descriptions of the original researchers’ sampling decisions. Third, for a reviewer to determine whether an original author met the premises of Mill’s (1843) Canons well enough to warrant accurate causal and correlational inferences, an adequate description of sampling decisions must be provided. Finally, if editors and readers alike are to make judgments about the inferential validity of a researcher’s conclusions, they need access to descriptions of sampling decisions that determine the appropriate inferences to be made on the basis of a study’s outcome. Without access to descriptions of sampling decisions, the process of self-correction in science may be stopped dead in its tracks. In short, the devil lies in the details. These facts lead us back to a duty to describe.
Are the Sampling Decisions of Published Psychological Studies Adequately Described?
As we contemplated these facts, we wondered whether the studies we have published described enough details to permit direct, systematic, or conceptual replication; inclusion in a meta-analysis; or evaluation of our inferential validity. We first reviewed several 8 of the extant inventories designed to provide a “standard way for authors to prepare reports of trial findings, facilitating their complete and transparent reporting, and aiding their critical appraisal and interpretation” (see the Consolidated Standards of Reporting Trials [CONSORT] Statement at http://www.consort-statement.org/). These standards did not suit our purposes. Although each addresses descriptive completeness, none of the demands of Mill’s (1843) Canons and accurate inference, replication, or meta-analyses are used in the standards to guide their structure. Moreover, although they include items relevant to these types of studies, other items—such as the formatting details—are not applicable (American Psychological Association, 2009), and some of the standards addressed only intervention designs.
We examined Mill’s (1843) Canons and created a comprehensive inventory of the information that a naïve researcher requires to undertake a replication study. We took a deductive approach by first generating a list of items that an original researcher should know, and then we removed items unrelated to Mill’s Canons (i.e., researcher hypotheses). Next, we examined the literature and retained only items empirically demonstrated to affect the outcome of studies (for examples from the literature in which they have mattered, see Table S1 in the Supplemental Material available online). 9 If these sampling decisions, known and easily described by the original researcher, have made a difference in some outcomes, then they might also make a difference in other studies. Until the field has a full working knowledge of when and whether they are relevant, it is wise to describe them. For example, few researchers explicitly acknowledge that the time of semester that a study is run might change their sample of student participants, but such an influence has been clearly documented (Aviv, Zelenski, Rallo, & Larsen, 2002; Cassidy & Kangas, 2014).
We modified each item until it appeared directly observable, objective, operationalized, and easily coded; ensured that items were mutually exclusive; and incorporated a scoring metric to guide assessment. The focus of every item is on descriptive sufficiency, and the sole aim is to rate the clarity of description relative to these requirements. We then created a coding manual in which we described each item in enough detail to permit undergraduate research assistants to code the items reliably (see Supplement C in the Supplemental Material).
We applied this inventory to our own work and discovered that we consistently underdescribed sampling decisions made in our experiments and correlational studies. We all too often did not describe variables that could affect the outcome of a study, for example, the sex of the experimenter(s) running our subjects or participants (e.g., Sorge et al., 2014), the exact procedures used (e.g., Bargh et al., 1996), when a study was conducted (e.g., Cassidy & Kangas, 2014), or exactly how we cleaned our data for analysis (for a potent example of the dangers of nondisclosure of data practices, see the exchange between Reinhart & Rogoff, 2010, and Herndon, Ash, & Pollin, 2013). As an example, in one recent case, we could not unearth why our attempts to directly replicate the results of a study performed by a colleague in our laboratory failed because we could not determine how our methodological and training sampling decisions differed from hers—she had not recorded apparently critical decisions, and she could not be reached for a consultation. 10 Moreover, we found that many of the details central to direct replications were not in memory or in the records that remained of our studies. If others requested those details, we simply could not supply them—all potential reasons for a failure to replicate our efforts because of our initial assumption that such details did not matter. Furthermore, without these details available, a replicating researcher may mistakenly conclude the original study was an error.
With that in hand, we asked, “How well are others describing the sampling decisions that support the execution of direct, systematic, or conceptual replication study; inclusion in meta-analysis; accurate inferences; and the analysis of inferential validity of a study?”
Method
The Replicability and Meta-Analytic Suitability Inventory (RAMSI)
The full RAMSI consists of five groups of items known to influence the outcome of a given study that fall within the inferential framework of Mill’s (1843) Canons: Methods and Procedures (nine items); Participant Recruitment and Characteristics (nine items); Assessor Recruitment, Characteristics, and Training (13 items); Experimenter Recruitment, Characteristics, and Training (10 items); and Results and Analysis (three items). Each item represents one sampling decision made by the original researcher (for a complete list and description of each of these items, see Supplement A in the Supplemental Material). Because some items apply only to quasi-experimental and experimental designs (e.g., Specific Group Procedures because correlational studies do not contain separate groups), we created two subforms of RAMSI—one for correlational studies and one for experimental and quasi-experimental studies.
In RAMSI, we used a 3-point rating metric for each item (0, 0.5, and 1) and a “not applicable, skip” option when relevant using the following general definitions:
0: Item not found within the study description;
0.5: Item found, but it lacks information necessary for direct replication;
1: Item found, sufficient information to be directly replicable; and
N/A, skip: Item not applicable to study.
The coding manual for RAMSI provided definitions and description for each item (see Supplement C in the Supplemental Material). In the present study, we used a web-based version on the website, DatStat Illume 4.11. 11 When a coder rated an item “N/A” or “0,” the web-based inventory skipped all subsequent items related to that topic and assigned an “N/A” or “0” as appropriate. The rating of each item represented how completely the original researcher described the sampling decision that the item addressed.
Articles sampled
We used RAMSI to estimate the average level of relevant descriptive detail of reports published during 2010 in the top five, middle five, and bottom five journals as ranked by the top 100 ISI Web of Science 5-year impact factor. We limited the articles to those describing empirical research, written in English, published in 2010, and involving human participants. 12 We also excluded qualitative studies and meta-analyses. A total of 1,083 articles met inclusion criteria (see Table 2 for a list of journals included and number of articles per journal). We retrieved articles and supplemental materials, when applicable, through the University of Arizona Library resources (e.g., EBSCO, SAGEPub, ISI Web of Science). Articles were assigned unique numbers for coding and tracking purposes.
Performance on Replicability and Meta-Analytic Suitability Inventory (RAMSI) Sections and Overall by Journal
Note: Table 2 illustrates the mean (J# % = journal number mean) and standard deviation (SD = journal’s standard deviation) percentage of points earned for each section of RAMSI across all articles in each individual journal. These were also calculated across all journals in each journal rank. Mean percentage and standard deviation of total points earned, independent of specific sections assessed, is also provided for articles within each journal. Hence, article total percentage will not equal the average of the percentages earned across the five sections (each consisting of varying number of items), as total points earned divided by total points possible for all articles is averaged in the calculation of “total points.” Journals in analyses (J# = number associated with that journal, n = number of articles included from that journal) include the following: J1 = Journal of Experimental Psychology General (n = 40); J2 = Journal of Personality and Social Psychology (n = 140); J3 = Development and Psychopathology (n = 55); J4 = Journal of Cognitive Neuroscience (n = 206); J5 = Psychological Science (n = 230); J6 = Personal Relationships (n = 36); J7 = Journal of Clinical Psychology (n = 79); J8 = Journal of Personality Assessment (n = 47); J9 = Applied Psycholinguistics (n = 27); J10 = Journal of Behavioral Decision Making (n = 26); J11 = Journal of Creative Behavior (n = 9); J12 = Teaching of Psychology (n = 40); J13 = American Journal of Family Therapy (n = 12); J14 = Psychological Reports (n = 111); J15 = The Arts in Psychotherapy (n = 25).
To calculate reliability scores, two assessors coded about 10% of articles in each journal. An online random number generator (www.random.org) determined the articles to be double-coded. After obtaining the descriptive and reliability data, we tested the hypothesis that higher ranked journals contain more complete research descriptions than lower ranked journals.
Article assessors and data collection
Full details of assessor demographics, recruitment, and procedures, as well as data collection procedures, are available in Supplement D in the Supplemental Material.
Data scoring
Total and section scores, determined as a percentage of points “earned” out of total points possible, were calculated for each article. In other words,
All items rated “not applicable” received a point value of 1 because authors made it clear in their report that the item was not applicable and, thus, not an undescribed detail. Subsequently, all items depending on this item were excluded from the sum of total points possible (for specific item point-decision rules, see RAMSI in Supplement A in the Supplemental Material).
Data analyses
We measured assessor reliability as the percentage agreement between assessors. We coded individual items as either agree or disagree using an “if, then” statement: agreement = 1, and disagreement = 0. Then, we calculated agreement across articles by individual item, section, and total article score using Microsoft Excel.
Each article coded for reliability received two separate sets of ratings. Of the data included in the main analyses, articles were randomly assigned such that the first or second rating was used (50% first rating, 50% second rating). We did this because we had no theoretical reason to think the first rating would be more valid than the second or vice versa. It was also used to avoid inflation of effect sizes by selecting article ratings consistent with hypotheses (e.g., Fiedler, 2011) and to distribute unknown errors.
We tested hypothesized relations among the scores of the three journal groups for the five sections of RAMSI and article total score using one-way, independent-sample analyses of variance (ANOVAs). To detect differences between journal group pairs, when applicable, we used SPSS 20 to conduct post hoc comparisons with Tukey’s honestly significant difference and Scheffe’s test.
Results
Assessor agreement
Overall, average assessor agreement across all items in the 10% of articles rated by two assessors was 78%. See Table S2 in the Supplemental Material for item-by-item assessor agreement statistics. It is worth noting that some of the items had low agreement, suggesting that undergraduate students still early in their training find some items particularly difficult to code. Most of the disagreements were between adjacent score points (e.g., assigning an article a .5 vs. 1 or 0 vs. .5) rather than a large discrepancy (e.g., assigning an article a 0 vs. 1). The primary point remains; there is a great deal of description missing from published articles.
Descriptive statistics
Table 3 shows the average percentage scores and standard deviations for each of the journals, by journal rank for the five sections, and total score for RAMSI. There was wide variance of reporting of the sampling decisions surveyed, independent of journal, with 23%–99% either underdescribed (i.e., a score of .5) or undescribed (i.e., a score of 0; for frequency of each score point assigned by item by journal, see Table S3 in the Supplemental Material).
Replicability and Meta-Analytic Suitability Inventory Alternative Brief (RAMSI-AB) Items
Note: In this table, we list and briefly describe each of the items included in the RAMSI-AB. In the RAMSI-AB, one can mark each item as either in the manuscript, not reported, or not applicable. If unreported, it is suggested that the author explain this reasoning in the cover letter submitted with the manuscript to the editor. The RAMSI-AB represents a subset of important sampling decisions to consider reporting, so authors are advised to reference the RAMSI Alternative as needed and record these when feasible.
The smallest relative variance was in total scores for Journals 8, 9, and 11. Total scores for Journals 7, 12, and 15 had the largest variance. Examination of score by RAMSI section revealed a wide range of scores both across individual journals and journal rank (for boxplots of article total score by journal, see Figure S1 in the Supplemental Material; for boxplots of score distribution for each RAMSI section by journal rank, see Figure S2 in the Supplemental Material).
Inferential statistics
Using one-way, independent-sample ANOVAs, we compared journal group percentage scores on each section of RAMSI and for total article score (for all ANOVA tables, see Table S4 in the Supplemental Material).
Method
Using an ANOVA, we detected significant differences among journal groups on mean total percentage score for the Method section, F(2, 1080) = 22.38, p < .001, r = .20. Post hoc comparisons indicated that mean score for the top-ranked journal group (M = 46.7, SD = 12.0) was significantly greater than the midranked (M = 40.9, SD = 11.2) and the low-ranked (M = 43.9, SD = 12.3) journal groups. The group mean for the Method section in low-ranked journals was also significantly greater than that of the midranked journal group.
Participants
Using an ANOVA, we detected significant differences among journal groups on Participants total percentage score, F(2, 1080) = 8.43, p < .001, r = .12. Post hoc comparisons indicated that the mean score for the top-ranked journal group (M = 50.0, SD = 21.9) was significantly greater than the midranked (M = 43.9, SD = 20.9) and the low-ranked (M = 45.1, SD = 21.8) journal groups; the mid- and low-ranked journal groups did not differ significantly.
Assessor
Using an ANOVA, we detected significant differences among journal groups on Assessor total percentage score, F(2, 1080) = 3.42, p = .03, r = .08. Post hoc comparisons indicated that the mean score for the top-ranked journal group (M = 45.0, SD = 30.3) was significantly greater than the midranked journal group (M = 35.7, SD = 44.4); the low-ranked journal group (M = 44.1, SD = 47.4) did not differ significantly from either.
Experimenter
There were no detectable differences among journal groups on Experimenter total percentage score, F(2, 1080) = 1.60, n.s., r = .05 (top-ranked journal group: M = 11.3, SD = 30.3; midranked journal group: M = 13.6, SD = 31.6; low-ranked journal group: M = 15.5, SD = 32.4).
Results
There were no detectable differences among journal groups on Results total percentage score, F(2, 1080) = 1.62, n.s., r = .05 (top-ranked journal group: M = 67.9, SD = 21.3; midranked journal group: M = 64.9, SD = 31.4; low-ranked journal group: M = 69.0, SD = 29.1).
Article
Using an ANOVA, we detected significant differences among journal groups on Article total score, F(2, 1080) = 4.48, p = .01, r = .09. Post hoc comparisons indicated that mean score for the top-ranked journal group (M = 32.3, SD = 13.8) was significantly greater than the midranked journal group (M = 28.9, SD = 15.1) but not the low-ranked journal group (M = 31.6, SD = 16.4). The mid- and low-ranked groups did not significantly differ.
Discussion
“The question is not whether various … effects are real and can be replicated—because they are and often have been—but rather why some researchers reproduce these effects and others do not. The question is important for advancing the knowledge of “how [these] influences operate, and it draws needed attention to the precise contexts and conditions required to produce [them]. More work remains.” (Bargh, 2014, p. 36)
Although statistically the top-ranked journals provided more information about methodological and participant sampling decisions than the mid- or low-ranked journals, differences were small. Overall, descriptions were surprisingly incomplete. In the articles in all three journal rankings, the authors most fully described details regarding results (65%–69%) and described the least regarding experimenters (11%–16%). Top-ranked journals provided about 32% of the information regarding an article’s overall total sampling decisions, and the middle- and low-ranked journals provided 29% and 32%, respectively. In other words, roughly 70% of the sampling decisions were under- or undescribed, independent of journal rank.
In general, the descriptive completeness of the articles evaluated here appears neither to meet the basic standards set by Mill’s (1843) Canons, to allow for full assessment of a researcher’s inferential conclusions, nor to meet the needs and requirements of replication or meta-analytic researchers. Moreover, although some statistically significant differences among reporting in top-, mid-, and low-ranked psychological journals appeared, the differences were relatively small; the lack of descriptive completeness across the journals sampled provides little substantial evidence of higher editorial standards by journal rank.
There is no simple fix for the problem at hand (see the Cochrane Review by Turner, Shamseer, Altman, Schulz, & Moher, 2012, on the effects of the CONSORT Statement on completeness of reporting in the medical field and the need for both author and journal involvement). McShane and Böckentholt (2014, this issue), for example, have called attention to the fact that the heterogeneity of effect sizes across replication attempts may lead to the mistaken interpretation that a replication effort failed. These authors have advised replicating researchers to gain estimates of heterogeneity of effect sizes (e.g., by having several labs simultaneously run the same study worldwide—the Many Labs approach; Klein et al., 2014) and then to use adjusted power analyses to guide the N in their participant sample. In short, whereas McShane and Böckentholt (2014) have called for researchers to adjust for heterogeneity of effect sizes because of unknown error, our call is for researchers to provide information that would allow for the identification of sources of that heterogeneity. Taken together, McShane and Böckentholt have provided a means of dealing with heterogeneity potentially attributable to under- or undescribed sampling decisions, and we provide a means of minimizing such heterogeneity. In our opinion, combining these approaches strengthens both.
We encourage researchers to use checklists, such as the RAMSI (or one of the alternative versions presented later), to guide the descriptive content of their reports. As much as comprehensive checklists have been shown to reduce complications in hospital operating rooms (de Vries et al., 2010), our inventory has the potential to reduce descriptive incompleteness in psychology.
Potential applications of the RAMSI
We offer three versions of RAMSI that researchers and editors, among others, may find useful in the Supplemental Material. The first (see Supplement A in the Supplemental Material) is the original version of RAMSI that we used and described in this article. It was created so that the current literature could be evaluated.
The second version, the RAMSI Alternative (RAMSI-A; see Supplement E in the Supplemental Material), could be helpful for both publication and general use. It guides users through each RAMSI item, provides the option of marking an item as already described in the manuscript (if used for publication) or as “N/A” if an item is not applicable to a given study, and contains space to provide a description of the sampling decision. It can be used by researchers to record crucial sampling decisions when conducting a study (for a demonstration of its use as applied to this article, see Supplement D in the Supplemental Material).
The RAMSI Alternative Brief (RAMSI-AB), created in light of the data from the present study, is a nine-item form notably shorter than the 44-item RAMSI and RAMSI-A (see Table 3). We omitted items that authors routinely and fully describe, that are obvious to report (inferential statistics, method of analysis, procedures used, etc.), and that are covered in other recent recommendations (as noted later in this section). Several related individual items were collapsed into single items because there were not substantial differences in level of published descriptions among them (e.g., assessor training vs. assessor reliability). Furthermore, because many psychological studies do not use assessors, assessor and experimenter items were placed into a single category. Thus, RAMSI-AB includes the core sampling decisions that are typically most under- or undescribed as informed by the data, literature, and Mill’s (1843) Canons. The RAMSI-AB is one possible and easily implemented set of recommendations regarding description of sampling decision that extends beyond recent recommendations about methodological disclosure alone.
Likely, the RAMS-AB would be most useful when dealing with a specific manuscript to track which items were included, not applicable, or omitted, and it could serve as a quick overview of such decisions to editors and reviewers alike. When items are not reported, we make the additional recommendation that the author(s) explain the reasoning behind this omission in the cover letter submitted with the manuscript for publication. Authors would then be on record about which sampling decisions replicating studies would be free to vary as desired.
We recommend that researchers use checklist items to record the sampling decisions they make when designing a study and before running it. We further recommend that after completing a study, researchers make conscious, deliberate, and intentional decisions—informed by Mill’s (1843) Canons and what is known in the literature—to determine which items should be described to help foster inferential validity, replicability, meta-analytic inclusion, and, thereby, self-correcting science. Finally, though we acknowledge the possibility that some details might prove to be unimportant, we recommend that researchers describe items reported in publications or online supplemental materials in enough detail to warrant direct replication by independent researchers.
Objections and rebuttals
Some may argue that the problem identified here is trivial and may assert that this article is unnecessary because a simple solution is already at hand. A reviewer (received April 26, 2012) of an earlier version of this article dismissed these ideas by commenting, I do not believe that the barrier to conducting these replications lies in the inability of researchers to attain original materials or sufficient details about the published study that they are attempting to replicate. Except in unusual cases, it is not difficult to contact authors to request their exact materials.
The reviewer’s belief is correct. It is not difficult to request exact materials. Furthermore, with the exception of authors who are no longer in the field, retired, or dead, the problem does not lie in making contact. Instead, the problem lies in the authors’ responses to those contacts. There are more than 50 years of data demonstrating that most authors simply do not respond to requests for methodological details, exact materials, or raw data. Many who do respond cannot supply requested information because the information was never recorded in the first place, has been forgotten, or was lost. In his undergraduate classes at Harvard, Robert Rosenthal vividly illustrated the latter by stating there are “an unusual number of fires and floods in Psychology Departments” (Kevin Thomas, personal communication, December 2012).
To our knowledge, Wolins (1962) was the first to document this problem. Of the 37 authors contacted for further information, only 24% provided it without insisting on having control of subsequent publications. The introduction of e-mail has not fixed the situation. Wicherts and colleagues (Wicherts, Bakker, & Molenaar, 2011; Wicherts, Borsboom, Kats, & Molenaar, 2006) requested study information from contemporary authors and found that 42.9% provided the requested information. More recently, LeBel et al. (2013) requested additional methodological descriptions from authors published in four major psychology publications; 13 they received an overall response rate of 46.4%. Others have documented widespread failures to adhere to ethical guidelines and journal policies for disclosure at time of publication (e.g., Alsheikh-Ali, Qureshi, Al-Mallah, & Ioannidis, 2011). 14 It appears that the reviewer’s belief is correct, but the implication that contact solves the problem is not.
Others might argue that journals are already implementing standards addressing this issue. The editorial board members of Nature were among the first to recognize the problems that incomplete descriptions create. In an April 2013 editorial, it was recognized that, although problems of replication start in the laboratories of researchers, journals contribute “when they fail to exert sufficient scrutiny over the results that they publish, and when they do not publish enough information for other researchers to assess results properly” (Nature Publishing Group, 2013, p. 398). As of May 2013, the editorial board of Nature implemented editorial practices “to ease the interpretation and improve the reliability of published results” by giving more space to method sections and ensuring that authors describe “key methodological details” through use of a checklist (Nature Publishing Group, 2013, p. 398). A similar series of policy changes has been recently implemented in Psychological Science (influenced by LeBel et al., 2013), requiring authors to confirm reporting of observation exclusions, all manipulations, all measures analyzed, and sample size decisions in articles (Eich, 2014). 15
These changes are important but incomplete. They are neither anchored in the requirements of Mill’s (1843) Canons nor focused on ensuring the inferential validity of a study, its replicability, or its inclusion in meta-analyses. Thus, they do not adequately foster self-correcting science. They also do not fully take into account what is known from the literature regarding sampling decisions that may influence a study’s outcome (e.g., sex of the experimenter, exact procedures used, when a study was conducted). Moreover, the data collected here demonstrate that far more than the handful of methodological sampling decisions tagged as needing to be described by past researchers are underreported or undescribed. Most glaring, this list includes crucial description of 80%–99% of sampling decisions involving experimenters and assessors, study settings, and time parameters of a study. Less glaring, though still troublesome, the studies in our sample also left underreported or undescribed 60%–90% sampling decisions relevant to participant demographics, recruitment, and participation eligibility criteria. These data suggest that although authors describe what they think matters, crucial sampling decisions are ignored, and authors, editors, or readers cannot rely on intuition alone to determine when and whether such details are important to record. All of these omissions may inhibit not only replication and meta-analysis but also any causal and correlational inferences based on research findings. Given that these sampling decisions are taken into account with RAMSI, RAMSI-A, and RAMSI-AB, they are more comprehensive than past suggestions.
Some also might argue that the level of descriptive detail called for by Mill’s (1843) Canons and embodied in RAMSI is too stringent because publishing space comes at too significant a cost, making this level of detail unrealistic, and researchers cannot be expected to describe so much about their studies.
The technological advances of the past 20 years, however, afford us inexpensive opportunities to post near unlimited supplemental material online. Even adherence to the considerably briefer RAMSI-AB would be an improvement on current standards, encouraging authors and editors to consider reporting decisions actively. Moreover, placing information, such as that contained in RAMSI-A, online (ideally with online supplemental material maintained by the publishing journal to ensure that it does not disappear and that it remains linked to the published study) could save replicating researchers time, effort, and expense. In some cases, it could also save original researchers embarrassment as well as save politicians from advocating or establishing harmful public policy (see again the exchange between Reinhart & Rogoff, 2010, and Herndon et al., 2013). In the present analysis, multiple articles merited the directly replicable score of “1” for each of the inventory items, indicating that this level of description can and already is achieved by at least some authors in some journals. Furthermore, all RAMSI-A items are easily known to and recordable by an original researcher—problems arise when the researcher erroneously assumes that an item is unimportant and fails to report it. In short, the argument that it is too expensive or too difficult to provide full descriptions of pertinent sampling decisions has lost its weight in today’s technological climate, especially in light of the damage that not providing them creates.
Finally, some might object that the different versions of RAMSI are incomplete. The items are limited to variables demonstrably relevant, at least in some cases, to the outcome of a study and to those sampled by, known to, easily recorded by, and in the control of a researcher. Researchers in the field do not yet have a good understanding of how to predict whether or when or even which of these sampling decisions will be relevant to the outcome of a given study. 16 Despite the incompleteness of RAMSI-AB and even RAMSI-A, implementation of such standards of reporting of sampling decisions represents a step toward a solution to the problem by helping researchers in the field build a base knowledge, better allowing examination of when and whether these sampling decisions even matter.
Final thoughts
We offer these thoughts and data in the hope that many of the problems facing a self-correcting science founded in valid inferences will continue to receive careful and thoughtful attention and intervention. Moreover, we hope that researchers in the field will keep an eye on the logic underpinning their designs, strategies used by other areas of science, and journals such as Nature and Psychological Science as more rigorous publication guidelines are instituted. No doubt there will be many strong and well-thought-out options forthcoming. In our opinion, researchers in the field would benefit by basing such options on intentional, principled, and conscious choice informed by research and foundational principles expressed by John Stuart Mill and others rather than by leaving the description of sampling decisions to chance, uniformed choice, or intuition alone.
Footnotes
Acknowledgements
We acknowledge and thank Alison Ledgerwood for her hard work, patience, and suggestions during the development of this article. We also thank the many undergraduate research assistants who contributed to this project.
Declaration of Conflicting Interests
The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
