Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)

Method

Design

This was a retrospective observational study with a cross-sectional design. Sampling units were individual articles. Measured variables are shown in Table 1.

OPEN IN VIEWER

Table 1.

Measured Variables

Variable	Applicable study designs
Article characteristics
Subject area, year of publication, study design, country of origin (based on corresponding author’s affiliation), human/animal subjects, 2017 journal impact factor (according to Thomson Reuters Journal Citation Reports)	All
Articles
Accessibility and retrieval method (can the article be accessed, and, if so, is there a public version or is paywall access required?)	All
Protocols
Availability statement (is availability, or lack of, explicitly declared?)
Content (what aspects of the study are included in the protocol?)
Materials
Availability statement (is availability, or lack of, explicitly declared?)	Study designs involving primary data a
Retrieval method (e.g., on request or via online repository)
Accessibility (can the materials be accessed?)
Raw data
Availability statement (is availability, or lack of, explicitly declared?)	Study designs involving primary data, a study designs involving secondary data (commentaries with analysis and meta-analyses)
Retrieval method (e.g., on request or via online repository)
Accessibility (can the data be accessed?)
Content (have all relevant data been shared?)
Documentation (are the data understandable?)
Analysis scripts
Availability statement (is availability, or lack of, explicitly declared?)
Retrieval method (e.g., on request or via online repository)
Accessibility (can the scripts be accessed?)
Preregistration
Availability statement (is availability, or lack of, explicitly declared?)	Study designs involving primary data, a study designs involving secondary data (commentaries with analysis and meta-analyses)
Retrieval method (which registry was used?)
Accessibility (can the preregistration be accessed?)
Content (what was preregistered?)
Funding
Disclosure statement (are funding sources, or lack of, explicitly declared?)	All
Conflicts of interest
Disclosure statement (are conflicts of interest, or lack of, explicitly declared?)	All
Replication
Statement (does the article claim to report a replication?)	All
Citation history (has the article been cited by a study that claims to be a replication?)	Study designs involving primary data a
Evidence synthesis
Meta-analysis citation history b (has the article been cited by and included in the evidence-synthesis component of a meta-analysis?)	Study designs involving primary data a
Systematic review citation history b (has the article been cited by and included in the evidence-synthesis component of a systematic review?)	Study designs involving primary data a

Note: The variables measured for an individual article depended on the study-design classification. For articles that were not available (the full text could not be retrieved) or were not in English, only article characteristics were obtained. The exact operational definitions and procedures for data extraction/coding are available in the structured form at https://osf.io/x9rmy/yh.

a

Encompasses the following study-design classifications: field studies, laboratory studies, surveys, case studies, multiple types, clinical trial, and other designs. ^bMeta-analysis and systematic review variables were coded as exclusive variables.

The study protocol was preregistered on September 26, 2018, and is available at https://osf.io/q96eh. All deviations from this protocol are explicitly acknowledged in Section A of the Supplemental Material available online. We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study. All data (https://osf.io/5qmz7), materials (https://osf.io/c89sy), and analysis scripts (https://osf.io/gfjtq) related to this study are publicly available. To facilitate reproducibility, the manuscript was written by interleaving regular prose with analysis code and is available in a Code Ocean container (https://doi.org/10.24433/CO.1618143.v2) that recreates the software environment in which the original analyses were performed.

Sample

We obtained a random sample of 250 psychology articles published between 2014 and 2017. We used a random-number generator to sample 250 articles from all 224,556 documents in the Scopus database (as of September 22, 2018) designated with the document type “article” or “review” and with an All Science Journal Classification (ASJC) code related to psychology (i.e., ASJC codes 3200–3207). This classification included 1,323 journals and all subject areas shown in Table 2. The sample size was based on our judgment of what was both informative and tractable. The evaluated time period was selected to represent recently published articles (relative to the start of data collection) and to facilitate comparisons with a similar assessment conducted in the social sciences for the same time period (Hardwicke, Wallach, et al., 2020).

OPEN IN VIEWER

Table 2.

Sample Characteristics for the 250 Randomly Sampled Articles and the 228 English-Language and Accessible Articles That Were Eligible for In-Depth Data Extraction

Characteristic	Eligible articles(n)	All articles(n)
Subject area
Clinical psychology	57	65
General psychology	41	43
Developmental and educational psychology	40	44
Applied psychology	34	38
Social psychology	29	33
Experimental and cognitive psychology	16	16
Neuropsychology and physiological psychology	10	10
Psychology (miscellaneous)	1	1
Year of publication
2014	54	58
2015	60	68
2016	49	54
2017	65	70
Study design
Survey/interview	73	77
Laboratory study	52	53
No empirical data	40	48
Observational study	33	40
Clinical trial	15	15
Multiple study types	6	6
Case study	4	4
Commentary with analysis	3	3
Meta-analysis	2	2
Country of origin
United States of America	99	99
United Kingdom	20	20
Canada	12	13
Germany	11	15
The Netherlands	11	11
28 other countries a (accounting for < 10 per country)	75	92
Subjects
Humans	174	184
Animals	7	7
Both	1	1
Neither humans nor animals involved	46	56

Note: The 2017 median impact factor for eligible articles was 2.23 (range = 0.33–15.07); the median impact factor for all articles was 2.06 (range = 0.22-15.07). The publication-year median impact factor for eligible articles was 2.09 (range = 0.23-20.77); the publication-year median impact factor for all articles was 2.00 (range = 0.22–20.77). No 2017 journal impact factor was available for 111 articles (97 eligible articles), and no publication-year impact factor was available for 113 articles (99 eligible articles).

a

For all countries, see https://osf.io/kg7j5.

Procedure

Data were collected between September 27, 2018, and September 23, 2019. Data extraction for the measured variables shown in Table 1 involved close manual examination of the articles supplemented with keyword searching guided by a Google form (https://osf.io/x9rmy) based on previous investigations in biomedicine (Iqbal et al., 2016; Wallach et al., 2018) and the social sciences (Hardwicke, Wallach, et al., 2020). As indicated in the second column of Table 1, the exact variables measured for a given article depended on the study design. Attempts to access each article were made by searching with the Open Access Button, Google, and, if necessary, at least two of the online libraries of the coders’ affiliated universities. For articles that were not available (the full text could not be retrieved) or were not published in English, only article characteristics were obtained.

Three investigators performed the initial extraction and coding; articles were assigned to each investigator on demand by randomly selecting from the remaining articles (J. D. Wallach, n = 32; J. E. Kosie, n = 93; R. T. Thibault, n = 125). A single investigator obtained the subject area, year of publication, and citation histories from Scopus and 2017 journal impact factors from Thomson Reuters Journal Citation Reports. All other variables in Table 1 were coded a second time by one of two other investigators (T. E. Hardwicke, n = 198; M. C. Kidwell, n = 52), and any coding differences were resolved through discussion. Interrater reliability assessments are shown in Table S1 in the Supplemental Material available online. Assuming all articles were published on July 1 of their respective publication year (i.e., halfway through the year), the time between publication and recording citation information ranged from 448 to 1,544 (Mdn = 996) days for number of citations and 456 to 1,900 (Mdn = 1,178) days for number of citing articles that were replications, systematic reviews, and/or meta-analyses. Conflict-of-interest statements (Table 3) were categorized by two investigators (T. E. Hardwicke and J. D. Wallach) in an exploratory manner (i.e., not preregistered).

OPEN IN VIEWER

Table 3.

Frequency of Different Types of Conflict of Interest Reported in the 12 Statements Reporting One or More Conflicts of Interest

Type of conflict of interest	Frequency of appearance in statements (n)
Industry-related
Authorship/editorship royalties	4
Research funding from industry	4
Served on industry advisory board	4
Consultancy for industry	3
Ownership of relevant commercial products, patents, or procedures	3
Speaking fees from industry	3
Employed by industry	2
Honoraria from industry	2
Industry equity holder	2
Travel or hospitality awards from industry	2
Other undefined payments from industry	1
Nonindustry-related
Research funding from government	4
Research funding from foundations, charities, and/or NGOs	2
Consultancy for foundations, charities, and/or NGOs	1
Honoraria from foundations, charities, and/or NGOs	1

Note: Because each of the 12 relevant conflict-of-interest statements may contain more than one type of conflict of interest, the frequency column sums to greater than 12. NGO = nongovernmental organization.

Results

Article availability (open access)

Among the 237 English-language articles, we obtained a publicly available version for 154 (65%, 95% CI = [59%, 71%]; Fig. 1a), whereas 74 (31%, 95% CI = [25%, 38%]) were only accessible through a paywall. Nine articles (4%, 95% CI = [0%, 10%]) were not available at all (Fig. 1a).

OPEN IN VIEWER

Fig. 1.

Assessment of transparency and reproducibility-related research practices in psychology. The x-axis shows the percentage of the total number of articles (N) assessed for a given indicator (which was contingent on the study design; see Table 1). Raw counts are shown inside bars.

Discussion

Our evaluation of transparency and reproducibility-related research practices in a random sample of 250 psychology articles published between 2014 and 2017 shows that, although many articles were publicly available, crucial components of research—protocols, materials, raw data, and analysis scripts—were rarely made publicly available alongside them. Preregistration remained a nascent proposition with minimal adoption. The disclosure of funding sources and conflicts of interest was modest. Replication or evidence synthesis via meta-analysis or systematic review was infrequent (although, admittedly, only a relatively short time had elapsed since the articles had been published). Although there is evidence that some individual methodological reform initiatives have been effective in specific situations (e.g., Hardwicke et al., 2018; Nuijten et al., 2017; for review, see Hardwicke, Serghiou, et al., 2020), the findings of the current study imply that their collective, broader impact on the psychology literature during the examined period was still fairly limited in scope.

For most of the articles (65%) we examined, we could access a publicly available version (open access). This is higher than recent open-access estimates obtained for biomedicine (25%; Wallach et al., 2018) and the social sciences (40%; Hardwicke, Wallach, et al., 2020), as well as a large-scale automated analysis that suggested that 45% of the scientific literature published in 2015 was publicly available (Piwowar et al., 2018). Limiting access to academic publications reduces opportunities for researchers, policymakers, practitioners, and the general public to evaluate and make use of scientific evidence. One step psychologists can take to improve the public availability of their articles is to upload them to the free preprint server PsyArXiv (https://psyarxiv.com/). Uploading a preprint does not preclude publication at most journals (Bourne et al., 2017), although specific policies regarding open access can be checked on the Sherpa/Romeo database (http://sherpa.ac.uk/romeo/index.php).

The reported availability of research materials was modest in the articles we examined (14%), which is comparable to recent estimates in the social sciences (11%; Hardwicke, Wallach, et al., 2020) and lower than in biomedicine (33%; Wallach et al., 2018). Several reportedly available sets of materials were in fact not available because of broken links, an example of the “link-rot” phenomenon that has been observed by others trying to access research resources (Evangelou et al., 2005; Rowhani-Farid & Barnett, 2018). We also did not find any study protocols (an additional document detailing the study methods); however, it is unclear to what extent this results from a difference in norms between, for example, biomedicine (in which prespecified protocols are increasingly promoted; Ioannidis, Greenland, et al., 2014) and psychology (in which there may not be an expectation to provide methodological details in a separate protocol document). We did not examine whether sufficient methodological information was provided in the Method sections of articles, as this would have required domain-specific expertise in the many topics addressed by the articles in our sample. The availability of original research materials (e.g., survey instruments, stimuli, software, videos) and protocols enables the comprehensive evaluation of research (during traditional peer review and beyond; Vazire, 2017) and high-fidelity independent replication attempts (Open Science Collaboration, 2015; Simons, 2014), both of which are important for the verification and systematic accumulation of scientific knowledge (Ioannidis, 2012). Furthermore, reusing materials and protocols reduces waste and enhances efficiency (Chalmers & Glasziou, 2009; Ioannidis, Greenland, et al., 2014). Psychologists can share their materials and protocols online in various third-party repositories that use stable permalinks, such as the Open Science Framework ² (OSF; see Klein et al., 2018). One observational study found that when the journal Psychological Science offered authors an open-materials badge there was a subsequent increase in the sharing of materials (Kidwell et al., 2016).

Data-availability statements in the articles we examined were extremely uncommon. This is consistent with accumulating evidence that suggests that the data underlying scientific claims are rarely immediately available (Alsheikh-Ali et al., 2011; Iqbal et al., 2016), although some modest improvement has been observed in recent years in biomedicine (Wallach et al., 2018). Although we did not request data from authors directly, such requests to psychology researchers typically have a modest yield (Vanpaemel et al., 2015; Wicherts et al., 2006). Most data appear to be effectively lost, including for some of the most influential studies in psychology and psychiatry (Hardwicke & Ioannidis, 2018b). Vanpaemel et al. (2015), for example, could not obtain 62% of the 394 data sets they requested from authors of papers published in four American Psychological Association journals in 2012. The sharing of raw data, which is the evidence on which scientists base their claims, enables verification through the independent assessment of analytic or computational reproducibility (Hardwicke, Bohn, et al., 2020; Hardwicke et al., 2018; LeBel et al., 2018) and analytic robustness (Steegen et al., 2016). Data sharing also enhances evidence synthesis, such as through individual participant-level meta-analysis (Tierney et al., 2015), and can facilitate discovery, such as through the merging of data sets and reanalysis with novel techniques (Voytek, 2016). Psychologists can improve data availability by uploading raw data to third-party repositories such as the OSF (Klein et al., 2018). Data sharing must be managed with caution if there are ethical concerns, but such concerns do not always preclude all forms of sharing or necessarily negate ethical motivations for sharing (Meyer, 2017). Furthermore, when data cannot be made available it is always possible to explicitly declare this in research articles and explain the rationale for not sharing (Morey et al., 2016). Journal policies that use badges to encourage data sharing (Kidwell et al., 2016) or mandate data sharing (Hardwicke et al., 2018; Nuijten et al., 2017) have been associated with marked increases in data availability in the journals that adopted them.

Of the articles we examined, only one shared an analysis script, a dearth consistent with assessments in biomedicine (Wallach et al., 2018), the social sciences (Hardwicke, Wallach, et al., 2020), and biostatistics (Rowhani-Farid & Barnett, 2018). Analysis scripts (a step-by-step description of the analysis in the form of computer code or instructions for recreating the analysis in point-and-click software) provide the most veridical documentation of how the raw data were filtered, summarized, and analyzed. Verbal descriptions of analysis procedures are often ambiguous, contain errors, or do not adequately capture sufficient detail to enable analytic reproducibility (Hardwicke, Bohn, et al., 2020; Hardwicke et al., 2018; Stodden et al., 2018). Psychologists can share their analysis scripts on a third-party repository, such as the OSF (Klein et al., 2018), and educational resources are available to help researchers improve the quality of their analysis code (Wilson et al., 2017). Sharing the computational environment in which analysis code successfully runs may also help to promote its longevity and trouble-free transfer to other researchers’ computers (Clyburne-Sherin et al., 2018).

Preregistration, which involves making a time-stamped, read-only record of a study’s rationale, hypotheses, methods, and analysis plan on an independent online repository, was rare in the articles we examined. Preregistration fulfills a number of potential functions (Nosek et al., 2019), including clarifying the distinction between exploratory and confirmatory aspects of research (Kimmelman et al., 2014; Wagenmakers et al., 2012) and enabling the detection and mitigation of questionable research practices such as selective-outcome reporting (Franco et al., 2016; John et al., 2012; Simmons et al., 2011). Preregistration is relatively new to psychology (Nosek et al., 2018, 2019), but similar concepts of registration have a longer history in the context of clinical trials in biomedicine (Dickersin & Rennie, 2012), in which they have become the expected norm (Zarin et al., 2017). However, clinical trials represent only a minority of biomedical research, and estimates suggest that preregistration is rare in biomedicine overall (Iqbal et al., 2016; Wallach et al., 2018). Preregistration is also rare in the social sciences (Hardwicke, Wallach, et al., 2020). There is no doubt that the number of preregistrations (and the related Registered Reports article format) is increasing in psychology (Hardwicke & Ioannidis, 2018a; Nosek et al., 2018); however, our findings suggest that efforts to promote preregistration may not yet have had widespread impact on routine practice. It is important to note that because there is a time lag between registration and study publication, our measures may underestimate adoption. Although norms and standards for preregistration in psychology are still evolving (Nosek et al., 2019), several dedicated registries, such as the OSF, will host preregistrations, and detailed guidance is available (Klein et al., 2018).

Our findings suggest that psychology articles were more likely to include funding statements (62%) and conflict-of-interest statements (39%) than social-science articles in general (31% and 15%, respectively; Hardwicke, Wallach, et al., 2020) but less likely than biomedical articles (69% and 65%, respectively; Wallach et al., 2018). It is possible that these disclosure statements are more common than most other practices we examined because they are often mandated by journals (Nutu et al., 2019). Disclosing funding sources and potential conflicts of interest in research articles helps readers to make informed judgments about the risk of bias (Bekelman et al., 2003; Cristea & Ioannidis, 2018). In the absence of established norms or journal mandates, authors may often assume that such statements are not relevant to them (Chivers, 2019). However, because the absence of a statement is ambiguous, researchers should ideally always include one, even if it is to explicitly declare that there were no funding sources and no potential conflicts of interest.

Of the articles we examined, 5% claimed to be a replication study—slightly higher than a previous estimate in psychology of 1% (Makel et al., 2012) and a similar estimate of 1% in the social sciences (Hardwicke, Wallach, et al., 2020) but comparable to a 5% estimate in biomedicine (Wallach et al. 2018). Only 1% of the articles we examined were cited by another article that claimed to be a replication attempt; of these articles, 11% were included in a systematic review, and 7% were included in a meta-analysis. Replication and evidence synthesis through systematic reviews and meta-analyses help to verify and build on the existing evidence base. However, it is unclear what an ideal frequency of these activities would be because they depend on many factors, such as how often studies are sufficiently similar to be amenable to synthesis methods. Although the current findings suggest that routine replication and evidence synthesis is relatively rare in psychology, many high-profile replication attempts have been conducted in recent years (Open Science Collaboration, 2015; Pashler & Wagenmakers, 2012). In addition, because the articles we examined were published relatively recently, there may be some time lag before relevant replication and evidence-synthesis studies emerge. For example, in biomedicine at least, there is a geometric growth in the number of meta-analyses, and in many fields multiple meta-analyses are often conducted once several studies appear on the same research question (Ioannidis, 2016).

The current study has several caveats and limitations. First, our findings are based on a random sample of 250 articles, and the obtained estimates may not necessarily generalize to specific contexts, such as other disciplines, subfields of psychology, or articles published in particular journals. However, this target sample size was selected to balance informativeness with tractability, and the observed estimates have reasonable precision. Second, although the focus of this study was transparency and reproducibility-related practices, this does not imply that the adoption of these practices is sufficient to promote the goals they are intended to achieve. For example, poorly documented data may not enable analytic reproducibility (Hardwicke, Bohn, et al., 2020; Hardwicke et al., 2018), and inadequately specified preregistrations may not sufficiently constrain researcher degrees of freedom (Claesen et al., 2019; Bakker et al., 2020). Third, we relied only on published information. Direct requests to authors may have yielded additional information; however, as noted earlier, such requests to research psychologists are often unsuccessful (Hardwicke & Ioannidis, 2018a; Vanpaemel et al., 2015; Wicherts et al., 2006). Fourth, a lack of transparency may have been justified in some cases if there were overriding practical, legal, or ethical concerns (Meyer, 2017). However, no constraints of this kind were declared in any of the articles we examined. Last, the study can gauge the prevalence of the assessed practices only during a particular time period. The effect of reform initiatives introduced after the examined time period, such as the founding of the Society for Improving Psychological Science (http://improvingpsych.org), will not be represented in our findings.

The current findings imply the minimal adoption of transparency and reproducibility-related practices in psychology during the examined time period. Although researchers appear to recognize the problems of low credibility and reproducibility (Baker, 2016) and endorse the values of transparency and reproducibility in principle (Anderson et al., 2010), they are often wary of change (Fuchs et al., 2012; Houtkoop et al., 2018) and routinely neglect these principles in practice (Hardwicke, Wallach, et al., 2020; Iqbal et al., 2016; Wallach et al., 2018). There is unlikely to be a single remedy to this situation. A multifaceted approach will likely be required, with iterative evaluation and careful scrutiny of reform initiatives (Hardwicke, Serghiou, et al., 2020). At the educational level, guidance and resources are available to aid researchers (Crüwell et al., 2019; Klein et al., 2018). At the institutional level, there is evidence that funder and journal policies can be effective at fomenting change (Hardwicke et al., 2018; Nuijten et al., 2017), and these initiatives should be translated and disseminated where relevant. Heterogeneous journal policies (Nutu et al., 2019) may currently be disrupting efforts to establish norms and promote better standards in routine practice. The Transparency and Openness Promotion initiative promises to encourage the adoption and standardization of journal policies related to transparency and reproducibility (Nosek et al., 2015), but it remains to be seen how effective this initiative will be in practice. Aligning academic rewards and incentives (e.g., funding awards, publication acceptance, promotion, and tenure) with better research practices may also be instrumental in encouraging wider adoption of these practices (Moher et al., 2018).

The current study is one of several to examine the prevalence of transparency and reproducibility-related research practices across scientific disciplines (Hardwicke, Wallach, et al., 2020; Iqbal et al., 2016; Wallach et al., 2018). Here, we have sketched out some of the topography of psychology’s territory. Additional studies will be required to fill in areas of the map that have yet to be explored and increase the resolution in specific areas (e.g., subfields of psychology). Future studies can also add a temporal dimension by comparing new data with the baseline established here, allowing us to explore the evolution of this landscape over time.

Supplemental Material