The perceived role of null results in school psychology research and publication

Abstract

Researchers appear to assume that published research is limited to significant findings. If that is the case, it may be related to perceived or actual publication bias (i.e., journals publishing only significant findings) and/or the file-drawer problem (i.e., researchers not pursuing publication of null results). The lack of published null results can result in faulty decision-making based upon incomplete evidence. Thus, it is important to know the prevalence of, and the contributing factors to, researchers' failure to submit null results. Few studies have addressed this issue in psychology and none have targeted school psychology. Consequently, this study examined the file drawer problem and perception of publication bias among school psychologists. Survey data from 95 school psychology faculty indicated that participants published about half of the studies that they had conducted, suggesting that the file drawer problem is experienced by this population. While lack of time appeared to impact publication pursuit, participants' responses also suggested they believed in publication bias. Obtaining null results substantially impacted the decision to write up studies in pursuit of publication. Therefore, it seems that a sizeable percentage of school psychology research is not available for review by researchers or practitioners.

Keywords

file-drawer problem null results publication bias school psychology

Research is often used to support decision-making in a variety of domains, including school psychology interventions. Although it is important for research to be available, it has been argued that some research is unavailable due to the file-drawer problem (Reysen, 2006; Rosenthal, 1979; Sterling, 1959) and publication bias (Cooper, DeNeve, & Charlton, 1997; Daniels, Montori, & Dupras, 2002; Olson et al., 2002; Walster & Cleary, 1970). These are viewed as potential sources of bias that may influence the conclusions drawn from the literature regarding important issues, such as intervention efficacy.

The terms publication bias and the file drawer problem differ in their focus. Publication bias involves a systematic preference for publishing statistically significant results in journals (Dickersin, 1990). It is defined as the biased outcome that occurs when researchers and editors systematically select studies with significant results over those with null results. The file-drawer problem, in contrast, occurs when nonsignificant findings are filed away by researchers rather than being disseminated to the larger research community (Rosenthal, 1979).

The reality is that it is rarely possible to retrieve all of the research results for one topic area. One study found that of almost 500 studies submitted to a professional medical meeting, 55% were not published (Weber, Callaham, Wears, Barton, & Young, 1998). Another found that 50% of all the studies approved by the Institutional Review Board at a large, midwestern university were not written up (Cooper et al., 1997).

Some hypothesize that failure to publish null results occurs when studies have flawed methodology (Ferguson & Brannick, 2012), which helps to maintain high methodological rigor for published research (Sigelman, 1999). As such, there really would not be publication bias against null results. Others speculate that articles with significant results are more likely to be accepted for publication regardless of rigor (Coursol & Wagner, 1986; Lane & Dunlap, 1978; Reysen, 2006; Sterling, 1959; Walster & Cleary, 1970); researchers have found that overestimating effect size increases the chances of publication (Lane & Dunlap, 1978). Published studies also have higher estimates of effect size than dissertation research (McLeod & Weisz, 2004). Additionally, Fanelli (2012) reported that the percentage of positive results in articles across all disciplines in the ISI-Essential Science Indicators database has been steadily increasing (70.2% in 1990–1991 versus 85.9% in 2007). This trend toward positive results was especially pronounced in psychology and psychiatry.

What is published could be misleading

As a body of research grows, the amount of research that is unavailable in the field also grows (Rosenthal, 1979). While this is not a problem in and of itself, researchers across many fields have suggested that bias can be introduced into scientific knowledge via differential publication of results (McLeod & Weisz, 2004). This potentially affects our estimations of relations between variables and the strength of treatment effects (Cooper et al., 1997; Fanelli, 2012). It has been suggested that even a small selection of unpublished results may significantly affect conclusions drawn from a body of literature (Howard et al., 2009; Scargle, 1999). Such an outcome was found by Turner, Matthews, Linardatos, Tell, and Rosenthal (2008) when examining published versus unpublished effect sizes of antidepressant medications. Effect sizes calculated from journal articles on 12 antidepressant drugs were found to be 11% to 69% higher than effect sizes drawn from research conducted by the Federal Drug Administration.

Further, when nonsignificant findings are not published, researchers may waste resources conducting studies that have already failed (Fanelli, 2012). Replication of failed studies is likely to introduce the potential for an inflated, yet unidentified risk of Type-I error. These errors occur when researchers incorrectly reject the null hypothesis though the null hypothesis was actually true (Atkinson, Furling, & Wampold, 1982; Lane & Dunlap, 1978). This limits our ability to determine whether significance is truly due to treatment effect or chance findings (Walster & Cleary, 1970).

Certain researchers argue that editors and reviewers responsible for selecting research for publication are less likely to publish articles with nonsignificant results (Howard et al., 2009). This argument, however, has not been supported in research on editorial decisions in the medical field. When four editors from JAMA were asked to choose manuscripts to be published, there was no significant difference between those with positive results (defined by the authors as statistically significant findings) and those with negative results (i.e., nonsignificant findings) (Olson et al., 2002). Additionally, there was no preference for significance found when medical faculty reviewed real and fake abstracts, then chose the ones they would submit for a conference presentation (Daniels et al., 2002). However, practices within the domain of psychology may be different. For example, Atkinson et al. (1982) asked reviewers from two major psychology journals to review fabricated manuscripts identical in all aspects except for the significance of results. The reviewers were less likely to recommend the publication of the nonsignificant manuscript than the manuscript with significant results (39% versus 82%, respectively). However, this is just one study and it is dated; additional exploration regarding publication bias in psychology fields is warranted.

Why researchers do not publish

Research supports the idea that, across disciplines, a large amount of the bias in what is published stems from researchers' decisions about whether or not to pursue publication, rather than editorial decisions regarding acceptance (Cooper et al., 1997; Coursol & Wagner, 1986; Dickersin, Min, & Meinert, 1992; Olson et al., 2002; Reysen, 2006; Weber et al., 1998). Dickersin et al. (1992) found that 90% of studies in their sample remained unpublished because the investigator chose not to write it up or submit it for publication. Rotton, Foos, and Van Meek (1995) found that 74.2% of researchers in their sample had filed away at least one article in their career and estimated that 15.39% of their sample's research had been filed away.

One reason that researchers decide not to pursue publication is a lack of statistical significance, or deemed importance, of the results (Cooper et al., 1997; Coursol & Wagner, 1986; Rotton et al., 1995). Cooper et al. (1997) found that that 50% of studies sampled were not written up in part due to noninteresting results; 31.3% were reported as having not been written up due to non-statistical significance. Of those that were written up, 22% were not prepared for journal submission due to nonsignificance. Overall, 74% of studies with significant results were submitted for publication in a book or journal, but only 4% of nonsignificant results were submitted for publication and 7% for presentation at a professional meeting. Reysen (2006) found that 11.7% of sampled studies with nonsignificant results were submitted for publication, although over half of those submitted were accepted.

Why, then, do some researchers abandon their projects that yield null results? When researchers' preconceived notions about nonsignificant results and their own submission practices were surveyed, researchers generally agreed that they failed to write up a non-statistically significant study because they believed that it was unlikely to be published (Reysen, 2006). Additionally, a majority of researchers believed there is an implicit, if not explicit, policy against non-statistically significant results at the editorial level. These beliefs, whether accurate or not, may contribute to researchers' failure to publish null results.

Much of the research addressing the issue of publication bias in psychology focuses on the potentially skewed conclusions that may be drawn from meta-analyses on topics affected by this bias. The focus of such meta-analytic research is commonly on statistical and visual tests for identifying and mathematically correcting for the bias (e.g., Howard et al., 2009; Rosenthal, 1979; Vevea & Woods, 2005). However, such corrections cannot be relied on to accurately account for the impact of the outcomes of studies that are not accessible (Cooper et al., 1997).

Furthermore, while much of the previous literature in psychology and in the social sciences is dated, Franco, Malhotra, and Simonovits (2014, 2016) investigated perceived publication bias and the file drawer problem more recently. Their findings indicated that in the social sciences, studies with null results are rarely written up (35%) and only 20% are eventually published (Franco et al., 2014). In evaluating psychology experiments, specifically, Franco et al. (2016) found that there was a substantial amount of underreporting of findings, particularly those that resulted in nonsignificant findings and lower effect sizes. Thus, it appears as though even when researchers choose to write up their findings, they omit some of the results in an effort to make their findings appear more favorable. This implies the presence of the file drawer problem and a perception by authors that publication bias exists.

Relevance to school psychology

In summary, few studies address the file drawer problem within the psychological world and there are no presently published research studies addressing this problem specifically for school psychologists. Given the emphasis placed upon using evidenced-based interventions in school psychology (as evidenced by resources provided by the International School Psychology Association, 2019 and National Association of School Psychologists, 2010) it is relevant to determine the extent to which school psychology researchers inhibit submission of findings when their results are nonsignificant. If we are to use evidence-based interventions in the field, it is critical that our evidence-base contains all results. If the published evidence for a particular intervention is biased toward statistically significant findings, we risk adopting practices whose effects are over-inflated in the literature. We also miss the opportunity to learn about potentially effective interventions that did not work initially, but might be altered to become more effective. In using evidence-based practice, we ideally want to have access to all of the evidence available, not just the evidence that supports the use of a particular practice. Additionally, concerns about publication likelihood with regard to significant outcomes may stifle creativity and innovation in school psychology research.

Current study

The goal of this research was to determine the extent to which school psychology studies are filed away and to assess the reasons school psychologists choose not to submit their research for publication. We set out to answer three questions: a) to what extent do school psychology researchers publish their research results, b) for what reasons do school psychology researchers fail to publish their work, and c) how do school psychology researchers view null results?

Method

Participants

Participants consisted of 95 faculty members in American Psychological Association (APA)-accredited or National Association of School Psychology (NASP)-approved school psychology programs in 31 states in the United States. Consistent with the field of school psychology, our sample included a majority of females (60 females, 33 males, and two did not report a gender). Participants worked as professors for a mean of 12.08 years (range 1–38, SD = 9.108) and as school psychologists for a mean of 4.39 years (range 0–25; SD = 4.62); 64% had tenure. The terminal degrees awarded in the programs for which participants worked were Master's degrees only (52.6%), Doctoral degrees only (10.5%), and both Master's and Doctoral degrees (36.9%). Nearly 64% described the training program they were currently working in as practice-based, whereas 36% described their program as research-based.

To select a representative sample of participants, faculty members were selected via systematic recruitment from each APA-accredited and/or NASP-approved school psychology program in the United States. Program websites were used to obtain email addresses for prospective participants. The goal was to send the survey to three faculty members in each program (if contact information was available for that many). When more than three were listed, the authors systematically chose faculty members from each program by alphabetizing them by their last name and then choosing every other individual. Upon moving to the next program, the authors began their selection of participants with the next letter of the alphabet after the one previously chosen. Surveys were emailed to 458 school psychology professors; 19 of the email addresses were no longer in service. Of the 439 who received the email, the response rate was 21.6%. Email responses from a small number of potential participants indicated a variety of reasons for lack of participation. For example, one individual indicated that she did not conduct research and eight did not identify as school psychologists.

Materials

A three-part questionnaire about research practices was developed, derived from previous surveys within this domain (Cooper et al., 1997; Reysen, 2006; Strein, Cramer, & Lawson, 2003; Weber et al., 1998). The first section asked demographic questions (i.e., gender, state of employment, and program characteristics). The second section asked participants to describe their research behaviors (i.e., research topic areas, methodological designs used, number of publications, and reasons they did not write up or submit research for publication). Finally, the questionnaire had eight Likert-scale questions (strongly disagree [1] to strongly agree [5] on a five-point scale) regarding beliefs about research publication. For example, participants were asked to identify whether they ‘believe journals are likely to publish null results’.

Procedure

A pilot study to refine the questionnaire was conducted with faculty in the fields of psychology, school psychology, counseling, and sociology at the authors' university. Based upon their feedback, minor wording changes were made. Then, participants were sent an email invitation to participate, containing information regarding the purpose of the study, informed consent information, and a link to the revised questionnaire via SurveyMonkey. Two follow-up emails were sent to encourage participation in the study.

Results

Data were first analysed descriptively. The amount of research that was published by participants was examined in order to get a sense of the overall proportion of studies likely to be filed away. Participants reported having completed data-collection and analysis for a mean of just over 23 studies, ranging from 0 to 200 (SD = 34.17; Mdn = 12). The amount of research conducted was compared to the amount of research actually published. The number of published studies ranged from 0–150, with a mean of just over 13 (SD = 24.02, Mdn = 5). Respondents were found to have published an average of 52% of the research that they produced, indicating that more studies were conducted than were published within this sample. There was, however, much variation across participants (ranging from 0% to 100%, SD = 27.16). Specifically, 86.2% of the sample reported filing away at least one research study in the course of their career. Consequently, it is clear that not all research in the field is available.

Experimental and quasi-experimental research designs were the most common, used by a total of 40% of the sample. Particularly relevant to evidence-based intervention, 23.2% of respondents reported that their primary area of research was either academic or therapeutic interventions. This number, however, is likely to underestimate research in interventions conducted by the sample, as virtually all of the categories listed could potentially include intervention (e.g., research in remedial education).

Reasons for not presenting or publishing studies

In order to further examine the reasons that school psychology researchers decide to abandon their research projects, participants were asked to indicate reasons for failure to pursue studies at two decision points: 1) after having collected and analysed the data, without producing any write-up of the work for dissemination, and 2) after having written about the study in some form, but prior to submitting for publication. With regard to the first decision-point, 64 participants (67.4%) indicated that they had worked on research projects for which they had completed data collection and analysis, but then chose not to write up in any form (e.g., presentation). This subset of participants was then asked to indicate all of the reasons that contributed to their failure to write up research results. Not having enough time was the most prevalent obstacle to writing up research, with 81.3% of respondents (N = 52) selecting this reason. The second most prevalent reason was thinking that the research would not be accepted by journals (45.3%), followed by having obtained null results (26.6%). Design or operational problems were indicated by 25% of respondents, and results that approached, but did not reach, significance was selected by 15.6%.

In terms of the second decision-point, 56.9% of respondents (N = 54) reported writing up studies that were not submitted for publication. Again, these participants endorsed reasons that contributed to this decision. Not enough time was selected by 55.6%, and 33.3% indicated that they did not think that the manuscript would be accepted by publishers. As presented in Table 1, common reasons related to not writing up a study in any form, such as nonsignificant results and design or operational problems, were less prevalent at this second juncture.

Table 1.

Reasons why researchers did not write up or submit studies.

	Did not write up		Did not submit
Reason	N	Percent	N	Percent
Not enough time	52	81.3	30	55.6
Thought it would not be accepted for	29	45.3	18	33.3
publication
Non-significant results	17	26.6	5	9.3
Design or operational problems	16	25	5	9.3
Publication was not an aim in the project	14	21.9	13	24.1
Problems with others (e.g., research assistants)	10	15.6	7	13
Results approached, but did not reach	10	15.6	4	7.4
significance
Too much trouble	7	10.9	5	9.3
Small effect size	4	6.3	5	9.3
Similar papers were already published	4	6.3	4	7.4
Results inconsistent with hypotheses	3	4.7	0	0
Discouraged by others	2	3.1	2	3.7
Failure to replicate previous findings	1	1.6	0	0
Other Reasons	1	1.6	1	1.9

Note: Participants could endorse as many reasons as they deemed appropriate; N = number of participants.

Researchers' beliefs about publication

Participants were also presented with questions intended to elicit ratings of their beliefs about publication and its pursuit (results are reported in Table 2). Notably, 79% of respondents agreed or strongly agreed with the statement, ‘I believe journals are less likely to publish null results’. Similarly, 76.8% of participants agreed that they believe journals should publish studies with null results. Despite this, 85.2% agreed that they would be more likely to pursue publication of a study with significant findings. These responses suggest that school psychology researchers’ decisions about pursuing publication of their work are biased toward significant findings, with this decision seemingly influenced by a widely-held belief that journals are less accepting of null results.

Table 2.

Researcher beliefs about publication practices.

Statement	Mean Rating	Strongly Agree	Percentage of respondents		Disagree	Strongly Disagree
Statement	Mean Rating	Strongly Agree	Agree	Neither Agree Nor Disagree	Disagree	Strongly Disagree
“I would be more likely to pursue publication of significant results.”	4.18	36.8	48.4	7.4	1.1	3.2
“I believe journals are less likely to publish null results.”	4.01	29.5	49.5	9.5	6.3	2.1
“I believe journals should publish null results.”	3.92	20	56.8	11.6	6.3	1.1
“I would look more closely at methodology when null results are obtained.”	3.62	8.4	52.6	23.2	10.5	0
“I would be more likely to pursue publication if there were a suggestive trend.”	3.48	5.3	53.7	24.2	9.5	4.2
“I would collect more data if I obtained results that approached, but did not reach significance.”	3.26	4.2	40	31.6	16.8	3.2
“I would pursue publication of a study rejected from a conference.”	3.11	8.4	34.7	20	26.3	7.4
“I would be discouraged from submission if my research received negative reviews.”	2.65	1.1	25.3	18.9	42.1	9.5

Pearson correlations were used to examine whether decision-making regarding null results was related to ratings on the Likert scale items. All significant correlations found were between decisions about not writing up results at all and researcher beliefs, rather than between pursuing written publication and beliefs.

A positive relationship was found between not writing up a study because of nonsignificant results and endorsement of the statement, ‘I would be more likely to pursue publication of significant results’ (r₆₁ = 0.30; p = 0.017). Another positive relationship was found between the belief that journals are less likely to publish null results and endorsement of the statement, ‘I would be more likely to pursue publication of significant results’ (r₉₀ = 0.43; p < 0.001). These findings suggest a connection between beliefs and decision-making.

Not writing up a study because of results that approached but did not reach significance was positively correlated with the belief that journals are less likely to publish null results (r₆₁ = 0.25, p = 0.048), and was negatively correlated with the statement, ‘I would collect more data if results approached, but did not reach, significance’ (r₆₁ = −0.35, p = 0.005). The latter finding suggests that a lack of willingness to collect more data when results approach significance relates to researchers abandoning projects prior to writing up a study at all. In contrast, researchers who are more willing to collect additional data if their results approach significance are less likely to abandon research projects at the first decision point.

To further compare the impact of null results at both decision points, a paired samples t-test was employed. The comparison was significant (t94 = 2.78; p = 0.007). Thus, receiving null results was significantly more prevalent a reason for deciding not to write up results in any form (M = .18) than for deciding not to submit for publication a study was already written up (M = 0.05).

Discussion

The present results help to paint a clearer picture of the reasoning behind school psychology faculty members' decisions to publish research. The findings suggest that the file drawer problem is at play; the majority of participants have filed away at least one study. On average, researchers published approximately half of the research that they conducted. When it comes to deciding whether or not to write up research, believing that journals will not accept their work seems to play an important role, though researchers may not have evidence that their perceptions are true.

Obtaining null results played a large role in decisions about whether to write up a manuscript at all, indicating that researchers may abandon projects with null results prior to dissemination of the findings in any format. These findings in school psychology are similar to findings from earlier research in psychology in general (e.g., Cooper et al., 1997; Coursol & Wagner, 1986; Reysen, 2006; Rotton et al., 1995). A substantial portion of the research completed by this sample is in the domain of intervention; thus, it is likely that some interventions that are viewed as having a strong evidence-base by practitioners also have contradictory or null findings that are not published. Withholding null results from the research pool poses a danger to our knowledge-base in that it allows the potential for an inflated, yet largely unrecognized risk of Type-I error. Thus, it has the potential to affect our estimations of relation between variables and of the effect size of treatments.

These findings call attention to the potential for bias within school psychology literature and highlight the importance of finding a solution to the file-drawer problem in this field. Literature in other fields offers various ideas about how to resolve the problem. Some suggest the creation of research registers consisting of data from unpublished research studies (Cooper et al., 1997; Howard et al., 2009; McLeod & Weisz, 2004; Scargle, 1999). Another solution could be the development of journals and databases devoted to null results, such as The Journal of Articles in Support of the Null Hypothesis (see Reysen Group, 2015), which is devoted to publishing null findings from methodologically rigorous psychology studies (Reysen Group, 2015). Others believe in a need to abandon traditional inferential statistics for Bayesian statistics (Howard et al., 2009; Scargle, 1999), an approach that focuses on probability rather than significance (Kruschke, 2010). Another possible solution is to change the process for publication. Rather than submitting an entire manuscript to a publisher, it has been suggested that researchers might withhold the results and discussion sections of their studies (Howard et al., 2009; Lane & Dunlap, 1978; Walster & Cleary, 1970). Doing so would force editors to make decisions based on relevance and methodology.

Decision-making about publishing

Participants generally believed that journals should publish null results; however, they also believed journals are less likely to publish null results. These beliefs, held by nearly 80% of the participants, seem likely to contribute to reluctance to pursue publication of nonsignificant findings. Therefore, regardless of actual editorial preferences, these perceptions of editorial decision-making seem to promote filing away research with null results.

Also notable is that not enough time was the most frequently endorsed reason for failing to write up or pursue publication. Of the participants who abandoned research projects prior to writing them up at all, 81.3% endorsed not enough time as a reason. This is in contrast with 55.6% of participants who decided to abandon projects before manuscript submission for the same reason. The percentage of those who didn't write up studies because it wasn't their initial aim is much less than those who didn't write up due to not having enough time. Thus, despite having apparent intentions of disseminating their research, it appears that the time demands imposed by the manuscript creation and submission processes is a barrier. Inherent barriers that exist within both the writing process (e.g., APA-style) and the submission process (e.g., journal specifications), while necessary, take significant time. It seems that these formal requirements have presented an obstacle for many researchers in disseminating their findings. In light of the time requirements of publication, researcher beliefs about editorial preferences, and the pressures to publish associated with academia, it is not surprising that school psychology researchers may prefer to write up and submit significant findings.

When researchers believe that publication is primarily for significant findings, there can be other implications for how they collect their data and interpret their results. It is possible that researchers themselves find more methodological flaws in their studies that fail to attain significance than those that reject the null, regardless of actual methodological quality. A small number of previous studies indicate that non-statistically significant research manuscripts have been more critically viewed by both editors (Atkinson et al., 1982) and practicing psychologists (Cohen, 1979) in terms of methodology than manuscripts with significant results. This could contribute to fewer attempts to submit non-statistically significant results to journals for publication, under the guise of methodological flaws identified by authors rather than editors.

Other potential responses to the pressure to reject the null hypothesis are additional sampling, statistical analyses, and other techniques that help researchers to move their results into the domain of statistical significance when they are approaching the 0.05 level (Gerber & Malhotra, 2008; Walster & Cleary, 1970). The results of the current study indicate that a concerning number of participants would consider engaging in additional sampling under these circumstances. Such activities pose serious risk for contributing additional error into the literature, and are particularly concerning since they seem to be a direct result of perceived publication bias and pressure to attain significance.

Limitations and future research

Limitations of this study include a modest sample size and low response rate. Additionally, this study sampled only school psychology faculty members at NASP-approved and APA-accredited institutions for higher education in the United States. Assessment of school psychologist researchers practicing in other geographical contexts would be beneficial. Future research should explore more directly the relationship between researcher beliefs about null results and their publication practices. For example, the survey asked about researchers' overall decision-making, but did not address reasons for not publishing specific types of projects.

Future research should also investigate the perceptions and practices of journal editors. The participants tend to believe that editors are less likely to publish null results. However, this study did not survey editors or journal reviewers, and consequently, the findings are only based upon the perceptions of researchers. Therefore, it is important to know whether publication bias actually exists at the editorial level. It may be the case that the studies with null findings that are rejected are plagued by methodological and statistical problems rather than experiencing publication bias. Providing researchers with data regarding actual editorial decision-making could contribute to a change in beliefs, as well as a change in submission practices. For example, if it were known that publication bias did not exist, and researchers believe they used sound methodology, they may be more likely to submit research with nonsignificant findings. Related, assessment of the existence of publication bias in the bodies of literature from which evidence-based practice in school psychology draws would allow practitioners to better evaluate the literature.

Finally, it is important that school psychology address the obstacles to dissemination that prevent researchers from sharing their findings. Low time-investment, easy-to-use databases that allow for researchers to make their methods and results public seem to have the potential to vastly improve the comprehensiveness of the available literature and the accuracy of estimations of effects.

Implications

In summary, the vast majority of school psychology researchers reported believing that journals are less likely to publish articles with null results, and reported that they would be more likely to pursue publication of significant results. A vast majority also agreed that journals should publish null results. This suggests that although researchers see the value of including null results in the body of published research, they believe that journals are biased against publishing research with null results, and as such, selectively pursue publication of significant results. It is important to note that these decisions are based upon researchers' perceptions, rather than evidence that journal editors are actually biased against null results. In addition to the issue of null results, there seems to be a significant amount of research that is not published for other reasons, such as not enough time. Taken together, a significant amount of research that has been conducted in the field of school psychology is not available to those who may benefit from access to it. This is problematic given the current state of the school psychology field and its focus on evidence-based practices. If we are not aware of the full extent of data collected, especially with regard to interventions, we may hinder our ability to support students in an optimal way. For example, one could implement an intervention under the guise that is effective, when in fact, null findings indicating otherwise have been filed away. Consequently, it is important for the school psychology field to address the perceptions of researchers to ensure that more of the research that has been conducted is available to researchers and practitioners.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Note

This study (Proposal #2011-11) was approved by the Human Subjects Research Committee at Alfred University, New York.

Jessica S. Iwachiw received her BA degree in psychology from State University of New York at Geneseo, as well as her MA, CAS, and PsyD in school psychology from Alfred University. She has worked as a school psychologist in the Douglas County School District in Colorado since 2014. Her research interests include equine-assisted psychotherapy and the human-animal bond.

Amy Lynn Button received her BA degree in psychology from St Bonaventure University and her MA, CAS, and PsyD from Alfred University. She served as a school psychologist for the Dansville Central School District and has done preschool-level psychoeducational assessments for area school districts. She is currently a professor in the Department of Psychology at Alfred University. Her primary research interests include services and outcomes for students with disabilities.

Jana G. Atlas received her BA degree from SUNY Binghamton, MSW from SUNY Albany, and PhD from Wayne State University. Her experience includes working as a psychologist for Tioga County Mental Health Clinic, Owego, NY and maintaining a private counseling practice. She is currently a Professor in the Division of Counseling and School Psychology at Alfred University. Her research interests include eating disorders, animal-assisted therapy, and students with disabilities in higher education.

References

Atkinson

D. R.

Furlong

M. J.

Wampold

B. E.

(1982) Statistical significance, reviewer evaluations, and the scientific process: Is there a (statistically) significant relationship?. Journal of Counseling Psychology 29(2): 189–194. doi: 10.1037/0022-0167.29.2.189.

Cohen

L. H.

(1979) Clinical psychologists' judgments of the scientific merit and clinical relevance of psychotherapy outcome research. Journal of Consulting and Clinical Psychology 47: 421–423. doi: 10.1037/0022-006X.47.2.421.

Cooper

DeNeve

Charlton

(1997) Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods 2: 447–452. doi: 10.1037//1082-989X.2.4.447.

Coursol

Wagner

E. E.

(1986) Effect positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology: Research and Practice 17(2): 136–137. doi: 10.1037//0735-7028.17.2.136.

Daniels

C. E.

Montori

V. M.

Dupras

D. M.

(2002) Effect of publication bias on retrieval bias. Academic Medicine 77(3): 266 doi: 10.1097/00001888-200203000-00019.

Dickersin

(1990) The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 263: 1385–1389. doi: 10.1001/jama.1990.03440100097014.

Dickersin

Min

Meinert

C. L.

(1992) Factors influencing publication of research results: Follow-up of applications submitted to two institutional review boards. Journal of the American Medical Association 267: 374–378. doi: 10.1001/jama.1992.03480030052036.

Fanelli

(2012) Negative results are disappearing from most disciplines and countries. Scientometrics 90: 891–904. doi: 10.1007/s11192-011-0494-7.

Ferguson

C. J.

Brannick

M. T.

(2012) Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analysis. Psychological Methods 17: 120–128. doi: 10.1037/a0024445.

10.

Franco

Malhotra

Simonovits

(2014) Publication bias in the social sciences: Unlocking the file drawer. Science 345 doi:10.1126/science.1255484.

11.

Franco

Malhotra

Simonovits

(2016) Underreporting in psychology experiments: Evidence from a study registry. Social Psychological and Personality Science 7: 8–12. doi: 10.1177/1948550615598377.

12.

Gerber

A. S.

Malhotra

(2008) Publication bias in empirical sociological research: Do arbitrary significance levels distort published results?. Sociological Methods and Research 37: 3–30. doi: 10.1177/0049124108318973.

13.

Howard

G. S.

Lau

M. Y.

Maxwell

S. E.

Venter

Lundy

Sweeny

R. M.

(2009) Do research literatures give correct answers?. Review of General Psychology 13(2): 116–121. doi: 10.1037/a0015468.

14.

International School Psychology Association. (2019). Evidence-based and behavioral interventions. Retrieved from https://www.ispaweb.org/resources/evidence-based-behavioral-intervention/.

15.

Kruschke

J. K.

(2010) Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science 1(5): 658–676. doi: 10.1002/wcs.72.

16.

Lane

D. M.

Dunlap

W. P.

(1978) Estimating effect size: Bias resulting from the significance criterion in editorial decisions. Britain Journal of Mathematics and Statistical Psychology 31: 107–112. doi: 10.1111/j.2044-8317.1978.tb00578.x.

17.

McLeod

B. D.

Weisz

J. R.

(2004) Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Counseling and Clinical Psychology 72: 235–251. doi: 10.1037/0022-006X.72.2.235.

18.

National Association of School Psychologists (2010) Standards for graduate preparation of school psychologists, Bethesda, MD: Author. Retrieved from www.nasponline.org.

19.

Olson

C. M.

Rennie

Cook

Dickersin

Flanagin

Hogan

J. W.

et al. (2002) Publication bias in editorial decision making. Journal of the American Medical Association 287: 2825–2828. doi: 10.1001/jama.287.21.2825.

20.

Reysen

(2006) Publication of nonsignificant results: A survey of psychologists' opinions. Psychological Reports 98: 169–175. doi: 10.2466/pr0.98.1.169-175.

21.

Reysen Group. (2015). About JASNH. Retrieved from http://www.jasnh.com/about.html.

22.

Rosenthal

(1979) The ‘file drawer problem’ and tolerance for null results. Psychological Bulletin 86: 638–641. doi: 10.1037/0033-2909.86.3.638.

23.

Rotton

Foos

P. W.

Van Meek

(1995) Publication practices and the file drawer problem: A survey of published authors. Journal of Social Behavior & Personality 10(1): 1–13. Retrieved from http://psycnet.apa.org/psycinfo/1995-34825-001.

24.

Scargle, J. D. (1999, March). Publication bias (The ‘file-drawer problem’) in scientific inference. Symposium conducted at the Stanford University Sturrock Symposium, Stanford, CA. doi: arXiv:physics/9909033.

25.

Sigelman

(1999) Publication bias reconsidered. Political Analysis 8: 201–210. doi: 10.1093/oxfordjournals.pan.a029813.

26.

Sterling

T. D.

(1959) Publication decisions and their possible effects on inferences drawn from tests of significance – or vice versa. American Statistical Association Journal 54(285): 33–34. doi: 10.1080/01621459.1959.10501497.

27.

Strein

Cramer

Lawser

(2003) School psychology research and scholarship. School Psychology International 24: 421–436. doi: 10.1177/01430343030244005.

28.

Turner

E. H.

Matthews

A. M.

Linardatos

Tell

R. A.

Rosenthal

(2008) Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 358: 252–260. doi: 10.1056/NEJMsa065779.

29.

Vevea

J. L.

Woods

C. M.

(2005) Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods 10: 428–443. doi: 10.1037/1082- 989X.10.4.428.

30.

Walster

G. W.

Cleary

T. A.

(1970) A proposal for a new editorial policy in the social sciences. The American Statistician 24(2): 16–19. doi: 10.1080/00031305.1970.10478884.

31.

Weber

E. J.

Callaham

M. L.

Wears

Barton

Young

(1998) Unpublished research from a medical specialty meeting: Why investigators fail to publish. Journal of the American Medical Association 280: 257–259. doi:10.1001/jama.280.3.257.