Animal researchers’ views on the publication of negative results and subsequent policy adoptions

Introduction

Scientists have identified the lack of publication of negative results of preclinical studies as a primary problem in biomedical research for over a decade.^1,2 The low dissemination of studies presenting negative findings has partially been blamed for the reproducibility crisis in biomedical research ³ and has consequently been highlighted in editorials in prestigious journals such as Science ⁴ and Nature ⁵ as an important scientific issue to address. Publishing negative results is crucial for mitigating publication bias, which occurs when positive findings are disproportionately represented in literature. This bias skews the scientific record, making it difficult to form accurate conclusions and often leading to wasteful redundancy in research efforts. By including negative outcomes, the scientific community gains a more comprehensive understanding of research topics, facilitating progress by acknowledging what does not work. This approach fosters a culture of openness, improves resource allocation and upholds the integrity of scientific inquiry by presenting a balanced view of research findings.

In animal research, publishing negative results is especially important to avoid unnecessary repetition of experiments, ensuring the ethical use of animals. It contributes to the refinement and reduction of animal use by informing the scientific community about methodologies or hypotheses that have been disproven, thus guiding future research toward more promising directions. This practice enhances the welfare of animals used in research by maximizing the value of the data obtained from each experiment and adhering to the principles of responsible scientific conduct.

However, negative studies are still seldom published in regular journals, and dedicated journals such as the Journal of Negative Results in BioMedicine are closing due to the low number of submissions. The clinical research community has recognized the challenge of finding suitable platforms for publishing negative results. An online tool, as mentioned by Bernard et al., ⁶ assists researchers in locating appropriate venues for such findings. However, only general-purpose repositories like Zenodo, Dryad or OSF.io often offer free avenues for submitting negative outcomes, underscoring the need for more dedicated spaces in scientific publishing landscapes.

Materials and methods

Wieschowski et al. ⁷ highlight that approximately one-third of animal studies go unreported, with van der Naald et al. ⁸ adding that around 74% of animals used in preclinical research are never mentioned in publications. This indicates a significant gap in the reporting of animal research findings. Given this situation, we asked the biomedical research community about their perceptions of publishing negative results. In particular, we were interested in determining how frequently researchers generate negative results, their opinions on the key causes for them, why they are typically not published, how they think they should be published, and the advantages and disadvantages of making them public. We used the EUSurvey platform to survey more than 200 researchers directly working with animals. (The survey transcript, its synopsis, study results and de-identified survey data are publicly available on the Open Science Framework, https://osf.io/9p7kz/)

The survey was distributed to all members listed in the ‘Members’ section on the website of the Federation of European Laboratory Animal Science Associations. In addition to this, we informed specific contacts about our survey: 12 in Sweden, 11 in Spain, seven in the UK, two each in Norway and the Netherlands and one each in Portugal, Ireland, Italy, Belgium and Germany. A total of 237 researchers completed the survey in its entirety (of a total of 241 respondents). The detailed profile of respondents is available on the Open Science Framework (OSF) link mentioned above. The most common profile of respondents was that of female academic researcher (59%), either established (32%) or in a leading position (29%), predominantly working in Spain or Germany (40%) and mainly dealing with mice (80%) or rats (46%).

Results

On average, respondents estimated that about one-third of their experiments finished with negative results, and that of these experiments, less than one-quarter could be published (generally along with other experiments yielding positive results). Both facts together imply that about one in four experiments are never reported. These numbers agree with those reported in Wieschowski et al. and van der Naald et al.^7,8

As a result, our survey shows that most biomedical researchers (about 95%) think that the animal research community should promote sharing research data and reports from negative experiments. From now on, we will put the percentage of respondents supporting a given statement in parentheses. Most questions were answered using a five-point Likert scale, but we typically condensed them into a three-point scale to ease the presentation of the results. Readers who are interested in the detailed results can refer to the OSF link provided above. A study summary is shown in Appendix 1 at the end of this article. The most common arguments for the promotion of sharing included:

Avoiding repetitions of experiments (49%) that have already been carried out elsewhere but are unknown because their results have not been made public will reduce the money, time and animal lives spent on redundant research (32%).

About 10% of the researchers raised concerns about the validity of having access to negative results. They argued that a negative result could be due to a bad experimental design, important flaws in the execution of the experiment, or its statistical analysis. Some researchers questioned the usefulness of having such negative results available (less than 5%).

According to the surveyed researchers, the leading causes of negative results are a true lack of difference between groups or correlation between variables (80%), the variability of the observed variables (which may be underestimated before experimenting (80%)), a low sample size (67%), animal, environmental or other unknown factors (66%), execution, methodological or procedural mistakes (38%), the lack of control of other external variables (33%) and the gap between in vitro and in vivo experiments (27%).

The reasons argued for not publishing negative experiments included the general belief that papers reporting negative results are not accepted by journals (especially high-impact journals (93%)), that even if they are published, they do not get a large number of citations (72%) and that there are no institutional or community incentives to publish (67%). This is combined with pressure to publish positive results (publishing negative results would take precious time and energy from them without them being obliged to do so (58%)). Additionally, there is a lack of a culture of publishing negative results (94%), with researchers afraid of losing their scientific reputation and future grant opportunities (58%). Ultimately, there is always the honest doubt that a negative result reflects an absence of differences between groups or correlation between variables, or the absence of positive results is due to an experimental or design mistake hiding the true positive result underneath (49%). The presence of positive and negative controls in the experiment is critical to determine whether the experiment was well conducted. It is essential to recognize that the questions in this section were phrased as, ‘What are the reasons you think other researchers do not publish their negative results?’, which is a more indirect approach than directly asking, ‘Why do you not publish negative results?’. This indirect method of questioning depersonalizes the query, which can frequently lead to more honest and accurate responses.

Based on career status, established (R3) and leading (R4) researchers chose not to publish negative results at a higher rate (56–59%) compared with student (R1) or early-career (R2) researchers (40–41%). This difference was statistically significant with an analysis of variance p-value of 0.035. However, no noticeable differences were observed between genders concerning this choice.

Researchers favour publishing negative results in a platform that is easily accessible for discovery (89%), offers citable content (80%), ensures quality control before publication (79%), is user-friendly for navigation and submission (78%), allows free publishing (77%), supports full papers in peer-reviewed journals (70%), is affiliated with a reputable institution (66%), includes peer-review processes (58%) or provides a non-peer-reviewed, free-format repository (57%). They also value platforms that enable comments (57%) and are part of an open-science database (52%).

Researchers agree that funding agencies and scientific institutions should require that negative results be published (79%). They also agree that scientific societies should promote this practice (97%). However, there was no agreement on whether this decision to publish negative results or not should be left to the researcher (50%). There was also no agreement either on whether the laboratories or the individual researchers should be rewarded in some way for publishing negative results (less than 40% in all related questions).

When asked how they would prefer to share negative results, participants indicated that they preferred writing a full paper and submitting it to an open-access, peer-reviewed journal without any publishing cost (70%). For them, it would be important that this journal is hosted at some recognized institution (66%) and that it allows comments from other researchers (57%).

Discussion

From this survey, we conclude that: 1) scientists consider the sharing of negative results to be essential for science, society, the 3Rs, and an ethical aspect of research; 2) they are willing to do it with a standard as high as for positive results; 3) but they see severe drawbacks, mainly in the scientific culture, publishing options and lack of institutional or community incentives. Despite the encouragement by scientists and scientific journals of the publication of negative results, in practice, all these drawbacks result in a deficient proportion of negative results being published and a quarter of experiments never being reported.

The fact that established (R3) and leading (R4) researchers exhibit a greater tendency not to disclose results compared with their student (R1) or early-career (R2) counterparts may have several reasons: 1) a higher likelihood of encountering negative or inconclusive results throughout their careers, 2) the conduct of preliminary or pilot studies that may not yield definitive outcomes, 3) reduced publication pressure, given their established status, 4) a distinct approach to sharing findings, reflective of generational or disciplinary cultures. Additional considerations might also play a role, such as: 5) a greater involvement in confidential or proprietary research, limiting the scope for public disclosure, 6) a more strategic approach to publication, choosing to report only the most impactful findings, 7) the potential for more complex, long-term projects that result in less frequent but more significant publications, and 8) a possible perception that specific findings do not add substantially to the existing body of knowledge, thereby choosing not to report them.

Concerning the publication of negative results, Heinl et al. ⁹ suggested adopting a more accepted format called ‘registered reports’ in psychology. Under this format, a study plan is submitted to a journal before starting the research, it is then peer-reviewed and, if accepted, the results are guaranteed to be accepted irrespective of whether they are positive, negative or inconclusive. This could certainly be an option. However, it is balanced against concerns that some researchers may use the negative results to inflate their CV.

Although there could be situations where the results of negative experiments may be problematic, we believe that proper reporting of the hypotheses, experimental design and the execution of such experiments would help distinguish experiments with significant flaws from those underpowered or whose results appear to be truly negative. In this regard, positive and negative controls in the experiment help to identify experiments with execution problems that invalidate their conclusions.

Still, it is essential to acknowledge that the findings of this study are primarily based on self-reported impacts and impressions of the participants, which may differ from objectively measured outcomes. This reliance on subjective reports rather than quantifiable data could potentially influence the accuracy and generalizability of the results.

Now, it is the turn of legislators, funding agencies, institutions, scientific societies and ethical committees to provide practical ways to promote the publication of negative results. The mandate for data management and sharing plans represents a move towards greater openness in scientific research. However, these plans are not tailored specifically to animal research and, in practical terms, they are still in their infancy. Specific measures could be undertaken, like requiring registration of animal experiments before data collection and subsequent follow-up of the results, providing practical ways to publish negative results, or adopting appropriate incentives for the laboratory or individual researcher.

Medical research is well ahead of animal research concerning the publication of negative results. ¹⁰ Many relevant institutions have already recommended the need for reporting negative results of clinical trials: the International Committee of Medical Journal Editors, World Health Organization and Committee on Publication Ethics. ¹¹ Despite this strong support, it has also been an essential bias towards positive results by scientific editors and researchers themselves. ¹¹ The main difference between clinical trials and preclinical studies is probably in the mandatory requirement to register all clinical trials in centralized registries controlled by government agencies (e.g. clinicaltrials.gov in the case of the Food and Drug Administration, or clinicaltrialsregister.eu in the case of the European Medicines Agency). When compared with animal alternatives such as the Preclinicaltrials.eu, ¹² animalstudyregistry.org, ¹³ we observe two important differences: 1) in clinical trials, there is an exhaustive tracking of the status of the trial (not yet recruiting, recruiting, withdrawn, completed, terminated, etc.) and the dates in which they change; 2) in clinical trials there is an explicit tracking of the outcome of the experiment either provided by the applicant or automatically matched by the agency through the registration numbers reported in scientific articles. Some stakeholders point to a potential drawback of preclinical registries, which is the possibility of scooping some studies and giving advantages to competitors. To address this concern, registries such as the OSF allow researchers to apply an embargo period during which the information in the registry is not made public or by making the information public after the consent of the main investigator. ¹⁴ This is not a problem in clinical trials because patents protect drugs.

Registries such as clinicaltrials.gov and clinicaltrialsregister.eu currently hold 459,627 and 43,630 studies, respectively. Compared with their preclinical counterparts, at the time of writing, we see 149 registered studies in animalstudyregistry.org and 127 in preclinicaltrials.eu.

This difference is even more striking if we consider that many preclinical studies never reach the clinical stage. The discrepancy can be attributed to two main factors. First, registering clinical trials has been established for a more extended period compared with preclinical studies. Second, the registration of clinical trials is mandatory, whereas it remains voluntary for preclinical studies. Despite the potential impact of animal registries^1,14,15 on improving many aspects of preclinical animal research, after four years of experience, they have not yet had any significant impact.

In summary, we believe that the four keys to improving the culture of sharing negative results in preclinical research with animals are: 1) the obligation to preregister studies, 2) a thorough tracking of their status, 3) of their results, and 4) a realistic way of publishing negative results. Despite the possible inconveniences and drawbacks of such a practice, achieving this would result in important scientific, ethical and economic benefits for society and science.