Implementation of the NIH Sex-Inclusion Policy: Attitudes and Opinions of Study Section Members

Introduction

Sex as a biological variable has been systematically excluded from fundamental research and clinical studies ^{1 –3} until two landmark injunctions by the National Institutes of Health (NIH). The first of these policy changes was mandated by Congress in 1993 through the NIH Revitalization Act, which required the inclusion of females in NIH-funded clinical research. ⁴ After several decades of developing best practices to improve the inclusion and retention of women in clinical research, male and female participants are approaching parity. ⁵ However, no such policies governed the inclusion of males and females in basic science or preclinical research.

Following years of advocacy, ^2,6 the NIH issued a notice in 2016 requiring NIH grantees to “consider sex as a biological variable (SABV)” (NIH NOT-OD-15-102). ⁷ Unlike the federal mandate for inclusion, the more recent policy asks only for justification and/or description of sex as part of the ordinary scientific report. The anticipation is that when investigators “consider” sex as a biological variable, they may be more inclined to “study” sex in the context of signaling pathways and physiological research, and in disease settings.

While commentaries both for and against the policy exist, ^{8 –11} data surrounding its implementation are lacking. ¹² In this study, we examined the attitudes of NIH study section members as a large demographic of individuals charged with the review of the policy as part of grant evaluation, and who are themselves grantees, thus providing a proxy for investigators across all disciplines of science. A survey was administered after the initial round of grant reviews in which applicants were required to consider SABV and 1 year later, thus providing the framework to assess change in comprehension and compliance. In aggregate, the data indicate an increasing approval of the policy along and improved compliance by investigators.

Materials and Methods

Results

A total of 1161 study section members, representing over 175 standing study sections and continuing special emphasis panels, participated in the 2016 and 2017 surveys (Table 1). Briefly, the majority of respondents were white (80% vs. 81%) males (51% vs. 56%) employed in academia (96% vs. 97%), who served as permanent members of study section (58% vs. 56%) with 3 years or more experience on study section (76% vs. 75%).

Next, we queried respondents on their awareness, understanding, and attitudes toward the SABV policy (Fig. 1 and Supplementary Tables S1–S3). The majority of respondents became aware of the SABV policy through the NIH (60% vs. 57%) and not through their home institution (13% vs. 12%, Supplementary Table S1). In 2016, the majority of respondents felt confident in their understanding of the SABV policy (83.5%), which increased significantly to 88.4% by 2017 (p = 0.018, Fig. 1 and Supplementary Table S3).

OPEN IN VIEWER

FIG. 1.

Comparison of respondents' perceived comprehension and attitudes toward the SABV policy between 2016 and 2017. SABV, sex as a biological variable.

In both years, over 70% of respondents indicated that the NIH provided adequate educational resources regarding the SABV policy (70.8% vs. 74.5%; p = 0.165), whereas the majority reported that they were unaware of or were not provided any educational resources by their home institutions (75.2% vs. 75.6%; p = 0.886, Supplementary Table S2). In both years, more than 90% of the respondents indicated that they discussed the policy at their respective study section meetings (92.2% vs. 92.3%; p = 0.951, Supplementary Table S2).

To assess study section members' attitudes toward the SABV policy, we asked if it was important to consider SABV in the experimental design of NIH-funded research, and if it would improve rigor and reproducibility (Fig. 1). These two questions highlight a direct application and goal of the policy, respectively. A majority of respondents thought that it was important to consider SABV in the experimental design (63.4% vs. 67.6%; p = 0.141) and that considering SABV would improve rigor and reproducibility (54.0% vs. 5%; p = 0.208, Supplementary Table S3). In both years, respondents whose research portfolios involve the study of sex differences were significantly more likely to think that the consideration of SABV is important (73% vs. 59% in 2016; p = 0.0037, and 77% vs. 63% in 2017; p = 0.0001) and would improve rigor and reproducibility compared to their peers (66% vs. 48% in 2016; p = 0.0002, and 66% vs. 53% in 2017; p = 0.0012, Table 2).

Self-reported biological sex of the respondent did not influence these views in 2016, yet in 2017, females were significantly more likely to think that the consideration of SABV is important and would improve rigor and reproducibility compared to males, respectively (73% vs. 63%; p = 0.005, and 64% vs. 53%; p = 0.002, Table 3).

The NIH SABV policy only applies to grants that include the use of human or vertebrate subjects. ⁷ As such, respondents who reviewed grants meeting these criteria (99% vs. 97%; p = 0.130, Table 1) were asked what proportion of grants they reviewed accounted for the consideration of SABV in the research strategy and adequately addressed the incorporation of SABV into the experimental design, analysis, and reporting, as instructed by NIH guidelines (Fig. 2 and Supplementary Table S4). In both cases, there was a statistically significant increase in the proportion of grants that addressed the SABV policy between 2016 and 2017 (p < 0.0001 for each endpoint).

OPEN IN VIEWER

FIG. 2.

Comparison of respondents' perception of SABV policy implementation between 2016 and 2017.

A majority of study section members indicated that the consideration of SABV was consistently weighed into an applicant's approach score (55% vs. 61%; p = 0.0518). In addition, there was a statistically significant increase in the proportion of grants that provided consistent justification for the use of a single sex between 2016 and 2017 (44% vs. 50%; p = 0.0330).

Respondents had the option to leave open-ended comments at the end of each survey (n = 130 in 2016 and n = 257 in 2017, Supplementary Table S5). There were no differences in demographics between respondents who left comments in 2016 and 2017, with the exception that there were significantly more study section chairpersons who commented in 2017 compared to 2016 (8.6% vs. 2.3%, respectively; p = 0.018). Following initial review, comments were categorized into one or more of the following major themes: budget, animal use, policy, research area, and grant review.

In the budget theme, comments highlighted the need for increased federal funding for research citing that the SABV policy places, “[an] unreasonable demand given the limited budgets allowed for NIH grants,” while comments pertaining to animal use indicated that the SABV policy “requires increasing the number of experimental animals.” Comments that addressed the implementation of the policy itself were mixed, as some lauded the policy, as “a simple and needed criterion,” while others were more critical citing it as, “[a] waste of time, not at all getting to the root of reproducibility problems.”

Finally, respondents provided comments directly related to study section and the grant review process. Comments spanned multiple topics such as reviewer education (“Our review officer [redacted] was really good in explaining the new policy to us. That is how I learned about how to incorporate it into my proposals and my reviews of others proposals,”), policy implementation (“The applicants have gotten the message and now include sex as a biological variable in their proposals.”), and discussion of the policy itself (“SABV was strongly emphasized in the premeeting discussions, and it was mentioned for every discussed application.”).

However, respondent comments revealed variability in the way the policy impacts grant scores citing that, “It is difficult to determine whether the individual reviewers modified their score on Approach based on whether [sic] the criteria for sex as a biological variable,” and “Reviewers have different opinions on how much weight consideration of SABV should have on the score of the application…,” While another respondent shared that, “Consideration of SABV was not usually a score-driving consideration.”

In addition to our qualitative analysis, semiquantitative in silico sentiment analysis was utilized to interpret the respondents' comments. In general, the overall comment sentiment analysis scores for both years were slightly negative [median (quartile 1–quartile 3): −0.262 (−0.828–0) vs. −0.181 (−0.726–0); p = 0.358, Table 4]. When analyzed by theme, comments pertaining to animal use or budget/funding issues received the lowest sentiment analysis scores, followed by comments related to the SABV policy itself.

Because we identified biological sex and experience conducting sex-based research as two potential factors that influence views toward the SABV policy, we analyzed comments by these variables. While there were no significant differences in 2016, sentiment analysis scores significantly differed for males and females (median = −0.2193 vs. 0; p = 0.032), and those whose work involves the study sex differences compared to those whose work does not (median = 0 vs. −0.2429; p = 0.007) in 2017 (Supplementary Table S6). Table 5 provides representative examples of positive, negative, and neutral comments in the policy category theme.

Discussion

Sex, as it relates to the biological determination of males and females, XX or XY, has largely been left out of biological studies not only in terms of inclusion but also in the analyses and reporting of data. ^5,14,15 The NIH SABV policy was created to address these inequities through the promotion of sex-inclusive research studies, in an effort to enhance the rigor and reproducibility of the biomedical research enterprise. ¹⁶

As the policy is relatively new, it is difficult to determine its direct impact on research output. Instead, we chose to survey the attitudes and opinions of NIH study section members who serve both as policy adjudicators and representatives of the biomedical research community. The results presented in this study suggest that awareness, perceived comprehension, and implementation of the NIH SABV policy are improving over time, but that negative attitudes toward the policy persist and additional training and/or guidance may be warranted.

Our data indicate that the SABV policy is accepted and understood by a majority, but not all, of study section members. We identified potential biases that may influence attitudes toward the SABV policy, such as field of study and sex. Researchers who currently explore the study of sex differences were significantly more likely to agree that it is important to consider SABV and that it improves rigor and reproducibility. This may indicate a potential knowledge gap for investigators who are less familiar with the role of sex in health and disease.

Also, sex of the study section member appears to influence attitudes toward the policy as female study section members in the 2017 cohort were significantly more likely to view the SABV policy as favorable compared to men. While women are in the minority of study section membership (44% of our respondents in 2017), they may see value of the policy at a personal level, as sex-based research is directly implicated in women's health. ^6,12,15

The ability to leave open-ended comments provided study section members with the opportunity to discuss issues not directly assessed in the survey. Sentiment analysis, utilized in an attempt to remove potential author bias, revealed negative feeling toward the policy. However, open-ended responses were entirely voluntary, and it may be that those with polarized feelings toward the policy were more likely to reply. Nevertheless, these comments provide a contextual framework in which the policy can be evaluated.

Budgetary and/or funding concerns were woven throughout many of the comments, citing that increased funds were needed to appropriately consider SABV. Also, many comments voiced concerns for animal welfare, stating that the SABV policy requires an increase in experimental animals. This is likely derived from the assumption that to consider SABV, sample sizes must be doubled and female animals at varying stages of the estrous cycle should be utilized. While estrous cycle staging may be required to conduct experiments specifically designed to explore hormonal influences on a given trait or condition, several studies reported that there is no increase in intragroup variability between males and females, regardless of estrous cycle stage, ^{17 –19} thus allowing for the inclusion of both male and female subjects, without an increase in sample size (for a more detailed discussion of this topic, consult a recent review by Beery ²⁰ ).

In terms of policy implementation, study section members expressed differing viewpoints as to how the SABV policy was to be evaluated and scored within the context of grant review. This highlights an opportunity for increased reviewer education, although many scientific review officers and section chairs were praised for their efforts to educate their members on the policy. Finally, it came to our attention during review of the comments that numerous individuals misused the terms sex and gender, referring to gender in a specific scientific context (i.e., cell or animal models) when sex is the appropriate term. ²¹ This could suggest that researchers may benefit from additional training and/or education on sex- and gender-based research practices.

In terms of how the policy has been accepted by grant applicants, study section members found that the number of grants that addressed SABV both in their research strategy and specifically within the experimental design, analyses, and reporting has significantly increased since the policy was introduced. Likewise, there was a significant increase in the number of grants, which provided consistent justification of the use of a single sex. These data are encouraging and demonstrate adherence to the policy. Future studies are needed to examine whether this initial outcome translates into sex-based or sex-inclusive research findings with direct impacts on health and disease.

While this is the first assessment of study section members' attitudes toward the SABV policy, it is not without limitations. First, there was a limited response rate for both surveys (10% and 15% for 2016 and 2017, respectively). Study section members may have felt uncomfortable participating in a survey related directly to the NIH review process. To ensure anonymity and the confidentiality of the review process, we did not ask any questions related to specific applications or analyze data by study section membership. This study was not funded by the NIH; however, this information was not included in the consent materials, which may have deterred study section members from participating.

In addition, this study is not a true follow-up as study section membership differed between the two study cohorts. This limitation was mitigated by the fact that the demographics and experience of respondents were not significantly different between 2016 and 2017. It is also important to note that this study assessed respondents' perceived knowledge of the SABV policy, and not through a direct assessment of policy content. Thus, respondents who indicated that they felt confident in their understanding of the SABV policy may not actually have content knowledge surrounding the purpose and scope of the policy. Likewise, respondents were tasked with estimating the percentage of grants reviewed, which incorporated the SABV policy several months after attending study section. We acknowledge that these data, at face value, serve only as estimate and do not reflect the actual number of grants that adhere to the SABV policy. Due to confidentiality of the grant review process, it would fall under the purview of the NIH to conduct an exact analysis.

Finally, while sentiment analysis was utilized in an attempt to remove reviewer bias of open-ended comments, implicit bias may still persist into thematic coding of the comments and the representation of such within this article. Blinded, independent review was used to mitigate this risk.

In conclusion, inclusion of sex as a biological variable in fundamental research funded by the NIH is a policy that is accepted by a majority of study section members. Moreover, study section members are knowledgeable about the policy with information largely obtained from NIH. Most study section members believe that the policy is improving rigor and reproducibility and a statistically significant number of grants are addressing sex as a biological variable appropriately in their submissions. Yet, respondent comments revealed that negative attitudes toward the policy do exist.

As women are only now approaching parity in clinical research, ⁵ 25 years after the implementation of the NIH Revitalization Act, we encourage the research community to view policies that examine the influences of both sex and gender on health and disease as scientific imperative. This will help to ensure that the reproducibility of scientific reports increase over time and that new discoveries using sex as a biological variable are on the horizon.