Abstract
Summary
Social work has grappled with quality of its knowledge base for over a hundred years. In the tradition of critical self-examination, this study was conducted to describe the methods employed in quantitative studies published in social work journals and to assess the extent to which they conform to widely accepted standards of quality. A random sample of 120 articles published in the top thirteen social work journals from 2010 to 2015 were reviewed to identify indicators of quality and rigor. Additional data on citations per year were collected through July 2022.
Findings
Most studies were cross-sectional and descriptive in nature (62%). While the majority of articles addressed the limitations of the study, only 40% contained reference to theory and 15% described how missing data were handled. Citations per year averaged 4.82, and citations per year exhibited a small, statistically significant positive association with quality indicators and reviewer global evaluations, but not with journal impact factor. Results did underscore the role of doctoral programs in promoting quality research.
Applications
Social work education programs should continue emphasizing quality social work research methodologies, particularly given the move toward postpositivism as a profession. We note a cycle of disincentivization may be occurring, where social work researchers take their best work to higher impact factor, non-social work journals. We also suggest lack of traditional indicators of rigor may be the result of “concerted resistance” to borrowed epistemologies; we call for consensus-based measures of rigor that reflect the realities of the social work profession.
The profession of social work devotes considerable resources to its research endeavors, seeking to make unique contributions to the social science and social service landscape. In the United States, organizations such as the National Institute of Mental Health (NIMH) and Society for Social Work and Research (SSWR) have sought to emphasize the importance of quality research contributions and improve the legitimacy and rigor of social work research (Glisson, 1995; Jenson, 2008; Witkin, 1995). While these resources and other efforts have encouraged progress and highlighted specific areas for improvement, past analyses have identified persistent limitations in the quality of social work research (Barusch et al., 2011; Fraser et al., 1991; Rubin, 2000). In light of this, regular examination of social work research quality is important. This study was conducted to provide insight into the status of quantitative work that has been published in the past decade in social work journals.
Persistent concerns with research quality
The social research enterprise has evolved significantly since its origins in the early twentieth century. From 1930 to 1945, most social work research consisted of fact-finding efforts and surveys; as the use of social surveys became increasingly scrutinized, social work research floundered (Irving, 1992; Rubin, 2000). The role, validity, and relevance of research met significant challenges as the profession matured, with social work scholars arguing the prioritization of “science” would compromise the “heart” of the profession (Boehm, 1961). Such reluctance to wholeheartedly embrace and engage in the scientific process no doubt contributed to the already strained relationship between research and social work, resulting in a significant deficit in systematic research publications from social workers (Fraser et al., 1991). Some scholars argued that the very nature of social problems and topics confronted by practicing social workers complicated the ability to produce high-quality, relevant research (Glisson & Gillespie, 1992; Guo & Hussey, 2004; Rubin, 2000).
Social work's struggle to fully devote itself to rigorous research has undermined its professional identity. Howard and Garland (2015) labeled the years between 1984 and 2014 the “Era of emulation,” reflecting the discipline's tendency to rely on methods and approaches from other social science fields (p. 178). Citing concerns with measurements, variables, and analytic techniques, Glisson and Gillespie (1992) suggested that “even the staunchest defenders of social work research would not claim that it has contributed significantly to or documented convincingly the effectiveness of practice” (p. 2). Other scholars have cited a lack of objectivity and insufficient effort to minimize bias as weaknesses in social work publications (Garrow & Hassenfeld, 2017; LeCroy, 2010; Rubin, 2000; Swigonski, 1993).
In 1991, a Task Force on Social Work Research formed by the NIMH in the United States published a state-of-the-field report. Their analysis indicated that a majority of articles used nonrepresentative samples, either completely avoided or only utilized univariate inferential statistical tests, and lacked internal and external validity (Glisson, 1995). Consequently, the NIMH report declared a “crisis in social work research … based on the paucity of resources devoted to research, the dearth of researchers and research productivity, and the quality of the research being produced” (Witkin, 1995, p. 426).
Even when social workers engage in the scientific method, the discipline has struggled to exemplify basic tenets of quality research. Concerns have been raised about the absence of sound measurement approaches, representative samples, current analytic methods, and replicable research designs (Glisson, 1995; Guo & Hussey, 2004; Thyer, 1996). Likewise, social work scholars have argued that the profession's commitment to research quality is insufficient, in both qualitative studies (historically favored by the discipline) and quantitative studies (which have increased of late) (Dellgran & Hojer, 2012; Hardy, 2014; Holosko, 2010; Shaw, 2003; Sheppard, 2016). Some have gone so far as to say that “the social work research enterprise lacks some of the most common elements of good science” (LeCroy, 2010, p. 321). Such accounts highlight the precarious status of social work's research contributions and jeopardize the profession's standing as a legitimate social science discipline.
In addition to quality concerns, social work research has struggled with proper documentation and presentation (Campbell, 2010). Most of the problems in reporting lie in omitting essential details “necessary to judge bias, consider replication, or initiate meta-analyses and systemic reviews” (Wharton, 2017, p. 492). In response to this critique, researchers have called for more thorough and widely accepted guidelines for reporting research (Jenson, 2005; Wharton, 2017). Such guidelines should be specifically tailored to social work's unique person-in-environment approach (Wharton, 2017). In contrast, Shaw and Norton (2008) argue that specific standards are unnecessary, advocating general guidance on quality assessment as an approach better suited to the social work community.
Efforts to improve quality
Published reviews of social work research have responded to common shortcomings by constructing general guidelines on how to properly report on both qualitative and quantitative research methods (Campbell, 2010; Simpson & Lord, 2015; Walby & Luscombe, 2017). Some generally accepted components of proper quantitative research reporting include adequately addressing theoretical and ethical considerations, acknowledging limitations, discussing the assumptions of statistical tests, discussing validity and reliability, and outlining the treatment of missing data (Campbell, 2010; Guo et al., 2009; Holosko, 2010; Shaw, 2007; Shaw & Norton, 2008; Simpson & Lord, 2015). In addition to these specific quality indicators, Shaw and Norton (2008) highlighted the importance of including an overall measure for the reviewer's assessment of the research.
Many researchers suggest that the lack of relevant, high-quality research in the profession reflects the failure of social work research education (Lein et al., 2017; Phillips et al., 2012; Rubin, 2000; Sheppard, 2016; Sundell & Ahsberg, 2018). The ability to “anticipate social and environmental changes and make relevant complementary adjustments in response” is a key component of research education, one that social work has struggled with in comparison to other health and social service professions (Lein et al., 2017, p. 68). Additionally, there are doubts that social workers possess the “quantitative literacy” necessary to effectively produce and consume quantitative research (Sheppard, 2016, p. 1534).
Although the quality deficit in social work research is admittedly well-documented and persistent, there have been notable improvements over time. First, the production of social work research has increased significantly in the past 30 years (Howard & Garland, 2015). While the quality of these contributions may not yet meet the highest standards, this increase in output does suggest a prioritization of research that has been historically lacking in social work. Additionally, while more advanced statistical models, such as structural equation modeling, hierarchal linear modeling, and propensity score analysis, may need to be further utilized, social work researchers have been employing more sophisticated quantitative methods (Guo, 2015). Examination of social work research has revealed promising trends in quality; however, research practices should be regularly monitored to ensure that progress is being made and to offer insight into how the social work profession can improve its contribution to the scientific community (Sundell & Ahsberg, 2018). Thus, we aim to describe the methods employed in quantitative studies recently published in social work journals and assess the extent to which they conform to widely accepted standards of quality. We also aim to understand if educational preparation influences quality of research, and if quality of research influences impact of research. To that end, this study provides descriptive data and tests two hypotheses: (1) we hypothesize quality index scores for research conducted by authors with social work PhDs will be significantly higher than by authors without; and (2) we hypothesize that quality measures will be significantly associated with citations per year.
Methods
This study used a retrospective descriptive approach based on review of a random sample of published articles. Both quantitative and qualitative data were recorded on a template and analyzed using descriptive and bivariate procedures. In this section, we describe the researchers, the sample, the data collection procedures, and the data analysis.
Researchers
Three senior social work faculty initiated the study and participated in data collection and analysis. All had PhDs in the field and were trained in the United States. One was the author of statistics textbooks and one taught quantitative methods. Upon the retirement of two of the original research team, three new researchers joined the project: one senior faculty member, one doctoral candidate, and one advanced MSW student, who completed the data analysis and writing.
Sample
Using the Journal Citation Reports, we drew a proportional stratified random sample of articles published between January 2010 and January 2015 in the top 13 journals (by impact factor) published in English that list social work as their first subject category. So, for instance, the journal Trauma, Violence and Abuse was not included in the sample because it lists criminology as its first subject. Likewise, Social Work in Public Health lists public health as its first subject, so was not included. While social work academics may publish in non-social work journals, these works are not readily distinguishable as “social work research,” thus not part of a distinctly “social work” knowledge base, and not included in our sample.
The inclusion criteria for articles within the selected journals called for studies that involved the collection and analysis of quantitative data, whether from human subjects or other sources; mixed-method studies were included. Excluded were: book reviews, conceptual pieces, case study narratives, and articles that reported on qualitative research. From a total of 6,044 articles published in these journals during the period under consideration, we drew a random sample of 120. The sample was sufficient to detect a medium effect size for a two-tailed alpha of .05 for correlational and group comparisons’ tests. Ultimately, twelve were not included in the final analysis either because they had been used to conduct reliability checks or did not meet inclusion criteria.
Data collection
Data were collected using a 16-page template developed and revised by the researchers. The template included five sections: description of the article, review of the literature, method, results, and discussion and conclusion. Data gathered included descriptive information about the article (e.g., substantive area or research design), dichotomous measures designed to assess the presence or absence of certain content (e.g., “Is theory mentioned in the literature review?” or “Does the article discuss implications for practice?”). Researchers also made evaluative determinations about the quality and contribution of the article (e.g., “Was the research design appropriate to address the aims of the research?” or “To what extent did the article persuade you that the study made an important contribution to social work?”). The quality indicators included in the template were derived from best practices outlined in research textbooks and author expertise (Abu-Bader, 2021). Qualitative content included quotes that were collected verbatim from the articles. For instance, in noting an article's practice implications the reviewer was directed to provide an illustrative quote or quotes. Likewise, space was allowed for general comments on key aspects of the methodology, such as sampling, measures, and statistical analysis.
Two independent reviewers completed these templates, a process that took from 45 to 90 minutes for each article. Following each review, Google Scholar was used to collect the total number of citations for each article as of July 2022. In data analysis, this number was divided by the number of years since the article was published (calculated as number of months since publication through July 2022 and divided by 12) for a measure of “citations per year.” A research assistant then reviewed the templates to ensure all requested information was noted before entering the data into an SPSS file. To establish the level of inter-rater agreement, reliability checks were completed using four articles. Each reviewer independently reviewed the articles and a Kappa was computed. The resulting Kappa (.70) indicated “substantial” agreement between the reviewers on all categorical measures (McHugh, 2012); the overall agreement between reviewers was 89.6%.
Data analysis
Descriptive measures, such as frequencies and means, were examined for all questions, and two measures of quality were used: the reviewer's global assessment and a Quality Index. The global assessment was measured using a single item, “How would you evaluate this article?” with a 5-point Likert-type scale ranging from “1 = poor” to “5 = excellent.” The Quality Index was constructed from eight dichotomous quality indicators, where yes = 1 indicated presence of the quality indicator in the article. Scores were summed to form the Quality Index ranging from “0” to “8.” Indicators included: addressed theory, addressed ethical considerations, addressed methodological limitations, addressed measurement limitations, discussed assumptions of statistical tests, described how missing data were handled, reported on reliability of measures, and reported on validity of measures. Citations per year were severely skewed based on tests of normality and skewness. To address this, a logarithm transformation was completed; this transformed the data into a distribution approaching normal.
In addition to the descriptive data, bivariate correlations were computed among quality indicators, and independent t-tests were conducted to compare articles written by authors with a PhD in social work to those without. All statistical analysis was conducted in SPSS v27.
Results
The sample
Articles were retrieved from thirteen social work journals. The social work journals reviewed here had a mean impact factor of .951, with a range from .538 to 1.375; journals are summarized in Table 1. One hundred and eight articles were included in the data analysis. Topics most frequently addressed included: social work practice (20%) and physical health (20%), followed by children (17%) and child welfare (12%) (see Table 2 for all topic frequencies). The use of experimental and quasi-experimental designs was rare (6%), while a majority (62%) of the studies reported cross-sectional and descriptive designs. Nonprobability sampling was the norm (73%), and the dominant source of data was interviews (35%) and written surveys (34%). Study types, designs, and methods are summarized in Table 3.
Journals sampled.
a2014 Journal Citation Reports®.
Frequency of topics addressed.
aBecause some articles addressed multiple topics, percentages will not add up to 100.
Research characteristics.
The relative merits of a PhD.
Impact of quality index.
*p < .05; **p < .01.
Indicators of quality
Of the dichotomous quality indicators collected, only one was used in a majority of articles: A majority (77%) of the articles addressed the limitations of the study. Over half (60%) of the articles contained no reference to theory. Only 15% described how missing data were handled. The prevalence of quality indicators is summarized in Figure 1.
Prevalence of quality indicators.
Scores on the Quality Index ranged from 0 to 7 (out of a possible 8, with a mean score of 3). Results supported our hypothesis that authors who had social work PhDs would score significantly higher on the quality index (see Table 4). Those with PhDs had a mean score of 3.47, compared to a mean of 2.66 for those without, t(102) = 2.40, p = .02. Likewise, PhD authors were significantly more likely to articulate hypotheses, χ2(1, N = 104) = 4.52, p = .03, and to address the validity of their measures, χ2(1, N = 104) = 5.85, p = .02. Among articles authored by PhDs, 19% included hypotheses and 32% addressed the validity of measures. The figures for articles authored by non-PhDs were 6% and 13%, respectively.
Citations per year averaged 4.82, with a range from 0.10 to 44.29 (SD = 6.08). The data supported our hypothesis that citations per year would be significantly associated with scores on the quality index (r = .194; p < .05). Additionally, there was a positive association (r = .209; p < .01) between reviewer evaluation and citations. Citations were not significantly associated with journal impact factor (see Table 5).
Discussion and implications
The findings of our study suggest that in the 30 years since Fraser and colleagues’ (1991) analysis, the quality and rigor of social work research has not greatly improved. Social work's embrace of postpositivist research has not been accompanied by increased rigor in published work. The association between citations and quality indicators is encouraging, suggesting that more rigorous research is achieving further reach—perhaps because consumers of social work research are also becoming more research savvy. However, similar to Fraser et al.'s (1991) findings of largely elementary research methods, in our study we found very few authors referencing theory, power, statistical assumptions, missing data, and other quality indicators. This may reflect a lack of commitment to quality throughout the research pipeline, from education to publication, and/or the failure of traditional indicators of rigor to reflect the concerns of the profession.
One could argue the most straightforward explanation of our findings is simply that social work researchers use sloppy methods; however, we note the pressures to produce research coupled with the low impact-factors of social work journals could make competent researchers less conscientious while drawing the best articles they produce to non-social work journals. Our findings suggest doctoral programs do teach budding scholars to conduct and report on high-quality research, as articles by authors holding PhDs were significantly more rigorous than those without.
Social work academics are familiar with the call to “publish or perish”; publications are used as both the metric of productivity and the avenue for advancement. This emphasis on research productivity stems in part from the neo-liberalization of higher education, and in part from the historically ingrained efforts to professionalize the field. High pressure to publish within academia can jeopardize the quality and integrity of published research (Fanelli, 2010; Gibbs, 2001; Rubin, 2000). While the stress on academics to produce research may increase productivity, there is evidence that it also increases bias and encourages researchers to report positive outcomes (Fanelli, 2010). The pressure to publish manifests most visibly in the tenure review process, which serves as the formal mechanism for career security and advancement for social work academics; use of H index, citation factors, and lengthy curricula vitae as metrics for research productivity emphasizes quantity over meaningful contribution.
Our findings suggest the use of journal impact factors as proxies for quality may be flawed, as in our sample the quality index was not related to journal impact factor. Rubin (2000) suggests that guidelines for tenure and promotion in institutions of higher education be reprioritized to promote proper engagement in the research process rather than simply rewarding the quantity of publications. We did find an association between citations and both reviewer assessment and quality index, though the small effect size suggests these measures should not be used interchangeably. Problematic over-reliance on citation indices and journal impact factors has garnered resistance across fields, demonstrated by the European-based Coalition for Advancing Research Assessment call to discontinue use of metrics like journal impact factors and H indices as “proxies for quality and impact,” in order to “reduce the dominance of a narrow set of quantitative journal- and publication-based metrics” (Coalition for Advancing Research Assessment, 2023).
Another consideration is the quality of the data being analyzed in published articles. Particularly when analysis is conducted with administrative data, complex statistical techniques may be needed to address high missingness or other data problems; these techniques require higher levels of statistical skill and are not always employed. In this study it was not feasible to assess the quality of data being analyzed in the selected articles—in fact, research reporting often does not include the level of detail needed to ascertain the quality of data used for analysis.
The push for publication has ripple effects as social work journals must respond to a glut of submissions as researchers “produce.” Most social work journals rely on voluntary peer in an editorial review process that assumes reviewers are conscientiously applying the same quality indicators to all articles. At the same time, pressure to respond to submissions and complete reviews in a timely fashion might preclude the completion of thorough and thoughtful reviews. Second, the volume of journals has increased considerably in the past three decades; in addition to increasing demand for reviewers, this rise in avenues for publication does not necessarily reflect a proportionate increase in articles meriting publication. If this is the case, articles which may have previously been screened out of the publication process due to low rigor can now find their way into scholarly publication.
These findings may also reflect a widespread tendency for social work researchers to place their best work in nonsocial work journals, as has been a trend historically (Green & Secret, 1996). Given the pressure to demonstrate academic productivity using metrics, this strategy is certainly understandable; the social work journals used for this study had a mean impact factor of .951. The low impact factors of social work journals may lead researchers to publish their best works in non-social work journals, perpetuating lowered rigor in social work journals, leading to low citations and impact factors, which again disincentivizes publication of high-quality work in social work journals—continuing the cycle.
An alternative explanation for our findings is that the decades of efforts to improve the quality and rigor of social work research have run into concerted resistance. These efforts are predicated on a shared understanding of what constitutes “good research,” however, we have not owned the task of defining quality social work research that is based on—not just aligned with—social work values. Our lack of consensus about what constitutes quality and rigor is rooted in our profession's lack of a unified epistemological stance, which leaves definitions of knowledge and who qualifies as a knower contested. Within the field, some researchers and institutions have embraced postpositivism as the cornerstone of evidence-based practice, while others have pushed back with arguments that multiple research paradigms are needed (Morris, 2006). In the absence of our own articulated definition, we have borrowed other professions’ definitions of evidence and quality—ironically undercutting the unique contribution of social work research in the quest for legitimacy.
The breadth of issues social work research is concerned with does not avail itself to a single research methodology. Social work deals with society's “wicked problems,” complex problems for which simple solutions are simply not available (Rittel & Webber, 1973). The end in addressing wicked problems is not to find the solution, but ever better solutions. Social work research reflects the complex domains of social work, which are human and multifaceted, rarely captured comprehensively via randomized control trial. It is a mark of quality research that the methodology selected is appropriate for the questions at hand; once the method is selected, the execution of the research, regardless of the type of design, should be done competently and conscientiously. As a profession, perhaps it is time to stop debating whether social work is art or science, and commit ourselves to producing quality research that reflects the best of both realms.
Limitations
The primary limitation of this study is that it necessarily excludes research published outside of social work journals. This study also relied on a random sample of articles from social work journals; we did not create a comparison group of articles in another field. Without this, we are only able to speak to the quality of social work articles as compared to previous analysis, not how they compare to rigor in other fields. While the random selection of the sample should minimize bias, there is always a possibility of undetected bias in our methodology.
Conclusion
Directions for future research
To further explore the avenues social work researchers use for publication, future research should identify researchers with a social work identity—perhaps those with both an MSW and PhD in social work—who have been highly cited. Rather than reviewing social work journals, information could be culled from curriculum vitae to determine where highly-cited social work researchers are publishing, and analyze the rigor of their work. This might help tease out the extent to which the profession's most committed researchers have migrated to non-social work journals to disseminate their work, revisiting Green and Secret's (1996) previous study.
Recommendations
Our findings indicate that, despite the contribution of doctoral programs, social work research continues to lack rigor and quality as defined by traditional measures. We believe a fundamental issue is at play here: the absence of explicitly articulated, values-driven quality and rigor criteria specific to the field of social work, and a commitment to application of these criteria by researchers and journal editors. We recommend a forum of stakeholders—researchers, journal editors, and academic administrators—gather with the intent of reaching consensus on these criteria. While this may seem a formidable undertaking, existing structures such as SSWR in the U.S. or the European Social Work Research Association are well-positioned to facilitate the process. From there, specific revisions to the review process in social work journals could go far toward promoting methodological rigor. Social work researchers might be invigorated by a call for adherence to criteria that are uniquely developed for the profession, rather than cribbed from other social science disciplines. Indeed, the long-term result might be more research that contributes directly to social work interventions in the “wicked problems” of our time.
Footnotes
Ethical approval
Institutional Review Board ethical approval was not required due to the public nature of the data analyzed for the study.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Declarations of conflict of interests
The authors confirm that they have no conflict of interest in respect of the material submitted in this article.
Authors’ contributions
Amanda Barusch and Soleman Abu-Bader developed the coding template, coded the articles, and conducted the analysis. Christina Gringeri and Elisabeth Koelling drafted the manuscript. Elizabeth Hendrix conducted further analysis and manuscript revisions. All authors contributed to the interpretation of results and provided input and final review.
