Construct-Irrelevant Variance on the ASWB Clinical Social Work Licensing Exam: A Replication of Prior Validity Concerns

The Association of Social Work Boards (ASWB) licensing exams play a pivotal role in the social work profession. Administered across the United States and Canada, the ASWB (2023a) states that their exams “test a social worker's competence to practice ethically and safely” (para. 2). Passing an ASWB exam is mandated by nearly all social work state and provincial regulatory bodies, along with completion of several thousand hours of postdegree supervised work experience, as a condition of social work licensure. As these exams play a critical gatekeeping role for social work practitioners, issues of validity, reliability, and fairness of these exams are of paramount importance to test-takers, the social work profession, and the broader public.

Despite the critical role played by the ASWB exams in the profession, scholars have raised concerns about their validity for over a decade (Albright & Thyer, 2010; DeCarlo, 2022; Perron, 2023; Victor et al., 2023). These studies have suggested that the exams do not accurately measure the intended knowledge and skills for social work practice. The accumulated evidence of validity flaws raises serious concerns about whether the ASWB exams serve as a sound measure of a social worker's readiness to practice safely and ethically.

Prior studies have given particular focus to the construct validity of the exams. That is, scholars have raised questions about whether the ASWB exams actually measure the capacity of social workers to practice safely and ethically, which is what the exams are supposed to measure (ASWB, 2023a; Perron, 2023). Albright and Thyer (2010) completed a foundational study in this area nearly 15 years ago. The researchers started with a set of 50 multiple-choice items developed by the ASWB to prepare test-takers for the clinical exam. They then removed the question prompt from each item—leaving only the four provided multiple-choice options—and administered the practice exam to 59 first-year MSW students. Because the question prompts were removed, leaving only the four multiple-choice options, the researchers expected the average score on the practice exam to be 25% based on chance. However, the students achieved an average score of 52%, selecting the correct response twice as often as anticipated. Albright and Thyer suggested that their findings likely indicate a capacity of ASWB test-takers to achieve relatively high scores by capitalizing on cues and patterns present within the multiple-choice answer options, rather than relying on their comprehension of social work principles and best practices. In other words, their study provided evidence of construct-irrelevant variance on the ASWB clinical exam.

Construct-irrelevant variance refers to the presence of extraneous factors that can influence test results, which are unrelated to the construct being measured (Haladyna & Downing, 2004). This variance can pose a significant threat to the validity of test scores, as it can lead to inaccurate inferences about the abilities or traits of the individuals being assessed. When construct-irrelevant variance is present, it undermines the ability of a test to accurately capture the targeted construct—in this case, the capacity of social workers to practice safely and ethically—resulting in misleading or biased outcomes. In order to ensure the validity of test results, test developers should minimize construct-irrelevant variance by carefully designing test items, providing appropriate accommodations, and considering potential sources of bias during test development and administration.

Current Study

Given the passage of time and potential exam changes, this replication study investigates whether the construct-irrelevant variance identified by Albright and Thyer (2010) persists in the current version of the ASWB clinical exam. By reassessing this issue with newly developed items, we aim to determine whether serious flaws related to construct validity remain, which would potentially render the exam an ineffective and inaccurate tool for evaluating the competencies of aspiring social work professionals. We use the same approach in this study, deploying ChatGPT, a large language model (LLM), to assess current ASWB-developed materials and determine whether the structure of multiple-choice options continues to serve as a source of construct-irrelevant variance on the clinical exam. ChatGPT has previously been used to evaluate licensing exams in both social work (Victor et al., 2023) and other professional disciplines including medicine (Gilsen et al., 2023) and engineering (Pursnani et al., 2023).

Method

Results

To establish a point of reference for considering the level of construct-irrelevant variance, we first determined the accuracy of ChatGPT when taking the traditional version of the clinical exam with both the question and multiple-choice options available for each item. Under those conditions, ChatGPT correctly answered 90% of the items on the clinical exam. We then conducted three rounds of testing on the modified version of the clinical exam, asking ChatGPT to select the most likely multiple-choice answer without the question available. The model accuracy was 70%, 76%, and 74% across these three rounds. Fleiss’ kappa across the three rounds was 0.81 indicating almost perfect agreement (Landis & Koch, 1977).

Discussion and Applications to Practice

The findings of this study—which replicate those of Albright and Thyer (2010)—indicate that ChatGPT, an artificial intelligence LLM, was able to perform far better than chance on the ASWB clinical licensing exam when the questions were hidden and only the four multiple-choice options were presented. Instead, ChatGPT achieved an average accuracy rate of 73.3% across three rounds of testing. In two of these instances, model accuracy was high enough that ChatGPT is likely to have passed the clinical exam without the questions provided.

These results raise significant concerns regarding the construct validity of the ASWB clinical exam, as it suggests that the exam might not be measuring the intended construct of social workers’ competence to practice safely and ethically. Instead, it appears that relatively high scores can be achieved on the clinical exam by relying on cues and patterns present within the answer choices themselves, a strong indicator of construct-irrelevant variance. This observation provides further support that use of the current ASWB clinical exam to evaluate the competence of social work professionals is not likely to be appropriate for its intended use. In light of these findings, the developers and administrators of the ASWB licensing exams must reassess the exam's structure, with the aim of eliminating potential biases and improving its construct validity. This may involve the use of alternative question formats, the implementation of more rigorous item-writing guidelines, or—our preference—the development of novel assessment methods that better capture the knowledge and skills required for safe and ethical social work practice.

Fortunately for the profession and the broader public, clinical social work licensure requirements already include extensive supervision by a licensed practitioner. In terms of assessing the construct of the ability to practice safely and ethically, supervisors are likely better equipped than a multiple-choice exam to determine the true capabilities of those seeking licensure. That is, observing performance in context is a more valid approach to assessing competence to practice safely and ethically (Haladyna & Downing, 2004). Supervisors are able to go beyond the evaluation of theoretical knowledge about professional ethics and safe practice, as assessed in licensing exams. Supervisors can provide real-time, situational guidance on safety and ethical dilemmas that arise in the field, fostering a nuanced, practical understanding of principles and practice guidelines that can be more contextually relevant and adaptive. This in-context approach can nurture the ability to apply ethical frameworks to complex, real-world situations in a manner that the abstract and/or information-limited questions on the licensing exam might not fully capture.

At least one jurisdiction has adopted an alternative pathway to clinical licensure that places more reliance on the clinical supervision model currently mandated by social work regulatory bodies. The Illinois Legislature recently passed House Bill 2365 which provides a pathway to clinical licensure that does not require passage of the ASWB clinical exam (Illinois General Assembly, 2023). Applicants are required to take the clinical exam once, and those who do not pass can opt for an additional 3,000 hours of supervised practice to obtain their license.

If testing is to remain a core feature of licensure, then we strongly advocate for increased transparency and independent evaluation of the ASWB exams to ensure conformity to the Standards for Educational and Psychological Testing. This publication, jointly produced by the American Educational Research Association, American Psychological Association, and the National Council of Measurement in Education (2014), details the specific criteria for developing and evaluating testing practices and for assessing the validity of score interpretations. The publication includes a special section that focuses specifically on professional credentialing. In particular, Cluster 1 Standards (11.1, 11.2, and 11.3) and Cluster 3 Standards (11.13–11.16) provide explicit guidelines concerning validity evidence for professional credentialing. Social work is not alone in this need for independent evaluation of its licensing exam, as the need for increased transparency and independent validation of exams has been called for in other mental health professions such as clinical psychology, counseling, and marriage and family therapy (Caldwell & Rousmaniere, 2022).

The recent announcement by the Association of Social Work Boards (ASWB, 2022a) regarding their decision to reduce the number of multiple-choice options on exams from four to three exemplifies the pressing need for transparency and independent evaluation in our field. The organization asserts that this exam modification stems from “psychometric expertise that confirms the validity” (para. 2) of this reduction in multiple-choice options. However, a significant issue arises as ASWB provides no publicly available evidence to back its claim.

The responsibility to accept ASWB's assurance about the validity of exam adjustments falls on test-takers, educators, and policymakers. However, a clear conflict of interest makes this faith-based acceptance worrisome. The conflict arises because the exam developer, ASWB, also assumes complete control over establishing and endorsing the exam's validity. This arrangement carries a potential bias, given ASWB's inherent interest in promoting the effectiveness and validity of its exam which serves as the organization's primary revenue and profit generator. In 2021, the most recent year for which revenue data are available, ASWB reported profits of over US$7 million, up approximately US$4 million from the year before (ASWB, 2021). Such a conflict of interest introduces considerable bias into the evaluation process, ultimately undermining the exam's validity. This conflict of interest highlights the importance and need for independent third-party verification to address the conflicts and bolster confidence in the validity of any modifications made to the test format.

While using ChatGPT for this study had advantages, it also presented limitations. First, as an AI language model, ChatGPT's decision-making might not mirror that of human test-takers. The model selects answers based on patterns and statistical associations that likely differ in some ways from how humans would arrive at an answer. This could mean that the selections the model makes when faced with incomplete information may not align with a human test-taker's selections under the same conditions. Secondly, ChatGPT has not received explicit training in social work knowledge and skills which could also be considered a disadvantage in generalizing these findings to authorized test-takers. Workers who have received advanced social work training and are engaged in clinical practice might make different decisions under these circumstances when drawing on their content knowledge or work experience.

In their original article documenting construct-irrelevant variance on the ASWB clinical exam, Albright and Thyer (2010) ask in the subtitle: quis custodiet ipsos custodes? In English, “who is watching the watchers?” Fifteen years later, the answer does not appear encouraging. Major validity and transparency concerns remain unresolved (DeCarlo, 2022; Victor et al., 2023), and the use of the exams in the licensure process continues to exclude workers from the profession based on race, age, and first language (ASWB, 2022b). Given that the ASWB has yet to correct the long-standing validity challenge replicated in this study—in addition to other identified flaws—we encourage state legislatures and regulatory bodies to temporarily suspend the use of the ASWB clinical exam as a requirement for licensure until an acceptable and independently validated alternative is established.