Abstract
Limited research describes approaches for applying a public health lens to fatal child maltreatment classification. Specialized terminology and tools could help improve consistency in classifying deaths resulting from caregiver behavior. A six-criterion classification tool was developed via expert panel review of over 100 child deaths by the Alaska Division of Public Health’s Child Death Review (CDR) program. Next, accuracy and acceptability were assessed by inviting staff from other CDRs using a national listserv to classify 21 brief case scenarios with the tool. Among the 47 respondents, sensitivity was 0.87, specificity 0.77, and accuracy 0.84. Variability by tool criterion ranged from 97% to 74% accurate. Most respondents (66%) reported the tool as being helpful for classifying deaths and moderate reliability was found. Study participants found it difficult to consistently apply specific criteria which resulted in a modification of the tool to improve the potential for universal adoption.
Introduction
Child maltreatment prevention as a public health concern was deemed a national priority by the U.S. Surgeon General in 2005 (Office of the Surgeon General (US), 2005). Yet, estimating the number of deaths due to child maltreatment remains a challenge as jurisdictions and sectors differ in their operational definition of maltreatment and rates often rely on a single source (Overpeck et al., 1998). Likewise, internationally the classification of child maltreatment fatalities has been challenging (Tursz et al., 2010). Although child maltreatment is often seen as a child welfare and law enforcement issue, multiple sources have called for the implementation of a public health approach to improve surveillance and prevention (Covington, 2013; Fortson et al., 2016a, 2016b; Richmond-Crum, Joyner, Fogerty, Ellis, & Saul, 2013; Schnitzer et al., 2013).
State child welfare agencies report maltreatment fatalities to the National Child Abuse and Neglect Data System (North CarolinaANDS). (About NCANDS, n. d.; US, 2020), but some research indicates that these data alone, often relying on a single source of data and agency vantage point, underestimate fatal child maltreatment deaths (T. Covington, 2013; Palusci et al., 2010; Schnitzer et al., 2008; Schnitzer et al., 2013; United States Government Accountability Office: Report to the Chairman, Committee on Ways and Means, House of Representatives, 2011). The multidisciplinary approach of Child Death Review (CDR) programs, which combines multiple data sources, has been proposed as the preferred method for identifying and classifying fatal child maltreatment (Schnitzer et al., 2008). However, CDR data are inconsistent and incomparable within and across jurisdictions due to varying definitions of child maltreatment (T. M. Covington, 2011; Quinton, 2017). The U.S. Government Accountability Office (GAO) documented the national undercount of fatal child maltreatment in official sources in 2011, calling for better data collection and analysis (United States Government Accountability Office: Report to the Chairman, Committee on Ways and Means, House of Representatives, 2011). This led to the passage of the Protect Our Kids Act in 2012 and the Commission to Eliminate Child Abuse and Neglect Fatalities (CECANF) which in 2016 recommended a national surveillance system using CDR data (Commission to Eliminate Child Abuse and Neglect Fatalities, 2016). Despite efforts by the Centers for Disease Control and Prevention to standardized definitions of child maltreatment for public health data (Rebecca, et al., 2008), studies have shown that CDR members inconsistently classify maltreatment deaths, especially neglect-related ones (Palusci et al., 2010; Schnitzer et al., 2011). Variability in state laws and expert opinions on what constitutes child neglect further complicates this issue (Jennissen et al., 2018).
A 2017 study assessed the reliability of maltreatment classification by CDR teams in Alaska and confirmed the challenge of consistently classifying neglect. (Parrish et al., 2017) As a result of these findings, a team of experts convened to review and classify maltreatment in 100 child deaths to determine how to objectively classify maltreatment. Child death review, child maltreatment classification, and medical examiner national experts were invited to assist the Alaska team Alaska’s expert team consisted of child welfare, public health, epidemiology, maternal child health, injury, and local child death review experts. Facilitated by the senior MCH epidemiologist, an iterative review process that adapted a Nominal Group Technique (Boddy, 2012) was used to identify classification bins (groups). First, the deaths were reviewed and classified into two groups, those with and without complete team agreement. Those with complete team agreement were then sub-grouped into Medical Examiner classification as homicide and Child welfare classification as maltreatment. Those without complete agreement were reviewed again and further grouped into themes based on the circumstances of the death. During this review iteration, the team (both national and local experts) expanded the terminology from “fatal child maltreatment” to “child fatalities resulting from caregiver behavior”. Through iterative discussion and review, different caregiver behavior threshold standards for each theme were applied to the cases until complete agreement was reached for classification resulting in the final three criteria of the tool (USA, 2020). The final sub-groups that emerged were caregiver behaviors based on legal laws, widely accepted public health recommendations, medical care, and supervision criteria. During this final review, to reach consensus the team ultimately limited to classifying deaths among children younger than six years old as general consensus states that children of these ages require constant sight or sound supervision (Peterson et al., 1993)
Original Alaska Caregiver Behavior Classification Tool Criteria.
The ACBC tool was designed to help improve consistency in classifying and quantifying deaths related to child maltreatment by CDR teams using a multi-disciplinary public health approach. The ACBC tool adopts expansive language to distinguish public health classified deaths from official maltreatment deaths meeting state child welfare agency definitions. This critical distinction ensures that deaths classified as maltreatment adhere to legally defensible classifications while facilitating a broader inclusion of deaths resulting from caregiver behavior. This broader tier-based classification model can allow for targeted interventions as well as broad public health prevention campaigns, each of which are targeted at the behavior of the caregiver. The purpose of this study was to evaluate the accuracy of classification relative to the a priori determined classification when utilizing the ACBC tool by a national sample outside the state of Alaska and inter-rater reliability of classifying among respondents. A secondary purpose was to assess the utility and acceptability of the tool by CDR coordinators.
Methods
Sample
A non-probability sample was obtained in 2022 by inviting participation using the National CDR Listserv 1 and direct email to state CDR coordinators.
Case Scenarios
Informed by 100 child death case files reviewed during development of the ACBC tool and national expertise of team members, we created 21 fictitious scenarios of deaths of children less than six years old that represent broad non-state specific types of scenarios. Each scenario was designed to emphasize a specific criterion and was as brief as possible to minimize the time required to complete the survey (see Appendix 1 for scenarios).
Survey
Data was collected online via anonymous survey. The survey collected basic demographic information before respondents were asked to make a determination using the classification tool and provided case scenarios. In the scenario review portion, respondents were instructed to review each scenario and determine if a caregiver’s behavior created or failed to mitigate a hazard which resulted in the child’s death responding “Yes,” “No,” or “Unknown” to each of the six criteria contained in the classification tool. In making determinations, the participants were advised that the caregiver behavior must be supported by evidence and directly related to the death following the premise that if the behavior was removed, the death likely would not have occurred. This was followed by a question rating the utility of the ACBC tool and an area for comments. Finally, respondents were asked to specify how helpful the tool was overall and to explain any challenges experienced in using each criterion to evaluate caregiver behavior.
Classification Criterion
The ACBC tool introduces the terminology “child fatalities resulting from caregiver behavior”, to ensure that “maltreatment” deaths are captured under a broader public health definition. The goal was to establish a clear threshold for assessing caregiver behavior that is informed by public health recommendations and distinguish deaths from those classified as maltreatment by child welfare, law enforcement or death certifiers. Rather, this nested approach includes the highly specific definition used by official sources while introducing a more sensitive public health definition. The six criteria were created from themes that were identified during the Alaska and expert team’s review of the 100 child deaths. The first two criteria assess determinations made by the Medical Examiner, law enforcement, and child welfare, while the next four consider caregiver behavior relative to local laws, public health recommendations, prescribed medical treatment, and caregiver supervision. Table 1 presents the original six criteria used in this study, for the final evidence-based criteria informed by this study see Appendix 2 for the complete final tool.
Statistical Analysis
We conducted both qualitative and quantitative analyses to understand user assessment of the ACBC tool. We limited our analysis to only the respondents who completed all 21 scenarios. A total of 80 respondents initiated the survey, with 47 completing all 21 scenarios (58.8%). Of those who started but did not complete the survey (n = 33), five completed 5-20 scenarios, eight completed 1-5 scenarios, and twenty did not complete any. We identified and report themes from open-ended responses related to challenges using the six criteria. We also report the frequency distributions of tool helpfulness as rated on a scale from “Not at all helpful” to “Very helpful.” Next, we quantified respondent accuracy (relative to the expert team’s determined a priori classifications) in two ways: first, we identified whether the respondent classified the scenario as resulting from caregiver behavior based on any of the six criteria and calculated the average sensitivity, specificity, and overall accuracy across raters, and second, we classified the percent correct of each of the six criteria by scenario and by respondent to identify areas of large incongruence. Finally, we assessed inter-rater reliability for n-m classifiers with Kappa statistics with an alpha level of .05. All statistical analyses were performed using R version 4.1.1. This study was reviewed and designated exempt by the University of Alaska Anchorage Institutional Review Board.
Results
For each of the 47 respondents, the sensitivity, specificity, and accuracy relative to the a priori classification was calculated across all scenarios and averaged across all respondents. The average sensitivity was 0.87 (SD 0.097; range 0.67–1.00), average specificity was 0.77 (SD 0.188; range 0.17–1.0), and the average accuracy was 0.84 (SD 0.081; range 0.71–1.00). We identified variability in classification by criterion, with some scenarios being particularly challenging to classify (See Appendix 3 for variability of correct scenario classification by criterion).
Demographics, Sensitivity and Specificity Among Respondents.
Reliability of Classification of “Yes”—Overall and by Criterion.
*p < .05.
Among the 47 respondents who completed all 21 scenarios, 41 (87.2%) rated the overall helpfulness of the ACBC tool process, of which 65.9% (n = 27) reported the process as helpful or very helpful, with over 95% indicating the process as somewhat helpful or better (n = 39). Respondents commented on the positive utility of this tool: “This process was very good and made you think not only about the PH [public health] recommendations but also the lack of knowledge of local laws. Would love to see this implemented nationally.” Another commented that this tool might be most useful for less experienced CDR team members: “I can see this assisting board members, especially newer ones. My experience with fatality review and the risk factors in my state perhaps restricted my need to use the criteria to evaluate the caregiver behavior.” To see all key themes identified, please see Appendix 4.
Discussion
This study documents the development of a potential method for reducing variability and enhancing statistical reliability in the classification of child-maltreatment deaths among state and county child and infant fatality review teams. Development of the ACBC tool and selection of the six criteria was a lengthy process refined over several years by case reviewers in Alaska and a team of national experts. The six criteria were intended to capture most possible circumstances related to caregiver behavior with each criterion independent of the others. This review, however, detected issues with the developed and tested tool. First, some confusion on the word “OR” in criterion 1 led some respondents to not distinguish between homicides by a caregiver and those by another adult. Second, multiple respondents were unable to complete criterion 3 as they reported not knowing local state laws or differences in state or local laws. Third, respondents were unclear how to classify scenarios if information was missing or not included in the scenario, and some defaulted to “No” while others responded “Unknown.” Based on these lessons learned, the ACBC tool was modified and simplified (see Appendix 2 for final tool). This new modified tool should be further refined and researched to help identify a set of criteria that establishes clear caregiver behavior thresholds and reduces ambiguity.
Our results suggest that it is possible to classify child deaths caused by a wide range of caregiver behaviors using broad definition with high accuracy relative to the expert team classification. Reliability between reviewers, however, was only moderate, indicating some continued challenges in developing consistency. The criteria were most reliable and valid when the following was evident: (1) Determination of abuse or neglect by death certifiers; (2) Determination of abuse or neglect by law enforcement and/or child protective services; (3) Failure of caregivers to follow public health standards for injury prevention and child health; and (4) Failure of caregivers to follow medically prescribed advice. Criteria that had less reliability and validity included: (1) Failure to ensure children follow federal, state, or local legally mandated safety devices or behaviors (e.g., helmet use), and (2) Failure to protect from a recognized hazard due to failure to supervise within sight or sound.
Additionally, challenges in reviewing hypothetical scenarios were noted. Comments from respondents explained some of the difficulty in applying criterion of “failure to ensure children follow legally mandated safety devices or behaviors.” The location of death was not specified in the hypothetical scenarios and some respondents were not clear if laws existed in their jurisdictions leading many respondents to frequently select “Unknown”. The use of hypothetical scenarios was a large limitation of the study, but did provide for an initial assessment and reception of this type of classification model. Further research is needed to test the adoption and use of it within CDR teams in different jurisdictions. This initial study, however, provides positive reception and basic proof-of-concept of a method for directly applying a tiered definition that implements public health prevention caregiver thresholds and standards to support consistent classification of deaths resulting from caregiver behavior.
Several studies have demonstrated that lower levels of parental supervision are associated with increased injury risk in young children, even though there are challenges in quantifying the dimensions of supervision, such as proximity, continuity and quality of attention. (Morrongiello et al., 2006; Schnitzer et al., 2015) The findings in this study suggest that when reviewers are explicitly asked if caregiver behavior failed to protect from a recognized hazard due to failure to supervise within sight or sound, there is high reliability that supervision behavior can be captured and counted.
Using the standardized tool may help to eliminate reviewer bias. It can also help mitigate some of the bias that may result from community norms in child health and safety (e.g., religious, cultural, or other locally accepted practices). The tool can also be useful in assessing how the mechanisms of injury deaths (e.g., ATV use) are counted and how caregiver behaviors relate to these mechanisms (e.g., allowing young children to ride alone on an ATV). This association is a critical point that illustrates the importance of the public health model and approach.
Finally, respondents found the process to be mostly to very helpful when assessing the relationship between child maltreatment and caregiver behavior during these hypothetical reviews. Respondents commented that the process would be especially helpful for when caregiver behavior was more nuanced than homicide. Respondents also commented that this process clarified where more information was needed in order to make a determination, especially with behaviors contrary to public health recommendations and local laws.
While minor modifications to the ACBC tool resulted, this study supports the need for clear guidelines for reviewing circumstances of death related to caregiver behavior. Applying a public health approach to child maltreatment classification should also adopt new terminology to reflect a broader tiered definition. We proposed and used the term “child fatalities resulting from caregiver behavior,” which allows for capturing not only maltreatment deaths assigned by official sources but a broader spectrum of deaths, especially those often thought of as neglect or negligence related. Evaluating and classifying caregiver behavior as neglectful or negligent can have legal implications and is associated with blame and punitive approaches. This broader terminology can avoid confusion in measurements from various sources and focus on the on the mutable behavior that is amenable to public health prevention efforts.
We created the ACBC tool based on well-established evidence-based public health recommendations as the standard for evaluating deaths and adopted the terminology of “child fatalities resulting from caregiver behavior”. A possible approach for adoption would be to have an abstractor trained in applying the ACBC criteria attend CDR reviews and based on the review, make determinations using the guidelines. Consistency checks could be made periodically by having multiple abstractors classify the same deaths. This potential approach could help reduce variation in classifications between individuals and be implemented by other jurisdictions facilitating cross-jurisdiction comparisons.
Another potential option would be to incorporate the ACBC into the National Violent Death Reporting System (NVDRS). The NVDRS is an abstractor-based system that leverages multiple data sources to enhance classifications (Paulozzi et al., 2004). The established standardized framework could be adopted for abstractors to follow. This integration could help address current inconsistencies and variability in the CDR data alone. However, more research is needed to evaluate the adoption of an abstractor-based approach with the CDR or NVDRS data systems, or possible hybrid leveraging the strengths of each collection method. Such research should focus on the accuracy and reliability of classifications, as well as the practical implications of implementation.
Among the limitations of this study is the sample being drawn from a single state and hypothetical cases provided in place of actual cases. State variations exist for a variety of factors related to child mortality—for example, geographic differences in access to care following a potentially fatal incident (Wolf et al., 2017). Furthermore, recent research has indicated a large degree of variability in case information and quality both within different child review teams and across states (Mantell et al., 2024). Finally, the version of the tool that underwent final revisions has not been tested.
Increasing accuracy in the classification of maltreatment-related child fatalities by CDR teams is essential for improving the reliable quantification of these deaths while preserving the enhanced surveillance offered by multiple data sources and expert opinion. Utilizing classification tools like the ACBC can help standardize determinations across CDR staff while reducing variability. This has important implications for epidemiological and public health work, supporting more consistent data collection and ultimately enabling better informed prevention recommendations.
Supplemental Material
Supplemental Material - Child Fatalities Resulting From Caregiver Behavior: A Public Health Approach to Child Maltreatment Classification
Supplemental Material for Child Fatalities Resulting From Caregiver Behavior: A Public Health Approach to Child Maltreatment Classification by Jared W. Parrish, Melissa Bradley, Rachel Gallegos, Barbara Coopes, and Teresa Covington in Child Maltreatment
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ramuson Foundation, Title V MCH Block Grant (B04MC40114) and Travis Trust Fund.
Supplemental Material
Supplemental material for this article is available online.
Note
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
