Abstract
This pilot study investigated the effect of the Pediatric Symptom Checklist (PSC) on identification of and physician response to behavioral health (BH) concerns. Researchers reviewed 1211 charts of youth aged 4 to 16 years. Records were compared during baseline and an intervention consisting of implementation of the PSC to determine the rate of BH identification and pediatrician response. Access to PSC data resulted in a trivial difference in BH concerns identified by physicians and did not affect physician responses. This case study demonstrates that simply implementing BH screening in primary care may not result in improved outcomes for these children.
In primary care (PC), 15% to 25% of children present with a behavioral health (BH) concern.1,2 These problems are managed most effectively if detected and treated in childhood, 3 and PC has been deemed a major point of access to brief and early intervention. 4 Unfortunately, primary care physicians (PCPs) have struggled to fill this role 5 and many children with BH concerns are not detected early or referred for services. 6
One solution to early detection is to use BH screening. BH screening has been recommended as a cost-effective and efficient practice for early identification of BH concerns in PC.7,8 Previous research has focused on validating screening tools that identify patients at risk for psychopathology,9,10 including the Pediatric Symptom Checklist. 11
Despite the support for screening instruments, 12 it is not clear how BH screening affects actual physician behavior. Research to date on these issues has been mixed. One study found that there was a significant increase in rates of referrals when the PSC was implemented in health clinics serving a low-income, Hispanic community. 13 However, other studies have found that few children identified with BH concerns are referred for services14,15 and that screening alone may have little impact on physician practices in terms of referral or initiation of treatment.16,17 Thus, there is still much to learn about how BH screening affects physician identification and referral of children with BH concerns.
The purpose of this case study was to examine how the implementation of a BH screening instrument affected PCPs’ identification and response to BH concerns in a real-world setting.
Method
Participants and Setting
A privately owned pediatric PC clinic in a Midwestern city participated. Seven pediatricians (5 full time; 3 female), with an average of 11 years of experience (range 3-17 years) staffed this clinic. BH services were provided on-site 2 days per week by a psychologist.
Researchers reviewed electronic charts for 1211 patient visits with ages ranging from 4 to 16 years attending visits. Table 1 summarizes characteristics of the visits for the 2 groups (screening and no screening).
Descriptive Characteristics of the No Screening and Screening Groups (N = 1,211).
Percentage calculated as number of newly identified behavioral health concerns divided by total number of patients seen.
Percentage calculated as number of actions taken for newly identified behavioral health concerns divided by total number of patients newly identified.
Measure
Independent Variable
Implementation of the Pediatric Symptom Checklist (PSC) was the independent variable. The PSC is a 1-page, 35-item screening questionnaire designed to capture a parent’s impression of their child’s psychosocial functioning in a PC setting.11,18 Each item is assigned a score of 2 = often; 1 = sometimes; and 0 = never. Cut scores are provided for different ages. Parents are also asked 2 additional questions: if their child has a BH concern that he/she needs help with and if they would like additional services. It has been found to be reliable 18 and valid.18-20
Dependent Variables
The primary dependent variables were whether a BH concern was identified during the visit and whether the pediatrician addressed the BH concern. Newly identified BH concerns and physician responses were ascertained from a variety sources in the visit note (eg, physician made note of a BH concern, physician added a new BH diagnosis, physician prescribed a psychotropic medication for first time, noted psychology referral, counseled on behavior).
Procedure
A quasi-experimental research design was employed, consisting of 2 groups: no screening (5 weeks) and screening using the PSC (6 weeks). Prior to PSC implementation, pediatricians were provided with a description of the PSC. During PSC implementation, researchers approached caregivers and provided them with the PSC to complete. The researcher scored the PSC and placed it at the pediatrician’s station. Pediatricians initialed PSC forms after review. In all, 82.9% of the PSC forms scanned into patient charts were initialed. The administration of the PSC was managed by the research team to ensure that the majority of patients were screened; however, decision making regarding whether a child was at-risk for a BH concern and management of the BH concern was left to the PCP.
Charts were reviewed by the researchers. Coded data included whether the pediatrician identified a BH concern and type, pediatrician response to the BH concern, whether an existing BH concern had been identified and if services were being provided, whether the PSC was in the at-risk range, and whether the caregiver requested assistance or services on the PSC.
A second rater coded data for 20.3% of the sample. Interrater agreement was calculated as the number of instances of agreement divided by agreements plus disagreements multiplied by 100%. Percentage agreement for the primary dependent variables was as follows: identified BH concern = 94.7%; pediatrician addressed BH concern = 91.5%.
Results
Descriptive Analysis
All providers detected new behavioral health concerns in their patient population. The percentages for full-time providers ranged from 5.4% to 14.5% during the no screening phase and from 9.5% to 20.5% in the screening phase. All pediatricians identified a higher percentage of patients with a behavioral health concern during screening. Physician response to the concerns varied across full time providers ranging from 18.8% to 55.6% during no screening and 21.4% to 50% during screening.
In the no-screening group, 8.7% of visits included documentation of newly identified BH concern(s). In visits with newly identified BH concern(s), physicians responded to 37.3%. During the screening phase, new BH concern(s) were identified for 13.4% of the 544 patients, and of these, pediatricians responded to 29.3%.
Based on caregiver ratings on the PSC, 5.0% of patients (n = 27) exceeded the cut score indicating risk of a BH problem. Of the 27 identified as at-risk, 11.1% had a BH concern identified, 44.4% had a previous BH concern documented, and 48.1% had existing behavioral health services documented.
When responding to the 2 questions at the end of the PSC, 30 caregivers, regardless if the PSC was at risk, indicated that their child had a behavioral problem for which he or she needed help. Notes showed that pediatricians addressed six of these concerns. Twenty caregivers indicated interest in services to address their child’s behavioral problem, and of these, pediatricians documented taking action 5 times.
Figure 1 summarizes the types of newly identified BH. Most common problems identified were feeding, sleep, and attention deficit/hyperactivity disorder (ADHD) symptoms. The pediatricians were most likely to provide a behavioral health recommendation to address concerns (Figure 2).
Proportion of different types of newly identified behavioral health concerns by group (no screening n = 73, screening n = 58).
Proportion of types of actions taken by the pediatricians to address the newly identified behavioral health concerns by phase (phase A no screening, phase B screening).
Statistical Analysis
Behavioral Health Concern Identified
Results of a 2-variable chi-square test showed a significant difference in the proportion of BH concerns identified between the screening and no screening groups: χ2(N = 1211, 1) = 6.93, P = .008. In the no-screening group, behavioral health concerns were identified in 8.7% (n = 58) of visits and in the screening group behavioral health concerns were identified in 13.4% (n = 73) of visits. When reviewing the output from the chi-square test, we also made the observation that these 2 percentages were similar, yet the chi-square test significant. Thus, given the large sample (n = 1211), we decided to also consider effect size, which was indeed small (r = 0.075 and when converted Cohen’s d = 0.15). This suggests that although the chi-square test was statistically significant, the difference was not meaningful.
Pediatrician Response to Identified Behavioral Health Concern
A 2-variable chi-square test was also used to evaluate differences in the proportion of cases where BH concerns where identified and the pediatrician responded between the conditions. The difference in the proportion of patients with identified BH concerns where the pediatrician responded was not significantly different between groups: χ2(N = 131, 1) = 0.88, P = .348.
Discussion
Access to BH screening data in this pediatric office was not sufficient in increasing the number of identified BH concerns by pediatricians; nor did implementation of the PSC affect pediatrician response to identified BH concerns. During all 1211 visits in both the screening and no screening phases, pediatricians documented 10.8% of patients having an identified BH concerns and 8.8% of the patients having had a previously identified BH concern. However, physicians responded to only 33% of identified BH concerns. Of the patients whose caregivers completed the PSC, pediatricians documented identified BH concern(s) for 13.4%. Interestingly, only 5.0% of the patients whose caregivers completed a PSC exceeded the cut score (ie, fewer BH concerns were identified using the PSC than practice as usual). In the majority of cases where a child was identified as at-risk or parents indicated that they were interested in seeking assistance, pediatricians did not take action.
Pediatric settings have been identified as an optimal setting for behavioral screening, yet, our findings are consistent with research demonstrating that when BH concerns are identified in PC, problems are often undertreated and receive minimal follow-up.21,22
This study was conducted in a metro area clinic, with a population of primarily English speaking, and privately insured families. The clinic also has a colocated psychologist providing services. When considering this context, our findings are particularly discouraging given that preventative care adherence has been associated with patients being better insured, English speaking, and nonminority.23,24
Limitations
This study has some limitations that may have affected the results. First, implementation of the PSC by the researchers may have limited integration of the screening procedure into the day-to-day procedures. Second, there were no direct measures of procedural integrity. Pediatricians reviewed the majority of the completed PSCs; however, we do not know if pediatricians reviewed the PSC prior to, during, or following the appointment. Our review of records focused on the notes from the appointment during which the PSC was completed and any prior records. Pediatricians may have followed up on an elevated PSC score during a future encounter. Third, this was a sample of convenience and external validity is limited given that data were collected within 1 clinic. Finally, given that the data were collected from patients’ charts, it is possible that physicians discussed concerns with the patient, but failed to document information.
Future Directions
This study demonstrated that implementation of the PSC in this clinic did not improve the identification or response by pediatricians to identified BH concerns. This may be explained by some of the limitations, but it may also be an example of the gap between evidence-based practice and implementation.25,26 Research is needed to identify the essential, valued, feasible, and effective components of BH screening procedures within PC and evaluate outcomes over time. Our findings are aligned with the call for more information and training on the requisite systems for effective screening and follow-up within PC.27,28
The psychometric validity of BH screening measures has been well established in the research. If pediatricians are to commit to integrating screening into PC, however, more research is needed on implementation (eg, provider/staff education and materials, pay-for performance initiatives targeted at screening, automated identification such as chart flagging, improvements in office procedures) that results in improved access to care for children and adolescents identified at-risk, and ultimately, improved outcomes over time.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by grant numbers HRSA D40HP02597-08-00 and M01HP25184-01-00.