Abstract
The aim of this study was to examine the psychometric properties of the Psychosocial Risk Assessment in Pediatrics (PRAP). PRAP is a screening tool designed to assess pediatric patients who are at risk of experiencing elevated distress during health-care encounters. A cross-sectional observational study was conducted with 200 pediatric patients. Patient’s distress levels were observed during their health-care encounter using the Children’s Emotional Manifestation Scale (CEMS). Health-care staff and parents were asked to rate the patient’s level of cooperation and stress. Exploratory factor analysis supported a single latent factor structure of the PRAP tool. Cronbach’s α for internal reliability was .83. PRAP score was strongly correlated with CEMS score with r = .82 (p < .0001). The PRAP is a standardized, reliable, and valid method for health-care providers to assess a patient’s risk of experiencing significant distress during treatment or testing.
Studies have shown that children and adolescents undergoing even minor procedures can experience amplified feelings of stress and anxiety (Kazak et al., 2001; Melamed and Siegel, 1975; Roberts et al., 1981). Some of the short- or long-term effects of a stressful health-care encounter can include regression, separation anxiety, nightmares, eating problems, and increased fear and anxiety surrounding health care (Rennick et al., 2004). There are many contributing variables that have the potential to influence how a child reacts to a stressful health-care encounter. Having a better understanding of the most influential variables associated with a child’s ability to cope with health-care encounters is critical to conducting thorough and accurate assessments of the psychosocial needs of children in health-care.
Children begin experiencing stress and anxiety early on in development. Anxiety and fears are part of normal development for children. Children’s fears develop from stranger anxiety in infancy, fantastical fears in early childhood, concrete fears in the schoolager, and social fears as an adolescent. Children learn how to emotionally self-regulate their fears and anxiety as they develop. Infants rely on their caregivers to help regulate their emotional arousal (Morris et al., 2007). By the end of a child’s first year of life, as a result of maturation of the frontal cortex, children begin to learn how to emotionally self-regulate (Dawson et al., 1992). As children grow and develop their ability to self-regulate, their emotions become more refined (Blair and Diamond, 2008). While development of the ability to emotionally regulate is part of the normative developmental process for children, it is not clear why some children are better at this process than others. It is likely that certain individual, family, or environmental factors contribute to a child’s level of anxiety and ability to emotionally regulate their fears and anxieties.
A thorough review of the literature revealed several variables that consistently correlated with children’s risk of experiencing negative psychological sequelae during health-care encounters. These variables include anxiety (Bossert, 1994a, 1994b; Hart and Bossert, 1994), temperament (Kain et al., 1996; McClowry, 1990), exposure to invasive procedures (Al-Jundi and Mahmood, 2010; Kain et al., 1996; Rennick et al., 2004; Saylor et al., 1987), parental stress (Dahlquist et al., 1994; Kain et al., 1996; Mabe et al., 1991; Small and Melnyk, 2006), and parental support (Small and Melnyk, 2006; Wolfram et al.,1997). Wray et al. (2011) found that approximately 68% of parents of children who were hospitalized experienced significant levels of anxiety and stress during admission and at the time of discharge. Thus, it is important to look at parental variables as well as child and environmental variables when evaluating the factors related to increased distress in children in health care.
In order to reduce stress and increase coping for patients and families, a systematic and proactive approach for assessing psychosocial need and determining the type and level of intervention for a particular patient and family is greatly needed. Preidentification of patients ‘at risk’ of emotional distress and poor behavioral compliance during health-care encounters would allow targeted interventions to reduce distress, which could enhance staff productivity and increase the quality of the patient and family’s health-care experience.
Development of PRAP
The Psychosocial Risk Assessment in Pediatrics (PRAP) was developed by the investigators to evaluate a patient’s likelihood of experiencing distress with the demands of a procedure or exam. PRAP measures patients on 10 variables. The six variables that include communication, anxiety and coping, temperament, special needs, parent stress, and past health-care encounters, and additional considerations, are assessed by interviewing the parent using a standardized interview questionnaire. The other four variables that include invasiveness of procedure/encounter, parent support, and developmental impact are assessed using the clinicians’ observations and clinical judgment. Each variable consists of four different descriptions rated by level and intensity. Each of the scoring descriptions has been carefully operationalized for each variable. The score for the PRAP is obtained by reviewing the descriptions in each category and selecting the number that most closely describes the patient based on the responses from the parent’s interview and the assessor’s clinical judgment. Each risk factor is scored from zero to three. The numbers obtained for each category are added together to obtain a total score between 0 and 30. Higher scores indicate greater risk of manifestation of more negative emotional behavior. All variables were weighted equally for the purpose of this study.
Prior to the study, the PRAP was piloted by 12 child life specialists at one-free standing children’s hospital. A total of 650 PRAP assessments in both inpatient and outpatient areas were conducted. The majority of the child life specialists (75%) felt the PRAP was easy to utilize and implement in their area. They found the PRAP to be useful in assessing the patients they worked with and helpful in determining what interventions might be effective to utilize with a patient. Staff who rated the PRAP as more difficult to utilize worked in fast-paced environments with high patient volumes or areas that had a high number of unconscious or sedated patients. These child life specialists stated they would still utilize the PRAP on some of the patients they were referred to but did not feel it was practical to use with every patient who came into their area.
Four specific aims for the study were identified. First, PRAP will have a unidimensional structure. Second, PRAP score will be positively significantly associated with Children’s Emotional Manifestation Scale (CEMS) score, which will indicate convergent validity of the measure. Third, PRAP score will be significantly positively associated with parent and staff ratings of the patient’s level of distress and cooperation. Fourth, PRAP will demonstrate sufficiently positive, .70 or higher, interrater reliability and internal consistency to allow future generalized use.
Method
Participants
The study recruited 200 participants aged from 3 to 21 years. It was important to the researchers that the PRAP could be used with a variety of patients no matter what the age or developmental ability was. A large age range was recruited for the study in order to include patients who may be older chronologically but developmentally function at a much younger age. Patients with both typical and atypical development histories were included in the study. Typical development history was operationalized as hitting all developmental milestones to date with no concern or diagnosis for a developmental delay, disability, or mental health issue.
Procedure
Six certified child life specialists were trained on the PRAP. Training consisted of reviewing the PRAP training manual and attending a two-hour group training session. During the training session, the investigators were provided opportunities to practice using the PRAP by viewing in vivo videos of three PRAP interviews.
Institutional Review Board approval was obtained prior to conducting the study. Participants were recruited from the parents and caregivers of patients undergoing blood draws, dental cleanings, and surgeries requiring anesthesia induction at one pediatric medical center. Verbal consent was obtained from all parents. Patients’ consent was also obtained when patients were developmentally able. Patients who received support during the procedure from trained psychosocial support staff were excluded from the study. While the parent and patient were waiting for the procedure, a certified child life specialist assessed the psychological risk level of the patient using the PRAP. During roughly half (n = 97) of the interviews, another child life specialist listened in on the same interview to conduct their independent PRAP on the patient. The second rater remained blind to the first raters scoring. Participants experienced standard care during their scheduled procedure, which in these areas did not typically include psychosocial social support from a child life specialist or other professional. During the patient’s procedure, a separate researcher investigator, blinded to the PRAP score, used the CEMS (Li and Lopez, 2005) to rate the child’s distress behavior during the procedure. At the end of the procedure, parents and a health-care provider directly involved in the patient’s care during the procedure (phlebotomist, dental hygienist, nurse anesthetist, or anesthesiologist) were asked to fill out a distress and cooperation survey to rate the patient’s observed degree of distress on a five-point Likert-type scale. Previous studies have indicated that parent’s and health-care staff’s rating of patient distress and anxiety significantly agreed with the child’s self-reported distress level as well as behavioral observation ratings (Ambuel et al., 1992; Blake et al., 1996; Patel et al., 2011; Walco et al., 2005). End points of the scale were labeled not distressed/cooperative and very distressed/cooperative. Parents and staff were blinded to one another’s ratings.
Data analysis
Descriptive statistics were used to summarize sample demographics and outcome measurements. Exploratory factor analysis was conducted to explore the dimensions of PRAP. Convergent validity of PRAP was assessed by positive association with CEMS score as well as parent and staff observation ratings using general multiple linear regressions and correlation coefficients. Interrater reliability was evaluated by intraclass correlation coefficient (ICC) based on 97 ratings from two child life specialists. Internal consistency reliability was assessed by Cronbach’s α. All the analyses were carried out using Statistical Analysis System (SAS 9.2, SAS Institute Inc., Cary, NC). A two-sided significance level was set at .05.
Results
Participant characteristics
Of the 200 children in the study, 55% were male. The average age of the child in this sample was 8.1 years (SD = 4.2) as shown in Table 1, but their mean developmental age per parent report was slightly younger at 7.3 years (SD = 4.0) . Of the entire sample, 75% was Caucasian. There were 66 (33%) patients who underwent blood work, 67 (33.5%) for dental cleanings, and 67 (33.5%) for surgery induction. An overwhelming majority in the sample had prior experience with the procedure they were undergoing (82%). Most of the patients did not have an anxiolytic medication prior to the procedure (91%). The majority of patients at 69% (n = 138) were identified as typically developing. Based on parent report, 31% (n = 62) of patients were identified with some sort of special need, including autism, developmental delay, mental retardation, cerebral palsy, fragile X syndrome, and Rubinstein–Taybi syndrome.
Demographics.
Outcome measures
The average eight-item PRAP total score for participants was 8.2 (SD = 4.6). The average CEMS score for participants was 11.1 (SD = 5.5). The mean CEMS score for the participants identified with a special need was 14.37 (SD = 5.44). The mean CEMS score for the participants who did not have an identified special need was 9.64 (SD = 4.91). Participant with special needs exhibited significantly higher distress during their health-care encounter compared to participants who were typically developing, F(1, 198) = 37.19, p < .001. PRAP and CEMS scores did not vary significantly based on the type of procedure.
Validity
Factor analysis on PRAP
Exploratory factor analysis using PROC FACTOR was performed to explore the possible underlying structure of the PRAP without imposing any preconceived structure on the outcome. Of the initial 10 items, only 8 were included in the final analysis. Two items (additional consideration and parent/caregiver support to patients) were removed from the final analysis due to low loadings (<.3) and low communalities. A single latent factor structure of PRAP tool was suggested by the scree plot and Jeffrey Kaiser’s method.
The partial correlation matrix of all eight items used in factor analysis showed no high correlations (r > |.7|), but some redundancy in the data was evident. In the final factor analysis, only the first component has eigenvalue of 5.9 above one, explained more than 74% of total variance among the eight items in the instrument. And the ratio of the first two eigenvalues was 6.2:1 indicating dominance of a single factor.
This model had an overall Kaiser–Meyer–Olkin measure of sampling adequacy value of 79% indicating that the sample is large enough to detect the factors. The model had small residual values of correlation matrix and a root mean square residual of .09. The Barlett’s test of sphericity statistic was 610.8 with 28 degrees of freedom. This indicated the null hypothesis in that the correlation matrix is an identity matrix and is rejected at <.0001 level of significance. Thus, these results support the appropriateness of the factor analysis use for the study data. Table 2 reports loadings and communalities for the remaining eight items. The item with higher loading is more representative of the factor. Communality indicates how much percent of the variance in the item has been accounted for by this one latent factor.
Loadings and communalities in one latent factor structure including eight items.
Convergent validity of PRAP
Data indicated PRAP had excellent convergent validity (Table 3). PRAP had a statistically significant positive correlation with the CEMS score and parent and staff ratings of patient distress and cooperation (Table 4). The reliability and validity of the CEMS tool was evaluated with children from 7 to 12 years old. Looking just at the patients who fell outside the age range of 7–12 years, PRAP scores had a statistically significant positive correlation with parent and staff ratings of patient distress and cooperation (Table 5). Multiple regression analysis was applied to further study the relationship between CEMS score and PRAP score controlling for potential covariates including child’s age, developmental age, BMI, tolerance to past procedures, gender, race, premedication or not, past procedure or not, and procedure types. The parsimonious model with only significant covariates was shown in Table 6. In total, 68% of CEMS variance was explained by PRAP, child’s developmental age, Hispanic or not, and tolerance of past procedures. A one-point increase in the PRAP score resulted in a .82-point increase in the CEMS score controlling for developmental age, being Hispanic, and tolerance of past procedures. The partial η 2 was .48, which means that the PRAP by itself accounted for only 48% of the overall (the PRAP effect + error) variance. The standardized estimate of .70 indicated a strong correlation between these two variables. While being Hispanic and tolerance of past procedures were both statistically significant, the standardized estimate indicated a weak relationship.
Zero-order correlations between PRAP item score and CEMS score.
CEMS: Children’s Emotional Manifestation Scale; PRAP: Psychosocial Risk Assessment in Pediatrics.
Zero-order correlation coefficients among outcome variables.
CEMS: Children’s Emotional Manifestation Scale; PRAP: Psychosocial Risk Assessment in Pediatrics.
Note: All p values were less than .0001.
*Pearson’s correlation coefficient.
† Spearman correlation coefficient.
Zero-order correlation coefficients for parent and staff ratings of patient distress and cooperation in patients aged 3–6 years and 13–22 years.
PRAP: Psychosocial Risk Assessment in Pediatrics.
Note: All p values were less than .0001.
*Pearson’s correlation coefficient.
Multiple regression analysis predicting CEMS scores.
CEMS: Children’s Emotional Manifestation Scale; PRAP: Psychosocial Risk Assessment in Pediatrics; SE: standard error.
Multiple regression analysis was also used to test if the PRAP total score significantly predicted parents’ ratings of the patient’s level of distress and cooperation during the procedure. The results of the regression indicated that the PRAP score accounts for 44% of the variation in parental report of patient distress (R 2 = .44, F(2, 161) = 64.6, p < .0001) and 29% of the variation in parental report of patient cooperation (R 2 = .29, F(1, 198) = 81.8, p < .0001). The PRAP total score had a much stronger relationship to staff report on the patient’s distress and cooperation than when it is regressed on parental report on the patient’s distress and cooperation. About half of the variation (R 2 = .47, F(1, 198) = 178.2, p < .0001) in staff distress ratings can be explained by the PRAP score. There was also a strong correlation between the PRAP total score and staff report of the patient’s distress (r = .69, p < .0001). While the PRAP score does not account for as much of the variation in staff report of cooperation (R 2 = .36, F(2, 195) = 56.5, p < .001), there is still a moderately strong relationship between the PRAP and staff report of cooperation (r = .60, p < .0001).
Reliability
The interrater reliability assessed by ICC was .985 with 95% confidence interval from .980 to .988. PRAP score was strongly correlated with CEMS score with r = .82 (p < .0001). The Cronbach’s coefficient α for the PRAP was strong at .83, indicating high internal consistency reliability for the tool. Two items, parental support (α = .23) and additional considerations (α = .24) had a moderately low correlation with the total score when they were deleted from the scale. Anxiety had the highest correlation (α = .71) with the total score. Data in Table 3 show the correlations between all the PRAP item scores with CEMS score. Anxiety, developmental considerations, and invasiveness of procedure were most correlated with CEMS score with the correlation coefficients ranged from .65 to .68. Test–retest reliability was not evaluated in this study. Due to the variable nature of the factors assessed using the PRAP, it is unlikely that a patient’s PRAP score would remain constant from one health-care visit to the next.
Discussion
Despite long-standing recognition of the stressors associated with health-care encounters for children, an evidence-based approach for identifying patients at risk of experiencing increased distress during a health-care encounter have not been available. The PRAP is a standardized, reliable, and valid method for medical and allied health providers to objectively and efficiently assess a patient’s risk of experiencing significant distress during treatment. Good internal consistency and interrater reliability were established for the PRAP. In addition, strong construct validity for the PRAP was demonstrated. Notably, the PRAP total score was associated with patient acute distress as measured by the CEMS and parent and staff ratings of patient distress and cooperation. The PRAP offers a user friendly assessment tool that takes approximately 5–10 minutes to complete.
Results revealed that both the chronological and developmental age of the patient were associated with higher PRAP and CEMS scores during their health-care procedure. The developmental age of the patient was more highly correlated with patient PRAP and CEMS when compared with the patient’s chronological age. Thus, greater weight should be given to a patient’s developmental age versus chronological age.
Findings indicated a significant difference between the distress levels of participants with and without special needs such that participants with an identified special need scored significantly higher on the CEMS than typically developing participants. These patients can be medically complex and often require numerous services and more intense support than typically developing children (Golson et al., 2006). Future investigation into how children with developmental disabilities respond and cope during health-care encounters is warranted.
One limitation of the study is that the CEMS, which was used to measure participants’ distress levels, has only been validated with children aged from 7 to 12 years. The majority (59%) of the patients in the study fell in this 7–12-year age range. For children who did not fall into this age range, the study compared the parent and staff ratings of patient’s distress and level of cooperation to the patient’s PRAP score to verify the reliability and efficacy of using the PRAP with patients aged 3–6 years and 13–22 years (Table 5). Despite the favorable psychometric properties reported, further research is necessary to refine the measure further and ensure its reliability and validity in use with other pediatric populations. Future research evaluating how intervention strategies might mediate the distress of patients identified as at risk using the PRAP would be valuable to determine the level and type of intervention needed for each risk category in order to prevent negative effects. It is likely that environmental factors and variables associated with the health-care providers involved with the patient’s care would have a profound impact on the distress level of a patient. It was beyond the scope of this study to investigate how environmental and provider variables might affect a patient’s distress during a health-care encounter. Further investigation into how these variables would impact patient distress and coping is warranted. The PRAP might also help serve as an efficient and cost effective approach to determine how psychosocial resources and staffing should be allocated to meet the greatest need.
Footnotes
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
