Abstract
Introduction
ADHD is a common psychiatric disorder affecting 4 million children aged 3 to 17 years in the United States (Bloom & Dey, 2006). ADHD symptoms impair ADHD youth in multiple domains, including school functioning, substance use, family relationships, antisocial behavior, friendships, and lifestyle/health habits (Faraone et al., 1993; Greene et al., 2001; Greene et al., 1999; Wilens et al., 2004). Prospective, longitudinal studies show that even when ADHD symptoms remit in adolescence and adulthood, signs of impaired functioning frequently persist (Barkley, Murphy, & Fischer, 2010; Biederman, Mick, & Faraone, 2000). Such data led Rostain, Jensen, Connor, Miesle, and Faraone (2013) to propose that the standard of quality care of ADHD youth should include optimal outcomes throughout the day that go beyond modest reductions of ADHD symptoms. They suggested that future work develop reliable and valid tools to measure optimal outcomes in clinical trials.
One way to optimize ADHD outcomes would be to improve the functioning of children at home before they go to school. Although time sensitive rating scales have been developed for adult ADHD (Adler, Lynch, et al., 2011; Adler et al., 2013; Adler, Shaw, et al., 2011), clinical trials of ADHD treatments in youth typically neglect morning behaviors impaired by ADHD symptoms. Such behaviors are typically subsumed in the broad category of overall symptoms and/or functioning during the day. The child’s morning behavior routine, especially on school days, deserves special attention for several reasons. The time between waking and arriving at school requires children to adhere to a complex sequence of behaviors (e.g., waking, dressing, eating, self-hygiene/brushing teeth, gathering school books, etc.) that requires the time management and working memory skills that are frequently impaired by ADHD symptoms (Whalen et al., 2006). For example, Barkley and Cunningham (1979) showed that ADHD impairs early morning organization, self-care, preparing for the school day, and transportation to school. Failure to properly execute before-school routines puts ADHD youth at risk for being late to school and forgetting to bring completed homework and other materials to school, which could lead to academic and social difficulties as other children begin to view them as deviating from the norm. Moreover, parents may experience tremendous stress around facilitating the child’s before-school behavior and/or be late to work on days when their ADHD child is late to school. Notably, Whalen et al. (2006) showed that children’s ADHD symptoms reduced parenting effectiveness and that this effect was greater before school than after school. Given that the before-school often comprises 2 to 3 hr or up to 20% of the day, and that duress during this time period may impact the entire day, studies capturing treatment efficacy in the early morning are essential for developing optimal treatment strategies.
Despite these considerations, the naturalistic effects of ADHD medications on before-school activities are largely unknown (Barkley & Cunningham, 1979; Block et al., 2009; Cunningham & Barkley, 1978, 1979). The laboratory classroom paradigm has successfully documented the time course of treatment effects on ADHD symptoms and performance measures but that paradigm does not address the effects of medications on before-school behaviors (McGough et al., 2006). Only a few naturalistic clinical trials have assessed the efficacy of ADHD medications on before-school behaviors. One study of atomoxetine reported improvement in before-school ADHD symptoms (Whalen et al., 2006). In that study, children taking atomoxetine received less negative ratings on morning ratings of ADHD symptoms compared with stimulant-treated children. Notably, mothers reported higher levels of parent efficacy and satisfaction. Two open-label studies (Wehmeier et al., 2008) showed that ADHD-related impairments in morning behaviors improved with atomoxetine.
Two placebo-controlled clinical trials assessed impairments in morning behavior associated with ADHD symptoms using the Before-School Functioning Questionnaire (BSFQ). Wilens et al. (2013) studied 461 ADHD youth, age 6 to 12, who had had suboptimal response to stimulants in their prior treatment. Participants were randomized to either continued stimulant treatment plus guanfacine-extended release (GXR) in the morning (GXR AM) or evening (GXR PM), or continued stimulant treatment plus placebo. Parent-rated BSFQ scores indicated that morning functioning improved with GXR whether given in the morning or evening. In a crossover study of 30 ADHD youth 6 to 12 years of age, Wilens et al. (2010) compared the methylphenidate transdermal system (MTS) with a placebo transdermal system (PTS). Compared with the PTS, MTS significantly reduced the investigator-rated BSFQ total score but not the child self-rated BSFQ total score.
Although the ability of the BSFQ to separate drug and placebo is impressive, the scale has never been subject to a formal psychometric evaluation of its reliability and validity. To fill this gap in the literature, we used data from Wilens et al. (2010) to assess the following features of the BSFQ: reliability as measured by internal consistency, test–retest reliability, and concurrent validity. Given the success of the BSFQ in prior clinical trials, we hypothesized that it would show high levels of reliability and validity.
Method
Participants
Details of the study were previously reported by Wilens et al. (2010). Eligible participants, aged 6 to 12 years, had a diagnosis of ADHD by a clinical interview supplemented by a structured psychiatric interview. Excluded from the study were potential participants with a medical condition, or treatment of a medical condition, which would either jeopardize participant safety or affect the scientific merit of the study. Also excluded from the study were potential participants who had moderate to severe dermatological atopy, identified structural cardiac abnormalities, mental retardation (IQ < 70), organic brain disorders, and seizure disorders. Likewise, youth with a history of psychosis or bipolar disorder or youth with current clinically significant comorbid psychopathology such as anxiety disorders, Major Depressive Disorder, or Tourette’s syndrome were not enrolled. Due to the age limits of this study, the diagnoses of substance use, abuse, and dependence were not a part of the exclusion criteria. Participants with a history of no response or intolerable adverse effects to methylphenidate were excluded for ethical reasons. From November 2006 through November 2008, participants were recruited from advertisements in local and regional media as well as from clinical referrals including new and existing patients from the outpatient psychiatric clinic. The Partner’s Human Research Committee approved the study. Parents of participants completed an informed consent, and all participants aged 7 and older completed an assent prior to study entry.
Clinical Trial
This study was a randomized, controlled, 4-week crossover study. Eligible ADHD youth were washed out for a minimum of 1 week from their previous treatment (if applicable) before they started the medication protocol. Participants were randomized to either MTS or PTS. Participants began the trial on either MTS/PTS at 10 mg for 1 week and were then titrated upward to 20 mg for Week 2. Participants were then crossed over directly to the other treatment for the remaining 2 weeks, again initiated at 10 mg for Week 1 and 20 mg for Week 2. There was no washout period before the participants crossed over to the corresponding treatment.
All prescriptions were filled at the Massachusetts General Hospital (MGH) Pharmacy. The MTS and the PTS were both provided by Shire Pharmaceuticals. The 10 mg patch provided a delivered dose of 10 mg over 9 hr (12.5 cm2; 1.1 mg/h delivery rate). The 20 mg patch provided a delivered dose of 20 mg over 9 hr (25 cm2; 2.2 mg/h delivery rate; Prescribing Information—©2006 Shire Pharmaceuticals Ireland Limited). Participants’ parents were instructed on how to apply the patch and to alternate left and right hip placement each day. Parents were instructed to apply the patch to their child’s hip between 6 and 7 a.m. and remove the patch between 3 and 4 p.m. every day. Diary data showed that families applied the patch between 6 and 7 a.m. 79% of the time. Medical compliance was assessed by counting both used and unused patches at each follow-up visit and was >80% for all participants.
Assessments
ADHD–Rating Scale (ADHD-RS)
The primary outcome was the ADHD-RS (DuPaul, 1990; DuPaul, Power, Anastopoulos, & Reid, 1998). Physicians assessed each of the individual symptoms of ADHD in Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association, 1994; 0-3 on a scale of severity) across the day (total score ranged from 0 to 54). Psychometric properties have been established in children and the scale has been shown to be sensitive to stimulant drug effects (DuPaul, 1990; DuPaul et al., 1998). The time frame of the ADHD-RS was the past week. In an exploratory manner, after capturing the primary outcome, we examined more specifically morning ADHD symptoms, using the ADHD-RS to assess symptoms for the time period of 6 to 9 a.m. only (ADHD AM RS).
BSFQ
The BSFQ was created by Drs. Timothy E. Wilens and Paul G. Hammerness (Wilens et al., 2010; Wilens et al., 2013). The questionnaire has two components. The clinician-rated component based on a parent/child interview comprises 20 items that cover commonly reported areas of dysfunction in early morning activities associated with ADHD. Each item is rated on a severity scale of 0 to 3 (0 = none, 1 = mild, 2 = moderate, 3 = severe). The items address early morning, before-school activities (i.e., breakfast, hygiene, time awareness, getting to school, etc.). Before answering questions, parents were told to answer questions regarding the 6 to 9 a.m. period only (Wilens et al., 2010; Wilens et al., 2013).
The second component to the BSFQ is a self-report section completed collaboratively by the child, parent, and/or guardian. The self-report section assesses how the child felt, his or her relationship with parents and siblings, his or her success with morning activities or problems, and whether the child was proud of himself or herself over the past week from 6 to 9 a.m. The self-report questions range from 0 = no to 2 = a lot.
Clinical Global Impression–Severity (CGI-S) Ratings
The physician-rated CGI-S Ratings (National Institute for Mental Health, 1985) were used to measure the overall severity and improvement due to ADHD symptoms. Per the direction of the scale, to maintain its reliability and validity, information was collected about the entire day.
Behavioral Rating Inventory of Executive Functioning (BRIEF)
The BRIEF was completed by participant’s parents. This scale is a reliable and valid behavior rating scale of executive functioning in children and adolescents with a high internal consistency (Cronbach’s α: range = .80-.98), a moderate interrater reliability, and a high test–retest reliability correlation coefficient (range = .72-.85; G. Gioia, Isquith, Guy, & Kenworthy, 2000; Gioia, Isquith, Retzlaff, & Espy, 2002; Mahone et al., 2002; Vriezen & Pigott, 2002). The BRIEF is an 86-item questionnaire for parents and teachers of children that assesses everyday behavioral expressions of executive functions. Per the direction of the form, to maintain its reliability and validity, information was collected about the entire day.
The DSM-IV Global Assessment of Functioning (GAF) scale was also collected at each weekly visit. The GAF scale is a composite rating of an individual’s overall level of psychological, social, and occupational functioning during the entire day (1 = worst to 100 = best).
Data Analysis
We analyzed the BSFQ investigator ratings and self-ratings separately. We examined internal consistency reliability using Cronbach’s alpha, which assesses the internal homogeneity of the items comprising a scale. High levels of Cronbach’s alpha indicate that the items on the scale are measuring a unitary construct. We assessed test–retest reliability with the Pearson and intraclass correlation coefficients between baseline and Week 2 ratings for placebo participants in the first arm of the crossover. We assessed concurrent validity by computing correlations between the BSFQ scores and measures of ADHD or ADHD-associated features. We also computed a multivariate regression model to determine which of the concurrent validators was most predictive of the BSFQ scores.
Results
Thirty-six participants were screened and signed consent forms, but only 31 provided the data needed to assess the reliability and validity of the BSFQ. Demographic features of the sample (N = 31) are as follows: 84% were male; the mean age was 9.2 ± 1.8 years with a range of 6 to 12. In all, 88% were non-Hispanic Caucasians, 6% were Hispanic, and 6% were Asian American. The mean past GAF score was 53.0 ± 3.8, the mean GAF at baseline was 54.4 ± 1.9, 7% had repeated a grade in school, 7% had been placed in special classes, and 57% had required extra help in school. Regarding lifetime psychiatric history, 10% were positive for depression, 70% for oppositional defiant disorder, 17% for agoraphobia, 3.3% for obsessive compulsive disorder, 20% for generalized anxiety disorder, 3.3% for posttraumatic stress disorder, 13.3% for separation anxiety disorder, and 17% for conduct disorder.
Reliability
We first present the results for Cronbach’s alpha, which estimates reliability by assessing the intercorrelations among items. Table 1 gives the detailed results for the investigator-rated BSFQ. The overall alpha is .91 with the lower 95% confidence interval is 0.86 providing a “worst case” assessment of reliability. For each item, the table gives the correlation of the item with the total score minus the item and the alpha of the total score that excludes the item (the item-rest correlation). When an item is performing poorly, it will have a low item-rest correlation and the alpha coefficient with the item excluded will be markedly lower than the overall alpha. As is evident from Table 1, no single item is functioning poorly. Table 2 gives the detailed results for the self-rated BSFQ. The overall alpha is .81 and the lower 95% confidence interval is 0.71. The item-rest correlations are lower than observed for the investigator-rated BSFQ, but no single item is functioning especially poorly.
Reliability of the Parent-Rated Before-School Functioning Scale.
Reliability of the Self-Rated Before-School Functioning Scale.
We used the participants given placebo during the first arm of the crossover to compute test–retest reliability between ratings made at baseline and those made at Week 2. The data for these participants are shown in Figure 1, which give scatterplots of the Week 2 scores (vertical axis) and baseline scores (horizontal axis). For the investigator-rated total score (Figure 1a), the Pearson correlation was .60 (p = .02) and the intraclass correlation was .39 (p = .02). For the self-rated total score (Figure 1b), the Pearson correlation was −.18 (p = .54) and the intraclass correlation was −.18 (p = .74).
Test–retest reliability of the BSFQ: (a) parent-rated BSFQ total scores at baseline and Week 2, and (b) self-rated BSFQ total scores at baseline and Week 2.
Concurrent Validity
We computed correlations of the BSFQ total scores with other measures of ADHD and ADHD-associated clinical features. These results (Table 3) show high concurrent validity of the investigator-rated BSFQ. All correlations are high and significant. In contrast, the child self-rated BSFQ shows low levels of concurrent validity with few significant correlations. When we used a multivariate regression model to predict investigator-rated BSFQ scores from all of the concurrent validators in Table 3, only the ADHD-RS-AM score remained significant, t(17) = 4.0, p = .001. The only other predictor to approach significance was the BRIEF Inhibition score (p = .07). All other predictors were not significant (all ps > .20). In the multivariate analysis of the self-rated BSFQ, no predictors were significant (all ps > .11). Consistent with the reliability and validity differences we observed between the investigator and self-rated versions of the BSFQ, the correlation between the two scales was low (.12) and not significant (p = .25).
Concurrent Validity of the Before-School Functioning Questionnaire.
Note. GAF = Global Assessment of Functioning; ADHD-RS = ADHD–Rating Scale; ADHD-RS-AM = ADHD–Rating Scale morning ratings; CGI = Clinical Global Impressions Scale; BRIEF = Behavioral Rating Inventory of Executive Functioning.
Another indicator of validity is the ability of the scale to detect the effect of medications known to treat ADHD. In the prior report from this data set, Wilens et al. (2010) showed that investigator ratings on the BSFQ significantly improved after treatment with the MTS compared with PTS. Self-ratings did not significantly discriminate drug and placebo. Here we additionally report that, for self-ratings, the standardized mean difference (SMD) effect size (Cohen’s d) was low and negative (−.26) suggesting that placebo was non-significantly better than drug. In contrast, for the investigator ratings, the SMD was high (.93) and similar to the effect sizes for ADHD-RS (d = 0.90) and ADHD-RS-AM (d = 0.76) ratings.
Discussion
Our results show that the investigator-rated BSFQ is a reliable and valid measure of the effect of ADHD symptoms on the morning behavior of children prior to arriving at school. As measured by Cronbach’s alpha, the internal consistency reliability was very high, suggesting that the items of the BSFQ investigator–rated scale measure a homogeneous construct. Test–retest reliability was lower but was still substantial given the 2-week period between assessments. The concurrent validity of the investigator-rated BSFQ was also high as measured by its correlations with current functioning, ADHD symptoms and behavioral measures of executive dysfunction. Moreover, the magnitude of drug-placebo differences for the investigator-rated scale was high and similar to the magnitude of drug effects on the ADHD-RS. In contrast to this favorable evaluation of the investigator-rated BSFQ, the child/self-rated BSFQ showed lower levels of reliability, no significant evidence of concurrent validity, and no significant treatment effect.
The dramatic differences in the psychometric properties of the investigator and child/self-rated BSFQ are consistent with the fact that, in the prior report from this study, the investigator-rated version separated from placebo whereas the self-rated version did not (Wilens et al., 2010). This finding is also consistent with a study of stimulant-treated children supplemented with extended release guanfacine or placebo. In that study, Wilens et al. (2013) showed that the parent-rated BSFQ separated from placebo, but despite the age range being more broad (e.g., ages 6-17 years), the child self-ratings did not. Thus, the BSFQ is valid when a parent makes ratings directly on the BSFQ form (the guanfacine study) or when an interviewer elicits answers from the parent. Taken together, these two studies of youth provide strong evidence for substantial reliability and validity of the investigator-rated BSFQ (based on parent responses), some evidence for the parent-rated version and strong evidence for low reliability and validity of the self-rated version.
Our evidence for the validity of the BSFQ rests on its high and significant correlations with ratings of ADHD symptoms, impairment and behavioral measures of executive functioning. Notably, the BSFQ shares about 50% of its variance with ADHD-RS ratings when these latter ratings are based on behavior throughout the day. This finding indicates how important morning behaviors are in the parent’s overall assessment of the child. As expected, the BSFQ was even more strongly associated with ADHD-RS rating when these latter ratings were based on morning behaviors (74% of shared variance). The lack of complete overlap between these measures suggests that the BSFQ provides information above and beyond what is provided by the ADHD-RS-AM. The BSFQ’s correlations with BRIEF subscales were smaller than its correlations with the ADHD-RS. These lower correlations are consistent with the fact that not all ADHD youth show executive functioning deficits (Biederman et al., 2004). When we used a multivariate regression model to predict investigator-rated BSFQ scores from all of the concurrent validators, only the ADHD-RS-AM score remained significant. This finding suggests, quite sensibly, that the effects of other concurrent validators on BSFQ ratings are mediated by the ADHD symptoms that occur before school.
We have no simple explanation for the low reliability and validity of self-ratings. Compared with their parents, children may have little insight into their morning behaviors and may not see them as impairing. The self-rated scale also differs from the parent/investigator-rated scale in two important dimensions. First, the self-rated scale is shorter (14 items vs. 20 items). Other features being equal, longer scales are typically more reliable and valid than shorter scales. Second, the content of the items differs between the two scales. For example, the self-rated scale includes four items about getting along or arguing with parents and siblings. These items do not appear on the other version. Our findings are similar to investigations that show improved symptom and functional outcomes when assessed by parent report compared with child report (Reitman, Hummel, Franz, & Gross, 1998). Interestingly, in guidelines from the American Academies of Child Psychiatry (Pliszka, 2007) and Pediatrics (Wolraich et al., 2011), assessment of ADHD itself relies upon parent-completed but not child self-report. We leave it to future research to determine whether a reliable and valid self-rated form of the BSFQ could be created by using the same item set used for the parent scale.
Our conclusions are tempered by several methodological limitations. First and foremost, we used data from a study that was designed as a clinical trial rather than an evaluation of the psychometric properties of the BSFQ. Had we designed the study as a psychometric evaluation, we would have used a larger sample and would have conducted test–retest reliability over a shorter period of time (e.g., a few days). In this regard, use of clinical trial data may have underestimated our estimates of reliability, validity, and their statistical significance. In particular, although our comparison of baseline and Week 2 scores for placebo participants simulates test–retest reliability, any variance due to the placebo effect would have reduced our estimate of reliability. We also could not compute interrater reliability. We view this as a minor limitation because (a) it is impossible to compute interrater reliability for self-ratings and (b) investigator ratings by two investigators would have been filtered through the same parent given that typically only one parent is available for ratings. Also of note, our study was limited to children 6 to 12 years of age. These results may not generalize to adolescents or adults. Another scale would be needed to assess early morning behaviors relevant for adults. A time sensitive scale has been developed to rate ADHD symptoms in adults (Adler, Shaw, et al., 2011), but we know of no scale that goes beyond ADHD symptom ratings to assess adult-specific morning behaviors. Our scale was tested in largely Caucasian children who were part of a clinical trial and may not generalize to more comorbid children from diverse cultural backgrounds. Finally, although we had sufficient power to detect statistical significance, our sample was small, which means our negative findings should be viewed with caution.
Despite these limitations, our systematic analysis provides evidence for the reliability and validity of the investigator-rated BSFQ. Given the clinical and practical impact of ADHD on morning symptoms and functioning, we suggest that this scale be used in future trials of ADHD medications for youth. Further work is needed to create a reliable and valid self-rated BSFQ.
Footnotes
Authors’ Note
Stephen V. Faraone analyzed the data and wrote the first draft of the manuscript. Paul G. Hammerness and Timothy E. Wilens provided review, comments, and revisions. All authors listed on the manuscript have seen and approved the submission of this version of the manuscript and take full responsibility for the manuscript. Neither Shire nor Ironshore had a role in study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication. Dr. Timothy E. Wilens has published a book with Guilford Press, Straight Talk About Psychiatric Medications for Kids, and has co-edited ADHD Across the Lifespan (Cambridge University Press) and Comprehensive Clinical Psychiatry (Elsevier). Dr. Hammerness has published books with Harlequin Press: Organize Your Mind, Organize Your Life, and Greenwood Press: ADHD-Biographies of a Disease.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: In the past year, Dr. Faraone received consulting income, travel expenses, and/or research support from Ironshore, Shire, Akili Interactive Labs, Alcobra, VAYA, and Neurovance and research support from the National Institutes of Health (NIH). His institution is seeking a patent for the use of sodium-hydrogen exchange inhibitors in the treatment of ADHD. In previous years, he received consulting fees or was on Advisory Boards or participated in continuing medical education programs sponsored by Shire, Alcobra, Otsuka, McNeil, Janssen, Novartis, Pfizer, and Eli Lilly. Dr. Faraone receives royalties from books published by Guilford Press: Straight Talk About Your Child’s Mental Health and Oxford University Press: Schizophrenia: The Facts. Dr. Timothy E. Wilens receives or has received grant support from the following sources: NIH (National Institute on Drug Abuse [NIDA]) and Shire. Dr. Timothy E. Wilens is or has been a consultant for NIH (NIDA), Euthymics, Theravance, and TRIS (Tris Pharma); and the National Football League Program for Substances of Abuse (ERM Associates) and Minor/Major League Baseball.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The data collection was funded by an investigator-initiated grant through Shire Pharmaceutical Company (Registration ID: NCT00586157, 
) to Timothy E. Wilens. The analyses and writing of this manuscript was funded by a grant to S.V. Faraone by Ironshore Pharmaceuticals & Development, Inc. Dr. Hammerness has conducted continuing medical education (CME) activities supported by grants from Shire.