Abstract
It is a pleasure to present a terrific group of papers in this issue of Journal of Neurotrauma. I would like to draw special attention to the article by Dr. Mikolic and colleagues titled “External Validation of the Post-Concussion Symptoms (PoCS) Rule for Predicting Mild Traumatic Brain Injury Outcome.” In this article, the authors investigated the accuracy of the recently developed PoCS 1 in a cohort of patients that was fully independent of the group of patients used to develop and initially validate the rule. The PoCS uses information available in the emergency department including age, sex, prior history of multiple traumatic brain injuries (TBIs), prior TBI in the past year, history of mental health disorder, presence of headache in the emergency department, cervical sprain, and hemorrhage on computed tomography scan to predict high, medium, and low risk of persistent self-reported symptoms at 3 months. Importantly, for those judged to be “medium risk” (the largest fraction), additional information at 7 days that can be obtained via phone call, including headache, sleep disturbance, fatigue, light sensitivity, and a high overall symptom burden, can further stratify participants into high versus low risk.
The authors found that the PoCS had a high sensitivity (93% or 85% depending on the cutoff used) for persistent post-concussion symptoms but a low specificity (28% or 37%), with an overall modest performance (Area under the receiver operating characteristic curve; AUC: 0.61). The sensitivity was similar, but the specificity was substantially lower than in the derivation and initial validation cohorts (54% and 45%, respectively). 1
This kind of work is incredibly important. It is common for prediction rules and prognostic scoring systems to be “overfitted,” meaning that they do well with the initial data but do not generalize to external data. There is no way to tell how well a given prediction rule or prognostic score will do until it is externally validated. The authors should be congratulated for their focus on rigor and reproducibility. Most of the prediction rules and prognostic scoring systems in our field have never been tested in external cohorts, so their reproducibility is completely unknown.
Many in the field who are interested in potentially performing interventional trials of candidate treatments starting in the acute phase after concussion will likely find this article especially impactful. Such clinical trials are often complicated by the challenge of deciding on inclusion criteria. If one includes a lot of participants who will recover well on their own, it dilutes the power of the trial and needlessly exposes people to potential risks. On the flip side, if one tightly restricts the inclusion criteria, many people who might potentially benefit will not be offered the intervention, and the generalizability of the results may be limited. While we still have a way to go before we have a fully optimized approach to predicting post-concussion symptoms, the new results from Mikolic et al. take a solid step in the right direction. Many more multicenter studies with robust sample sizes will be required to make meaningful progress in the field.