Agree to Disagree: Reliability of Seizure Frequency Assessments

Commentary

“Garbage in, garbage out” is a central problem in research and clinical care. Even with the most compelling research question, or the most advanced candidate therapy, our ability to understand effects is only as good as our ability to reduce noise and reliably measure key outcomes.

In epilepsy care, we clearly have a problem. Seizure reduction is a critical goal of antiseizure therapeutics. Yet, substantial sources of random and systematic bias could easily obscure the truth. Patients have wide variability across time in seizure frequency. But beyond natural variation, many other downstream measurement challenges exist—patients do not recognize many of their seizures, ¹ eyewitnesses may later poorly recall basic seizure characteristics, ² patients or caregivers may not communicate all episodes to the clinician, clinicians may not adequately document seizure frequencies even with an accurate history, and clinicians may come to different conclusions regarding the likelihood that any given event was in fact a seizure. With so many potential pitfalls, testable hypotheses exist regarding the reliability of seizure frequency, such as whether the major source of variability occurs at the clinician or the caregiver level, and whether reliability drops off depending on what exact seizure metric is used.

Dinger et al ³ recently studied this important problem. Their objective was to assess interrater reliability in seizure frequency both between clinician raters, and also comparing clinician versus caregiver raters. They used a large multicenter dataset of children who underwent vagus nerve stimulation for drug-resistantepilepsy. ⁴

First, they conducted a retrospective chart review of a single center in Toronto from 2007 to 2022 including 254 patients (1524 rated pairs). They compared 2 independent clinicians’ seizure frequency ratings at several timepoints (baseline, and then 3, 6, 12, 36, and 60 months). Agreement was generally strong, even though their Rater 1 tended to score seizure frequencies higher than their Rater 2. For example, intraclass correlation coefficients between clinician raters pooling all timepoints together were 0.94 for absolute seizure frequency and 0.91 for relative seizure frequency (both excellent, perfect would be 1). Agreement beyond chance for 2 dichotomized measures for change (50% or 90% seizure reduction compared to baseline) was also quite good (Kappa = 0.94 and 0.88 for 50% and 90% reduction, respectively). The major source of disagreement was incomplete or nonspecific chart data, namely wide-ranging documented seizure frequencies thus leading clinicians to each choose a different value. Other lesser sources of disagreement were errors reading the chart or variability in summarizing multiple seizure types.

Second, they conducted prospective evaluations across multiple centers from 2018 to 2025 including 140 patients (540 rated pairs). They compared ratings from one clinician versus the patient's caregiver at each timepoint. Results were notably worse. For example, the same intraclass correlation coefficients were 0.17 and 0.03 (almost no correlation), and Kappas were 0.28 and 0.26 (weak). Clinicians tended to interpret a somewhat rosier seizure frequency reduction than caregivers.

What can we learn from these data?

A major roadblock was not necessarily between-clinician unreliability, but rather total absence of the data needed to assess reliability in the first place. For example, in the retrospective cohort, 27% of desired timepoints could not be scored due to missing baseline or follow-up values. Worse, in the prospective cohort only 65% of patients had any assessments, and 49% of absolute seizure frequencies were missing. This presents a serious threat to validity to the degree that data are missing not at random. This story has also played out elsewhere—natural language processing to extract seizure frequencies. ⁵ In that case, algorithms accurately identified portions of text describing seizure frequencies … but only when present in the medical record. They say an ounce of prevention is worth a pound of cure. Documenting seizure frequency represents a quality metric set forth by the American Academy of Neurology. ⁶ Thus, these findings emphasize the need for standardized note templates to ensure that clinician documentation enables seizure tracking for both research and clinical purposes, whereas such a high rate of missingness in prospective data collection requires a more complex set of solutions regarding retention of human subjects to improve research quality.

Seizure changes represent only one of many outcomes by which to judge the success of antiseizure therapeutics, as it is well-known that epilepsy is a multidimensional condition in which seizures are only one of many contributors to biopsychosocial well-being. ⁹ Still, measuring seizure frequency is a key input for tracking response to treatment in both research and clinical settings. Hopefully concerted efforts to systematically document seizure frequency in the medical record alongside advances in tracking technologies could improve assessments.