Abstract
Rationale:
The serotonergic and cholinergic systems are jointly involved in regulating sleep, but this balance is theorized to be disturbed in depressed individuals.
Objective:
The goal of this study was to use biological probes in healthy participants, to model the serotonergic/cholinergic imbalance of depression and its associated abnormalities in sleep structure.
Methods:
We tested 20 healthy female participants 18–30 years of age on four non-consecutive nights. Participants were given galantamine (a cholinergic agent), buspirone (a serotonergic agonist), both drugs together, or placebo before sleeping.
Results:
Buspirone suppressed tonic rapid eye movement (REM): There was a significant increase in REM latency (p < 0.001). Galantamine increased tonic REM sleep, leading to more time spent in REM (p < 0.001) and shorter REM latency (p < 0.01). Galantamine and buspirone given together were not significantly different from the placebo night by REM sleep measures, but disrupted sleep more than either drug alone.
Conclusions:
These findings are partially consistent with the cholinergic literature about sleep in depression, notably short REM latency, higher percentage of total sleep time spent in REM and increased sleep fragmentation. The prolonged REM latency and reduced percentage of REM with buspirone resembled the effect of selective serotonin reuptake inhibitor antidepressants on REM sleep.
Keywords
Introduction
One of the most commonly-reported symptoms of depression is disturbed sleep, with up to 90% of patients suffering from depression reporting symptoms of insomnia or hypersomnia (Riemann et al., 2001). Modern polysomnographic sleep research has repeatedly found altered sleep structure associated with depression, such as decreases in slow wave sleep (SWS), and increases in the duration and intensity of rapid eye movement (REM) sleep (Palagini et al., 2013). Recent longitudinal studies have found that abnormal REM sleep measures such as elevated REM density (RD), which is the number of rapid eye movements (REMs) per minute of REM sleep; and shorter REM latency (RL), the amount of time it takes to enter REM sleep after sleep onset, can predict the onset and course of depression in adolescents and adults (Augustinavicius et al., 2014; Modell et al., 2005). It is possible that the altered sleep structure is not necessarily caused by depressive episodes, but may reflect an underlying neuropathology; the neural mechanisms governing sleep regulation are inextricably linked with those regulating cognitive and affective systems (Riemann et al., 2001).
The cholinergic-aminergic imbalance hypothesis (Janowsky et al., 1972; Jouvet, 1972) is one framework for understanding the links between sleep and depression. The serotonergic and cholinergic systems are believed to be normally in balance, jointly controlling REM sleep. This balance is disturbed in depressed individuals, leading to abnormalities in REM sleep (Benca et al., 1996; Buysse et al., 1990; Foster, 1976).
Serotonin (5-HT) has been heavily implicated in the pathophysiology of depression and its effects on REM sleep are also well known. REM sleep is inhibited by the serotonergic neurons originating in the dorsal raphe nucleus (DRN) which projects into the cholinergic cells of the pontine nuclei (Jun et al., 2006) where the REMs are generated. Here, 5-HT acts primarily at the 5-HT1A receptor to suppress REM sleep and reduce the percentage of REM sleep (REM%) (Palagini et al., 2013).
The cholinergic system is also shown to be involved in the regulation of REM sleep. Cholinomimetics were found to shorten RL and increase RD, an effect that was found to be stronger in healthy relatives of depressed patients than in controls, implying that sensitivity to cholinergic stimulus could be a trait marker for depression (Riemann et al., 1994; Sitaram et al., 1982).
Most research in this area uses cross-sectional samples; however, not all patients with mood disorders display these REM sleep abnormalities. No studies have simultaneously probed the system with both cholinergic and serotonergic agents, in a repeated measures design with the same participants.
The goal of this study was to use biological probes in normal individuals to gain a clearer picture of the link between a serotonergic/cholinergic imbalance and its associated abnormalities in sleep structure. We used the 5-HT1A agonist buspirone to suppress REM sleep and a drug that increases acetylcholine (ACh), galantamine, to increase REM sleep in a crossed design, with a placebo control night.
Buspirone is a 5-HT1A agonist most commonly used either to treat anxiety symptoms or as an antidepressant, sometimes given in conjunction with other antidepressants to augment their effects (Altamura et al., 2013; Robinson et al., 1990). When buspirone is administered orally, it suppresses REM sleep, meaning that it must act more strongly at the post-synaptic receptors in the laterodorsal tegmentum (LDT) and the pedunculopontine (PPT) region than at the autoreceptors in the DRN, which would otherwise reduce 5HT transmission (Wilson et al., 2005). This finding supports the hypothesis that the post-synaptic 5-HT1A inhibitory receptors in the LDT/ PPT are being activated, thereby suppressing REM sleep. Buspirone also increased sleep fragmentation.
For our cholinergic probe we used galantamine, commonly prescribed for the treatment of mild-to-moderate dementia in Alzheimer’s disease. Galantamine is a short-acting anti-acetylcholinesterase that enhances central cholinergic activity. Acetlycholine is an agonist at muscarinic receptors (M1 and M2), as well as an allosteric modulator of nicotinic receptors (Jiang et al., 2013; Riemann et al., 1994). Animal studies have found that tonic REM (electroencephalogram (EEG) changes and atonia) is primarily mediated by muscarinic receptors, while phasic pontine-geniculate-occipital (PGO) waves associated with REMs are mediated by nicotinic receptors (Gillin et al., 1993). Previously, Riemann et al. (1994) found that galantamine, taken before sleeping, shortens RL, increases RD and suppresses SWS in healthy participants; and recommended its use for future pharmacological challenge studies. Galantamine is now available in a long-acting form that is ideal for sleep research and was used as our cholinergic probe in this study.
Our hypothesis was that our cholinergic probe would alter sleep in a way that mimics sleep patterns commonly found in major depressive disorder (MDD); while our serotonergic probe would cause the opposite effects, similar to the effects of selective serotonin reuptake inhibitors (SSRIs) on sleep. We also hypothesized that giving both probes together might cause them to mitigate each other’s effects.
Predictions
The 5-HT1A agonist buspirone would suppress REM sleep, leading to a reduced REM% in the night and longer RL.
The anti-cholinesterase galantamine would increase RD and shorten RL.
There would be an interactive effect, in that galantamine and buspirone given together may mitigate each other’s effects on tonic REM sleep (RL and REM%) but the REM periods would still have an elevated RD from the galantamine.
Both drugs would increase sleep fragmentation, leading to lower sleep efficiency (SE) and a larger amount of wake after sleep onset (WASO).
Materials and methods
Research protocol and experimental design
We recruited 22 female participants age 18–27, through the University of Ottawa’s online recruitment system, the Integrated System for Participation in Research (ISPR). Participants were chosen from the same gender and age group, in order to reduce variance in the dependent variables. Participants were asked to fill out three questionnaires: Sleep Disorders Questionnaire (Douglass et al., 1994), Beck Depression Inventory 2 (BDI-2) (Beck et al., 1996) and Epworth Sleepiness Scale (Johns, 1991). Participants were also asked about their general health and their immediate family’s psychiatric history, to determine eligibility. Exclusion criteria included a family history of major mental illness, a score above 12 on the BDI-2 (Kjaergaard et al., 2014), use of psychotropic drugs, significant health problems, and pregnancy or lactation.
Eligible participants were screened for psychiatric illness by the investigator, using the Diagnostic and Statistical Manual of Mental Disorders, 4th edition, (DSM-IV) Structured Clinical Interview for DSM-IV, Non-Patient version (SCID-NP) (APA, 2000). They were assessed by a physician prior to their participation in the procedure, via physical examination, measurement of blood pressure and pulse rate in the sitting position, and clinical laboratory tests (lactate dehydrogenase (LDH), Aspartate Aminotransferase (SGOT), bilirubin, serum creatinine, thyroid free triiodothyronine (FT3), thyroid-stimulating hormone (TSH), urinalysis and a complete blood count (CBC) including platelets). Participants with normal results on these measures underwent a screening nocturnal polysomnogram (NPSG), which ruled out sleep disorders such as sleep apnea. Because this study also included 8 hours of electrocardiogram (EKG) on two channels, it also served to exclude any participants with cardiac arrhythmias or other EKG abnormalities. The screening NPSG also served as an acclimatization to the sleep laboratory, to eliminate the ‘first-night effect’ from the subsequent drug trial nights (Israel et al., 2012).
Participants were recorded for four non-consecutive study nights in the sleep laboratory. Brief (non-validated) laboratory questionnaires were given, to assess sleepiness and perceived sleep quality at bedtime and in the morning.
On each of the study nights, participants received two double-blind test medications in identical-colored capsules: galantamine 16 mg (slow release), buspirone 15 mg, or placebo in a randomized, double-blind design. The order of the treatments was randomized with a latin square design, to avoid order effects of the treatments.
In the past, galantamine has been sold commercially in both immediate-release and slow-release forms. As of 2008, the manufacturer removed the immediate-release forms from the market, due to the possibility of gastrointestinal (GI) side-effects in elderly patients; accordingly, the geriatric starting dose is now either 8 mg or 16 mg of the slow-release galantamine (CPHA, 2008). Because our participants were all healthy young adults, the slow-release formula was selected in order that its effects on REM sleep would still be occurring in the latter part of the night and the 16-mg dose was chosen to maximize the potential response (Riemann et al., 1994). Buspirone’s suggested therapeutic dose is 20–30 mg in two to three divided doses, mainly to avoid daytime sedation. The starting dose is suggested at 5 mg three times a day, to avoid sedation (CPHA, 2008). Because sedation is not an issue in a sleep study and since we needed to guarantee measureable effects throughout the night, a dose of 15 mg at bedtime was selected.
The half-lives of the drugs (galantamine at 8 hours and buspirone at 2–10 hours) require a 48-hour wash-out period after each night in the sleep lab, in order to allow five half-lives to elapse before the next test drug (CPHA, 2008). Therefore, the participants slept for at least one night at home between each of the lab nights 2, 3, 4 and 5 (so the median number of nights at home between study nights was seven; mean 11.2 (SD 12.4)). To further reduce variance between participants, they were asked not to use any drugs, alcohol or caffeine after noon, for the duration of the study. Strenuous physical activity was not permitted within 4 hours of bedtime on the sleep study nights, to avoid possible induction of initial insomnia. The medications were taken orally in the laboratory, 1–2 hours before bedtime and at least 2 hours after the last meal, for optimal timing and absorption. Participants chose their bedtime between 9 pm and midnight, according to when they usually fell asleep at home.
Sleep methodology
Sleep was recorded by a 10-electrode EEG array (C3-A2, C4-A1, O1-A2, O2-A1, EOG, EKG lead II and EMG) and scored visually from the C3/A2 electrode trace, using standard American Academy of Sleep Medicine (AASM) measures (Iber et al., 2007). Standard nocturnal polysomnography instrumentation was used on Night 1, to screen out participants suffering from a clinical sleep disorder such as sleep apnea or periodic limb movement disorder. This equipment included a microphone to detect snoring, a motion sensor, finger pulse oximetry, respiration belts, pressure and temperature airflow sensors on the upper lip, and electrodes on the anterior tibial areas for leg movements. None of our participants was found to have clinical sleep disorders.
On Night 2 through Night 5, only sensors for sleep scoring, EMG, EEG, and EOG electrodes were used. Sleep waveforms were recorded on the Somnologica® computerized sleep recording system (Embla Systems, Ottawa, ON, Canada) using digital amplification and filtering. Sleep staging on 30-second epochs was done visually by a single experienced researcher using the Stellate Harmonie® system, version 6.1 (Stellate Systems, Montreal, QC, Canada). Individual REMs were identified using Harmonie’s proprietary software, which searches for inverse mV/sec slope rates in the two EOG channels, employs a noise-reduction algorithm and requires that a total duration criteria be met for the REM to be counted. The algorithms for this eye movement detection are published (Agarwal and Gotman, 2001).
Ethics approval was granted by Health Canada and by the Research Ethics Board of the Royal Ottawa Mental Health Center. The study ran from June 2010 to January 2014.
Statistical analysis
All variables were assessed for normal distribution via normality plots and measures of skew, and kurtosis using the SPSS statistical program. Variance-stabilizing transformations (logit and natural log) were done if required. These data were appropriate for a repeated measures analysis of variance (ANOVA) with repeated measures on participants. Because these measures are known to vary with age, we first ran a MANCOVA, with age as a covariate. Age was found to have no significant effect on the measures, likely due to the small age range of the participants. Participants’ menstrual cycles were sufficiently randomized by the study design that they did not need to be considered in the analysis.
We then performed a multivariate repeated-measures MANOVA on the following variables: RL, RD, the percentage of time asleep spent in REM (REM%), the percentage of sleep time spent in stage N1 (N1%), which is a usually transitory stage often described as a light doze; the percentage of sleep time spent in N2 (N2%), which is fully but not deeply asleep; the percentage of sleep time spent in N3 (N3%), which is a deeper stage of sleep previously referred to as slow wave sleep; WASO; sleep onset latency (SOL); total sleep time (TST); percentage of time spent awake (Wake%); and the number of awakenings during the night. There were four levels of the within-group variable, ‘drug condition’: baseline (placebo), buspirone, galantamine and the combination of the two drugs. The MANOVA was significant, leading us to perform univariate ANOVAs on each measure. Pairwise comparisons of the drug nights versus placebo nights were made when the univariate ANOVA was significant. The significance level required for post-hoc comparisons was adjusted by the Bonferroni method.
Results
The overall repeated-measures MANOVA found a significant effect of drug condition on sleep measures (F33,147 = 4.574; p < 0.001). The result of the univariate ANOVA performed on WASO is illustrated in Figure 1 and the univariate results for the rest of the individual sleep measures are shown in Table 1.

WASO in minutes (error bars show SD) for each drug condition. WASO was significantly higher in the combination night compared to the other three nights; and higher in both the buspirone and galantamine nights, compared to the placebo night.
Results from the univariate ANOVA post-hoc analyses.
Indicates a significant difference, compared to the placebo night, in pair-wise comparisons.
ANOVA: analysis of variance; N1%, N2%, N3%: percentage of time asleep spent in Stage N1, N2 or N3; NS: not significant; RD: REM density; REM: rapid eye movement; REM%: percentage of time asleep spent in REM; RL: REM latency; SE: sleep efficiency, which is TST divided by time in bed; SOL: sleep onset latency; TST: total sleep time; Wake%: percentage of time in bed spent awake.
Buspirone
The 5-HT1A agonist buspirone suppressed REM sleep: There was a significant increase in RL (p < 0.001) on the buspirone night, compared to the placebo night. REM% and RD trended lower on the buspirone night, but these differences were not significant. Buspirone also disrupted sleep, approximately doubling WASO, Wake%, N1% and the number of awakenings during the night.
Galantamine
The anti-cholinesterase galantamine did not significantly increase RD. It did increase tonic REM sleep, leading to a higher REM% (p < 0.001) and lower RL (p < 0.01) on the galantamine nights, compared to placebo. Galantamine also disrupted sleep, significantly increasing WASO, Wake%, N1% and the number of awakenings during the night, as well as lowering N3%.
Buspirone and galantamine combination
Galantamine and buspirone given together negated the effects of each other on REM sleep measures: The drug combination nights were not significantly different in REM% or RL, compared to the placebo night.
Galantamine and buspirone together increased sleep fragmentation and lowered sleep quality more than either did when taken alone: The combination drug night had a higher number of awakenings (p < 0.05), WASO (p < 0.001), N1% (p < 0.001) and reduced N3% (p < 0.001); as well as a significantly lower sleep efficiency (p < 0.01), compared to the placebo night. As shown in Figure 1, the amount of WASO was significantly higher in the combination night than in either of the single-drug nights.
Sleep disruption and dreaming
Two participants had to be removed from the analysis, due to galantamine disrupting their sleep too much: One awoke after 3 hours of sleep with symptoms of nausea and vomiting on both the galantamine nights and withdrew from the study; the second also awoke after approximately 3 hours of sleep on both of the galantamine nights and could not get back to sleep before morning. Of the remaining 20 participants, seven reported nausea in the morning after taking galantamine with buspirone and three reported nausea in the morning after galantamine alone. Responses on the morning questionnaire indicated that the subjective sleep perception accurately corresponded to their sleep recording, in all drug conditions.
Galantamine was found to dramatically increase the number of dreams reported, as well as their unpleasantness. On the placebo night, 20 participants reported a total of 10 dreams, one of which was “a stressful disaster.” On the buspirone night there were 16 dreams, one of which was “scary.” On the galantamine night participants reported 32 dreams, with five participants reporting dreams that were “weird”, “scary” or “nightmares.”
Galantamine also appeared to induce dreams of false awakenings, with five participants reporting ‘inception-style’ layered dreams or dreaming that they awoke in the lab and found something wrong (e.g. they were “sick and mute” or “afraid of the technician”). The combination night had a similar effect of increasing dreams and false awakenings, with 42 dreams reported in total, five participants reporting dreams of false awakenings, and six reporting dreams that were unpleasant or nightmares.
Discussion
This study demonstrates that the effects of a serotonergic agonist and a cholinergic agent on sleep support the predictions of the cholinergic-aminergic imbalance hypothesis of depression.
Our serotonergic agonist, buspirone, suppressed REM sleep by increasing REM latency. Total REM% was not significantly lower, but this may have been because buspirone was not available in a slow-release preparation and was wearing off in the second half of the night, when the majority of REM sleep occurs. Previous studies using this drug found that it decreased the amount of time spent in REM sleep (Wilson et al., 2005). In animals, direct perfusion of 5-HT1A agonists in the LDT/PPT reduced REM sleep, which is consistent with findings that 5-HT1A receptors in these areas are post-synaptic and inhibitory (McCarley et al., 1995). Our findings supported the hypothesis that, in humans, it is primarily the post-synaptic 5-HT1A receptors in the LDT/ PPT that are being activated by buspirone, and subsequently, suppressing REM sleep.
Our cholinergic agent, galantamine, decreased N3%, increased REM% and decreased RL, but it had no significant effect on RD. To explain its different effects upon tonic and phasic REM activity, it is helpful to examine animal models of REM sleep. A study by Velazquez-Moctezuma et al. (1989) found that M2 agonists increase RD significantly, when injected directly into the LDT/PPT. Their model for the REM control system hypothesizes that phasic REM events (RD), as well as the associated EEG spikes in the PGO waves, are promoted by cholinergic projections from the parabrachial regions of the pons (LDT/PPT). They postulate that REM sleep is tonically suppressed (lower total REM% and increased RL) by the noradrenergic and serotonergic neurons in the DRN and locus coeruleus, by way of their inhibition of the LDT/PPT. The LDT/PPT also has internal cholinergic trigger or ‘burst’ activity, mediated by a cholinergic M2 receptor. Therefore, the timing of REM sleep is controlled separately from the bursts of phasic activity in REM: Tonic REM sleep is hastened by M1 agonists and delayed by M1 antagonists, while phasic RD is increased by M2 agonists and reduced by M2 antagonists.
This model explains our findings and implies that, while galantamine increases ACh, it acts mainly at the M1 receptors. Galantamine also has an alerting effect, due to its allosteric action on the nicotinic acetylcholine receptor alpha four subunit (CHRNA4), which could be the mechanism of its disruption of sleep (Jiang et al., 2013). Nicotinic receptors have also been linked to the generation of PGO waves in animals; and galantamine’s nicotinic properties have been proposed as a mechanism for increased RD (Riemann et al., 1994). While we did not see effects on RD in contrast to Riemann’s proposal that there is a nicotinic effect via galantamine, we did find that there was an increase in alertness shown by a higher Wake%, more numerous awakenings, and less Stage N3 ‘deep’ slow-wave sleep.
Our prediction regarding the interaction between the two drugs was validated. Galantamine and buspirone given together tended to cancel out each other’s effects on REM sleep latency and REM%, while their deleterious effects on sleep quality appeared to be additive: Both drugs increased sleep fragmentation, leading to lower sleep efficiency and a higher percentage of time in bed spent awake. The higher cholinergic tone led to more stage REM sleep, similar to the finding that depressed patients have a higher percentage of total sleep time spent in REM. The way that buspirone counteracted this effect was similar to how antidepressants act in depressed patients: By lowering the proportion of sleep spent in REM.
Sleep was fragmented by both drugs, something to be conscious of when prescribing these medications clinically. Galantamine, in particular, was found to disturb sleep; although it is a drug that patients often take at bedtime, to avoid the side effect of nausea. It is unclear from this study whether sleep disruption is a long-term side effect, or if it would cease to be a problem after a patient had adapted to the drug. Nightmares and insomnia can be disturbing to patients, as well as their caregivers. Being aware of these effects on sleep might encourage patients to compensate with other means of improving sleep quality or quantity.
The finding that a cholinergic agent caused nightmares and interrupted sleep is consistent with previous findings: Galantamine and other cholinergic agonists can also increase the rate of dreaming, especially lucid dreams, as well as increasing the risk of sleep paralysis (Corbo et al., 2003; Rogers et al., 1998). A recent study with Alzheimer patients found that galantamine does not have a significant effect on insomnia and there is only a “mild” increase in reported nightmares (Stahl et al., 2004). Our findings contradict the latter, possibly because our participants were younger than the usual geriatric patient population and may have had stronger adverse reactions. Another possible reason for the discrepancy in reported nightmares is that Stahl’s numbers for dreams were taken from physicians’ accounts, whereas our participants were asked about sleep quality and dreams immediately upon waking; the Alzheimer’s patients may have forgotten their unpleasant dreams or considered them not important enough to report, artificially lowering the numbers. In the future, similar research efforts might benefit from administering a validated dream questionnaire to their participants, to more thoroughly assess the effects of drugs on dreaming.
We found no change in crude REM density. While it is possible that a more subtle method of analysis that examines the patterns of discrete events, such as a Markov analysis (Boukadoum and Ktonas, 1988; Douglass et al., 1992), would find an effect of the buspirone or galantamine on the rapid eye movements in REM sleep, it is more probable that RD is controlled by some neurotransmitter system other than ACh, under normal physiological conditions. It has been suggested that dopamine (DA) affects the relationship between ACh and RD (Tandon et al., 1999). A repeated measures study that manipulated ACh and DA levels simultaneously could offer valuable insight on the effects of that interaction on sleep.
This research had several limitations. Our participants did not include those of male gender, older participants, nor patient populations; so our results cannot be generalized to these populations without further study. We were also unable to record the serum levels of galantamine and buspirone before each study night, to ensure there was complete washout of the drugs. Future studies may wish to include this as part of their procedure, for extra rigor.
Conclusion
This study provided a direct test of the aminergic-cholinergic imbalance model of depression in human participants, for the first time. Our findings demonstrate that the effects of buspirone and galantamine on sleep largely support the predictions of the cholinergic-aminergic imbalance hypothesis: Buspirone suppressed REM sleep, while galantamine increased it. These findings were consistent with the literature about the effects of depression on sleep structure, except for the absence of RD findings. With both drugs, sleep was fragmented, which is something to be conscious of when prescribing these medications.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the University of Ottawa, Institute of Mental Health Research.
