Speech-based machine learning for detecting neuropsychiatric symptoms in Alzheimer's disease

Abstract

Background

Neuropsychiatric symptoms (NPS) are common in Alzheimer's disease (AD) and mild cognitive impairment (MCI), yet their detection relies on subjective assessments. Speech features offer a promising objective biomarker for NPS, reflecting emotional and cognitive states. However, existing studies are limited in terms of scale and duration.

Objective

This study aims to characterize acoustic features associated with NPS in early cognitive decline using Automated Assessment Model-Mini-Mental State Examination framework, and to evaluate machine learning classifiers for identifying indicators of NPS.

Methods

Speech data from 647 clinically diagnosed AD or MCI patients were collected and split into training and test sets in a 6:4 ratio. The training set was used for feature selection and model development, while test set was used for performance evaluation. The Synthetic Minority Over-Sampling Technique was applied to address class imbalance. Twelve machine learning models were trained to classify NPS categories. The best-performing models were evaluated, and SHapley Additive exPlanations (SHAP) were used to analyze feature importance.

Results

The ExtraTrees model outperformed the others in identifying patterns associated with NPS categories, with cross-validated AUCs ranging from 0.869 to 0.901. SHAP revealed spectral_entropy_std and kurtosis_energy as key features across multiple NPS categories.

Conclusions

This study demonstrates that short speech samples obtained during the MMSE can identify acoustic patterns associated with NPS in clinically diagnosed AD and MCI using machine learning. Given the single-center design and absence of external validation, model outputs should be interpreted as directional signals to raise clinical awareness rather than as definitive diagnostic determinations.

Keywords

Alzheimer's disease machine learning mild cognitive impairment neuropsychiatric symptoms speech analysis

Introduction

Neuropsychiatric symptoms (NPS) are highly prevalent across dementia and mild cognitive impairment (MCI), and occur in up to 80% of patients with Alzheimer's diseases (AD).^1,2 NPS includes both affective symptoms (such as depression, anxiety, and apathy) and behavioral symptoms (such as agitation and hallucinations).^3,4 These symptoms not only severely impact patients’ quality of life but also increase the burden on caregivers and accelerate disease progression.⁵ As early warning indicators of disease, NPS holds significant value for early diagnosis and intervention.⁶ Therefore, achieving accurate and timely identification of these symptoms has become a critical challenge in both clinical and research fields.

Despite the predictive value of NPS in the early stages of dementia, their identification and assessment still face numerous challenges.⁷ Currently, the evaluation of NPS primarily relies on clinical scales and proxy-report tools, such as the Neuropsychiatric Inventory (NPI).⁸ However, these methods are highly subjective, relying on caregiver reports and being influenced by recall bias and cultural factors, which can ultimately compromise the accuracy and timeliness of NPS identification and diagnosis.⁹ In recent years, some studies have employed natural language processing (NLP) techniques to automatically extract NPS information from physicians’ free-text records.¹⁰ While NLP has shown promising classification performance in electronic health record data, its results remain limited by the variability in documentation styles and inconsistencies in physicians’ descriptions.¹¹ Moreover, NPS in the early stages are often mild or insidious and may go unrecognized or unreported by patients and caregivers, further complicating early detection.^12,13 The limitations of traditional methods have prompted researchers to explore more objective and automated detection approaches to enable more sensitive and continuous evaluation of NPS.¹⁴

With recent advancements in speech analysis technology, researchers have begun exploring the use of speech features as objective biomarkers for NPS.¹⁵ Speech analysis techniques can automatically extract paralinguistic features (such as intonation, voice quality, and temporal characteristics), which are closely related to emotional states and cognitive functions.^16–19 These features provide real-time insights into patients’ emotional and cognitive states without relying on subjective reports. Studies have shown significant associations between specific speech features and NPS, such as apathy, depression, and anxiety.²⁰ For example, apathetic patients often exhibit slower speech rates and reduced speech proportion, while anxious patients display significantly increased speech quality characteristics, such as signal-to-noise ratio.²¹ These findings suggest new possibilities for identifying and assessing NPS using speech analysis technology. Additionally, compared to traditional questionnaires and clinical interviews, speech data collection is easier and can be implemented through mobile devices, making it suitable for large-scale screening and continuous tracking.²²

However, current studies on identifying NPS in MCI using speech features are limited, with small sample sizes and short durations of speech.^23,24 Grounded in our previously developed Automated Assessment Model-Mini-Mental State Examination (AAM-MMSE) framework, this study used standardized spoken prompts to administer and capture the full MMSE process, ensuring consistency and clinical replicability.²⁵ By applying a large-scale speech model to remove background noise and extract patient speech features, we aim to aim to characterize acoustic features associated with NPS in both MCI and clinically diagnosed AD, and to evaluate the feasibility of machine learning classifiers in identifying patterns that may serve as indicators of NPS. Integrating this approach with traditional clinical assessments may ultimately support earlier recognition of behavioral symptoms. We emphasize that model outputs should be interpreted as a composite acoustic signature, reflecting the intertwined dimensions of cognitive and behavioral symptomatology, rather than as NPS-specific diagnostic classifiers.

Methods

Participants and diagnosis procedures

Patients who attended the Memory Clinic of the Department of Geriatrics at the First Affiliated Hospital of Chongqing Medical University between January 1, 2023, and December 31, 2024, were consecutively enrolled. All participants were clinically diagnosed as having either MCI, based on Petersen's criteria (2004), or AD, according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Revised (DSM-IV-R) criteria and the 2011 National Institute of Aging-the Alzheimer's association (NIA-AA) diagnostic guidelines for AD. Each patient underwent a comprehensive neuropsychological test covering multiple cognitive domains, as well as an evaluation using the NPI. The flow diagram of patient enrollment is summarized in Figure 1.

Figure 1.

The flowchart of the proposed model encompassing three key phases: (1) data acquisition and preprocessing; (2) construction and assessment of the classification model; (3)feature important assessment.

Participants were eligible for inclusion if they were aged 50 years or older, voluntarily agreed to participate in the study, and had adequate hearing and vision to complete the neuropsychological assessments. Individuals were excluded if they had a history of alcohol or substance abuse, major psychiatric disorders, intracranial neoplasms, or previous stroke. The study protocol received ethical approval from the Medical Ethics Committee of the First Affiliated Hospital of Chongqing Medical University (approval number: 20212901; approval date: May 10, 2021). Informed consent was obtained from all participants or their legal representatives after a comprehensive explanation of the study was provided. A comprehensive battery of neuropsychological tests was administered to evaluate general cognitive functioning and NPS.

Global cognitive status was assessed using the MMSE.^26,27 NPS were evaluated using the NPI.^28,29 For analysis, each NPS category (e.g., depression, anxiety, apathy) was treated as a binary outcome variable based on clinical assessment. These symptoms represent clinically documented NPS that commonly co-occur in individuals with AD and MCI. It is important to clarify what these outcome variables represent in the context of this study. They reflect the presence or absence of clinically documented symptoms within a population of individuals with clinically diagnosed AD or MCI. Model predictions for each category should therefore be interpreted as indicating the degree to which speech features align with the acoustic patterns associated with that symptom in this specific clinical population, not as a standalone diagnostic tool for the symptom itself. This distinction is critical for appropriately interpreting the findings.

Speech data collection and processing

Speech data were directly collected during the patient's first clinical visit using our team's self-developed AAM-MMSE systems.²⁵ This model employs standardized spoken prompts to administer the MMSE and records the entire assessment process, without requiring additional verbal responses from the patient, thereby minimizing psychological burden. The recorded speech data (approximately 8 min) were processed to remove background noise following the methods described in the study by Du et al.³⁰

During preprocessing, recordings with excessive noise or severe degradation were excluded. Remaining mild background noise was attenuated using DeepComplexCRN (available at: https://github.com/huyanxin/DeepComplexCRN, accessed on 12 June 2025). The audio was then resampled to 16 kHz, and a 32-ms Hann window with 50% overlap was applied during short-time Fourier transform. Each participant contributed 10–12 recordings across MMSE subtasks, forming a reproducible dataset for acoustic analysis. Notably, DeepComplexCRN served strictly as an offline denoising module to reduce environmental noise before feature extraction. In contrast, CosyVoice2 was utilized not for denoising but as a representation model whose embeddings exhibit intrinsic robustness to residual noise and speaker variability.

Speech features were then extracted from the processed speech data. This approach to speech analysis does not consider the semantic content of the participant's speech, thereby enhancing its applicability in clinical settings. The features were extracted across several key dimensions, including mean_energy (the average energy of the speech signal, reflecting overall vocal intensity), power_sum (the total power of the signal over time, indicating the cumulative vocal strength throughout the recording), and speaking_ratio (the proportion of time during which the participant was speaking, used to quantify speech activity versus silence or pauses), among others. An overview and explanation of all extracted speech features are provided in Table 1. Supplemental Table 3 presents the mapping between clinical constructs and speech features.

Table 1.

Overview and explanation of extracted speech features.

Speech features	Explanation
mean_energy	Average energy of the signal, reflecting vocal intensity
std_energy	Standard deviation of energy across frames, indicating variability
max_energy	Maximum frame-level energy observed
min_energy	Minimum frame-level energy observed
energy_range	Difference between max and min energy, representing dynamic vocal range
skew_energy	Skewness of the energy distribution, indicating asymmetry
kurtosis_energy	Kurtosis of the energy distribution, reflecting peakedness
delta_mean	Mean of the first-order derivative, measuring temporal change in energy
delta_std	Standard deviation of first-order derivative features
delta2_mean	Mean of the second-order derivative, reflecting acceleration in energy change
delta2_std	Standard deviation of second-order derivative
power_sum	Total power over the signal duration
power_ratio	Ratio of power in voiced vs unvoiced segments
pause_ratio	Proportion of time spent in silence
speaking_ratio	Proportion of time spent speaking
spectral_entropy_mean	Mean of spectral entropy, measuring signal complexity
spectral_entropy_std	Standard deviation of spectral entropy
spectral_flatness_mean	Mean of spectral flatness, indicating tonality vs noisiness
spectral_flatness_std	Standard deviation of spectral flatness
spectral_contrast_mean	Average spectral contrast across frequency bands
spectral_contrast_std	Variability in spectral contrast

Model establishment and performance assessment

The patients in each group were divided into a training set and a test set in a 6:4 ratio. The training set was used for feature selection and model development, while the test set was used to evaluate the performance of the established classification models. In clinical research, sample imbalance between positive and negative cases is a common issue that can impair the predictive performance of classification models. To address this, the Synthetic Minority Over-Sampling Technique (SMOTE) was applied to the training datasets using the imblearn library in Python. Model optimization was conducted via Bayesian optimization, implemented through the bayesian-optimization package. To predict NPS in patients with MCI or clinically diagnosed AD, multiple classification algorithms were employed, including Logistic Regression (LR), Random Forest (RF), ExtraTrees, Gradient Boosting, AdaBoost, k-nearest neighbor (k-NN), DecisionTree, Support Vector Machine (SVM), Naive Bayes, Linear Discriminant Analysis (LDA), Quadratic discriminant analysis (QDA) and Light Gradient Boosting Machine (LightGBM). All modeling procedures were carried out using Python's sklearn package. Model performance was assessed using accuracy, precision, recall, F1 score, and area under the curve (AUC). The flowchart is shown in Figure 1.

Feature importance assessment

Interpreting machine learning models remains a significant challenge, especially when justifying the auxiliary diagnostic outcomes for specific patient groups. To address this, we employed (SHAP), a widely recognized method available through the Python shap library. SHAP enables both global and local interpretation of model predictions.³¹ In our study, it was used to compute and visualize feature importance based on the final classification model.

Results

Patient characteristics

Between January 1, 2023, and December 31, 2024, a total of 1485 patients presenting with memory impairment as their primary symptom were recruited for this study. Of these, 125 patients with non-Alzheimer's disease (Non-AD) dementia, 60 patients with other diseases, and 115 patients with unclear diagnoses were initially excluded. Additionally, 430 patients who were either unable or refused to complete the full cognitive assessment were excluded. Moreover, 108 patients were excluded due to excessive noise in their voice recordings, which prevented the extraction of reliable acoustic features. Ultimately, 647 patients with a clinical diagnosis of AD (based on NIA-AA criteria) or MCI (based on Petersen's criteria) were enrolled. A summary of the patient enrollment process is presented in Figure 2.

Figure 2.

Flowchart illustrating the process of patient recruitment and enrollment.

A total of 647 dementia patients were enrolled in this study, among whom 130 patients (20.09%) were diagnosed without NPS (Non-NPS group), 517 patients (79.91%) with NPS (NPS group). Detailed data is shown in Table 2.

Table 2.

Demographic and general evaluative outcomes of dementia patients with or without NPS.

Variable	Non-NPS (n = 130)	NPS (n = 517)	p
Female, N (%)	88 (67.69%)	338 (65.38%)	0.6935
Dementia
MCI, N (%)	79 (60.76%)	151 (29.21%)	<0.0001
clinically diagnosed AD, N (%)	51 (39.24%)	366 (70.79%)	<0.0001
Age (years)
Median	73	74	0.4867
Interquartile	69–78	69–79	0.4867
Education (years)
Median	9	9	0.9723
Interquartile	6–11	6–12	0.9723
MMSE
Median	21	15	<0.0001
Interquartile	16–24	10–22	<0.0001

Performance of classification models

The speech features in the training sets were used as input variables for each classification model, with the output variable indicating the presence or absence of each NPS category. The performance of each NPS category classification model, verified by 5-fold cross-validation in the training set, are shown in Supplemental Table 1.

After successfully establishing the models, we assessed their performance on the test set to determine the most effective classifier. Model performance was evaluated by comparing AUC values, given that hyperparameter tuning in this study was explicitly designed to maximize AUC. The results are presented in Supplemental Table 2. The ExtraTrees model consistently emerged as the best-performing model for all NPS categories. The classifiers demonstrated excellent performance in cross-validation, with AUC values ranging from 0.869 to 0.901 (Table 3). The AUC values for each NPS category were as follows: delusion (AUC = 0.871), hallucination (AUC = 0.901), depression (AUC = 0.881), anxiety (AUC = 0.877), apathy (AUC = 0.869), irritability (AUC = 0.880), aberrant motor behavior (AMB) (AUC = 0.877), sleep disturbance (AUC = 0.878), and appetite and eating disorder (AED) (AUC = 0.886). The specific performance of the ExtraTrees model for each NPS category is shown in Figure 3.

Figure 3.

The confusion matrix of ExtraTrees models. (a) delusion, (b) hallucination, (c) depression, (d) anxiety, (e) apathy, (f) irritability, (g) aberrant motor behavior (AMB), (h) sleep disturbance, (i) appetite and eating disorder (AED).

Table 3.

Performance evaluation table of ExtraTrees in training set and test set.

		Training set					Test set
NPS category	sample size, N (%)	Accuracy	Precision	Recall	F1 Score	AUC	Accuracy	Precision	Recall	F1 Score	AUC
delusion	198, (30.60%)	0.765	0.750	0.794	0.771	0.842	0.792	0.667	0.633	0.649	0.871
hallucination	126, (19.47%)	0.844	0.824	0.875	0.849	0.930	0.877	0.673	0.673	0.673	0.901
depression	168, (25.97%)	0.784	0.761	0.832	0.794	0.888	0.800	0.638	0.620	0.629	0.881
anxiety	220, (34.00%)	0.693	0.674	0.740	0.700	0.782	0.773	0.701	0.600	0.647	0.877
apathy	203, (31.38%)	0.744	0.722	0.803	0.755	0.848	0.773	0.628	0.620	0.624	0.869
irritability	228, (35.24%)	0.687	0.667	0.743	0.701	0.768	0.785	0.702	0.656	0.678	0.880
AMB	180, (27.82%)	0.787	0.767	0.825	0.791	0.870	0.819	0.641	0.631	0.636	0.877
sleep disturbance	149, (23.03%)	0.816	0.783	0.874	0.826	0.908	0.838	0.600	0.623	0.611	0.878
AED	80, (12.36%)	0.913	0.900	0.930	0.914	0.976	0.892	0.629	0.595	0.611	0.886

AMB: aberrant motor behavior; AED: appetite and eating disorder.

Visualization of feature importance

To quantify the contribution of individual speech features in identifying patterns associated with NPS among patients with dementia, we applied SHAP values to interpret the feature importance derived from the classification models. Since the max_energy feature showed identical values across all subjects and lacked discriminatory power, it was excluded from the analysis. The remaining 20 speech features were ranked according to their mean absolute SHAP values for each task. The SHAP value of each feature in ExtraTrees model were showed in Figure 4, and the average SHAP value of each feature in ExtraTrees model is showed in Supplemental Figure 1.

Figure 4.

SHAP value of each feature in ExtraTrees model. (a) delusion, (b) hallucination, (c) depression, (d) anxiety, (e) apathy, (f) irritability, (g) aberrant motor behavior (AMB), (h) sleep disturbance, (i) appetite and eating disorder (AED).

We further analyzed the top 30% speech features for each NPS-related classification task based on SHAP value rankings.³² The summary of this analysis is presented in Figure 5, where red bubbles indicate features with positive correlations, blue bubbles indicate negative correlations, and larger bubbles represent higher-ranking features. From the results, we observed that two features, spectral_entropy_std and kurtosis_energy, were consistently ranked among the top contributors across six NPS categories, suggesting shared acoustic patterns across these symptoms. Additionally, std_energy and delta_mean were found to be highly important in five NPS categories.

Figure 5.

The top 30% speech features for each NPS category. positive correlations in red; negative correlations in blue. AMB: aberrant motor behavior; AED: appetite and eating disorder. (Color figure available online).

Discussion

In this study, we applied machine learning classifiers to non-invasive speech features collected during the MMSE to characterize acoustic patterns associated with NPS and to evaluate the feasibility of this approach for identifying indicators of NPS. Among these models, the ExtraTrees classifier achieved the highest AUC score (ranging from 0.869 to 0.901) across tasks, indicating its superior performance in detecting speech-based patterns associated with NPS. In this context, model outputs reflect acoustic patterns associated with each symptom category, rather than serving as standalone diagnostic classifiers.

Clinical implications

NPS are highly prevalent in dementia and have a significant impact on patient outcomes and caregiver burden.^28,33 However, early-stage NPS, such as subthreshold depression, sleep disturbance, or apathy, often remain undetected in routine clinical settings-especially during the MCI or early AD stages, when symptoms may not yet significantly interfere with daily functioning.^34,35 This diagnostic gap underscores the need for unobtrusive, scalable methods to facilitate early detection.

This study demonstrates the feasibility of using short speech record during the standardized cognitive evaluation of the MMSE to passively and non-invasively identify acoustic patterns associated with these symptoms. Importantly, this approach requires no additional time or burden on patients or caregivers, as it leverages speech naturally produced during routine cognitive assessment. Notably, the results were consistent with recent study showing that acoustic and linguistic markers can predict neuropsychiatric changes even in prodromal stages of cognitive decline.^15,36

Given that NPS are frequently under-reported or mischaracterized in clinical records,^4,11 automated speech-based tools may offer an effective adjunct for raising awareness of potential NPS, particularly in primary care or memory clinic settings where psychiatric expertise may be limited.³⁷ By enabling earlier recognition of behavioral signals that might otherwise go unnoticed, these tools hold promise for improving patient outcomes and potentially slowing the progression of cognitive and behavioral deterioration.

It is critical to emphasize that the model outputs described herein should be interpreted strictly as supportive signals to aid clinical awareness and screening, not as diagnostic indicators, particularly in real-world clinical settings. In practice, this means that a positive output should never be used in isolation to make a clinical diagnosis or treatment decision; rather, it serves only to flag a patient for further targeted assessment or closer observation. Given the exploratory nature of this single-center study, any clinical application of these findings should adhere to defined guardrails. The suggestion should only be considered if it aligns with the clinician's gestalt, and initial use should be confined to low-stakes, non-binding decisions. These precautions are essential until the model's performance is externally validated. Furthermore, these findings are grounded in the premise that the study population consisted of individuals with clinically diagnosed AD and MCI (based on NIA-AA and Petersen's criteria), rather than neuropathologically confirmed cases. The acoustic patterns identified are therefore associated with NPS as they present in clinically diagnosed populations, a context that reflects real-world practice but does not guarantee specificity to AD neuropathology. This distinction does not diminish the clinical relevance of the findings, as clinical diagnosis remains the basis for real-world decision-making; rather, it clarifies the boundaries of what the study can claim.

Feature importance and shared biomarkers

By applying SHAP-based analysis to ExtraTrees model, we identified speech features that were consistently important across multiple symptom classifiers. Specifically, spectral_entropy_std and kurtosis_energy were among the top 30% of features for six NPS categories, while dstd_energy and elta_mean were important in five categories. These shared features may represent cross-cutting digital biomarkers of neuropsychiatric burden. Previous studies have shown that vocal prosody, variability, and energy fluctuations are linked to affective states and psychiatric disorders.^38,39 Our findings extend these insights by identifying features with cross-symptom utility in a controlled clinical context.

Spectral entropy measures the complexity and information content of speech signals. Fluctuations in spectral entropy may indicate disrupted neural integration or neurotransmitter imbalances affecting cognitive and emotional regulation. Studies have shown that spectral entropy can serve as a predictive indicator of working memory changes, reflecting alterations in brain functional states.^40,41 Kurtosis energy describes the peakedness of the energy distribution in speech signals; changes in kurtosis may relate to abnormalities in muscle tone or respiratory control during phonation, which have been associated with affective disorders and neurodegeneration.⁴²

To date, the potential of std_energy and delta_mean as reliable acoustic markers spanning multiple NPS has not been systematically investigated. While previous research on speech-based digital biomarkers has predominantly emphasized global prosodic features, such as pitch, pause duration, or speaking rate, the relevance of finer-grained temporal and energy-related parameters has received comparatively limited attention. The present findings indicate that both std_energy and delta_mean demonstrate stable and clinically relevant discriminative value across at least six distinct NPS, supporting their potential as cross-cutting acoustic markers in this domain.^15,43

Methodological strengths and interpretability

In this study, we employed 12 classification machine learning classifiers to identify NPS in individuals with MCI or clinically diagnosed AD using non-invasive speech data. The strong performance of the ExtraTrees classifiers can be attributed to the model's ability to handle high-dimensional, nonlinear relationships commonly present in acoustic features. As a tree-based ensemble method, ExtraTrees introduces greater randomness during node splitting compared to Random Forests, which helps reduce model variance and improve generalization. This is important given the limited sample sizes and potential noise in clinical speech recordings.⁴⁴

The effectiveness of ExtraTrees in our study aligns with recent findings in related fields. For instance, in a recent study evaluating machine learning algorithms for Parkinson's disease diagnosis using speech features found that the ExtraTrees model achieved outstanding predictive performance, with an accuracy of 94.34%.⁴⁵ Similarly, Wu et al. applied ExtraTrees in a multilingual classifier for early Alzheimer's detection and reported high classification accuracy across languages.⁴⁶ Menne et al. also highlighted the utility of tree-based methods in identifying speech markers of major depressive disorder.¹⁷ These studies support our findings and collectively suggest that ExtraTrees is well-suited for speech-based screening of NPS in neurodegenerative populations. Our results underscore the potential of integrating machine learning models like ExtraTrees into digital screening tools that enhance clinical decision-making and enable earlier intervention strategies.

To address concerns about model interpretability, we integrated SHAP to elucidate the contribution of individual features to model predictions. This approach allows for transparent identification of the most influential speech characteristics for each NPS, which not only enhances the explainability of the model but also facilitates potential clinical acceptance and integration.^47,48 By providing interpretable outputs, SHAP-based analysis supports a more informed decision-making process and bridges the gap between complex machine learning techniques and real-world clinical utility.

Comparison with previous work

While previous studies have applied speech analysis to detect depression, anxiety, or cognitive impairment individually,^15,39 few have systematically targeted a wide array of NPS using speech produced during a standardized cognitive task, such as the MMSE. This structured context allows for consistent elicitation of speech across individuals, minimizing variability due to content or topic. Additionally, treating each symptom as an independent classification task enables us to disentangle their potentially shared yet distinct underlying mechanisms.

Recent work has also highlighted the under-documentation of NPS in electronic health records,¹¹ which presents a critical barrier to timely recognition and treatment. Our approach–automated, explainable, and deployable during routine cognitive screening-offers a scalable solution to complement existing diagnostic practices.

Limitations and future directions

This study has limitations. First, the single-center design and lack of external validation mean that our findings should be considered hypothesis-generating rather than definitive. While internal validation strategies (e.g., cross-validation, bootstrapping) provide a high-confidence estimate of potential performance, they are known to yield optimistic performance estimates. The reported AUCs (0.869–0.901) are therefore likely overestimates of true performance in new settings; optimism-corrected estimates suggest the generalizable AUC may be lower. Multi-center external validation is a critical next step.

Second, a key interpretive consideration is that our acoustic models did not adjust for global cognition (e.g., total MMSE score) or diagnostic stage. This represents a structural confounding issue inherent to the study design: NPS are not independent of disease progression, and therefore the extent to which the observed predictive performance reflects NPS-specific signals versus overall disease severity remains unclear. This limitation cannot be fully resolved through post-hoc statistical adjustments. Critically, this structural confounding is further compounded by the fact that our study population was defined by clinical diagnosis rather than neuropathological confirmation, meaning that the relationship between speech features, NPS, and disease stage is embedded within clinically defined categories. NPS are core features of AD/MCI and are intrinsically linked to disease progression. The strong associations observed in this study suggest that speech acoustics may serve as a sensitive composite signature of NPS in the context of neurodegeneration, encompassing both cognitive and behavioral dimensions. Future studies with prospective designs, such as cohorts stratified by disease stage, longitudinal tracking of symptom onset, or comparators with non-AD neurodegeneration, are needed to further disentangle the unique contributions of speech features to individual NPS domains.

Third, the clinical interpretability of our findings is constrained by the absence of an external validation cohort. To address this, we propose a blueprint for future validation. An ideal external validation cohort should be prospectively collected from a demographically and clinically distinct population (e.g., a non-academic center with different disease prevalence). We propose that successful external validation would require the model to achieve a minimum performance threshold before it could be considered for broader clinical deployment. We are actively seeking collaborations to facilitate this essential validation, as the current model should not be adopted clinically without it.

Conclusion

In summary, this study demonstrates the feasibility of using short, spontaneous speech samples collected during MMSE administration to identify acoustic patterns associated with NPS in individuals with MCI or clinically diagnosed AD. Among the 12 machine learning models evaluated, the ExtraTrees classifier achieved the best performance, and SHAP-based interpretation provided insights into key acoustic markers, such as spectral_entropy_std, kurtosis_energy, std_energy and delta_mean, that may reflect shared neurobiological mechanisms across multiple symptoms. Given the single-center design and absence of external validation, model outputs should be interpreted as directional signals to raise clinical awareness rather than as diagnostic determinations. These findings are grounded in the premise that diagnoses were clinical rather than neuropathologically confirmed; thus, the acoustic patterns identified are associated with NPS as they present in clinically diagnosed populations. Future multi-center external validation is needed.

Supplemental Material

sj-docx-1-alz-10.1177_13872877261456293 - Supplemental material for Speech-based machine learning for detecting neuropsychiatric symptoms in Alzheimer's disease

Supplemental material, sj-docx-1-alz-10.1177_13872877261456293 for Speech-based machine learning for detecting neuropsychiatric symptoms in Alzheimer's disease by Yingxi Chen, Haodong Huang, Yurou He, Shiyu Chen, Ying Tan, Jiaqi Song, Lihua Chen, Xiaoqin Wang, Yang Lü and Weihua Yu in Journal of Alzheimer's Disease

Footnotes

Acknowledgements

The authors have no acknowledgments to report.

ORCID iDs

Yang Lü

Weihua Yu

Ethical considerations

The protocol was approved by the Medical Ethics Committee of The First Affiliated Hospital of Chongqing Medical University (approval number: 20212901; (approval time: 10 May 2021). The study conforms with the World Medical Association Declaration of Helsinki published on the website of the Journal of American Medical Association.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Author contribution(s)

Yingxi Chen: Conceptualization; Data curation; Funding acquisition; Writing – original draft.

Haodong Huang: Conceptualization; Methodology; Software; Writing – original draft.

Yurou He: Investigation; Resources; Writing – review & editing.

Shiyu Chen: Conceptualization; Resources; Writing – review & editing.

Ying Tan: Formal analysis; Resources; Writing – review & editing.

Jiaqi Song: Methodology; Resources; Writing – review & editing.

Lihua Chen: Methodology; Resources; Writing – review & editing.

Xiaoqin Wang: Investigation; Resources; Writing – review & editing.

Yang Lü: Formal analysis; Funding acquisition; Writing – review & editing.

Weihua Yu: Conceptualization; Funding acquisition; Writing – review & editing.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by grants from STI2030-Major Projects (No. 2021ZD0201802), Chongqing Talent Plan (cstc2022ycjh-bgzxm0184), Science Innovation Programs Led by the Academicians in Chongqing under Project (2022YSZX-JSX0002CSTB), Key Project of Technological Innovation and Application Development of Chongqing Science & Technology Bureau (CSTC2021jscxgksb-N0020), Program for Youth Innovation in Future Medicine, Chongqing Medical University (W0166) and the fund for top talent cultivation project of Chongqing Medical University (BJRC202403).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Supplemental material

Supplemental material for this article is available online.

References

Saggu

Bai

Aida

, et al. Monoamine alterations in Alzheimer's disease and their implications in comorbid neuropsychiatric symptoms. Geroscience 2025; 47: 457–482.

Argyriou

Fullard

Krivinko

, et al. Beyond memory impairment: the complex phenotypic landscape of Alzheimer's disease. Trends Mol Med 2024; 30: 713–722.

Schwertner

Pereira

, et al. Behavioral and psychological symptoms of dementia in different dementia disorders: a large-scale study of 10,000 individuals. J Alzheimers Dis 2022; 87: 1307–1318.

De Vito

Kunicki

Joyce

, et al. Parallel changes in cognition, neuropsychiatric symptoms, and amyloid in cognitively unimpaired older adults and those with mild cognitive impairment. Alzheimers Dement 2025; 21: e14568.

Park

Lee

Huber

, et al. Empirical classification of neuropsychiatric symptoms and association of classes with diagnostic progression and cognitive decline in MCI and AD populations. Biol Psychiatry 2025; 97: 1059–1066.

Sone

Beheshti

Tagai

, et al. Neuropsychiatric symptoms and neuroimaging-based brain age in mild cognitive impairment and early dementia: a multicenter study. Psychiatry Clin Neurosci 2025; 79: 158–164.

Gitlin

Marx

Stanley

, et al. Assessing neuropsychiatric symptoms in people with dementia: a systematic review of measures. Int Psychogeriatr 2014; 26: 1805–1848.

Anantapong

Jiraphan

Aunjitsakul

, et al. Behavioural and psychological symptoms of people with dementia in acute hospital settings: a systematic review and meta-analysis. Age Ageing 2025; 54: afaf013.

Lai

. The merits and problems of Neuropsychiatric Inventory as an assessment tool in people with dementia and other neurological disorders. Clin Interv Aging 2014; 9: 1051–1061.

10.

Kersloot

van Putten

FJP

Abu-Hanna

, et al. Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies. J Biomed Semantics 2020; 11: 14.

11.

Eikelboom

Singleton

van den Berg

, et al. The reporting of neuropsychiatric symptoms in electronic health records of individuals with Alzheimer's disease: a natural language processing study. Alzheimers Res Ther 2023; 15: 94.

12.

Cummings

Ritter

Rothenberg

. Advances in management of neuropsychiatric syndromes in neurodegenerative diseases. Curr Psychiatry Rep 2019; 21: 79.

13.

Saeedi

Hetjens

Grimm

MOW

, et al. Acoustic speech analysis in Alzheimer's disease: a systematic review and meta-analysis. J Prev Alzheimers Dis 2024; 11: 1789–1797.

14.

de Medeiros

Robert

Gauthier

, et al. The Neuropsychiatric Inventory-Clinician rating scale (NPI-C): reliability and validity of a revised assessment of neuropsychiatric symptoms in dementia. Int Psychogeriatr 2010; 22: 984–994.

15.

König

Mallick

Tröger

, et al. Measuring neuropsychiatric symptoms in patients with early cognitive decline using speech analysis. Eur Psychiatry 2021; 64: e64.

16.

Berardi

Brosch

Pfarr

, et al. Relative importance of speech and voice features in the classification of schizophrenia and depression. Transl Psychiatry 2023; 13: 298.

17.

Menne

Dörr

Schräder

, et al. The voice of depression: speech features as biomarkers for major depressive disorder. BMC Psychiatry 2024; 24: 794.

18.

Ambrosini

Giangregorio

Lomurno

, et al. Automatic spontaneous speech analysis for the detection of cognitive functional decline in older adults: multilanguage cross-sectional study. JMIR Aging 2024; 7: e50537.

19.

Sluis

Angus

Wiles

, et al. An automated approach to examining pausing in the speech of people with dementia. Am J Alzheimers Dis Other Demen 2020; 35: 1533317520939773.

20.

Amini

Hao

Yang

, et al. Prediction of Alzheimer's disease progression within 6 years using speech: a novel approach leveraging language models. Alzheimers Dement 2024; 20: 5262–5270.

21.

Soleimani

Ouyang

Cho

, et al. Speech markers of depression dimensions across cognitive status. Alzheimers Dement (Amst) 2024; 16: e12604.

22.

Gregory

Linz

König

, et al. Remote data collection speech analysis and prediction of the identification of Alzheimer's disease biomarkers in people at risk for Alzheimer's disease dementia: the Speech on the Phone Assessment (SPeAk) prospective observational study protocol. BMJ Open 2022; 12: e052250.

23.

Wang

Kuang

Guo

, et al. Automatic detection of putative mild cognitive impairment from speech acoustic features in mandarin-speaking elders. J Alzheimers Dis 2023; 95: 901–914.

24.

Huang

Yang

Che

, et al. Automatic speech analysis for detecting cognitive decline of older adults. Front Public Health 2024; 12: 1417966.

25.

Chen

Zhang

, et al. A fully automated Mini-Mental State Examination assessment model using computer algorithms for cognitive screening. J Alzheimers Dis 2024; 97: 1661–1672.

26.

Folstein

McHugh

. Mini-mental state”. A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 1975; 12: 189–198.

27.

Rosen

Mohs

Davis

. A new rating scale for Alzheimer's disease. Am J Psychiatry 1984; 141: 1356–1364.

28.

Cummings

Mega

Gray

, et al. The Neuropsychiatric Inventory: comprehensive assessment of psychopathology in dementia. Neurology 1994; 44: 2308–2314.

29.

Litvan

Mega

Cummings

, et al. Neuropsychiatric aspects of progressive supranuclear palsy. Neurology 1996; 47: 1184–1189.

30.

Wang

Chen

, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint, https://doi.org/10.48550/arXiv.2412.10117 (2024).

31.

Lundberg

Nair

Vavilala

, et al. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nat Biomed Eng 2018; 2: 749–760.

32.

Guo

Xue

Tao

, et al. Prognostic analysis of sepsis-induced myocardial injury patients using propensity score matching and doubly robust analysis with machine learning-based risk prediction model development. Front Med (Lausanne) 2025; 12: 1555103.

33.

Horn

Hickey

Nietert

, et al. Study partner characteristics moderate the prediction of cognitive decline in aging. Alzheimers Dement 2025; 21: e70240.

34.

Vickrey

Hays

Maines

, et al. Development and preliminary evaluation of a quality of life measure targeted at dementia caregivers. Health Qual Life Outcomes 2009; 7: 56.

35.

Guan

Rehman

Nathan

, et al. Neuropsychiatric symptoms: risk factor or disease marker? A study of structural imaging biomarkers of Alzheimer's disease and incident cognitive decline. Hum Brain Mapp 2024; 45: e70016.

36.

Badal

Reinen

Twamley

, et al. Investigating acoustic and psycholinguistic predictors of cognitive impairment in older adults: modeling study. JMIR Aging 2024; 7: e54655.

37.

Alexopoulou

Köhler

Mallick

, et al. Speech-based digital cognitive assessments for detection of mild cognitive impairment: validation against paper-based neurocognitive assessment scores. J Alzheimers Dis 2025; 108: S118–S131.

38.

Cummins

Scherer

Krajewski

, et al. A review of depression and suicide risk assessment using speech analysis. Speech Commun 2015; 71: 10–49.

39.

Fraser

Meltzer

Rudzicz

. Linguistic features identify Alzheimer's disease in narrative speech. J Alzheimers Dis 2016; 49: 407–422.

40.

Kang

Nunes

Sharma

, et al. Utilizing speech analysis to differentiate progressive supranuclear palsy from Parkinson's disease. Parkinsonism Relat Disord 2023; 115: 105835.

41.

Tian

Zhang

, et al. Spectral entropy can predict changes of working memory performance reduced by short-time training in the delayed-match-to-sample task. Front Hum Neurosci 2017; 11: 437.

42.

Yang

Chakraborty

, et al. Identifying psychiatric manifestations in schizophrenia and depression from audio-visual behavioural indicators through a machine-learning approach. Schizophrenia (Heidelb) 2022; 8: 92.

43.

Arrieta

Baz

García-Ribas

. FORTCARE-MCI study protocol: evaluation of Fortasyn Connect in the management of mild cognitive impairment in primary care. Front Neurol 2024; 15: 1434210.

44.

Geurts

Ernst

Wehenkel

. Extremely randomized trees. Mach Learn 2006; 63: 3–42.

45.

Karapinar Senturk

. Early diagnosis of Parkinson's disease using machine learning algorithms. Med Hypotheses 2020; 138: 109603.

46.

Ding

Chetty

Noori Hoshyar

, et al. Speech based detection of Alzheimer’s disease: a survey of AI techniques, datasets and challenges. Artif Intell Rev 2024; 57: 325.

47.

Han Wang

Cui

Lee

, et al. Applied machine learning in intelligent systems: knowledge graph-enhanced ophthalmic contrastive learning with “clinical profile” prompts. Front Artif Intell 2025; 8: 1527010.

48.

Lundberg

Erion

Lee

S-I

. Consistent individualized feature attribution for tree ensembles. arXiv preprint, https://doi.org/10.48550/arXiv.1802.03888 (2018).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.24 MB