Early detection of Alzheimer's disease based on leveraging multimodal features of the clock drawing test

Abstract

Background

Although multi-task handwriting analysis has the potential to improve early detection of Alzheimer's disease (AD), the educational bias inherent in its text-based tasks poses a significant obstacle to its widespread adoption across different regions.

Objective

Using the clock drawing test, we aim to design a deep neural network to extract features from static images and process signals to achieve high-precision recognition of early AD.

Methods

Early Detection of Alzheimer's Disease based on Leveraging Multimodal Features of the clock-drawing test (EDADLMF) is proposed. Firstly, to utilize the behavioral features inherent in the clock drawing test task, we propose a Dual Stream Clock Drawing Feature Extraction module,which employs a convolutional neural networks to capture the spatial features of static clock face images, while concurrently employing a multi-layer perceptron to map low-dimensional process signal into a high-dimensional feature space. Furthermore, we propose a Feature Fusion module with the Squeeze-and-Excitation attention mechanism to adaptively enhance key features and fuses complementary information from different modalities. Thirdly, to enhance the model's focus on hard-to-classify samples, a PolyLoss function is introduced to assign greater weights to difficult samples.

Results

Comparative experiments on benchmark demonstrated that EDADLMF outperforms the compaired methods on accuracy (92.59%), precision (93.65%), recall (92.65%), and F1-score (92.59%), and the case study indicates that the developed prototype system has well effectiveness.

Conclusions

The clock drawing test, combined with process signals and image data, exhibits better screening accuracy and could serve as a practical alternative to initial MRI scans.

Keywords

Alzheimer's disease clock drawing test deep learning multimodal fusion

Introduction

Motivation and contribution

Alzheimer's disease (AD) is an irreversible neurodegenerative disorder. The Alzheimer's Association¹ in the United States classifies the disease by severity into stages: preclinical AD, mild cognitive impairment, mild AD (also known as early AD), moderate AD, and severe AD. Notably, the early AD represents a “golden window for intervention”, as timely detection of early AD behaviors can lay the foundation for subsequent targeted interventions. Currently, the early identification of AD primarily relies on medical examination results,² such as brain computed tomography,³ magnetic resonance imaging (MRI),⁴ positron emission tomography (PET),⁵ and other medical data,^6–9 from which identification models are established through data mining analysis. Furthermore, medical diagnosis, treatment, or prediction also takes into account factors such as specific gene carriers, family history, and vascular factors.^10–12 However, the use of specialized medical testing for early AD diagnosis faces multiple limitations. First, it depends on expensive equipment and specialized institutions, resulting in high costs and limited accessibility. Consequently, diagnosis typically occurs only after patients actively seek medical care, leading to significant delays. Secondly, because the clinical manifestations of early AD closely resemble normal brain aging and lack specificity, patients are often reluctant to seek medical care, resulting in persistently high rates of missed diagnoses. Thus, a large number of cases are already in the middle or late stages by the time they receive a definitive diagnosis. As an alternative, early AD detection through daily behavior observation, such as subtle body movements,^13–15 speech,^16,17 and handwriting tasks,^18–20 has attracted widespread attention. The clock drawing test (CDT)^21,22 is less influenced by educational and cultural backgrounds, making it widely adopted in both clinical settings and research.

Current methods of using artificial intelligence to identify AD through the CDT primarily focus on two approaches: 1) Unimodal analysis based on static images.^23–31 These methods rely on the final clock drawings produced by subjects, employing architectures such as convolutional neural networks (CNN) to extract visual features from the images (e.g., figure closure, digit layout, pointer accuracy). Cognitive function scores and classifications are then generated based on this static information. 2) Single-modal analysis based on the process signal during the CDT.^32–37 This method captures temporal behavioral data during the drawing process (such as pen tip velocity, acceleration, pause duration, and pressure), with two primary analysis approaches. First, directly applying machine learning algorithms (such as support vector machine (SVM), random forests (RF), and variational quantum classifiers) to classify these features. Second, one-dimensional time-series data is first converted into a two-dimensional image using techniques such as recurrence plots, followed by feature extraction and classification via CNN. However, unimodal approaches can only reflect a single aspect of cognitive impairment, making it difficult to comprehensively capture the complementary relationship between static structure and dynamic behavior. Therefore, to develop a more comprehensive cognitive assessment model, researchers have recently begun exploring multimodal fusion methods that combine static clock images with dynamic drawing characteristics. This approach aims to overcome the performance limitations of single-modal methods and improve the accuracy and robustness of early AD detection.

The human brain does not comprehend the world through isolated sensory channels (such as vision, hearing, taste, and touch), but rather functions as a highly coordinated multimodal information processing system. As revealed by neuroscience research, the holistic perceptual effect of multisensory integration far exceeds the sum of the individual perceptual effects of each sense.³⁸ This synergistic effect stems from the complementary and corroborative nature of information across different modalities. For example, in CDT, multimodal approaches achieve a paradigm shift “from outcome-oriented” to “process-outcome integration” by integrating static images with dynamic process information, providing a more comprehensive basis for evaluating executive functions and visuospatial abilities. Even when subjects do not exhibit typical abnormalities in a single modality, cross-modal cross-validation and compensatory mechanisms can uncover more subtle indicators of cognitive status, thereby enhancing diagnostic sensitivity and specificity. Currently, multimodal analysis methods aim to construct more comprehensive cognitive assessment models by integrating temporal dynamic information during the CDT (such as pen tip trajectory, velocity, and acceleration) with the static structural features of the image.^39,40 Figure 1 depicts the process signal (velocity, acceleration, pressure, etc.) of clock testing overlaid with the color channel of the image to achieve multimodal feature fusion of process signal and image data.³⁹ This method often fails to capture the complex relationships within the dynamic features and lacks effective mechanisms for cross-modal semantic alignment and collaborative reasoning. Consequently, the model struggles to extract highly discriminative, fine-grained AD biomarkers throughout the entire CDT process.

To utilize the behavioral features inherent in the clock drawing test task, this paper proposes proposes a Early Detection of AD based on Leveraging Multimodal Features of the clock-drawing test (EDADLMF). First, in order to extract data features of different modalities, a CNN was employed to extract spatial features from the clock face drawings produced by participants, capturing fine visual cues such as clock face structure, numeral layout, and pointer angles. Concurrently, a multi-layer perceptron (MLP) is constructed to model dynamic process features during the drawing process, thereby delineating underlying patterns of cognitive-motor coupling. Second, in order to model the complementarity of different modalities, the introduction of a channel attention mechanism enables adaptive weighting of the two types of heterogeneous features, achieving complementary integration of cross-modal information while suppressing redundancy. This enhances the sensitivity to early pathological manifestations. Finally, to overcome the performance bottleneck caused by the model's tendency to overlook hard-to-classify samples in small datasets, we introduce a loss function that focuses on difficult samples. The function enhances the classification accuracy for early AD by assigning higher training weights to hard-to-classify samples.

Overall, the main contributions of this paper are as follows:

1) We propose a novel EDADLMF for early detection of AD using CDT behavioral features. First, in order to extract features from different modalities, we propose a Dual Stream Clock Drawing Feature Extraction module. Second, to model the complementarity among these modalities, we propose a Feature Fusion module with the squeeze-and-excitation attention mechanism (SEFF).

2) We introduce the PolyLoss function optimized for hard-to-classify samples. This approach effectively optimizes the classification boundary, and improves recognition performance while maintaining model simplicity and training stability.

3) The effectiveness of this method has been validated through experiments on publicly available datasets such as DARWIN and NHATS, with a recognition accuracy of 92.59% on the DARWIN dataset, representing a significant improvement of 2% to 12% over existing state-of-the-art methods.

Related work

In research on early AD screening using the CDT, the analysis methods are typically classified into three categories based on the type of data they process: static image analysis, process signal analysis, and multimodal information fusion. This review uses this classification framework to organize and summarize existing studies.

Static image

Early AD assessment methods based on static images primarily employed two technical approaches: one involved rule-based reproduction of clinical scoring, and the other utilized deep learning for end-to-end learning of diagnostic features.

1) Explainable models based on automated scoring. This method aims to simulate clinicians’ assessment processes using the CDT scoring scale(such as the modified Sulman scoring rules, where scores of 0–2 indicate cognitive impairment and scores of 3–5 fall within the healthy range) through algorithmic simulation. It generates interpretable quantitative scores and then infers cognitive status based on score thresholds. Its core advantage lies in its decision-making process, which aligns with clinical standards, offers high transparency and interpretability, and is easily understood and accepted by clinicians. For example, Hu et al.²⁹ systematically compared the performance of advanced models such as ResNet101, EfficientNet, and vision transformer (ViT), with the ViT model achieving an accuracy rate of up to 77.5%. Raksit et al.²⁴ employed the attentional pairwise interaction network (API-Net). Inspired by the human mechanism of discerning differences through comparison, this network processes image pairs as input for contrastive learning, excelling at distinguishing visually highly similar samples. This approach demonstrated remarkable performance in fine-grained image recognition, achieving an 80.33% recognition accuracy, further validating the effectiveness of deep learning models in accomplishing such interpretable scoring tasks.

To achieve more objective and interpretable scoring, a combination of semantic segmentation and classification network architectures is employed. First, semantic segmentation is performed on the clock face diagram to extract key components, followed by quantitative analysis of these independent components. For example, Park et al.²⁵ utilize a U-Net architecture to achieve pixel-level segmentation of clock faces and hands, utilizing a CNN model pre-trained on the MNIST dataset for digit recognition. They ultimately decompose the evaluation into four dimensions: outline, digits, hands, and center. The system achieves high recognition accuracy across all components (contour is 89.95%, number is 89.04%, pointer is 87.21%, center is 96.80%). Jiang et al.²⁷ cluster clock stroke segments into three component categories (outline/numerals/hands), then classify strokes within each category using a 1-D CNN, and finally produce a 12-component quantization result based on component completeness. Both systems systematically evaluate critical elements, including clock face closure, completeness, numeral sequence of numerals (1 through 12), spatial distribution, presence of hands, and pointing accuracy. Their scoring logic directly corresponds to scales used by clinicians, combining automation with interpretability.

2) End-to-end black-box diagnostic models. This type of method completely bypasses explicit clinical scoring rules and directly enables end-to-end learning from images to AD. For example, Kang et al.²⁶ randomly selected 13,468 clock image samples from the NHATS dataset to classify AD and healthy controls (HC). They employed ResNet50 to extract features from the clock images, achieving an average accuracy of 74.67% through classification via a fully connected layer. Natthanan et al.²³ achieved an AD recognition accuracy of 75.0% on a test set of 918 painted clock samples using the VGGNet16 framework. Carmen et al.³⁰ train a CNN model using a larger clinical sample (n = 7009), achieving an accuracy of 75.65%. They employ saliency maps and Grad-CAM to visualize the regions of interest for the model, providing an initial approach to understanding its decision-making basis. To further enhance performance and efficiency, Samad et al.²⁸ apply the lightweight MobileNetV2 network to the Framingham Heart Study (FSH) dataset, achieving a high recognition accuracy of 81.3% ± 4.3%. This demonstrates the potential of network architecture optimization for improving performance. Significant progress has been made in AD assessment methods using static images, establishing a crucial foundation for the automated screening of cognitive impairment. With the advancement of digital technology, CDT have gradually transitioned from traditional paper-based formats to digital platforms. Digital clock drawing test (dCDT) not only captures high-precision static images but also comprehensively records dynamic stroke information during the drawing process, thereby providing multidimensional, granular data to support cognitive assessment.

Process signal

Process signal refers to the dynamic behavioral sequences and their inherent features recorded during the execution of handwriting tasks using digital tablets and stylus devices. These include time stamps, pen pressure, coordinate trajectories, velocity, acceleration, and other time-series data. These dynamic data provide a rich information foundation for in-depth analysis of cognitive function mapping during the painting process. Research indicates that prolonged airtime in the handwriting of AD patients is highly correlated with reduced integrity of the parietal-frontal white matter on MRI scans.⁴¹ Additionally, studies have shown that motor slowness (such as reduced writing speed) correlates with decreased anisotropy fraction in the corpus callosum and corona radiata, while diminished writing pressure occurs concurrently with basal ganglia atrophy. Ding et al.⁴² find that performance on the dCDT showed significant correlation with amyloid deposition in PET imaging, thereby reinforcing the view that this task can serve as a biomarker for diagnosing cognitive impairment. Cilia et al.³² designed a series of handwriting tasks based on the clock-drawing test and created the DARWIN dataset. They compare multiple machine learning algorithms on a feature subset of the CDT, achieving a maximum recognition accuracy of 72.4%. To enhance performance, Gattulli et al.³⁴ select the six handwriting tasks with the highest recognition rates from the dataset and combine them. They employ the RF algorithm for experimentation, successfully increasing the accuracy rate to 88.29%.

To overcome the performance limitations of traditional machine learning methods, researchers are investigating advanced algorithms for the detection of early AD. For instance, Akpinar et al.³⁶ apply the variational quantum classifier model to the DARWIN dataset and attain an early AD recognition accuracy of 75%, showing that the quantum algorithm model performs similarly to the classical machine learning method mentioned in reference.³² Cappiello et al.³⁵ concentrate on distinguishing AD from HC using process signal features from handwriting tasks. By optimizing algorithm parameters, they obtain a notably higher accuracy rate (88.69%±4.69%) for the quantum support vector classification (QSVC) model compared to the classical SVC (85.28%±7.46%), highlighting the promise of quantum algorithms.

During the processing of handwriting task signals, deep neural networks can automatically learn highly abstract features from the raw signals. For example, Pakize et al.³⁷ creatively use a recursive approach to transform a one-dimensional process signal into two-dimensional images. By applying lightweight neural networks to the DARWIN dataset, they further improved the accuracy of distinguishing AD from HC to 90.4%. Kang et al.³³ employ an architecture combining a self-attention mechanism with fully connected layers to better model long-range dependencies in handwriting data. Experimental results demonstrated that the model achieved an overall recognition accuracy of 94.3% across the entire dataset. However, it is noteworthy that on the CDT subset, the model's accuracy dropped to 75.9%. This contrast reveals the model's limitations in handling handwriting from complex cognitive tasks. CDT requires strong visual-spatial structure capabilities,⁴³ and its handwriting features (such as graphic layout and pointer angles) differ significantly from conventional continuous writing, which the model may not fully learn. Research on process signal based on writing tasks has gradually expanded from early feature engineering and traditional machine learning optimization to exploring cutting-edge algorithms, such as quantum computing and deep learning. However, relying solely on unimodal process features for detailed assessments of cognitive states may still be limited, prompting researchers to explore new approaches that integrate multimodal information.

Multimodal fusion

To improve the accuracy of CDT in detecting early AD, researchers have increasingly focused on cross-modal collaborative modeling that integrates static clock drawings and process signal. Cilia et al.⁴⁰ combined static image features with process signal to enhance classification performance. By mapping process signal such as velocity, acceleration, and pressure onto a binary image, a three-channel RGB color image is generated. The CNN is then employed to automatically extract the fused features, followed by decision-making using a classifier. Experimental results demonstrate that the color image incorporating fused dynamic features significantly outperforms methods relying solely on the original binary image (73.0% versus 67.8%), validating the effectiveness of multimodal fusion. To further enhance performance, the team³⁹ subsequently introduced additional dynamic features (adding “jerk”) in their follow-up research. They generate four-channel images (.tiff) and employ a pre-trained VGGNet16 network for feature extraction, combined with SVM for classification. Under this architecture, the recognition accuracy reached 73.2%. Gong et al.⁴⁴ employ ViT to extract static image features and MLP to map process features, introducing a gating mechanism to coordinate similarity attention and dissimilarity attention. This mechanism effectively integrates feature information from different modalities by dynamically weighting two types of attention. Their experimental results achieved an AD recognition accuracy of 81.25% on the CDT subset. The above research findings indicate that designing an efficient feature fusion architecture and incorporating complementary multimodal information are more critical for enhancing model performance than simply increasing feature dimensions. Table 1 describes the comparison of previous AD diagnostic works based on CDT.

Table 1.

Comparison of previous AD diagnosis works based on the CDT.

Author	Time	Dataset (size/name)	Feature extraction method	Deep learning model	Result (accuracy %)
Hu²⁷	2024	25,872	Extract static image features of CDT	Vision Transformer	Score (0–5)
Hu²⁷	2024	/ NHATS	Extract static image features of CDT	Vision Transformer	77.5%
Raksi²²	2023	3108	Extract static image features of CDT	API-Net	Score (0–10)
Raksi²²	2023	/ NHATS	Extract static image features of CDT	API-Net	80.33%
Park²³	2021	238/ Private dataset	Extract key components of static images (contours, numbers, pointers)	U-Net	Contours 89.95%
					Numbers 89.04%
					Pointers 87.21%
					Centers 96.80%
Jiang²⁵	2021	100/ Private dataset	Extract key components of static images (contours, numbers, pointers)	1-D CNN	Pearson correlation value 0.763
Kang²⁴	2024	13468	Extract static image features of CDT	ResNet50	AD vs. HC
Kang²⁴	2024	/ NHATS	Extract static image features of CDT	ResNet50	74.67%
Natthanan²¹	2022	918/ Private dataset	Extract static image features of CDT	VGGNet16	AD vs. HC
Natthanan²¹	2022	918/ Private dataset	Extract static image features of CDT	VGGNet16	75.0%
Jimenez-Mesa²⁸	2023	7009/ Private dataset	Extract static image features of CDT	CNN	AD vs. HC
Jimenez-Mesa²⁸	2023	7009/ Private dataset	Extract static image features of CDT	CNN	75.65%
Samad²⁶	2021	HC 3263	Extract static image features of CDT	MobileNetV2	81.3%
		AD 160
		/ FSH
Cilia²⁹	2022	HC 89	Extraction of process signal features during dCDT	Learning Vector Quantization	72.57%
		AD 85
		/ DARWIN
Gattull³¹	2023	HC 89	Extraction of process signal features during dCDT	RF	88.29%
		AD 85
		/ DARWIN
Akpinar³³	2023	HC 89	Extraction of process signal features during dCDT	Variational quantum classifier	75.0%
		AD 85
		/ DARWIN
Cappiello³²	2025	HC 89	Extraction of process signal features during dCDT	QSVC	88.69%
		AD 85
		/ DARWIN
Pakize³⁴	2024	HC 89	Extraction of process signal features during dCDT	CNN	90.4%
		AD 85
		/ DARWIN
Cilia³⁹	2021	HC 89	1) Extract static image features of dCDT 2) Extraction of process signal features during dCDT	CNN	73.0%
		AD 85
		/ DARWIN
Cilia⁴⁰	2022	HC 89	1) Extract static image features of dCDT 2) Extraction of process signal features during dCDT	CNN	73.2%
		AD 85
		/ DARWIN
Gong⁴¹	2025	HC 89	1) Extract static image features of dCDT 2) Extraction of process signal features during dCDT	VIT + MPL	81.25%
		AD 85
		/ DARWIN
Ours	-	HC 89	1) Extract static image features of dCDT 2) Extraction of process signal features during dCDT	CNN + MLP	92.59%
		AD 85
		/ DARWIN

Although the above research has achieved significant results, existing literature focuses on single-modal studies and lacks exploration of multimodal data fusion. Its limitation restricts improvements in the recognition accuracy of the CDT and its clinical application. Based on this, this paper proposes a multimodal fusion approach for the recognition of early AD, whose core lies in systematically extracting and integrating two types of features: static images and process signal. We not only utilize CNN to extract visual structural features from clock faces, but also employ MPL to model temporal patterns in behavioral sequences. More crucially, we introduce a channel attention mechanism to adaptively weight and fuse bimodal features, effectively enhancing the representation of discriminative features while suppressing redundant information.

Methods

This section we proposed the proposed EDADLMF. Figure 2 is the architecture of the proposed EDADLMF. The EDADLMF method consists of a Dual Stream Clock Drawing Feature Extraction module (DSCDEF), followed by a Feature Fusion module with the SEFF. First, the DSCDEF module processes input data in parallel. It utilizes CNNs to encode the spatial structural features of clock face images, while simultaneously learning high-order dynamic representations from low-dimensional process signal through multilayer perceptrons. Subsequently, the SEFF module concatenates these two heterogeneous feature vectors and applies a nonlinear transformation through a fully connected layer to achieve fusion. It then learns the interaction relationships between modalities via the fully connected layer and generates a deeply fused feature representation, ultimately serving as the basis for AD classification decisions. The pseudocode of EDADLMF method is shown in Algorithm 1.

In Algorithm 1, the image data $X^{i m g} \in R^{B \times 3 \times H \times W}$ and process signal data $X^{t a b} \in R^{B \times d}$ of each batch are input into EDADLMF to output the category and probability of each sample. Record all trainable parameter sets of the model as $Θ$ , which are divided into different subsets based on the model components:

Θ = {Θ_{i m g}, Θ_{M L P}, Θ_{a t t}^{i m g}, Θ_{a t t}^{M L P}, Θ_{g}, Θ_{c l s}}

Among these, $Θ_{i m g}$ contains a frozen subset that is not updated during training.

In Step 3, a dual stream feature extraction module DSCDEF based on clock drawing testing is composed of VGGNet16 and MLP, respectively, to extract AD related features from image data $X^{i m g} \in R^{B \times 3 \times H \times W}$ and process signal data $X^{M L P} \in R^{B \times d}$ .

Algorithm 1: Early detection of Alzheimer's disease based on leveraging multimodal features of the clock-drawing test (EDADLMF).

Input: Dataset

D = {X_{i}^{i m g}, X_{i}^{M L P}, y_{i}}

X_{i}^{i m g} \in R^{3 \times H \times W}

X_{i}^{M L P} \in R^{d}

, batchsize is B, class is C

Initialize: Learning rate

η = 0.001

, epoch, freeze

f_{i m g} . f e a t u r e s [: 4]

, etc.

Output: predicts class C and probability

Y \in R^{C}

Step1: for epoch=1,2,…, epochs do

Step2: for each mini-batch

{(X_{i m g}, X_{M L P}, y)}

with |batch| =B do

Step3:

Z_{i m g} \leftarrow f_{i m g} (X_{i m g}), Z_{M L P} \leftarrow f_{M L P} (X_{M L P}), Z_{i m g} \in R^{B \times 512}, Z_{M L P} \in R^{B \times 512}

//feature extraction

Step4:

{\tilde{Z}}_{i m g} \leftarrow A_{i m g} (Z_{i m g}), {\tilde{Z}}_{M L P} \leftarrow A_{M L P} (Z_{M L P}), {\tilde{Z}}_{i m g} \in R^{B \times 512}, {\tilde{Z}}_{M L P} \in R^{B \times 512}

//Inner-branch attention

Step5:

S \leftarrow c o n c a t ({\tilde{Z}}_{i m g}, {\tilde{Z}}_{M L P}), S \in R^{B \times 1024}

// Dynamic weight generation and sample-level weighted fusion

Step6:

a \leftarrow g (S), a \in R^{B \times 2}

// Linear—>ReLU—>Linear

Step7:

w \leftarrow s o f t m a x (a, \dim = 1), w \in [w_{i m g}, w_{M L P}], w \in R^{B \times 2}

// The sum of each row is 1.

Step8:

F \leftarrow w [:, 0] . u n s q u e e z e (1) * {\tilde{Z}}_{i m g} + w [:, 1] . u n s q u e e z e (1) * {\tilde{Z}}_{M L P}, F \in R^{B \times 512}

Step9:

O \leftarrow f_{c l s} (F), P \leftarrow softmax (O, \dim = 1), O \in R^{B \times C}

// P denotes the predicted probability

Step10:

L_{c e} \leftarrow P o l y L o s s_{F L} (P, Y), L_{r e g} \leftarrow λ | | Θ | |_{2}^{2}, L \leftarrow L_{P o l y L o s s} + L_{r e g}

// Optional regular expression items

Step11:

\nabla \leftarrow b a c k p r o p (L)

// Backpropagation

Step12: optimizer.step()

Step13: end for

Step14: end for

VGGNet16 uses ImageNet pre trained weights to fine tune the model through transfer learning, and freezes the parameters of the first four convolutional blocks. The input image data $X^{i m g} \in R^{B \times 3 \times H \times W}$ undergoes four convolutional blocks and pooling to obtain the feature $u \in R^{B \times 512 \times 14 \times 14}$ , which is then flattened through three convolutional layers and one pooling layer to obtain the clock image features related to AD recognition. Finally, it is mapped into a 512 dimensional vector through two fully connected layers, as shown in equation (1).

\begin{aligned} h_{1} = C o n v (u, W_{d}^{(1)}, b^{(1)}), W^{(2)} \in R^{512 \times 14 \times 14}, \\ h_{2} = C o n v (h_{1}, W_{d}^{(2)}, b^{(2)}), W^{(2)} \in R^{512 \times 14 \times 14}, \\ h_{3} = M ax p o o l (C o n v (h_{1}, W_{d}^{(3)}, b^{(3)})), W^{(3)} \in R^{512 \times 14 \times 14}, \\ h_{4} = D r o p o u t_{p} (R e L U (h_{1} W^{(1)} + b^{(4)})), W^{(4)} \in R^{4096 \times 25088}, \\ Z_{i m g} = D r o p o u t_{p} (R e L U (h_{1} W^{(2)} + b^{(5)})), \\ W^{(5)} \in R^{512 \times 4096}, Z_{i m g} \in R^{B \times 512} \end{aligned}

(1)

Where p is 0.5, $W_{d}^{(i)}$ is a convolutional kernel, $W^{(i)}$ is the training weight, and $b^{(i)}$ are the bias.

In the process signal feature extraction network, the process signal data $X^{t a b} \in R^{B \times d}$ is similarly projected into a 512-dimensional space, as shown in the following equation:

\begin{aligned} g_{1} & = D r o p o u t_{p^{'}} (R e L U (B N (X^{t a b} U^{(1)} + c^{(1)}))), \\ U^{(1)} \in R^{d \times 1024}, \\ Z_{M L P} & = D r o p o u t_{p^{'}} (R e L U (B N (g_{1} U^{(2)} + c^{(2)}))), \\ U^{(2)} \in R^{1024 \times 512}, Z_{M L P} \in R^{B \times 512} \end{aligned}

(2)

Where $p^{'}$ is 0.5, $U^{(i)}$ is the training weight, and $c^{(i)}$ is the bias.

In Step 4, the output features $Z_{i m g}$ and $Z_{M L P}$ from Equations (3) and (4) are respectively weighted through the channel attention mechanism, as shown in Equations (3) and (4).

\begin{aligned} s_{i m g} = σ (W_{2}^{a t t, i m g} Re L U (W_{1}^{a t t, i m g} (Z_{i m g})^{T}))^{T} \in R^{B \times 512} \\ {\tilde{Z}}_{i m g} = s_{i m g} ⊙ Z_{i m g}, {\tilde{Z}}_{i m g} \in R^{B \times 512} \end{aligned}

(3)

\begin{aligned} s_{M L P} = σ (W_{2}^{a t t, M L P} Re L U (W_{1}^{a t t, M L P} (Z_{M L P})^{T}))^{T} \in R^{B \times 512} \\ {\tilde{Z}}_{M L P} = s_{M L P} ⊙ Z_{M L P}, {\tilde{Z}}_{M L P} \in R^{B \times 512} \end{aligned}

(4)

In Steps 5–8, concatenate the feature-weighted $Z_{i m g}$ and $Z_{M L P}$ from Equations (3) and (4) into $S = [{\tilde{Z}}_{i m g},$ ${\tilde{Z}}_{M L P}] \in R^{B \times 1024}$ . Then, output a weight vector of length 2 per sample using the dynamic weight $g (\cdot)$ , as shown in Equation (5).

\begin{aligned} A = W_{g}^{(2)} R e L U (S W_{g}^{(1)} + b_{g}^{(1)}) + b_{g}^{(2)} \in R^{B \times 2}, \\ W = S o f t m a x (A, a x i s = 1), W = [w^{i m g}, w^{M L P}] \in R^{B \times 2} \end{aligned}

(5)

The Softmax function is normalized over each sample (row), satisfying the condition that for each sample i, $w_{i}^{i m g} + w_{i}^{M L P} = 1$ .

Next, the feature-weighted ${\tilde{Z}}_{i m g}$ and ${\tilde{Z}}_{M L P}$ from Equations (3) and (4) are combined with Equation (5) through weight allocation to obtain the fused feature F, as shown in Equation (6).

F = w^{i m g} ⊙ {\tilde{Z}}_{i m g} + w^{t a b} ⊙ {\tilde{Z}}_{M L P} \in R^{B \times 1024}

(6)

In Step 9, the fusion feature F from Equation (6) is fed into a MLP layer for feature extraction. Subsequently, the Softmax function is applied to predict probabilities and determine the final category, as shown in the following equation:

\begin{aligned} q_{1} = D r o p o u t_{p^{^{″}}} (R e L U (B N (F W_{c}^{(1)} + b_{c}^{(1)}))), W_{c}^{(1)} \in R^{1024}, \\ q_{2} = D r o p o u t_{p^{^{″}}} (R e L U (B N (q_{1} + b_{c}^{(2)}))), W_{c}^{(2)} \in R^{512}, \\ \hat{P} = S o f t m a x (O) = S o f t m a x (q_{2} W_{c}^{(3)} + b_{c}^{(3)}) \in R^{B \times C} \end{aligned}

(7)

Meanwhile, the model is optimized during training by minimizing the improved focal loss function, as shown in the following equation:

\begin{aligned} L_{P oly - F L} & = - \frac{1}{B} \sum_{i = 1}^{B} \sum_{c = 1}^{C} \log ({\hat{y}}_{i, c}) {(1 - {\hat{y}}_{i, c t})}^{γ + 1} \\ + ε_{1} (1 - {\hat{y}}_{i, c t})^{γ + 1} + λ | | Θ | |_{2}^{2} \end{aligned}

(8)

In classification tasks, focal loss serve as widely adopted training objective functions, whose effectiveness and practicality have been thoroughly validated across numerous computer vision tasks. To overcome the limitations of standard focus loss in handling difficult classification samples, this paper decomposes the focus loss function into a series of weighted polynomial basis functions based on the Taylor expansion theoretical framework, thereby deriving the unified PolyLoss function.⁴⁵ The focus loss function L_FL is expressed as:

L_{F L} = - (1 - P_{t})^{γ} \log (P_{t}) = \sum_{j = 1}^{\infty} 1 / j (1 - P_{t})^{j + γ} = (1 - P_{t})^{1 + γ} + 1 / 2 (1 - P_{t})^{2 + γ} \dots

(9)

Here, P_t denotes the predicted probability for a given target label, γ is the modulation factor used to adjust the weight of difficult samples, and (1-P_t) ^γ represents the loss weight. When P_t → 1, the value of (1-P_t) ^γ is small, indicating that the loss weight for correctly classified samples is low. When P_t → 0, the value of (1-P_t) ^γ approaches 1, and the loss weight is retained.

The Focus loss functionis optimized using gradient descent, which requires calculating the partial derivative of the loss function with respect to the predicted probability P_t, defined as:

\begin{aligned} - \frac{d L_{F L}}{d P_{t}} & = \sum_{j = 1}^{\infty} (1 + γ / j) (1 - P_{t})^{j + γ - 1} \\ = (1 + γ) (1 - P_{t})^{γ} + (1 + γ / 2) (1 - P_{t})^{1 + γ} \dots \end{aligned}

(10)

Analyzing the polynomial composition of the gradient expression reveals that the focal loss can be expressed as a PolyLoss form:

L_{P o l y - F L} = - (1 - P_{t})^{γ} \log (P_{t}) + ε_{1} (1 - P_{t})^{γ + 1}

(11)

Among these, ε is an adjustable hyperparameter, typically set to a non-negative value. When P_t→1, the value of ε(1-P_t)^γ+1 →0, resulting in a small loss weight and minimal contribution to the total loss. When P_t→0, the model assigns lower confidence to predicting the true category, indicating that this category represents a difficult-to-classify sample or a minority class. At this point, the value of ε(1-P_t) ^γ+1 is large. The increased value of the new term effectively imposes an additional penalty weight on these samples, prompting the model to focus more on these hard-to-learn or less-abundant samples. The hyperparameter ε controls the intensity of this additional penalty. This loss function framework based on polynomial expansion not only enhances the model's adaptability to diverse data distributions but also offers new insights into the underlying mechanisms of existing loss functions.

Results

Dataset

1) DARWIN. The DARWIN dataset^45,46 is constructed based on the collection of 25 handwriting tasks, comprising a total of 174 participants, including 85 healthy individuals and 89 AD patients. All AD patients are clinically diagnosed and underwent Mini-Mental State Examination (MMSE) and Montreal Cognitive Assessment (MoCA) testing to objectively evaluate their cognitive function status. During the subject recruitment process, individuals with uncorrected severe visual and auditory impairments, significant motor impairments, or those taking medications that may affect test performance are excluded to minimize the impact of confounding factors on the clock drawing test. The DARWIN dataset comprises text tests, connect-the-dots tests, circle-drawing tests, and clock-drawing tests.³⁹ The raw data for each handwriting task is recorded as (time, x, y, pre), where time is the timestamp, x and y are position coordinates, and pre is the pressure exerted by the pen tip on the tablet. The raw data are used to generate 2D static images and 1D process signal features with 18 attributes.

This paper selects selects the data from tasks 2, 21, and 24 as the experimental dataset, and their sample are shown in Figure 3. Task 2 requires connecting lines between two points four times, and task 21 involves redrawing complex patterns. The clock drawing test, task 24 in the DARWIN dataset, requires subjects to draw a clock displaying 11:05. We identify and exclude data points containing only single-modal process features to construct paired datasets, ensuring the model learns cross-modal mappings. The dataset is divided into training, validation, and testing sets in a ratio of 70:15:15.

2) NHATS. National Health and Aging Trends Study (NHATS)⁴⁷ is established by the U.S. National Center on Aging to systematically investigate changes in health and cognitive function among older adults. This study recruit participants aged 65 years and older. Through annual follow-ups, it collect multidimensional health information, including word recall tests (immediate/delayed) for memory assessment, evaluations of sleep quality, anxiety, and depression, as well as behavioral indicators such as demographic characteristics, medical history, and daily outings. Additionally, this dataset includes a clock-drawing test task with the instruction prompt: “Please draw a clock and write all the numbers on it, setting the time to 11:10”. This task can be used to assess participants’ executive function and visuospatial abilities.

This article selects 195 individuals with AD as positive samples, and randomly selects 200 individuals from a healthy population as negative samples. A total of 395 samples are divided into training set, validation set, and testing set in a ratio of 70:15:15. The original image has a pixel size of 2560 × 3312. The YOLO object detection model is used to recognize the clock in the image, and the recognized area is cropped and scaled to 256 × 256 pixels.

Implementation details

1) Image data preprocessing. In data collection, some test subjects draw multiple images and use image segmentation to process the data. When there are multiple clock images stored in the clock drawing image, only the one with the highest completion degree is retained, and the others are deleted. When there are non-clock images in the clock image, keep the clock image and delete irrelevant images. Figure 4 shows the mian data processing, where the red box represents deleted images and the green box represents retained images. In addition, data augmentation is used to improve the generalization ability of deep models. Adjust the input image size to 224 × 224 pixels, then randomly rotate it by 10° or horizontally mirror and flip it. The brightness, contrast, and saturation of the image are randomly adjusted within a range of ± 20%.

2) Process signal preprocessing. The DARWIN dataset contains 18 process signal features. To reduce model complexity, we comprehensively analyze the impact of relevant handwriting features on AD identification and selected a dataset containing 8 key parameters for the construction process signal, as shown in Table 2. To enhance model performance and prevent data leakage, the data is split into training, validation, and test sets (70:15:15). Subsequently, Z-score normalization is performed based solely on the mean and standard deviation of the training set, and this transformation is applied to both the validation and test sets to ensure consistency. To enhance generalization, data augmentation is introduced during the training phase: in each batch, a noise intensity σ was sampled from Uniform [0.001, 0.005], and independent zero-mean Gaussian noise is added to the features of each sample within the batch. During the verification and testing phase, no noise is added.

Table 2.

Description of the process signal characteristics.

Name / Unit	Statement
Total_time (ms)	Total time spent to perform the entire task.
Air_time(ms)	Time spent to perform in-air movements.
Paper_time(ms)	Time spent to perform on-paper movements.
Avg _v (mm·s⁻¹)	Average speed of on-paper movements.Speed is the variation of displacement with respect to time.
Avg _a (mm·s⁻²)	Average acceleration of on-paper movements. Acceleration is the variation of speed with respect to time.
Jerk(mm·s⁻³)	Average jerk of on-paper movements.Jerk is the variation of acceleration with respect to time.
Avg _p (mN)	Average of the pressure levels exerted by the pen tip.
Strokes	Num of pendown1.

3) Parameter Settings. The experiment is conducted on a Windows 11 Professional 64-bit operating system, utilizing a 12th Gen Intel^® Core™ i7-12700KF processor (3.60 GHz), 32 GB of RAM, and an NVIDIA GeForce RTX 4080 SUPER graphics card with 16 GB of dedicated video memory. The deep learning architecture employed is implemented using Python 3.8, PyTorch 2.4.1, and CUDA Toolkit 11.8. The model training strategy and hyperparameter configuration are detailed in Table 3.

Table 3.

Hyperparameter setting.

Parameters	Values
Activation function	ReLU
Optimizer	Adam
Initial learning rate	0.001
Batchsize	32
Epoch	100 epoch
Output layer	Softmax
Learning Rate Adjustment Strategy	StepLR
Early Exit Strategy	Validation set accuracy shows no improvement for 30 consecutive rounds.

Evaluation indicators

In this study, which distinguishes AD from HC based on the dCDT, accuracy, precision, recall, and F1 score are introduced as evaluation metrics.^43,48 Precision measures the proportion of samples predicted as positive that are actually positive; recall measures the model's ability to identify true positive samples; the F1 score is the harmonic mean of precision and recall, serving as a comprehensive measure of the model's overall performance. The formula for calculation is shown below.

\begin{aligned} A c c u r a c y = \frac{T P + T N}{T P + T N + F P + F N} \\ P r e c i s i o n = \frac{T P}{T P + F P} \\ R e c a l l = \frac{T P}{T P + F N} \\ F 1 S c o r e = 2 \times \frac{P r e c i s i o n \times R e c a l l}{P r e c i s i o n + R e c a l l} \end{aligned}

Ablation study

In this study, the feature extraction of static CDT images is based on the VGGNet16 network. Unlike natural image classification tasks that require deep networks to extract complex semantic features, the evaluation of the CDT primarily relies on low- and mid-level features. VGGNet16 effectively captures these key features through its hierarchical convolutional architecture: 1) edges and contours (shallow-layer features). Whether the clock face is closed, the hands are straight, or the numerals are clearly defined. These are most strongly responded to by the shallowest convolutional kernels. 2) Simple textures and shapes (mid-level features). Whether the writing of numbers is coherent, whether the proportions between the hands and the dial are imbalanced, and whether there are repeated strokes. Structural relationships (inferable from intermediate-level features): Whether the digits are correctly positioned, whether the hands indicate the correct time, etc. Although the deep VGGNet16 network can extract abstract and global semantic information, for the clock drawing test classification task, these high-level features are not only redundant but may also introduce noise on limited-scale handwriting datasets such as DARWIN, increasing the risk of overfitting.

Figure 5 illustrates the maximum activation feature maps of different convolutional modules after fine-tuning the model. As shown in Figures 5(b) and (c), the shallow convolutional blocks (blocks 1 and 2) primarily extract edge structures and fundamental geometric features, revealing distinct circular contours (corresponding to the outer edge of the clock face), straight-line segments (corresponding to the hands and numerals), and corner responses (corresponding to the positions of written numerals). These low-level features effectively reflect the basic compositional integrity of the image. In Figures 5(d) and (e), the middle convolutional blocks (blocks 3 and 4) further capture more discriminative textures and local patterns, revealing the coherence of digital handwriting, the proportional relationship between hands and the dial, and the consistency in stroke thickness. The feature maps clearly express the spatial combination of local structures, such as the uniformity of digit arrangement and the accuracy of hand angles. In Figure 5(e), although the features is extracted by the deep convolutional blocks are more abstract, they primarily manifest as integrated responses to structural relationships in the clock-drawing task. The feature maps reflect the overall rationality of the layout, including central symmetry, the relative positions of various elements, and the overall harmony of the composition. These features help identify possible structural deviations or spatial layout anomalies.

Conducting multimodal fusion experiments based on the CDT. The fusion strategy is the critical link in determining model performance. Table 4 shows the performance comparison of different fusion functions on the CDT subset. We compare the effects of five fusion functions-concat(·), max(·), avg(·), add(·), and multi(·)-on the early detection of AD using a test dataset. The concat function achieved recognition accuracy, precision, recall, and F1 score of 92.59%, 93.65%, 92.59%, and 92.61%, respectively, outperforming all other fusion methods in all metrics. The concat fusion method not only fully preserves the deep visual representations of image modalities and the behavioral characteristics of process signal through channel concatenation, but also achieves lossless convergence of two heterogeneous information streams via dimensional superposition. This enables the network to simultaneously capture spatial textures, color distributions, and multidimensional behavioral cues such as brushstroke velocity and acceleration in the painting clock test, thereby maximizing the preservation of the original modality's complementarity and discriminative power.

Table 4.

Performance comparison of different fusion functions on the CDT subset.

Fusion function	Accuracy	Precision	Recall	F1 Score
Concat(·)	92.59	93.65	92.59	92.61
Max(·)	85.19	88.89	85.19	85.10
Avg(·)	88.89	91.11	88.89	88.89
Add(·)	88.89	91.11	88.89	88.89
Multi(·)	88.89	91.11	88.89	88.89

The loss function plays a crucial role in model, as it not only determines the direction of optimization but also directly impacts the model's convergence and final classification accuracy. We employ the PolyLoss function with α=0.25 and γ=2. The hyperparameter ε is tuned to make the model focus more on hard-to-classify samples during training. The Performance comparison of ε of the PolyLoss function on the CDT subset are shown in Table 5. When ε=0, the PolyLoss function transforms into the focus loss function. The corresponding model achieves recognition accuracy, precision, recall, and F1 score of 88.89%, 91.11%, 88.89%, and 88.89%, respectively. At this point, the predicted probabilities for these samples are relatively low. Adjusting the hyperparameter ε enhances the recognition weight for difficult samples. When ε=1.6, the corresponding model achieved recognition accuracy, precision, recall, and F1 score of 92.59%, 93.65%, 92.59%, and 92.61%, respectively. Models employing the PolyLoss loss function can effectively improve recognition accuracy in clock drawing tests.

Table 5.

Performance comparison of ε of the PolyLoss function on the CDT subset.

ε	Accuracy	Precision	Recall	F1 Score
0	88.89	91.11	88.89	88.89
0.4	85.19	88.89	85.19	85.10
0.8	81.48	86.93	81.48	81.23
1.2	81.48	86.93	81.48	81.23
1.6	92 . 59	93.65	92.59	92.61
1.8	85.19	88.89	85.19	85.10
2.2	85.19	88.89	85.19	85.10
2.6	85.19	86.20	85.19	85.23
3.0	85.19	86.20	85.19	85.23

Bold values indicate the highest accuracy achieved by each model in the experiment.

Table 6 describes the performance comparison of different fusion depths on clock testing subsets. The accuracy, precision, recall, and F1 score of the model with a fusion depth of two fully connected layers were 92.59%, 93.65%, 92.59%, and 92.61%, respectively, outperforming all other models in terms of performance. Compared to 0-layer and 1-layer fusion, 2-layer fusion depth avoids premature information loss; compared to 3-layer and 4-layer fusion, 2-layer fusion depth maintains better feature relevance. This finding provides important guidance for designing multimodal fusion networks. Overly simplistic fusion may fail to fully exploit complementary information between modalities, while overly complex fusion may introduce unnecessary computational overhead and risks of overfitting. The 2-layer fusion depth achieves an optimal balance between expressive capability and generalization performance, making it the best architectural choice for this task.

Table 6.

Performance comparison of different fusion depths on clock testing subsets.

Integration depth	Accuracy	Precision	Recall	F1 Score
0	73.52	73.33	68.75	70.96
1	85.19	88.89	85.19	85.10
2	92.59	93.65	92.59	92.61
3	88.89	89.19	88.89	88.92
4	88.89	91.11	88.89	88.89

Comparison

This section systematically conducted multi-model comparative experiments to comprehensively evaluate the performance of different modality inputs and model architectures in automatic recognition of AD in the clock drawing test. The experiments were conducted on two independent datasets, DARWIN and NHATS, encompassing traditional machine learning models, such as SVM and RF, deep neural networks, and the proposed method of EDADLMF. This provides robust empirical evidence across two dimensions: horizontal architecture comparison and vertical data generalization.

Compare the recognition performance of single modality (image modality or process signal) and dual modality on different subsets of DARWIN, as shown in Table 7. The experimental results show that in the three subsets of task 2, task 21, and task 24, the CDT task has the highest recognition accuracy for AD. This is attributable to the CDT task's comprehensive sensitivity in assessing cognitive domains such as executive function, visuospatial abilities, and memory retrieval areas that are central to early impairment in AD.

Table 7.

Performance comparison of different models on DARWIN subsets.

Modal	Model	Accuracy			Precision			Recall			F1 Score
		Task			Task			Task			Task
		2	21	24	2	21	24	2	21	24	2	21	24
Static images	AlexNet	78.12	78.13	78.12	77.78	76.92	77.77	95.23	98.73	93.33	78.79	81.08	77.42
	VggNet16	81 . 25	81.25	84.37	83.58	84.24	84.45	81.25	81.25	84.37	81.25	81.25	84.33
	GoogLeNet	78.12	75.00	81.50	80.83	78.02	85.11	79.02	75.00	76.00	77.93	74.80	80.27
	ResNet18	78.13	78.13	81.25	78.32	78.32	81.25	78.13	78.15	81.25	78.15	78.15	81.25
Process signal	MobileNetV2	75.00	75.00	78.13	78.79	77.78	78.13	76.07	75.29	78.24	74.90	75.00	78.10
	MLP	76.47	82.35	82.86	81.25	82.61	80.00	72.22	90.48	80.00	76.47	86.36	80.00
	TabNet	67.65	82.35	71.43	71.20	77.27	72.00	66.67	94.44	73.00	68.24	85.00	71.00
	SVM	70.58	70.29	68.57	92.31	78.08	68.59	57.14	63.89	68.46	70.59	68.84	68.47
	RF	72.35	69.71	75.00	75.54	72.93	74.52	72.22	70.00	72.50	73.54	70.72	73.00
	KNN	65.59	69.41	68.57	69.42	73.92	69.17	64.44	66.67	68.79	66.42	69.42	68.47
	DT	60.29	67.35	66.47	62.65	70.64	64.92	67.78	68.33	66.38	64.48	68.77	64.44
Muiti-moda	Ours	85.19	85.29	92.59	85.75	85.63	93.65	85.19	85.29	92.59	84.97	85.39	92.61

Bold values indicate the highest accuracy achieved by each model in the experiment.

On the CDT subset in Table 7, the EDADLMF method achieved the best performance with a recognition accuracy of 92.61%, surpassing the best single-modal model by 8.22%. In the modality of the image, the recognition accuracy of VGGNet16 model is 84.37%, which is better than other methods. In the modality of the process signal, the MLP model achieved a recognition accuracy of 82.86%, outperforming other methods. The EDADLMF method significantly outperformed all comparison models. But the multimodal fusion EDADLMF method significantly outperforms all comparison models, demonstrating modal complementarity. When image quality is limited, process signal can provide effective complementary information for the EDADLMF method; conversely, when process signal are affected by noise interference, the imaging modality can serve as a reliable verification benchmark. This cross-modal complementary mechanism enables the EDADLMF method to exhibit enhanced robustness and generalization capabilities when confronted with fluctuating data quality, significantly outperforming single-modal models.

Since the NHATS dataset lacks process signal for the CDT, we systematically compare the classification performance of five classic CNN architectures (AlexNet, VGGNet16, GoogLeNet, ResNet18, and MobileNetV2) on this dataset, as shown in Table 8. VGGNet16 achieved the highest recognition accuracy of 66.03%, significantly outperforming other comparison models. It is worth noting that the overall classification performance on the NHATS dataset is lower than the results previously reported on the DARWIN dataset. Analysis of the data reveals that the performance differences primarily stem from the differing annotation standards applied to the two datasets. NHATS strictly classifies individuals with a definitive clinical diagnosis of AD as positive for the marker, while unconfirmed or missed cases are categorized as negative. Such marker noise may exert a certain influence on the model's classification performance.

Table 8.

Performance comparison of different models on NHATS dataset.

Modal	Model	Accuracy %	Precision %	Recall %	F1 Score %
Pictures	AlexNet	58.49	75.00	12.50	21.43
	VggNet16	66 . 03	76.03	66.04	65.20
	GoogLe-Net	60.38	75.62	60.38	59.13
	ResNet-18	64.15	76.39	64.15	64.22
	Mobile-NetV2	60.37	61.33	59.12	58.91

Bold values indicate the highest accuracy achieved by each model in the experiment.

Across two independent datasets, VGGNet16 achieved the highest recognition rates, demonstrating that its convolutional layers possess stable and transferable representational capabilities for edge and texture combinations in clock-drawn images. The NHATS dataset exhibits undetected noise, while the DARWIN labels are relatively clean. Despite this, VGGNet16 consistently outperforms the comparison architectures, demonstrating the model's robustness and generalization advantages under heterogeneous clock drawing tests. This further validates the sustained effectiveness of the fine-tuning strategy based on ImageNet pre-trained weights for clock drawing image tasks.

Case study

The prototype system developed based on clock drawing tasks for early AD screening needs to have functions such as data acquisition and processing module, detection module, and visualization module. Data acquisition is primarily used to collect data from test subjects during CDT, encompassing both static images and process signal. The data processing module is primarily used to retrieve local files, clock images, and process signal data. The CDT detection module serves as the core component of this system. It primarily employs EDADLMF to analyze collected data, identifying whether early AD is present within the data. The experimental platform is built on the Windows operating system. Algorithm implementation relies on the PyTorch deep learning framework, while the graphical user interface is developed using Python's built-in tkinter toolkit.

To evaluate the practical performance of the early AD disease identification system, a comparative testing plan was designed. Case A randomly selects 10 samples each of the AD and HC classes in the validation set. In Case B, volunteers were trained to replicate the clock drawings from Case A, thereby generating new static images and dynamic feature data. The purpose of Cases A and B was primarily to examine whether the process characteristics of the clock drawing test impact the detection results.

It should be noted that to evaluate the effectiveness of the algorithm model, a successful recognition is determined when the model outputs an AD probability greater than 50% and the actual label is 1, or when the AD probability is less than 50% and the actual label is 0. Since case b involves tracing the watch image, it can only replicate its static appearance, making it difficult to consistently capture process signal such as speed, acceleration, pressure, and stroke count. This system supports dual-modality input. When a single input is single-modality data, the other modality data is set to zero.

Table 9 presents the comparison of accuracy of different testing schemes in early AD recognition systems. In case A, the minimum recognition accuracy for the AD category exceeded 90%, while that for the HC category exceeded 80%, with an average accuracy of 85%. In case B, the AD category achieved a minimum recognition accuracy of over 90%, while the HC category reached a minimum accuracy of over 80%. The average accuracy across all categories was 70%. These results indicate that: 1) the process signal during the clock-drawing test influence the accuracy of identifying early AD. When relying solely on static images for diagnosis, the lack of a mapping relationship between static images and dynamic features adversely affects diagnostic accuracy. 2) The process signal generated during the test subject's clock-tracing task exhibit differences from authentic samples, leading to a decline in the system's recognition accuracy.

Table 9.

Comparison of accuracy of different testing schemes in early AD recognition systems.

Case	AD		HC		Total accuracy/%
Case	Correct number	Accuracy/%	Correct number	Accuracy/%	Total accuracy/%
A	9	90	8	80	85
B	8	80	6	60	70

In summary, the AD early detection system developed in our study can identify the designed situation with an accuracy rate of 85%. The system demonstrates excellent robustness, enabling data collection across different brands of tablets and digital styluses to assess cognitive impairment through participants’ clock drawing tasks. This indicates that the method is suitable for the early detection of AD.

Discussion

This article proposes a recognition method based on CDT multimodal features (EDADLMF) to achieve early AD auxiliary evaluation and overcome the influence of cultural background differences. First, we design a dual-stream feature extraction (DSCDEF) and a deep feature fusion (SEFF) modules to effectively integrate the static visual information from the CDT with dynamic process signal. Experiments demonstrate that this multimodal fusion strategy significantly outperforms single-modal approaches, validating the effectiveness of integrating complementary information to enhance the robustness of early AD identification. Second, to address the challenge of difficult-to-recognize samples, we introduce and optimize the PolyLoss function. By fine-tuning its hyperparameters, we further enhance the model's discriminative capability. Test results on the DARWIN subset and NHATS dataset demonstrate that this method exhibits strong generalization capabilities, achieving recognition accuracy and F1 scores of 92.59% and 92.61%, respectively, outperforming existing approaches. Third, a qualitative assessment is conducted through case studies, validating the effectiveness of multimodal data fusion.

To improve the robustness and systemic flexibility of early AD identification models, future research should concentrate on these three aspects. 1) Enhancing the handwriting process signal. Expanding on the existing CDT feature set, introduce more discriminative process signal such as pen pressure variation trajectories and velocity-time fluctuations to increase feature diversity and temporal discrimination capabilities. 2) Developing a comprehensive real-time analysis system. At present, the data collection and intelligent recognition systems operate independently. The next step will be to advance system integration, enabling real-time processing from handwriting data collection and feature extraction to AD risk assessment, thereby enhancing the system's practicality and response efficiency. 3) Expanding multimodal data fusion: Beyond the current utilization of static image and dynamic handwriting features, we plan to introduce multimodal behavioral and physiological signals such as speech, eye tracking, and facial micro expressions to construct a more comprehensive early screening model for Alzheimer's disease through cross modal representation learning. Furthermore, the most reliable evaluation method is dynamically selected based on the patient's baseline functional status, thereby enhancing the applicability of the model in clinical populations with different comorbidities.

Footnotes

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China under Grant No.62373116, and the Guizhou Provincial Science and Technology Projects, China, under grant QKHZC [2023]118, QKHJCQN [2025]055, and GHJD[2025]005.

ORCID iDs

Fengying Yang

Biao Xu

Jiacheng Lin

Donghua Zheng

Shangen Lan

Kexin Luo

Guanci Yang

Ethical considerations

This research was performed in accordance with Subcommittee of Human Medical Experimental Ethics of Guizhou University approved protocol HMEE-GZU-2023-7009.

Consent to participate

Not applicable

Consent for publication

Not applicable

Author contribution(s)

Fengying Yang: Funding acquisition; Methodology; Project administration; Resources; Supervision; Writing – review & editing.

Biao Xu: Investigation; Software; Writing – original draft.

Jiachen Lin: Formal analysis; Project administration; Writing – review & editing.

Donghua Zheng: Data curation; Validation; Writing – review & editing.

Shangen Lan: Investigation; Software; Writing – review & editing.

Kexin Luo: Investigation; Visualization.

Guanci Yang: Funding acquisition; Methodology; Project administration; Resources; Supervision; Writing – review & editing.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China, Guizhou Provincial Science and Technology Projects, (grant number 62373116, QKHZC [2023]118, QKHJCQN [2025]055, and GHJD[2025]).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

Data will be made available from corresponding author on reasonable request.

References

Alzheimer's Association. 2024 Alzheimer's disease facts and figures. Alzheimers Dement 2024; 20: 3708–3821.

Yang

, et al. Cognitive assessment in neurodegenerative diseases: A review of interactive methods. Information Fusion 2026; 127: 103836.

Gandhi

Murugesh

. Detection of neurodegenerative disease in brain using region splitting based segmentation with deep unsupervised neural networks. Expert Syst 2022; 39: e12775.

Narnaware

Mulla

. Early detection of multi-class Alzheimer’s disease using Hybrid capsule auto encoder model. Biomed Signal Process Control 2025; 110: 108330.

Wang

, et al. Brain tau PET-based identification and characterization of subpopulations in patients with Alzheimer's disease using deep learning - derived saliency maps. EJNMMI Phys 2025; 12: 55.

Zhang

Cui

Lü

, et al. A multimodal learning machine framework for Alzheimer's disease diagnosis based on neuropsychological and neuroimaging data. Comput Ind Eng 2024; 197: 110625.

Azzali

Cilia

De Stefano

, et al. Automatic feature extraction with Vectorial Genetic Programming for Alzheimer’s Disease prediction through handwriting analysis. Swarm Evol Comput 2024; 87: 101571.

Karimpoor

Churchill

Tam

, et al. Functional MRI of handwriting tasks: a study of healthy young adults interacting with a novel Touch-Sensitive tablet. Front Hum Neurosci 2018; 12: 30.

Zhang

Steingrimsson

Ambekar

, et al. Radiomics of PET using neural networks for prediction of Alzheimer's disease diagnosis. Stat Med 2025; 44: e70128.

10.

Sharma S

and Mandal PK

. A Comprehensive Report on Machine Learning-based Early Detection of Alzheimer's Disease using Multi-modal Neuroimaging Data. ACM Comput Surv 2023; 55: 1–44.

11.

Quan

Moreno-Gonzalez

Xie

, et al. A near‐infrared probe for detecting and interposing amyloid beta oligomerization in early Alzheimer's disease. Alzheimers Dement 2023; 19: 456–466.

12.

Odusami

Maskeliūnas

Damaševičius

, et al. Analysis of features of alzheimer's disease: Detection of early stage from functional brain changes in magnetic resonance images using a finetuned ResNet18 network. Diagnostics (Basel) 2021; 11: 1071.

13.

Vicedo

Nieto-Reyes

Bringas

, et al. Automatic apraxia detection using deep convolutional neural networks and similarity methods. Mach Vis Appl 2023; 34: 60.

14.

Negin

Rodriguez

Koperski

, et al. PRAXIS: towards automatic cognitive assessment using gesture recognition. Expert Syst Appl 2018; 106: 21–35.

15.

Yang

, et al. Human activity recognition based on multienvironment sensor data. Inf Fusion 2023; 91: 47–63.

16.

Giudicessi

Aduen

Fox-Fuller

, et al. The MAPP room memory test: examining contextual memory using a novel computerized test in cognitively-unimpaired individuals with autosomal dominant Alzheimer's disease. J Prev Alzheimers Dis 2024; 11: 463–468.

17.

Chien

Huang

, et al. The use of verbal and nonverbal memory tests for Alzheimer's disease screening in Taiwan Chinese. Am J Alzheimers Dis Other Demen 2023; 38: 250560268.

18.

Cilia

De Stefano

Fontanella

, et al. Handwriting analysis to support Alzheimer's disease diagnosis: a preliminary study. In: Vento

Percannella

(eds) Computer analysis of images and patterns. CAIP 2019, Salerno, Italy. Cham: Springer, 2019, pp.143–151.

19.

Hosseini-Kivanani

Salobrar-Garcia

Elvira-Hurtado

, et al. Predicting Alzheimer's disease and mild cognitive impairment with off-line and on-line house drawing tests. In: 2024 IEEE 20th international conference on e-science (e-Science). Osaka, Japan: IEEE, 2024, pp.1–10.

20.

Stanzani

Matacena

Baccari

, et al. The tree drawing test in evolution: an explorative longitudinal study in Alzheimer's disease. Am J Alzheimers Dis Other Demen 2022; 37: 240488613.

21.

Sejunaite

Gaucher

Lanza

, et al. Clock drawing test: types of errors and accuracy in early cognitive screening. J Alzheimers Dis 2023; 95: 1597–1608.

22.

Yang

Long

, et al. Rapid cognitive assessment: accuracy and discriminant validity of Mini-Cog and process-based Clock Drawing Test. J Alzheimers Dis 2025; 104: 200–208.

23.

Ruengchaijatuporn

Chatnuntawech

Teerapittayanon

, et al. An explainable self-attention deep neural network for detecting mild cognitive impairment using multi-input digital drawing tasks. Alzheimers Res Ther 2022; 14: 111.

24.

Raksasat

Teerapittayanon

Itthipuripat

, et al. Attentive pairwise interaction network for AI-assisted clock drawing test assessment of early visuospatial deficits. Sci Rep 2023; 13: 18113.

25.

Park

Lee

. Automatic, qualitative scoring of the clock drawing test (CDT) based on U-Net, CNN and mobile sensor data. Sensors (Basel) 2021; 21: 5239.

26.

Kang

Wang

Lim

. Classification model of clock drawing test based on contrastive learning using Multi-Channel features with Channel-Spatial attention. IEEE Access 2024; 12: 186466–186475.

27.

Jiang

Zhang

Zeng

, et al. Mobile-based clock drawing test for detecting early signs of dementia. In: Thirty-Fifth AAAI conference on artificial intelligence (AAAI-21), 2021, pp.16048–16050.

28.

Amini

Zhang

Hao

, et al. Remote diagnosis of dementia using AI methods on clock drawing images. Alzheimers Dement 2021; 17: e51911.

29.

Qin

Gonzalez

, et al. A novel vision transformer model produces clock drawing test scores as accurate as expert human coders. Sci Rep 2026; 16: 4032.

30.

Jimenez-Mesa

Arco

Valenti-Soler

, et al. Using explainable artificial intelligence in the clock drawing test to reveal the cognitive impairment pattern. Int J Neural Syst 2023; 33: 2350015.

31.

Amini

Zhang

Hao

, et al. An artificial intelligence-assisted method for dementia detection using images from the clock drawing test. J Alzheimers Dis 2021; 83: 581–589.

32.

Cilia

De Gregorio

De Stefano

, et al. Diagnosing Alzheimer's disease from on-line handwriting: a novel dataset and performance benchmarking. Eng Appl Artif Intell 2022; 111: 104822.

33.

Kang

Zhang

Guan

, et al. Early Alzheimer's disease diagnosis via handwriting with self-attention mechanisms. J Alzheimers Dis 2024; 102: 173–180.

34.

Gattulli

Impedovo

Pirlo

, et al. Handwriting task-selection based on the analysis of patterns in classification results on Alzheimer dataset. In: Proceedings of the IEEESDS’23: data science techniques for datasets on mental and neurodegenerative disorders. Zürich, Switzerland: Springer, 2023, pp.115–125.

35.

Cappiello

Caruso

. Quantum AI for Alzheimer’s disease early screening. Neurocomputing 2025; 647: 130565.

36.

Akpinar

Physics

University

, et al. Quantum machine learning in the cognitive domain: Alzheimer's disease study. arXiv preprint 2023; arXiv:2401.06697.

37.

Erdogmus

Kabakus

. The promise of convolutional neural networks for the early diagnosis of the Alzheimer's disease. Eng Appl Artif Intell 2023; 123: 106254.

38.

Cohen

Celnik

Pascual-Leone

, et al. Functional relevance of cross-modal plasticity in blind humans. Nature 1997; 389: 180–183.

39.

Cilia

Alessandro

De Stefano

, et al. Deep transfer learning algorithms applied to synthetic drawing images as a tool for supporting Alzheimer's disease prediction. Mach Vis Appl 2022; 33: 49.

40.

Cilia

D'Alessandro

De Stefano

, et al. From online handwriting to synthetic images for Alzheimer's disease detection using a deep transfer learning approach. IEEE J Biomed Health Inform 2021; 25: 4243–4254.

41.

Werner

Rosenblum

Bar-On

, et al. Handwriting process variables discriminating mild Alzheimer's disease and mild cognitive impairment. J Gerontol B Psychol Sci Soc Sci 2006; 61: P228–P236.

42.

Ding

Lyu

Karjadi

, et al. Association of the digital clock drawing test with amyloid and tau PET biomarkers in low age risk adults. Sci Rep 2025; 15: 11104.

43.

Luo

Yang

, et al. Croup and pertussis cough sound classification algorithm based on channel attention and multiscale Mel-spectrogram. Biomed Signal Process Control 2024; 91: 106073.

44.

Gong

Qin

El-Yacoubi

. Hybrid transformer for early Alzheimer's detection: integration of handwriting-based 2D images and 1D signal features. IEEE J Biomed Health Inform 2025; 29: 8293–8305.

45.

Leng

Tan

Liu

, et al. PolyLoss: A polynomial expansion perspective of classification loss functions. arXiv preprint 2022; arXiv:2204.12511.

46.

Cilia

Stefano

Fontanella

, et al. An experimental protocol to support cognitive impairment diagnosis by using handwriting analysis. Procedia Comput Sci 2018; 141: 466–471.

47.

Freedman

Schrack

Skehan

. National health and aging trends study user guide: Rounds 1–12 beta release, https://nhats.org/researcher/nhats/methods-dodsdocumention?id=user_guide (2023).

48.

Yang

Liu

, et al. Short-term prediction method of blood glucose based on temporal multi-head attention mechanism for diabetic patients. Biomed Signal Process Control 2023; 82: 104552.