A customized CNN model for signature authentication—Forensic implications

Introduction

A handwritten signature is a unique way for an individual to authenticate the documents and signify their approval. Signature verification is legally and publicly recognized worldwide. In personal identification systems, the biometric measures play a crucial role. Biometric traits are broadly classified into physiological (physical) and behavioral characteristics. Physiological biometrics include fingerprints, hand or palm geometry, retina, iris, facial features, and voice, which are commonly used for recognition and identification. Behavioral biometrics, on the other hand, include signatures, keystroke dynamics, and gait patterns. Among all biometric traits, the voice and signatures are the most developed technologies for individual recognition and identification.^1–3

The execution of a signature is a result of muscular and nerve coordination, influenced by the signer's physical and physiological characteristics as well as the signing process. ⁴ Authentication of a signature involves comparing the handwriting characteristics of a copied or forged signature with those of admitted or known signatures.^5,6 The document examiner must be able to distinguish genuine from forged signatures among the samples examined. In addition, the signature and handwriting of a person can provide an insight into age, sex, handedness, and ethnicity, facilitating forensic identification of an individual.^7–9

Until now, handwritten signatures in forensic examinations have been manually analyzed by forensic document examiners (FDEs). However, manual authentication of the signatures can be time-consuming and prone to errors. To enhance the accuracy and hence improve their opinion on signature identification one can, use the modern technique such as machine learning and deep learning algorithms of artificial intelligence (AI). This use of computer-based AI recognition system for handwritten signatures can lead to its automation in future. Moreover, AI-based examination can significantly reduce the time required and increase the accuracy of identifying forgeries in documents. This technology can explicitly be very much relevant in banking sectors, where issuing the check and financial documents are still using the handwritten signatures from the users.^1,10 Consequently, these sectors are vulnerable to fraud. Sections other than banking such as wills, property dispute, official documents and crimes related to handwritten signature needs such automatic signature analyzer, which can identify genuine or forged signatures without or less human aid with higher accuracy rate.

To date, researchers have used various AI model to identify the genuine or forged signatures. For instance, Yapici et al. ¹¹ presented a new application for Cycle-GAN as a data augmentation to address the problem of insufficient data in signature verification. The authors tested the proposed data augmentation on four different convolutional neural network (CNN) methods—VGG-16, VGG-19, ResNet-50, and DenseNet-121, on two widely used databases, GPDS and MCYT. By the process of transfer learning (fine tuning of pre-trained models), among the above-mentioned models DenseNet-121 obtained the most accurate results with an accuracy of 97%. In addition, Tyagi et al. ¹² proposed a novel unsupervised approach that is convolutional autoencoder–support vector machine (CAE–SVM), which extracts features using CAE and identifies writers using SVM. They investigated the potential of hyperspectral imaging (HSI) as a non-destructive and sophisticated method for detecting and preventing document forgery. This method captures the subtle spectral differences in handwriting samples using HSI. This may be working on the principle that each ink has different spectral signature, showing how it reflects light at difference wavelength. They tested the writing ink hyperspectral image dataset for blue and black inks and the proposed approach achieved the highest accuracy of 92.78% surpassing other methods. The study's findings highlight its effectiveness as a powerful forensic analysis tool for detecting and preventing document forgeries.

However, previous studies have predominantly relied on pre-trained deep learning-based models and fine-tune them for their specific dataset.

The customized models on primary datasets are advantageous and necessary because of the fact that the data are collected strictly according to the needs of the model and the study in the primary dataset. Moreover, in the primary dataset, as in case of the present study, various handwriting characteristics such as strokes, direction, inclination, curvature, and tremor, can be studied efficiently from the original images of signatures obtained from the participants. However, the fine tuning of pre-trained models may lack specificity and have a chance of domain mismatch. Therefore, the present research was aimed to customize a deep learning-based CNN model trained on a primary dataset of 1400 signatures (700 genuine and 700 forged) to identify the forged signatures. This model has wide applications in banking sector, forensic document examination, court rooms and beyond.

Material and methods

This section provides the details of proposed signature classification (forged or genuine), data collection, dataset description, preprocessing of images, and the structure of the CNN used for prediction (Figure 1). The software used in the preparation of this model is the Anaconda (Jupyter) notebook for scripting high level language Python along with the required library modules. All the experiments were done in the work station of CPU with 2.00 GHz processor and 16 GB RAM.

OPEN IN VIEWER

Figure 1.

The schematic diagram of the proposed approach for preparing CNN model.

Dataset description

The study employed a dataset comprising 1400 signature images, evenly distributed with 700 instances for each class, namely forged and genuine signatures. Each class was systematically partitioned into training, validation, and test subsets using a 7:3:4 ratio, respectively. Figure 2 visually depicts the selection of sample images extracted from the dataset and their splitting technique for reference.

OPEN IN VIEWER

Figure 2.

Dataset description, (A) sample images from the signature dataset, (B) scheme of dividing the dataset into training, validation, and testing subsets.

Proposed convolution neural network (CNN) model

The CNN is the neural network deep learning model that is frequently employed. The CNN uses Python frameworks such as Keras and Tensorflow. The CNN architecture generally contains five types of layers: the input layer, convolution layer, pooling layer, fully connected (fc) layer, classification (output) layer. The number of these layers were experimentally increased or decreased based on the accuracy in the output layer. The final architecture of the proposed CNN model (Figures 3 and 4) is constituted of 4 convolutional layers, 4 max-pooling layers, 1 flattened layer, 2 fully connected (dense) layers with dropout layer and 1 output layer. Each convolutional layer followed by a batch normalization + ReLU (rectified linear unit) activation function ¹⁵ that collectively prevents the model from overfitting. Experiments were performed by adjusting the number of layers and all other hyperparameters in the model, and the architecture yielding the highest validation accuracy was selected. Each layer of CNN is responsible for learning, extracting and deducing the information from the image and finally classifying it as a genuine or forged signature. Training and validation of dataset highly depend upon the hyperparameters in each epoch. The hyperparameters which decide the performance of deep learning model consist of batch size, learning rate, filters, grid size, pool size, number of epochs, etc. (Supplementary Table S2); all these hyperparameters can be adjusted according to the model's requirement.

OPEN IN VIEWER

Figure 3.

The architecture of the proposed convolutional neural network (CNN) for handwritten signature classification, demonstrating the reduction in pixels of images after each convolutional layer with filters and grid size in each layer.

OPEN IN VIEWER

Figure 4.

A schematic block diagram of CNN classifier for handwritten signature analysis.

Performance evaluating metrics

Evaluation of the trained CNN model is necessary by checking its working on the test dataset. The performance of the trained CNN model can be evaluated by different criteria consisting of accuracy, specificity, sensitivity, precision, recall, and F1-score. ¹⁶ The accuracy represents the complete share of the correctly classified samples. Specificity measures how effectively the model identifies negative cases, while sensitivity assesses the accuracy of true identifications. Precision indicates the number of predicted positive samples that are actually correct. The F1-score combines precision and recall to provide a balanced measure. Relying solely on accuracy can lead to biased results, particularly with imbalanced datasets. Therefore, these metrics are typically employed to ensure balanced and reliable evaluations of the model's performance. These criteria are computed by using the confusion matrix (Figure S1) where, genuine signatures treated as positive and forged signatures are treated as negative. All the values as provided in the following equations:

A c c u r a c y = T P + T N T P + F P + T N + F N

S p e c i f i c i t y = T N T N + F P

S e n s i t i v i t y = T P T P + F N

P r e c i s i o n = T P T P + F P

F 1 - S c o r e = 2 × P r e c i s i o n × S e n s i t i v i t y P r e c i s i o n + S e n s i t i v i t y

Results and discussion

The training curve and loss curve of the model with each epoch are shown in Figure S2. These curves provide the training accuracy and loss of information after each epoch of model training. After the completion of 20 epochs, the training accuracy and validation accuracy for each class were measured at 97.32% and 97.92%, respectively. Additionally, Figure S3 shows the activation mapping (heatmap) for 128 filters of the last convolution layer in the model. This activation mapping indicates that the model is learning the signature part of the image separately in some filters and prioritizing this part of the image for decision making in the classification process. Moreover, a comprehensive evaluation was conducted using a confusion matrix (Figure S1) on a test set comprising 400 images with 200 images of each class. This matrix illustrates the number of images from test dataset as true positive (TP), true negative (TN), false positive (FP), false negative (FN). Notably, the matrix shows that 180 forged signature images were correctly identified as forged, representing TN cases, while 158 genuine signature images were correctly classified, constituting TP cases. The overall confusion matrix for the test yielded a test accuracy rate of 84.5% for prediction. Therefore, the overall accuracy yielded by the present model are 97.32% and 97.92% corresponding to training and validation accuracy, respectively, and 84.5% to test accuracy.

The number of FN (actual genuine signatures predicted as forged) were more than FP, it infers that most of the genuine signatures are predicted as forged perhaps due to the presence of natural variations in the genuine signatures. ¹⁷

Furthermore, other evaluation matrices were also computed, including a precision, recall (sensitivity), F1-score, and specificity, which provided the values of 85%, 84%, 84%, and 90%, respectively. The present evaluation criteria collectively affirm the model's effectiveness and reliability in signature authentication as genuine or forged. Along with all these evaluation matrices, the area under the receiver operating characteristics (ROC) curve abbreviated as AUC is also analyzed. The plot of the ROC curves for testing dataset is shown in Figure S4 with the AUC (area under curve) score as 98% (0.98). The observed differences in accuracy between the training/validation and testing sets may indicate the presence of overfitting or underlying dataset variability, highlighting the need for improved generalization through larger and more diverse datasets.

The present study distinguished genuine signatures of the same person with forged signatures obtained by the process of simulation from different subjects. Supplementary Table S3 compares the performance of the proposed CNN model to the state-of-the-art used by other authors for nearly identical purposes. For instance, Maza-Merchán and Cordero ¹⁸ proposed shallow CNN model and trained on CEDAR handwritten signature dataset. With an Equal Error Rate of 1.716%, the model can identify the handwritten signatures of 55 users, confirming their authenticity against forgeries and outperforming other approaches using the same dataset. Another study was conducted by Lopes et al. ¹⁹ where they have used multiclass CNN inspired by the AlexNet architecture. After the training of the model on few genuine signatures, the model obtained the precision and recall of 85%. Supplementary Table S3 lists additional studies and compares their accuracy to the suggested model's accuracy.^20,21

The principal advantage of the current proposed model lies in its high validation and testing accuracy which was thoroughly evaluated through the use of unseen images during training. While both the current model and previous iterations demonstrate high accuracy in signature identification, further training on a diverse database with varying signature formations is essential for real world deployment in forensic examination scenarios. Additionally, future advancements may involve exploring preprocessing tools beyond ImageDataGenerator to enhance overall performance and prevent data overfitting. Furthermore, in forensic science, the natural variations ²² in the handwritten signature or any handwritten text have also been taken under consideration during the examination of documents. However, AI models may be misled by these natural variations. Therefore, the model training should account for such natural variation and classify the signatures as either forged or genuine accordingly.

The limitations of the present study include a small dataset for the current customized CNN, however, it's a primary dataset and the first-hand information especially collected for the present study. Moreover, the signatures were collected from a single population, a specific geographic area, and all the signatures were obtained in the same language. It is important to acknowledge that the model is presently limited to training on a single type of signatures. Using this model's architecture, future researches may expand its training on various types of signatures over a large dataset, thereby facilitating deployment in the field of forensic document examination. Nevertheless, in future if this model can get the potential of deployment in the forensic document examination, examiner can use the model to scan the signature and on spot get-to-know about its authenticity in less time and with high accuracy.

Conclusion

The customized CNN model demonstrated strong performance with training and validation accuracy of 97.32% and 97.92%, respectively. In addition, when model is tested on 400 unseen images of signatures and provided an accuracy of 84.5%. Other evaluation criteria were also employed, including precision, recall (sensitivity), F1-score, and specificity, which yielded values of 85%, 84%, 84%, and 90%, respectively. Since the model is population specific and is limited to training on a single type of signature, therefore, the future model can be trained on different signatures with different population, geographical area as well as in different languages, which can make the model more suitable for future real-time applicability in forensic document examination and other document verification fields.

Supplemental Material