Document image binarization using difference of concatenated convolutions

Abstract

Binarization is the most important stage in historical document image processing. Efficient working of character and word recognition algorithms depend on effective segmentation methods. Segmentation algorithms in turn depend on images free of noises and degradations. Most of these historical documents are illegible with degradations like bleeding through degradation, faded ink or faint characters, uneven illumination, contrast variation, etc. For effective processing of these document images, efficient binarization algorithms should be devised. Here a simple modified version of the Convolutional Neural Network (CNN) is proposed for historical document binarization. AOD-Net architecture for generating dehazed images from hazed images is modified to create the proposed network.The new CNN model is created by incorporating Difference of Concatenation layer (DOC), Enhancement layer (EN) and Thresholding layer into AOD-Net to make it suitable for binarization of highly degraded document images. The DOC layer and EN layer work effectively in solving degradation that exists in the form of low pass noises. The complexity of working of the proposed model is reduced by decreasing the number of layers and by introducing filters in convolution layers that work with low inter-pixel dependency. This modified version of CNN works effectively with a variety of highly degraded documents when tested with the benchmark historical datasets. The main highlight of the proposed network is that it works efficiently in a generalized manner for any type of document images without further parameter tuning. Another important highlight of this method is that it can handle most of the degradation categories present in document images. In this work, the performance of the proposed model is compared with Otsu, Sauvola, and three recent Deep Learning-based models.

Keywords

Binarization historical document images degradation difference of concatenated convolutions enhancement layer

1 Introduction

Historical document images exist in a variety of colors and shading. Physical appearance of these documents varies based on factors such as lighting, viewing angle and aging of the documents. These documents that are digitized using scanners and advanced cameras contain undesired textual noises like specks, dots, black borders and hole punch marks. Moreover, due to aging and improper handling various degradation are induced in these images. The degradation associated with document images are uneven illumination, contrast variation, bleed through degradation, faded ink or faint characters, smear or show through, blur and deterioration. Figure 1 shows examples of some degraded images taken form DIBCO and H-DIBCO dataset.

Fig. 1

Degraded images taken from DIBCO and HDIBCO dataset.

For efficient working of optical character recognition the target text area should be effectively segmented from these documents. Segmentation of text area in turn depends on selection of efficient image enhancement and binarization algorithms. Many works have been reported in the area of document image binarization, but the research area is still active since a single algorithm has not yet been devised to work on variety of document images. Binarization algorithms are classified into local binarization and global binarization algorithms. In local binarization algorithm different parts of the documents are binarized with different thresholds. In global binarization algorithm a single threshold is selected for binarization of the entire image. Binarization algorithms can be divided into 5 categories.

Normal thresholding

Histogram based Thresholding

Evaluation of criteria functions

Sliding window based

Neural Network based

In normal thresholding based methods [2], the threshold value is selected as the medium of distribution of pixel values in the image so that 50%of pixel value lies in each side of the medium.

In [3], an iterative method of threshold selection is implemented. Here an initial random choice of threshold values are made and then images are divided into two classes based on the selected threshold. The mean of each class is estimated and the average of the two mean values based on distribution is selected as the next threshold value. The method works in an iterative manner until there is no change in the threshold value for subsequent iterations.

The most standard approach in threshold selection is based on the location of valleys that exist in the histogram. Most commonly used method for document image binarization is the selection of threshold value based on histogram of an image. But the selection of threshold based on a particular criteria do not work effectively in entire document image scenario. A long history of research exists in the domain of document image processing, focusing on finding an apt threshold value for document image binarization. In reference [4], a transformed histogram is created to enhance the performance of normal histogram based threshold selection. The transformed histogram is created based on the scatter plot of gray values in the image. A weighted projection based on the plotted values on the gray level axis is carried out to create the transformed histogram. Location of valleys on the transformed histogram is used for threshold selection.

In the method devised in [23], the gray level histogram is normalized and probability distribution is estimated for binarization. Here the method adopted is to calculate an optimal threshold value which maximizes between class variances. Here the objective functions are designed based on between class variance, within class variance and total variance. Based on the objective functions, a threshold value is selected and probability values of pixels are calculated based on histogram estimation. Here the binarization problem is reduced to an optimization problem that searches for a threshold k that maximizes one of the objective functions designed based on histogram values. The method can further be extended for the separation of given documents into more than one class by selecting multiple threshold values.

The entropy based method was first introduced in [5]. Pun^′s algorithm analyzes the entropy of white and black pixels. Let entropy of black pixels be H_b and entropy of white pixels be H_w. The algorithm works in finding a threshold value ‘t’ that maximizes the function H = H_b + H_w. A variation of entropy based function is implemented in [6]. In [7], an entropy based thresholding method is implemented by assigning different probability distributions for objects and background. In [8], a histogram based entropy function is implemented for document image binarization. In [9], the entropy of pixels below and above a particular threshold value is calculated. The aggregate histogram value based on two entropy values are evaluated. Based on the evaluated aggregate entropy, the image is binarized into background and text area. Threshold values selected for binarization of images are based on maximization or minimization of criteria functions. The criteria functions were designed based on the parameters that optimally describes the image. In [10], selection of threshold values are designed based on maximization of a criteria function. Here the criteria function is designed based on continuous probability distributions obtained from the histogram. In [11], the threshold value for image binarization is selected by optimization of a criteria function. The function is designed based on the difference between two parameters. The first parameter is estimated based on the normal density of pixel distribution in the two classes and the second parameter is selected based on the conditional probability evaluated for a pixel to be in foreground and background image. In [12], the target binarization is designed based on the minimization of a global energy function carved modelling Markov random field model. The energy function works on two functions like Laplacian function and Smoothness term function. Laplacian of intensity value is used to differentiate ink from the background. In [13], binarization is carried out by an adaptive thresholding approach based on the image contrast. The method computes the local threshold value by using the mean value of the maximum (max(i, j)) and minimum (min(i, j)) intensities inside a local neighborhood window centered at a particular pixel (i, j). The method works only when the contrast threshold value is less than 15. This method fails in case of larger contrast values. In Niblack^′s algorithm [14] pixel-wise threshold is calculated based on the local mean ‘m’ and the standard deviation ‘s’ of all the pixels in a sliding rectangular window. In [15], the researchers Sauvola and Pietikainen proposed an improvement on Niblack^′s method with the aim to overcome the background noise problem. The two features based on which the algorithms work are the average of the gray level values and transient difference within the window. In [16], Gatos et al. proposed a multi-stage document image binarization method. The first step in the algorithm applies a low pass Wiener filter to reduce noise and correct the image contrast followed by Sauvola^′s method for preliminary image segmentation. Then, the background is estimated using intensity analysis. In the last step the final thresholding result is estimated by taking the differences of images resulted from the first and second step. The main drawback of this technique is that it allows only the enhancement of the textual part in the document images. In [25], Khurshid et al. proposed a modification on Niblack^′s method and entitled Nick^′s method. In [17], Su et al. presented a hybrid thresholding algorithm that combines different thresholding methods (Otsu^′s and Sauvola^′s methods, Gatos^′s and Su^′s methods, Lu^′s and Su^′s methods) to improve the document image binarization quality. In reference [18], a new method is proposed where the contrast value is estimated based on Michelson^′s contrast fomula.This method can only be applied to images if the contrast value is less than a particular threshold. In [19], Wolf proposed a method to normalize the contrast and mean gray value of the image and compute the threshold. In [20], instead of calculating the dynamic range of gray-value standard deviation from the whole image, local calculation is carried out by introducing the notion of two local windows, one encompassing the other. The values of local mean, the minimum gray-level and standard deviation are calculated using the primary local window. The dynamic range standard deviation is calculated using the larger window known as the secondary local window. Then the binarization threshold is computed from the above estimated values.

In [24], a contrast image based thresholding method is implemented. In this work, the image is binarized following 3 steps. They are construction of contrast image, detection of high contrast pixel and finally the threshold based binarization. In [21], a new methodology is proposed for improving document image binarization using three-stage algorithm. In the first stage, the background is removed using an iterative median filter. Then the miss-classified pixels are separated by a combination between Local Co-occurrence Mapping (LCM) and Gaussian Mixture Clustering. In the final stage isolated components are identified and suppressed using a morphology operator. In [22], Edge Level Thresholding(ELT thresholding) is carried out. It consists of three major modules: the ISEF edge detector (Shen-Castan (ISEF) Edge Detection), the MLS (moving least-squares) surface fit, and the thresholding module.

In this work a new CNN based model is proposed by modifying the structure of AOD-Net (All in one dehazing Network) [1]. AOD-Net has been implemented for dehazing the hazed image. The structure of AOD-Net is modified to include Difference of Concatenation (DOC) layer, Enhancement (EN) layer and Thresholding layer. These layers increase the overall quality of the output binary images when compared with existing CNN architectures. The concatenation layers help in reducing the structure of the proposed model by increasing the depth of input to each convolution layer. Convolutions with increased input depth, enable same updated weighted factors to be convolved with different intensity variations of the input image. Moreover, it will also reduce the complexity of the entire model by reducing the number of weight vectors considered for update through back propagation. The ‘DOC’ layer helps in increasing the intensity value of textual contents in the image. The ‘EN’ layer works to improve the overall efficiency of output images by incorporating a new method for image enhancement.

Section 2 discusses some neural network based methods for implementation of document image binarization. Section 3 depicts the design and structure of the newly proposed model. Section 4 exhibits the experimental analysis.

2 Related work

In [26], three neural based binarization techniques are proposed. Here a Self Organizing Map (S.O.P) is employed on the image to find the most representative gray levels or colors. Once the number of gray levels or colors are identified, methods like Kmeans,Sauvola or Niblack are used for binarization in case of gray scale images and Multi Layer Perceptron (MLP) for color images. The experimental analysis part reveals that the specified methods work better compared to other classical binarization techniques. In [27], binarization is carried out using Long Short-Term Memory (LSTM). Here 2D LSTM is employed to classify a pixel as text or background. The method initially starts by processing information based on local context and then propagates information globally to achieve better results. The paper reveals that the concerned method produces excellent results in DIBCO-2013 dataset. In [28], binarization is formulated as a pixel classification task where Fully Convolutional Network (FCN) architecture which works at multiple image scales including full resolution is proposed. Here a multiscale architectural variation for FCN is implemented that results in precise localization and preserves the benefit of down sampling. In this work the same network is trained on both historical paper documents and palm leaf documents. This method overrides the current state-of-the-art methods that had poor results in palm leaf documents. FCN can easily be adapted to any binarization ground truth environment as it does not incorporate any prior information. In this work a comparative analysis is carried out using different optimizing loss functions like Cross Entropy, Pseudo F-measure (P-FM), F-measure and combination of both Pseudo F-measure (P-FM) and F-measure. The Experimental analysis part shows that the proposed method can handle diverse domains of documents. In [29], convolutional auto-encoders are proposed to convert an input image into a binary image. The auto-encoder used here is termed as Selectional Auto-Encoder since the selection value of each pixel depends on whether the pixel belongs to foreground or background. Three topological variations of selectional autoencoders like Convolutional Auto-Encoder (CAE), Stacked What-Where Auto-Encoder (SWWAE) and Very deep Residual Encoder-Decoder Network (RED-Net) are used here for experimental analysis. Here F-measure (Fm) is used to perform a direct evaluation of the binarization process. Once trained this model, it performs binarization by applying global threshold. This approach has proven to outperform existing binarization strategies in various benchmark datasets.

In [30], a deep hierarchical Deep Supervised Network (DSN) architecture is learned for the prediction of text pixels at different feature levels. Here by using higher level features the network can easily differentiate the text pixels from background noises and also manages degradation. Binary images generated using this architecture produced clearer background and better preserved strokes. The DSN model provides better performance compared to Convolutional Neural Network (CNN) structures regarding convergence and the test performance. However, using a single DSN results in loss of low-level information like character edges and contours, which is not a suitable binarization problem. A new hierarchical DSN architecture is proposed in this work to sustain different levels of information simultaneously. The design of this network architecture has been resulted from the coarse-to-fine principle that is usually applied to extract structures in images and videos. The hierarchical DSN structure is designed and adopted based on the observation that the predicted maps at the last layers reveal low noise and the early layers show clearer foreground maps. The proposed method is analyzed with widely used DIBCO datasets and produces outstanding results compared to state-of-the-art models. In [31], a Convolutional Neural Network (CNN) is used to classify each pixel in the image as either background or foreground. Here experiments were carried out to find the best topology, training data and features for the network. The author claims that this approach outperforms all other widely used binarization methods. In [32], a convolutional encoder- decoder model with deep learning is proposed for document image binarization. The encoder constituting of a stack of convolutional layers produce a mid-level document image representation and the encoder produces a high resolution image having the same size as the original image. The encoder network is composed of a stack of transposed convolutional layers. The proposed method is compared qualitatively and quantitatively with other binarization methods on the public domain dataset. The proposed method has comparable performance and more generalization capabilities compared to state-of-the-art methods for document image binarization. In reference [33], an iterative deep learning frame for enhancement and binarization is proposed. Here the CNN is trained to learn the degradation of document images and produces clean and uniform images as output. The architecture used here is a combination of recurrent refinement and stacked refinement. Recurrent refinement section uses the same trained neural network architecture for document enhancement and stacked refinement section uses a stack of different neural networks for iterative output refinement. After a series of recurrent refinement and stacked refinement, uniform and enhanced images are produced which is then subjected to a binarization module. A comparative analysis of various local and global techniques on resultant enhanced images are exhibited in the paper. The newly proposed architecture produced an outstanding performance in various standard benchmark DIBCO dataset. In [34], a U-shaped architecture is proposed for binarization of gray scale document images. This U-shaped architecture has successfully been applied in a variety of image segmentation problems. The main advantage of U-shaped architecture is its ability to capture the context dependent local binarization methods. It is a CNN type architecture which provides outstanding performance in H-DIBCO-2017 and H-DIBCO-2018 dataset. In [39], a new model consisting of two networks are proposed. Texture Augmentation (TANet) part of the model tries to obtain a noisy version of the image by adding the noisy texture of the reference document image and the Binarization Network (BiNet) part tries to binarize the newly generated noisy image. Joint training of the two networks enhances the adversarial robustness of the system. The experimental analysis part of the work reveals that the method achieves outstanding performance in DIBCO datasets.

The proposed model in this work is an entirely innovative version of CNN with a reduced number of layers.The new concepts introduced into the proposed model are the layers Difference of Convolution (DOC) and Enhancement layer (EN). The working of the model is simplified by incorporating better dimensionality reduction of features and by reduced inter-pixel dependency. Convolution layer filters in the proposed work are selected in such a way as to reduce feature dimensionality and inter-pixel dependency.

3 Methods

3.1 Problem statement

The historical document images are highly degraded with different types of bleed through degradation and artifacts. Consider a highly degraded image I. Solution to be designed here is to estimate a binary Image I_b which is free from all sorts of degradation. The degraded image I is given by $I = I_{u} + e$ (1) Here ‘e’ is a degradation factor that corrupts the original image and I_u is a clear image which is free of all types of degradation. The objective of the proposed work is to design a convolutional network based model that produces the binarized clear image. The output image should be of the form $I_{b} = B (I_{u})$ (2) where B is the binarization function that works on the clear image I₍u) to produce the binarized version I_b.

3.1.1 Problem formulation

The proposed model M designed based on CNN architecture is given by $I_{b} = M (I)$ (3) Here the model M is applied on degraded Image I to produce the binarized image I_b. The model M is designed based on 5 convolution layers (c₁, c₂ . . . . c₅), three concatenation layers (con₁, con₂, con₃), Difference of Concatenation layer ‘DOC’, Enhancement layer ‘EN’ and Thresholding layer ‘b’. The output O (c) of each convolution layer is by $O (c) = [X] = [X_{1}, X_{2} . . . . . . . . . X_{k}]$ (4) [X] is the set of output feature maps produced by the convolution layer. Based on the ‘k’ kernels selected for convolution operation, the convolution layer produces ‘k’ feature maps as the output. The convolution operation to estimate each cell value in the output feature map X_k is given by $X_{k} [i, j] = \sum_{c = 1}^{d} \sum_{k = 0}^{m - 1} \sum_{l = 0}^{n - 1} X (i + k, j + l, d) K (k, l, d)$ (5) ’K’ is the kernel for convolution operation with size ‘m × n × d’. The value d depend upon the depth or number of channels in the input feature maps.

The output O_con1 of concatenation layer con₁ is given by $O_{con 1} = C (O (c_{1}), O (c_{2}))$ (6) Here O (c₁) , O (c₂) are the outputs of the convolution layers. c₁, c₂ and C is the concatenation function that combines O (c₁) and O (c₂). The concatenation function C for two input images ‘X’ and ‘Y’ is given by $C (X, Y) = Z$ (7) If ‘X’ is of size ‘m × n × d1’ and ‘Y’ is of size ’mxnxd2’ then the output image ‘Z’ will be of size ‘m × n × (d1 + d2)’. The output O_con2 of concatenation layer con₂ is given by $O_{con 2} = C (O (c_{3}), O (c_{2}))$ (8) Here O (c₂) , O (c₃) is the output of the convolution layers c₂, c₃. The output O_con3 of last concatenation layer con₃ is given by $O_{con 3} = C (O (c_{1}), O (c_{2}), O (c_{3}), O (c_{4}))$ (9) Here O (c₁) , O (c₂) , O (c₃) , O (c₄) are the outputs of the convolution layers c₁, c₂, c₃, c₄. In convolution layer c₄, the convolution operation is carried out with the difference of output images in O_con1 and O_con2. Based on equation(10) a clear version of input image $I_{u}^{'}$ is produced in the Enhancement layer. $I_{u}^{'} = O (c_{5}) * I - O (c_{5}) - 1$ (10) Here O (c₅) is the output of convolution layer c₅. $I_{u}^{'}$ is the degradation-free version of I_u produced by the model. In the EN layer, O (c₅) is multiplied (element-wise) with input image ‘I’ and the resultant image produced is subtracted from O (c₅). O (c₅) is almost a clear version of the input image, but the image appears to be blurred, as there is only a small difference in intensity values between the text content and background. Therefore to enhance the contrast between textual content and background, O (c₅) is modified as per equation (10). The intensity values of the background pixels in O (c₅) are either negative or positive values nearer to zero or zero itself. Here an element-wise multiplication of input image ‘I’ and O (c₅) is carried out to enhance the contrast between text pixels and background pixels. This element-wise multiplication will produce a large increase in pixel values of text content but produce only low intensity values for background pixels. When the intensity values in O (c₅) corresponding to background pixels are zero, the elementwise multiplication operation results in zero. Therefore multiplication operation does not produce any impact on the output of the background pixels with zero value. Finally O (c₅) is subtracted from the multiplication result produced by ‘I’ and O (c₅). This subtraction operation further reduces the intensities of background pixels, thereby changing the intensity values of all background pixels either to zero or negative values. In order to ensure that all background pixels are either zero or negative value, a constant ‘1’ is further subtracted from the above result.

Figure 2 shows the clear version $I_{u}^{'}$ of the degraded input image I produced by the model. A threshold value T is selected in the thresholding layer based on the mean value of $I_{u}^{'}$ . $T = Mean (I_{u}^{'})$ (11)

Fig. 2

Visual results of evaluation showing the effectiveness of the proposed model in producing the clear version of degraded input image. (a)Original Image (b)Clear version( $I_{u}^{'}$ ) of the degraded input image I produced by proposed model.

The thresholding layer ‘b’ works by performing thresholding based on the ‘T’ value estimated. The final binarized image is formulated based on the given equation.

I_{b} = T (I_{u}^{'})

(12) The entire model M is learned through back propagation using stochastic gradient method based on the difference of I_b produced at each epoch and I_bgt. Here I_bgt is the ground truth of the input image.

3.2 Model architecture

The CNN architecture defined in this work comprises 5 Convolution layers, 3 Concatenation layers, a Difference of Concatenation layer (DOC), Enhancement layer and a Thresholding layer. The input image size and filter sizes were fixed based on experimental analysis using different filters of varying sizes. The convolution layers in the model work with filters of size either ‘1 × 1’ and ’n × 1’. Here such a filter size has been selected in each convolution layer to reduce the dimensionality of input images and also to reduce the effect of neighboring pixels on each output pixel value. The Concatenation layer plays a vital role in reducing the depth of the model by increasing the depth of the input image in the convolution layer. Difference of Concatenation layer produces output which gives more weightage to edge components that represents textual part and less weightage to non-edge components that represent non-textual part of the input image. The input image is resized into a size of ‘480 × 640 × 3’. In Convolution layer1(Conv1), the input image is convolved using 3 filters of size ’1 × 1’ initialized with Gaussian random values. Here the filter size ‘1 × 1’ is selected to convert a three channel image into a single channel image. The filter takes into account the average values of each pixel location corresponding to 3 channels thereby eliminating the dependency on neighboring pixel values. As three filters of size ‘1 × 1’ is used in the convolution layer1, three output feature maps are created. In the Convolution layer2 (Conv2), the output of convolution layer1 is convolved with 3 filters of size ‘3 × 1’. Here also such a filter size is chosen for dimensionality reduction by reducing the dependency on neighboring pixels. The images of size ‘480 × 640 × 3’ formed as outputs of convolution layer1 (Conv1) and convolution layer2 (Conv2) are concatenated at ‘concatenation layer 1’ to form an image of size ‘480 × 640 × 6’. The output of convolution layer2 (Conv2) is convolved with 3 filters of size ‘5 × 1’ at convolution layer 3 (Conv3) to form an output image of size ’480 × 640 × 3’. The output images of convolution layer 2 (Conv2) and convolution layer 3 (Conv3) are concatenated at concatenation layer 2 to form an output image of size ’480 × 640 × 6’. In the DOC layer, differences of the output images produced at concatenation layer1 and concatenation layer2 are estimated. At convolution layer 4 (Conv4), 3 filters of size ’7 × 1’ is applied to the resultant image from the DOC layer producing an output image of size ‘480 × 640 × 3’. The output images at convolution layers 1,2,3 and 4 are concatenated at concatenation layer 3 to form a resultant image of size ’480 × 640 × 12’. The output image from concatenation layer 3 is submitted to convolution layer 5(Conv5) where the input image is convolved with 3 filters of size ‘3 × 1’ thereby producing an output image of size ‘480 × 640 × 3’. The output from convolution layer 5 is passed to the Enhancement layer (EN) where the image is enhanced based on equation (10) and ReLU operation. The image produced at the EN layer is a clear lighter version of the degraded input image. In the thresholding layer a simple thresholding is carried out based on the mean value of the image generated at the normalization layer. Figure 3 depicts the model architecture. The detailed summary of the proposed network is depicted in Table 1. Table 1 demonstrates the number of filters, filter size (kernel size), padding size, output tensor size of each convolution layer, axis of concatenation and output filter size of each concatenation layer and output tensor size of all other layers in the model.

Fig. 3

Architecture.

Table 1

Summary of the Proposed model: ‘L’ represents layer number, ‘K’ represents kernel size, ‘F’ is the number of filters, ‘P’ is the padding and ‘A’ denotes axis of concatination

L	Description	K	F	P	A	Output Tensor Dimension
1	Input Layer	-	-	-	-	480 × 640 × 3
2	Convolution Layer1	1 × 1	3	same	-	480 × 640 × 3
3	Convolution Layer2	3 × 1	3	same	-	480 × 640 × 3
4	Concatenation Layer1	-	-	-	-1	480 × 640 × 6
5	Concatenation Layer2	-	-	-	-1	480 × 640 × 6
6	Convolution Layer3	5 × 1	3	same	-	480 × 640 × 3
7	Difference of Concatenation Layer	-	-	-	-	480 × 640 × 6
8	ConvolutionLayer4	7 × 1	3	same	-	480 × 640 × 3
9	Concatenation Layer3	-	-	-	-1	480 × 640 × 12
10	Convolution Layer5	3 × 1	3	same	-	480 × 640 × 3
11	Enhancement Layer	-	-	-	-	480 × 640 × 3
12	Thresholding Layer	-	-	-	-	480 × 640

4 Experiments

4.1 Dataset and implementation

The proposed model is trained and validated with 50 images from Document Image Binarization Contest (DIBCO) datasets, 15 images from Persian Heritage Image Binarization Dataset (PHIBD12) Dataset, 100 images from Balinese Palm leaf Manuscript of ICFHR 2016 contest and 7 images from Bickley Diary Dataset. The DIBCO datasets used in this work are H-DIBCO 2014, H-DIBCO 2016, H-DIBCO 2018, DIBCO-2017 and DIBCO 2019. A total of 192 images were used for training and validation of the proposed model. Out of the 192 images, 135 images were selected for training purpose and 57 images for validation. As the benchmark dataset available for binarization is very small, image augmentation is carried out to boost the performance of the model. To increase the size of the dataset the input images are cropped into patches of size 128 × 128 × 3. The patch sizes were selected based on experimental analysis. Python code is implemented to create patches from the image. The patch creation code is run through 10 iterations to generate different set of training and validation patches from each DIBCO dataset. Each patch is scaled to a size of 480 × 640 × 3 to fit into the model input. Approximately 108000 patches were selected for training purpose and 45000 entirely different patches were used for validation purpose. The details regarding images and patches selected for training and validation are depicted in Table 2. A newly created palm leaf dataset constituting of grantha characters were also used for testing purpose. Palm leaf images constituting of grantha script were collected from the Oriental Research Institute in University of Kerala (University located in the southern State in India).

Table 2
Dataset description

Dataset Images Patches

Training Validation Training Validation

DIBCO 50 20 40000 16000

Bickley Diary 5 2 4000 1600

PHIB12 Dataset 10 5 8000 4000

Balinese Palm-Leaf 70 30 56000 24000

Total 135 57 108000 45000

Dataset	Images	Patches
DIBCO	50	20	40000	16000
Bickley Diary	5	2	4000	1600
PHIB12 Dataset	10	5	8000	4000
Balinese Palm-Leaf	70	30	56000	24000
Total	135	57	108000	45000

The architecture and learning process were implemented using Keras Library. We used Adam optimizer and MSE (mean square error) as loss function. Here ReLU is used instead of BReLU to get optimized results. The learning rate is set as 0.0001. Momemtum value is chosen as 0.9 and decay parameter as 0.0001. All these values for various parameters were chosen based on experimental analysis. The network almost took 20 epochs to converge.

4.2 Visual results

The performance of the model is validated with the 45000 patch images kept aside for validation purpose. To compare the performance of the proposed model with other existing efficient methods, the model is also tested with whole size images from the selected standard datasets. The ground truth for patch images were generated by cropping corresponding patches from the ground truth of validation images. The proposed method works effectively in enhancement and binarization of highly degraded documents. It works effectively for various types of degradation like uneven illumination, contrast variation, ink bleed degradation, smear through and show through effect, blurring and deterioration. Figure 4 shows that the proposed method works effectively in handling smear through and show-through degration. The image shown in Fig. 4 is taken from H-DIBCO 2016 dataset.

Fig. 4

Visual results of evaluation showing the effectiveness of the proposed model in handling smear through and show through degradation. (a) Original Image (b) GroundTruth (c) Binarized Image based on proposed model.

Figure 5 shows the binarized output of an image taken from DIBCO 2017 dataset. The input image is effected with ink bleed and show through effect degradation. The visual results in Fig. 5 shows that the proposed model can effectively handle the ink bleed and show through degradation.

Fig. 5

Visual results of evaluation showing the effectiveness of proposed model in handling ink bleed and show through degradation (a) Original Images (b) GroundTruth (c) Binarized Images.

Figure 6 shows the binarized outputs generated by the proposed model on degraded images taken from H-DIBCO 2018 dataset. The images shown in the figure are effected with degradation due to uneven illumination, contrast variation and deterioration. Images shown in the figure 6 reveals that the model works efficiently in handling various types of degradation.

Fig. 6

Visual results of evaluation showing the effectiveness of model in handling degradations like uneven illumination, contrast variation and deterioration (a) Original Images (b) GroundTruth (c) Binarized Images.

In some historical images the text area gets lighter and weaker due to aging. Figure 7 shows the effectiveness of the proposed method in handling lighter and weaker text content.

Fig. 7

Visual results of evaluation showing the effectiveness of the proposed method in handling lighter and weaker text content (a) Original Images (b) GroundTruth (c) Binarized Images.

Figure 8 depicts the working of the proposed model when tested with palm leaf images constituting of grantha characters.

Fig. 8

Visual results of evaluation of proposed method in grantha dataset:(a) Original Images (b) Binarized Outputs.

The proposed model produced outstanding performance when compared with Otsu and Sauvola binarization methods. Figure 9 shows the binarization output image produced on applying Otsu and Sauvola binarization methods to the input image. Output images produced by Otsu has lots of unwanted artifacts. As Otsu being a global thresholding method, it cannot work effectively to determine a single threshold that can be applied to an entire document image. Figure 9(c) shows that on applying the Sauvola method to the input image with a local window of size ’3 × 3’ the entire image content is lost. As the threshold selection in Sauvola method is not robust to low contrast and moreover is not invariant to contrast inversion, the method is more likely to produce results as shown in Fig. 9(c) while working with input images of low contrast variation. But on increasing the window size and enhancing the contrast of input images better output images can be produced. Figure 9(d) shows the output image produced on working with the proposed model. From the visual results it can be affirmed that the proposed model works efficiently when compared to Otsu and Sauvola binarization method.

Fig. 9

Visual results comparing the performance of Otsu binarization, Sauvola binarization and proposed method:(a) Original Images (b) Binarized image resulted from Otsu (c) Binarized Images resulted from Sauvola (d) Binarized image resulted from proposed model.

In the proposed work, mean value thresholding is performed in the thresholding layer. The thresholding method in the layer can be modified to any of the standard thresholding method. Experiments were carried out to modify the thresholding layer with Otsu and Sauvola binarization methods. Figure 10(b) shows the resultant output images produced when the thresholding layer in proposed model is modified with Otsu binarization algorithm and Fig. 10(c) shows the resultant images produced when thresholding layer is modified with Sauvola binarization method. Figure 10(d) shows the resultant images produced by the proposed model. From the resultant images in Figs. 9 10, it can be affirmed that Otsu and Sauvola combined with the proposed model works better in degraded images compared to the methods being applied in bare. These outputs reveal the effectiveness of layers upto enhancement layer in producing a clear version of degraded images that further enhances the efficiency of binarization algorithms.

Fig. 10

Visual results comparing the performance of proposed method with modification in proposed method based on Otsu binarization and Sauvola binarization:(a) Original Images (b) Binarized image resulted from Proposed model + Otsu (c) Binarized Images resulted from Proposed Model + Sauvola(d) Binarized image resulted from proposed model.

Figure 11 shows the comparative visual results of the proposed method with other recent Deep Learning based methods like CNN [29], AdBiNet [39] and UNet [34] on Bickley Diary dataset, Balinese palm leaf Dataset and PHIBD12 Dataset. The fifth row in the image that corresponds to proposed work depicts that the proposed model can produce more clear binarization results compared to other recent Deep Learning methods.

Fig. 11

Visual results comparing the performance of proposed method with other deep learning methods on Bickley dataset, Balinese palm leaf dataset and PHIBD12 dataset:(a) Original Images (b) Binarized images resulted from CNN (c) Binarized Images resulted from AdBiNet (d) Binarized Images resulted from UNet (e) Binarized image resulted from proposed model.

4.3 Quantitative results

To evaluate the effectiveness of the proposed model quantitatively, the model is experimented with images patches from the validation dataset. To compare the performance with other standard algorithms the model is also tested with the full size images from the selected benchmark datasets. The average performance measures are estimated on DIBCO datasets, Bickley Diary Dataset, PHIBD12 dataset and Balinese Palm leaf manuscript. The resultant images produced were evaluated based on measures like F-measure, Pseudo F-measure (Fps), Distance Reciprocal Distortion metric (DRD) and the Peak Signal-to-Noise Ratio (PSNR) metrics [38]. These metrics as being the measures for analysis of binarization results in DIBCO binarization context were chosen for quantitative analysis in this work. Moreover, in most of the work related to document image binarization, these measures were used as standard measures for evaluating the performance of proposed algorithms. The values of F-measure, Pseudo F-measure (Fps) and the Peak Signal-to-Noise Ratio (PSNR) should be very high and the values of Distance Reciprocal Distortion metric (DRD) should be very low for an adequate binarized result. The values of F-measure, pseudo F-measure (Fps) should approach to 100 for an apt binarized ouput. The average values based on the evaluated measures are tabulated and were quantitatively compared with some state-of-the-art methods. Table 3 shows the evaluation measures based testing on H-DIBCO 2014 Dataset. Table 4 shows the results based on H-DIBCO 2016 Dataset, Table 5 based on DIBCO 2017 Dataset, Table 6 based on H-DIBCO 2018 Dataset, Table 7 based on Bickley Diary Dataset, Table 8 based on PHIBD12 dataset and Table 9 based on Balinese palm leaf manuscript. In each table the performance of the proposed method is compared with binarization methods like Otsu, Sauvola, the best method selected from yearly DIBCO contest, CNN, UNet and AdBiNet. From the tabulated results it can be concluded that the proposed model produced outstanding performance when compared with other methods. But in case of H-DIBCO 2014 where the number of degraded images are less compared to the other DIBCO dataset, the average performance of the model is slightly low compared to the best method in the DIBCO-2014 contest and UNet. In other datasets where the degraded images are high in number the proposed model overrides all other methods.

Table 3
Experimental analysis on H-DIBCO 2014 dataset

Algorithm FM F _PS PSNR DRD

Otsu 91.78 95.74 17.63 2.647

Sauvola 86.83 91.80 18.72 4.896

BestMethod 96.88 97.65 22.66 0.902

DIBCO2014 [36]

CNN 85.0 89.8 16.9 7.6

UNet 94.8 97.0 20.8 2.7

AdBiNet 87.70 90.90 18.04 4.04

Proposed Work 96.52 97.44 23.23 0.9

Algorithm	FM	F _PS	PSNR	DRD
Otsu	91.78	95.74	17.63	2.647
Sauvola	86.83	91.80	18.72	4.896
BestMethod	96.88	97.65	22.66	0.902
DIBCO2014 [36]
CNN	85.0	89.8	16.9	7.6
UNet	94.8	97.0	20.8	2.7
AdBiNet	87.70	90.90	18.04	4.04
Proposed Work	96.52	97.44	23.23	0.9

Table 4

Experimental analysis on H-DIBCO 2016 dataset

Algorithm	FM	F _PS	PSNR	DRD
Otsu	86.6	89.9	17.8	5.6
Sauvola	84.6	88.4	17.1	6.3
BestMethod	87.61	91.28	18.11	5.21
DIBCO2016 [35]
CNN	78.85	81.23	15.21	4.85
UNet	87.70	88.15	19.51	4.21
AdBiNet	87.08	88.03	18.75	4.27
Proposed Work	91.26	94.35	19.6	2.7

Table 5

Experimental analysis on DIBCO 2017 dataset

Algorithm	FM	F _PS	PSNR	DRD
Otsu	82.52	86.87	16.42	7.49
Sauvola	86.61	88.67	17.80	5.56
BestMethod	91.04	92.86	18.28	3.40
DIBCO2017(UNet) [37]
CNN	87.32	89.02	17.75	4.57
AdBiNet	89.46	91.37	18.66	4.12
Proposed Work	94.8	97	20.8	1.7

Table 6

Experimental analysis on H-DIBCO2018 dataset

Algorithm	FM	F _PS	PSNR	DRD
Otsu	51.45	53.05	9.74	59.07
Sauvola	67.81	74.08	15.78	17.69
BestMethod	88.34	90.24	19.11	4.92
DIBCO2018 [38]
CNN	87.08	88.03	18.75	4.27
UNet	89.36	92.78	19.43	3.90
AdBiNet	88.21	88.82	18.27	5.44
Proposed Work	91.2	95.76	19.7	2.7

Table 7

Experimental analysis on bickley diary dataset

Algorithm	FM	F _PS	PSNR	DRD
Otsu	50.42	52.8	7.8	62.2
Sauvola	64.60	66.2	11.62	24.9
CNN	66.44	67.6	10.76	17.6
UNet	72.8	73.2	13.6	24.3
AdBiNet	67.71	68.4	9.79	20.6
Proposed Work	78.54	79.22	13.15	9.75

Table 8

Experimental analysis on PHIBD12

Algorithm	FM	F _PS	PSNR	DRD
Otsu	82	85	16.46	3.2
Sauvola	85.04	89.8	16.9	7.6
CNN	85.54	86.80	17.43	5.8
UNet	87.7	88.3	19.6	4.2
AdBiNet	86.93	90.85	17.50	4.61
Proposed Work	90.03	93.63	18.49	3.05

Table 9

Experimental analysis on balinese palm leaf manuscript

Algorithm	FM	F _PS	PSNR	DRD
Otsu	32.63	34.75	7.8	62.2
Sauvola	16.25	16.15	7.09	47.6
CNN	85.2	87.86	17.89	7.21
UNet	90	91.82	19.71	3.8
AdBiNet	91.97	90.47	19.29	4.98
Proposed Work	93.91	92.8	19.72	4.08

The combined unique features of the model like convolution kernel size, concatenation layers and Difference of Concatenation layer leads to the outstanding performance of the model compared to existing models. The use of ’1 × 1’ and ‘n × 1’ kernel size for convolution reduces the dependency of each pixel on neighboring pixels values, thereby reducing the overall smoothening effect of the output from convolution layers. The concatenation layer introduced in the model reduces the depth of the proposed model but increases the depth of input images to the convolution layer. The negative effect induced in the model due to the reduction in model depth is substantiated by increasing the depth of images for convolution. The introduction of the concatenation layer thereby reduces the complexity of the entire model. DOC layer plays a key role in enhancing the intensity values of edge components that represent textual part of the image.

5 Conclusion

The proposed model is a modified version of Convolution Neural Network constituting of a set of Convolution layers, Concatenation layers, Difference of Concatenation layer, Enhancement layer and Thresholding layer. The model produces outstanding performance compared to the existing methods like Otsu, Sauvola, Basic CNN, UNet and AdBiNet. The experimental analysis part with H-DIBC0 2014, H-DIBCO 2016, H-DIBCO2018, DIBCO 2017, DIBCO 2019, Bickley Diary, PHIBD12 and Balinese Palm Leaf dataset reveals the efficiency of the model visually and based on evaluation measures like F-measure, Distance Reciprocal Distortion metric and the Peak Signal-to-Noise ratio. In the case of H-DIBCO 2014 where the number of degraded images are less compared to the other DIBCO dataset, the average performance of the model is slightly low compared to the best method in the DIBCO-2014 contest and UNet. But in other datasets where the degraded images are high in number the proposed model overrides all other methods. The model works efficiently in handling degradations in most of the historical images. Moreover, the model works effectively in retrieving the weaker text content. This model with limited depth shows better performance compared to existing CNN-based models having more layers. The performance of the model can further be enhanced by increasing the number of layers. The proposed model can be modified to work in other research domains like de-raining, de-hazing, and denoising.

References

, Peng

, Wang

, Xu

and Feng

, AOD-Net: All-in-One Dehazing Network, 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, 4780–4788.

Doyle

, Operation useful for similarity-invariant patternrecognition, J Assoc Comput Mach 9 (1962), 259–267.

Ridler

T.W.

and Calvard

E.S.

, Picture thresholding using aniterative selection method, IEEE Trans Syst Man Cybern SMC:8 (1978), 630–632.

John

and Rosenfeld

, Histogram modification for thresholdselection,6. 10./TSMC, NASA STI/Recon Technical Report N. 78 (1979), 15466.. 10.1109/TSMC.1979.4310072.

Pun

, A new method for gray-level picture thresholding using theentropy of the histogram, Signal Process 2 (1980), 223–237.

Johannsen

and Bille

, A Threshold Selection Method Using Information Measures, Proceedings of the 6th International Conference on Pattern Recognition, Munich, Germany, 19–22 October 1982, 140–143.

Kapur

J.N.

, Sahoo

P.K.

and Wong

A.K.C.

, A new method for gray-levelpicture thresholding using the entropy of the histogram,, ComputVision Graphics Image Process 29 (1985), 273–285.

C.H.

and Tam

P.K.S.

, Aniterativealgorithmforminimum cross-entropythresholding,, Pattern Recogn Lett 19 (1998), 771–776.

Mello

C.A.B.

and Lins

R.D.

, Generation of Images of Historical Documents by Composition, Proceedings of the 2002 ACM symposium on Document Engineering, 2002.

10.

Reddi

S.S.

, Rudin

S.F.

and Keshavan

H.R.

, An optimal multiplethreshold scheme for image segmentation,, IEEE Trans Systems ManCybemet SMC-14 (1984), 661–665.

11.

Kittler

and Illingworth

, Minimum error thresholding,, Pattern Recognit 19 (1986), 41–47.

12.

Howe

N.R.

, Document binarization with automatic parameter tuning, International Journal on Document Analysis and Recognition, 2012, 247–258.

13.

Bernsen, Dynamic Thresholding of Gray Level Image, ICPR‘86: Proceedings of International Conference on Pattern Recognition, Berlin, 1986, 1251–1255.

14.

Niblack

, An introduction to digital imageprocessing, Prentice-Hall Englewood Cliffs, 1986.

15.

Jaakko

and Matti

, Adaptive Document Image Binarization, Pattern Recognition, 2000, 225–236.

16.

Gatos

, Pratikakis

and Perantonis.

, Adaptive degraded document image binarization, Pattern Recognition, 2006, 317–327.

17.

Bolan

, Shijian

and Lim

T.C.

, Combination of Document Image Binarization Techniques, In Proceedings of the 11th International Conference in Document Analysis and Recognition (ICDAR), 2011, 22–26.

18.

Omar

, Walid

and Dominique

, Degraded Historical Documents Images Binarization Using a Combination of Enhanced Techniques, arXiv:1901.09425v1, 2019.

19.

Christian

and Jean-Michel

, Extraction and Recognition of Artificial Text in Multimedia Documents, Formal Pattern Analysis Applications, 2004, 309–326.

20.

Mengling

and Yap-Peng

, Contrast adaptive binarization of low quality document images, IEICE Electronic Express, 2004, 501–506.

21.

Nikolaos

and Papamarkos

, Document Image Binarization using Local Features and Gaussian Mixture Modelling, Image and Vision Computing, 2015, 33–51.

22.

James

, Jennings

and Salkauskas

A.G.

, Thresholding using an illumination model, 1993, 270–273.

23.

Ostu

, A threshold selection method from gray-level histogram,, IEEE Trans. Systems Man Cybemet SMC-8 (1978), 62–66.

24.

Bolan

, Shijian

and Lim

T.C.

, Binarization of historical document images using the local maximum and minimum, International Workshop on Document Analysis Systems, 2010, 159–166.

25.

Khurram

, Imran

, Claudie

and Nicole

, Comparison of Niblack inspired Binarization Methods for Ancient Documents, 2009, 1–10.

26.

Hatem

, Eddie

and Belaid

, Neural based binarization techniques, 2015, 317–321.

27.

Zeshan

A.M.

, Joan

P.-P.

, Faisal

, Thomas

, Andreas

and Marcus

, Document Image Binarization using LSTM: A Sequence Learning Approach, 2015, 79–84.

28.

Chris

and Tony

, Document Image Binarization with Fully Convolutional Neural Networks, 2017, 99–104.

29.

Calvo-Zaragoza , JorgeGallego and Antonio-Javier , A selectional auto-encoder approach for document image binarization, Pattern Recognition, 2017, 86.

30.

Quang

, Yang

K.S.

and Hyung

L.G.

, Binarization of Degraded Document Images based on Hierarchical Deep Supervised Network, Pattern Recognition, 2017, 74.

31.

Calvo-Zaragoza

, Vigliensoni

and Fujinaga

, Pixel wise binarization of musical documents with convolutional neural networks, 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA), Nagoya, 2017, 362–365.

32.

Peng

J.X.

, Cao

and Natarajan

, Using Convolutional Encoder-Decoder for Document Image Binarization, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, 2017, 708–713.

33.

and Schomaker

, DeepOtsu: Document enhancement and binarization using iterative deep learning, Pattern Recognition, 2019, 379–390.

34.

Pavel

, Ilin

D.A.

and Dmitry

, U-Net-bin: hacking the document image binarization contest, Computer Optics, 2019, 825–832.

35.

Pratikakis

K.Z.

, Barlas

and Gatos

, ICFHR2016 Handwritten Document Image Binarization Contest (H-DIBCO 2016), 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), Shenzhen, 2016, 619–623.

36.

Konstantinos

, Basilis

and Ioannis

, ICFHR2014 Competition on Handwritten Document Image Binarization (H-DIBCO 2014), 2014, 809–813.

37.

Pratikakis

K.Z.

, Barlas

and Gatos

, ICDAR2017 Competition on Document Image Binarization (DIBCO 2017), 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, 2017, 1395–1403.

38.

Pratikakis

K.Z.

, Kaddas

and Gatos

, ICFHR 2018 Competition on Handwritten Document Image Binarization (H-DIBCO 2018), 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), Niagara Falls, NY, 2018, 489–493.

39.

Bhunia

A.K.

, Sain

and Roy

P.P.

, Improving Document Binarization Via Adversarial Noise-Texture Augmentation, 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, 2721–2725.

Document image binarization using difference of concatenated convolutions

Abstract

Keywords

1 Introduction

3 Methods

3.1 Problem statement

4.1 Dataset and implementation

Table 2 Dataset description Dataset Images Patches Training Validation Training Validation DIBCO 50 20 40000 16000 Bickley Diary 5 2 4000 1600 PHIB12 Dataset 10 5 8000 4000 Balinese Palm-Leaf 70 30 56000 24000 Total 135 57 108000 45000

References

Table 2
Dataset description

Dataset Images Patches

Training Validation Training Validation

DIBCO 50 20 40000 16000

Bickley Diary 5 2 4000 1600

PHIB12 Dataset 10 5 8000 4000

Balinese Palm-Leaf 70 30 56000 24000

Total 135 57 108000 45000