Color multi-focus image fusion algorithm based on fuzzy theory and dual-tree complex wavelet transform

Abstract

This paper puts forward a new color multi-focus image fusion algorithm based on fuzzy theory and dual-tree complex wavelet transform for the purpose of removing uncertainty when choosing sub-band coefficients in the smooth regions. Luminance component is the weighted average of the three color channels in the IHS color space and it is not sensitive to noise. According to the characteristics, luminance component was chosen as the measurement to calculate the focus degree. After separating the luminance component and spectrum component, Fisher classification and fuzzy theory were chosen as the fusion rules to conduct the choice of the coefficients after the dual-tree complex wavelet transform. So fusion color image could keep the natural color information as much as possible. This method could solve the problem of color distortion in the traditional algorithms. According to the simulation results, the proposed algorithm obtained better visual effects and objective quantitative indicators.

Keywords

Color multi-focus image image fusion dual-tree complex wavelet transform fuzzy theory

Introduction

In recent years, multi-focus image fusion research has been extremely important in the data fusion field. Most of the fusion algorithms are based on the gray multi-focus image fusion. However, the human visual system can only distinguish about 20 kinds of different gray levels at the moment of observing the gray images while can distinguish more than 1000 kinds of different color levels.^1,2 The human visual system's ability to distinguish color is far better than the resolution of grayscale. So, more information can be obtained from the scene in color images more quickly and more accurately. In the current technology, color images can be captured, transmitted, and processed quickly and as a result have received more attention. Color multi-focus images are created by the limitation of focus region when the sensors capture the color images, which influence the overall effect of the images.³

During the past few years, N. G. Kingsbury developed a dual-tree complex wavelet transform (DT-CWT), which allows perfect reconstruction. In the process of DT-CWT, the parameters will not have comparatively large variation even in the condition where the input image has slight offsets. Meanwhile, DT-CWT has a good performance of directional selectivity, which can separate factors from six different directions, making it possible to gain more detailed information from the original image and get the edge characters more clear.^4,5

Generally, the performance of color multi-focus image fusion algorithm is largely decided by the color space model and blending strategy. Aiming at these two problems, the DT-CWT was introduced in the process of image fusion. Combining the characters of DT-CWT and multi-focus image fusion, a color multi-focus image fusion algorithm based on fuzzy theory and DT-CWT is put forward in this work. As the channels of R, G, B in the color image are related, chromatic distortion may occur when fusing the color images. The algorithm introduced in this work changes the fusing process of the original color image to the IHS space, which leads to a good effect compared with the results of the simulation experience.

Theory of DT-CWT

In 1998, Kingsbury put forward the theory of DT-CWT, which kept the advantages of complex wavelet transform (CWT) while solving the problems in the process of realizing the perfect reconstruction filters.

DT-CWT has two parallel beamlets named a and b, which adopts the real wavelet transform. Beamlet a gives the real part of CWT while beamlet b gives the imaginary part. The low-pass filter and high-pass filter of beamlet a is defined as $h_{o} (n)$ and $h_{1} (n)$ and those of beamlet b are defined as $g_{0} (n)$ and $g_{1} (n)$ . The exploded view of DT-CWT is shown in Figure 1.

Figure 1.

The sample weighted graph constructed from Table 1.

The two-dimensional DT-CWT not only has the advantages of a one-dimensional wavelet, but also has the direction selectivity, making it more suitable in handling the strangeness of direction. One-dimensional wavelet transform is adopted in rows and columns by corresponding filters in the process of two-dimensional DT-CWT, and three sub-bands are generated in the first and second quadrants corresponding to six directions of ±15 °, ±45°, ±75°. Due to the good performance of translation invariance and direction selectivity of DT-CWT, detailed edge characters can be well kept in the process of decomposing original image in this work.

Fuzzy theory and “if–then” inference strategy

“If–Then” inference strategy contains the condition statement fuzzy logic, the formation of which can be shown as follows.

In the statement “If x is A then y is B”, A and B are fuzzy language variables and their relationship is fuzzy relation, which is marked as $A \to B$ . “x is A” is the precondition while “y is B' is the conclusion. The process of adapting “if–then” can be illustrated in three aspects as follows.

All possible preconditions are taken into consideration.

Confirm the degree of membership of each precondition of “if–then”.

According to the degree of membership of each precondition and possibility of all conditions, deduce the right solutions, that is to say, the degree of membership of the results.

The fuzzy reasoning is the process of deducing the new conclusions from all the possible preconditions. Uncertainties can be avoided effectively by applying this method in the image fusion.

Experimental result and analysis

Feature extraction

According to the mechanism of optical imaging, bandwidths of system functions of focus of image have a wider range than the unfocused one.⁶ As a result, pixel value of clear image is obviously larger than the blurry one.

To get more information on the sub-band coefficients in the process of fusion, the mean square value of the difference between central pixel and neighboring window's pixel can be defined as

F (x, y) = \sqrt{\frac{1}{W} \sum_{l = - L}^{L} \sum_{k = - K}^{K} [D_{j}^{ɛ} (x, y) - D_{j}^{ɛ} (x + l, y + k)] 2}

(1)

In the formula, $D_{j}^{ɛ} (x, y)$ is the high-frequency sub-band of DT-CWT. $ɛ = {H, D, V}$ , $(2 L + 1, 2 K + 1)$ is the size of window and W is its pixel count.

The mean value of dependency of neighboring window can be defined as

NF (x, y) = \frac{1}{W} \sum_{l = - L}^{L} \sum_{k = - K}^{K} [F (x + l, y + k)]

(2)

By taking the difference in characteristics between the six high-frequency sub-bands generated in the process of decomposing of DT-CWT as feature vectors, we have

F V^{i} (x, y) = {NF}_{A}^{i} (x, y) - {NF}_{B}^{i} (x, y)

(3)

In the formula, $i = 1, 2, \dots 6$ represents the six high-frequency sub-bands of DT-CWT.

The feature vector of each pixel is shown as follows

FV (x, y) = [F V^{1} (x, y), F V^{2} (x, y), \dots, F V^{6} (x, y)]

(4)

Fisher discriminator

By having the feature vectors, fisher discriminator is applied to the selection of sub-band coefficients of the focused and unfocused regions. Compared to the pulse coupled neural network (PCNN) and support vector machine (SVM), fisher discriminator is more convenient to carry out and has a quicker speed and circumstance of overload training can be avoided owing to the requirement of fewer training samples.⁷

The optimal mapping direction of fisher standard is decided by the ratio of maximum of between-class scatter and within-class scatter. $S_{w}$ and $S_{b}$ represent matrixes of between-class scatter and within-class scatter, respectively

S_{w} = \sum_{k = 1}^{K} \sum_{x \in classK} (x - u_{k}) (x - u_{k}) T

(5)

S_{b} = \sum_{k = 1}^{K} (u_{k} - u_{0}) (u_{k} - u_{0}) T

(6)

u_{0} = \sum_{k = 1}^{K} u_{k}

(7)

where

u_{k}

is the mean vector of category k,

u_{0}

is the mean vector of overall situation, and K is the number of categories. The optimal mapping direction is the eigenvector of maximum eigenvalue of

{S_{w}}^{- 1} S_{b}

. In these two kinds of discriminating problems, it is not necessary to calculate the eigenvector of

{S_{w}}^{- 1} S_{b}

and the optimal mapping direction can be defined as

W_{0} = {S_{w}}^{- 1} (u_{1} - u_{2})

(8)

Y_{1}

and

Y_{2}

are the mapping of the two kinds of classification in the optimal direction

W_{0}

. In the condition that

E [Y_{1}] > E [Y_{2}]

, the decision is

We kept the output of discriminator as the decision map, which chose sub-band coefficient in different directions and decomposition level of two images. The fusing rule can be defined under the rules as follows

D_{F, j}^{i} = (D_{A, j}^{i} \times DM) + (D_{B, j}^{i} \times ~ DM)

(10)

In the formula, $j = 1, 2, \dots N - 1$ . N represents the number of decomposition levels, $i = 1, 2, \dots 6$ indicates the six sub-bands in each decomposition level and ∼ is not, which is a logical operator. In the coarse scale decomposition level, DM's estimation is adopted by downsampling.

Fuzzy fusing rule

Most of the characteristic space's wavelet coefficient is influenced by smooth region of the original image, so that there is no distinct difference to judge whether it is the focus region or the unfocused one. In order to solve this uncertain condition, three different strategies are applied in the fusing rules, of which the first one adopts the following formula to get DM and it can be obtained by Fisher discriminator.

Y_{1} = (D_{A, j}^{i} \times DM) + (D_{B, j}^{i} \times ~ DM)

(11)

The second strategy adopts the following formula to get DM

In the formula, $N F_{A, B}$ can be gained by equation (2), that is

Y_{2} = (D_{A, j}^{i} \times dm) + (D_{B, j}^{i} \times ~ dm)

(13)

The third fusing rule is

Y_{3} = \frac{(D_{A, j}^{i} + D_{B, j}^{i})}{2}

(14)

The three fusing rules can be induced into a well-performed rule to realize the maximum of the information of the resulting images, and fuzzy theory discriminate can be applied to achieve this goal. “If–Then” inference strategy is adopted in this work and the discriminator can be built as follows if each fusing rule is marked as an output result.

If $NF$ is large AND $En$ is large then $Y_{2}$ is the output.

If $NF$ is large AND $En$ is small then $Y_{1}$ is the output.

If $NF$ is medium AND $En$ is large then $Y_{2}$ is the output.

If $NF$ is medium AND $En$ is small then $Y_{1}$ is the output.

If $NF$ is small then $Y_{3}$ is the output.

In the formula, $En = | E n_{A} - E n_{B} |$ and $NF = | N F_{A} - N F_{B} |$ , $E n_{A, B}$ is the energy of sub-band coefficients of local window.

Each rule can be represented by a relation function. Figure 2 shows the triangle relationships of $NF$ and $En$ , in which $T_{1}$ and $T_{2}$ are the norms.

τ_{1} = μ_{Large}^{1} (NF) \times μ_{Large}^{2} (En) τ_{2} = μ_{Large}^{1} (NF) \times μ_{Small}^{2} (En) τ_{3} = μ_{Medium}^{1} (NF) \times μ_{Large}^{2} (En) τ_{4} = μ_{Medium}^{1} (NF) \times μ_{Small}^{2} (En) τ_{5} = μ_{Small}^{1} (NF)

(15)

Figure 2.

Relation function of NF and En: (a) triangle relationship of NF; (b) triangle relationship of En.

AND represents the smallest relation. In order to get the output result of the discriminator, all the direction of the rules are combined together. According to the maximum membership principle,⁸ conclusion can be drawn as follows

If τ_{i} = {max τ_{j}}_{j = 1 \dots 5} AND i \to k then class is Y_{k}

(16)

where k is the number of the categories, j is the number of the rules, and

i \to j

indicates that rules i points to result

Y_{k}

In order to develop the fuzzy functional relation between $NF$ and $En$ , $T_{1}$ and $T_{2}$ need to be confirmed. In this work, $T_{1} = 0.05$ and $T_{2} = 0.1$ .

The fuzzy fusing rule is applied in the high-frequency sub-bands of every decomposition level. In cases of low-frequency sub-band, another DM of fisher discriminator is applied, which is shown as follows

S_{F, j} = (S_{A, j} \times {DM}^{\land}) + (S_{B, j} \times ~ {DM}^{\land})

(17)

In the formula, ${DM}^{\land}$ is obtained through downsampling of this decomposition level. After combining all the sub-band coefficients, inverse transformation of DT-CWT is carried out to get the final fusion image.

In cases of the fusion of two color multi-focus images, the framework of fusion method is shown in Figure 3. The method is made up by the following steps.

(1) Switch the two color multi-focus images A and B from RGB space to IHS space, and separate the luminance components $I_{A}$ and $I_{B}$ from these two images;

(2) Do DT-CWT decomposition to luminance components $I_{A}$ and $I_{B}$ and get their low- and high-frequency coefficients, respectively;

(3) Fuse the images under the fuzzy fusion rules and get the low-frequency coefficient $S_{F} (x, y)$ and high-frequency coefficient $D_{j, F}^{ɛ} (x, y)$ , $ɛ = {H, D, V}$ ;

(4) Perform inverse transformation of DT-CWT for fused low- and high-frequency coefficients respectively and get the result $I_{F}$ ;

(5) Take a weighted average of $S_{A}$ , $S_{B}$ and $H_{A}$ , $H_{B}$ , obtain $S_{F}$ and $H_{F}$ respectively, and conduct inverse transformation of IHS with $I_{F}$ ;

(6) Switch from IHS space to RGB space, and get the fused image.

Figure 3.

Color multi-focus image fusion framework based on IHS and DT-CWT.

Experimental results and analysis

In order to verify the effectiveness and correctness of the method introduced in this work, a 600 × 800 color multi-focus image of the same scene are simulated through this method and compared with the other three. The first one is based on the fusion algorithm of IHS space and Laplacian pyramid transform (LPT_IHS). The second one is based on the fusion algorithm of IHS space and DT-CWT (DT-CWT_IHS). The third one is based on the method of Wang et al.,¹⁰ which fuses R, G, B components from the original image after DT-CWT. In this method, the pixel of local area is set as 5 × 5 and self-defined threshold value of low frequent components are set as 0.1 and 0.01, while the high-frequency components are set as 0 and 0.1. These three fusion algorithms based on multi-resolution analyzes all applied “sym4” wavelet filters. The images are all decomposed into three levels. In order to make it easy to compare the fusion algorithms of the simulation, fusion rules adapt the low-frequency coefficient of the weighted average and bigger region energy for high-frequency coefficient. Figure 4(a) and (b) shows the color multi-focus original images and (c) to (f) are the results of different methods.

Figure 4.

Results of different methods: (a) original image focus on left; (b) original image focus on right; (c) fused image based on LPT; (d) fused image based on DT-CWT_IHS; (e) fused image from paper¹⁰; (f) fused image based on the proposed algorithm.

From Figure 4, compared with the original image, all fusion methods can fuse the focusing characteristics of different color multi-focus images effectively. The focus differences of the original images are all eliminated and the detailed information is kept. When compared in detail, fused image from paper¹⁰ (Figure 4(e)) has an obvious color distortion from the original image. The reason lies in that this method fuses the R, G, B components of the original image respectively, which indicates the information of brightness, chromaticity, and so on. This process can destroy the proportional relation between brightness and chromaticity and reduce the gray information of the result image. On the contrary, the fusion results of LPT_IHS and DT-CWT_IHS have been improved greatly, but there are still some fuzzy regions on the English letters and the book edges in the images. The method introduced in this work, whose result is shown in Figure 4(f), has well solved the problems raised by other methods due to the application of fuzzy theory.

In order to objectively evaluate the performance of different algorithms, the indexes adopted are correlation coefficient (CC), root mean square error (RMSE), and the color image quality measurement (CIQM) by Yang and Chiam⁹ in the evaluation system.

Table 1 gives the simulation indexes of the four fusion methods. Based on the three indexes introduced above, color multi-focus image fusion algorithm introduced in this work has a better overall fusion performance than the other three methods. The CC of LPT_IHS is the smallest and is the largest one, indicating its fusion performance is the worst. The performance of DT-CWT_IHS and the method of Wang et al.¹⁰ are both better than LPT_IHSs, indicating that DT-CWT can provide more detailed edge information of images, which is advantageous in keeping the useful information of the original images.

Table 1.

Fusion performance of different algorithms.

Fusion algorithm	Channel	LPT_IHS	Wang et al.¹⁰	DT-CWT_IHS	Proposed in this work
$CC$	R	0.8233	0.8772	0.8828	0.9135
	G	0.8017	0.8502	0.8618	0.8988
	B	0.7957	0.8426	0.8482	0.8806
$RMSE$	R	2.5285	1.9864	1.9208	1.7215
	G	2.5734	2.0252	1.9514	1.7322
	B	2.3552	1.9101	1.9346	1.8161
$CIQM$		1.2939	1.2017	1.3384	1.3686

CC: correlation coefficient; RMSE: root mean square error; CIQM: color image quality measurement.

The difference between DT-CWT_IHS and the method introduced in this work results from adopting different fusion rules, which shows that the rules raised in this work has the ability to improve the quality of fusion image. In addition, compared with DT-CWT_IHS, RMSE of this method reduced by 13.3%, indicating that this method can extract the sub-band coefficients more effectively.

The method of Wang et al.¹⁰ has an obvious lower CIQM than the other three, standing for the highest color distortion, which correspond to the subjective assessment. The method raised in this work has a higher value of CIQM and this allows it to get more information from the images and achieve better fusion results.

Conclusions

In this work, a color multi-focus image fusion algorithm based on fuzzy theory and DT-CWT is put forward, which perfectly solves the problems of uncertainty generated in the process of selecting the sub-band coefficients in the smooth region of the image. We separated luminance component and spectrum component of the color original images in the IHS space, and the fuzzy theory is introduced. The mapping relations between different category characteristics and pixel gray values of images are measured by fuzzy membership degree and the scheme of selection of coefficient are put forward based on the fuzzy theory. The simulation results shows that this method has a better performance in keeping the detailed information of the original images than LPT_IHS, DT-CWT_IHS, and the method of Wang et al.,¹⁰ which successfully solves the color distortion problems produced by fusing the R, G, B channel respectively for the color multi-focus images.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This paper was supported by the Central University basic scientific research funds of Harbin Engineering University (Grant No. HEUCF150816).

Author contributions

SY performed the experiments; JL analyzed the data; SY wrote the paper.

References

Kim

Sohn

. Visual fatigue prediction for stereoscopic image. IEEE Trans Circ Syst Video Technol 2011; 21: 231–236.

Petrovic

Cootes

. Objectively adaptive image fusion. Inform Fusion 2007; 8: 168–176.

Toet

Lucassen

. A universal color image quality metric. Proc SPIE 2003; 5108: 12–23.

Chaux

Pesquet

J-C

Duval

. Noise covariance properties in dual-tree wavelet decompositions. IEEE Trans Inform Theory 2007; 53: 4680–4700.

Zhang

. Absolute value activity and regional similarity based on image fusion. Comput Eng Appl 2006; 18: 8–10.

Shao

Jiang

Wang

. Stereoscopic video coding with asymmetric luminance and chrominance qualities. IEEE Trans Consum Electron 2010; 56: 2460–2468.

Kaaniche

Benazza-Benyahia

Pesquet-Popescr

. Vector lifting schemes for stereo image coding. IEEE Trans Image Process 2009; 18: 2463–2475.

Kumar

Gupta

Khanna

. Text extraction and document image segmentation using matched wavelets and MRF model. IEEE Trans Image Process 2007; 16: 2117–2128.

Yang YY and Chiam TC. Rule discovery based on rough set theory. In: Proceedings of 3rd international conference on information fusion, Paris, France, 2002, pp. 313–315.

10.

Wang

. Color multi-focus image fusion algorithm based on dual-tree complex wavelet transform. Comput Eng Appl 2007; 43: 12–14. 40.