Multiple exposure fusion based on sharpness-controllable fuzzy feedback

Abstract

Multiple exposure fusion (MEF) is attracting considerable attention in research on high dynamic range (HDR) imaging: Eliminating the need to generate an intermediate HDR image, MEF directly expands an image’s dynamic range and thus provides greater detail enhancement than traditional HDR techniques. However, in the fusion stage, the optimal weights of each pixel in the images input to the final synthesized image are challenging to determine and usually required manual tuning of parameters. In addition, many MEF algorithms have been proposed, but most have lacked a self-regulation mechanism. To tackle the above disadvantages, we apply fuzzy theory and present a novel MEF framework with a fuzzy feedback structure. In this work, over- and under-exposed images are generated from a single input image using local histogram stretching. This avoids the creation of ghost artifacts when multiple exposed images are fused in the dynamic scene containing object motion. In the fusion stage, fuzzy logic is used to determine pixel weights based on gradient and chrominance analysis, and a guided image filter is used to suppress noise and enhance edges in the weight maps. To ensure detail enhancement without excessive or insufficient sharpness, we developed a simple sharpness measure named the edge-map overlapping rate (EOR). With EOR and the feedback structure, users are allowed to manipulate the output synthesized image to their preferred sharpness level, and the above weights are appropriately redesigned by automatically regulating the magnitude of the fuzzy input. From experimental results, this work demonstrated excellent image quality and outperformed other existing HDR/MEF methods.

Keywords

Fuzzy logic fuzzy feedback high dynamic range (HDR)multiple exposure fusion (MEF)

1. Introduction

Common camera only can capture limited range of intensities, for example, 8-bit depth intensity levels for each color channel. However, the human visual system (HVS) responds to real-world scenes, which contains much greater dynamic range (about 20-bit depth intensity levels) than that of captured digital images. Therefore, HDR imaging is an essential technique used in photographs to overcome the limited dynamic ranges of cameras [1].

Traditionally, the HDR scene is first obtained by merging several differently exposed low dynamic range (LDR) images; to visualize the HDR scene on the conventional 8-bit display devices, the HDR image is then converted into a final LDR image by using tone mapping algorithms [2]. Compared to traditional HDR, multiple exposure fusion (MEF) is a more recently developed technique that produces details and directly expands the dynamic range of the final synthesized image without the need to generate an intermediate HDR image.

Although MEF methods generate more informative final synthesized LDR image than traditional HDR, the challenge comes from how to appropriately combine useful information from input images, and generate a final image that resembles the natural world scene attained by human visual system as close as possible. Most MEF methods can be classified into two categories: 1) multiple image-based MEF [3 –11]; and 2) single image-based MEF [17 –23].

For the methods belong to the first category, the input LDR images are captured by using the same imaging sensor at different exposures. Shen et al. [3] proposed a HDR image composition method to fuse multi-exposure images based on an extended Retinex model. Gu et al. [4] proposed an MEF algorithm based on gradient field, which is derived from the structure tensor using multi-dimensional Riemannian geometry. To preserve the detail information, the gradient field is modified with twice average filtering under an iterative process, until the gradient magnitude of each pixel is determined. Shen et al. [5] proposed an exposure fusion approach using a boosting Laplacian pyramid, which involves boosting the detail and base signal independently. Patel et al. [6] proposed a MEF method that improves the visual perception of HDR images. The Weber’s law states that people cannot perceive variations in bright areas as much as in dark areas. Therefore, they proposed an exposure control mechanism that compensates for the limitations of the HVS by adaptively adjusting the exposure level. Li et al. [7] proposed a multiscale MEF algorithm that uses guided image filtering (GIF) to smooth the Gaussian pyramids of the weight map of each input LDR image. Details are preserved in the brightest and darkest regions of the final synthesized image. Other multiple image-based fusion methods have also been proposed in [8 –11].

However, for the methods in the first category, fusing a sequence of differently exposed images is typically suitable for static scenes only. In dynamic scenes with camera movement or moving objects such as cars and people, multiple image-based MEF methods are susceptible to image artifacts such as ghosts and edge blurring.

To overcome ghost and blur problems, Li et al. [12] proposed a hybrid patching algorithm for MEF methods with moving objects. First, the motion regions of the differently exposed images are detected through consideration of spatiotemporal consistencies. To compensate for the inconsistent pixels in the motion regions, an optimization problem is formulated and then solved using both the intensity mapping method at the pixel level and the hole-filling method at the block level iteratively. However, this method is computationally expensive and the convergence speed depends heavily on the parameter setting. Li et al. [13] proposed a ghost removal algorithm, which includes a bidirectional normalization-based method for detecting the moving pixels of each LDR image. The correction of inconsistent pixels takes time because it involves a two-step approach using temporal and spatial correlation. Numerous deghosting methods that are designed for MEF have previously been proposed in [14 –16]; yet most of them require high computation cost.

For the methods belong to the second category, only single input LDR image is sufficient, and the images with other exposures are generated from the same image with post-processing. Typically, three exposed (normally-, under-, and over-exposed) LDR images are required. Therefore, it inherently removes the ghost artifacts that occur in multiple image-based MEF. Im et al. [17] proposed a ghost-free single image-based MEF method, in which the three LDR image are generated from the input image by local histogram stretching. An edge-preserving denoising method is proposed, and the weighting map is computed by a Gaussian-shaped function of the pixel value. Celebi et al. [18] proposed a fuzzy-based MEF method wherein a contrast limited adaptive histogram separation scheme is used to generate additional LDR images from a single input image. To account for pixel visibility, fuzzy logic is applied to determine the pixel weights when fusing the images. Raveendranath and Johnson [19] proposed a method for HDR image generation, in which differently exposed images are generated by varying the gamma value of the input image using power law transformation. However, the selection of the most suitable gamma value was not discussed in the study.

Therefore, single image-based MEF methods are an important subset of HDR imaging. For mobile devices, small low-cost sensors are commonly used to reduce manufacturing costs. However, they have limited light-capturing capacity and require long exposure times, which tend to be suffered from ghost artifacts in the dynamic scenes. On the other hand, the computational power of mobile devices is limited and cannot handle sophisticated deghosting processes, which limits the application of mobile HDR imaging. Single image-based MEF methods thus seem to be a promising solution to mobile HDR imaging. Because of its importance in dynamic-scene HDR and mobile HDR imaging, Gu et al. [20] proposed a single image-based HDR method using multiscale decomposition, where an input image is decomposed into one base layer (containing variations in intensity) and three detail layers (containing small-scale details). A local edge-preserving (LEP) filter is utilized to fuse these layers. Other single image-based HDR methods have also been proposed in [21 –23].

Although single image-based MEF methods have the advantage of being ghost-free, most of them are one-step approaches that lack a self-regulating mechanism. That is, most methods determine the final pixel weights without checking the visual quality of the synthesized output image. In addition, for most existing MEF methods, the sharpness of the final synthesized image tends to vary according to the content of input images and is sometimes detail-preserved but excessively sharp, or insufficiently sharp (i.e. blurring in details).

Considering the above shortcomings of the existing methods, we present a single image-based MEF method. Its advantages are summarized as follows:

We apply the concept of using fuzzy logic to determine pixel weights from our baseline method [18] and further modify [18] by accounting for the color vividness (in Section 2.2) to improve the visual quality. In addition, the GIF technique is utilized in the fusion stage to enhance details and reduce noise (in Section 2.3).

We propose a MEF framework that consists of a simple sharpness measure EOR and a feedback structure (in Section 2.4). This allows users to decide their preferable sharpness, and the above pixel weights are automatically and appropriately adjusted by the feedback structure. Therefore, compared to existing methods, this work demonstrates robust sharpness-control ability that avoids excessively or insufficiently sharp. The proposed MEF method is evaluated on various visual quality measures (in Section 3), and the results show that this work outperforms the state-of-the-art methods.

Through consideration of the EOR difference between the desired EOR and the current EOR, we propose an adaptive adjustment scheme to fine-tune the pixel weights, which enables the fuzzy loop to converge within a reasonable number of loops.

This paper presents a fuzzy-based HDR, which can appropriately determine pixel weights. The rest of the paper is organized as follow. Section 2 presents the proposed single image-based MEF method that belongs to the abovementioned second category. In Section 3, we provide the experimental results and compare them to the existing methods in Refs. [17 , 20]. Finally, Section 4 concludes our work.

2. Proposed single image-based MEF method

This section presents the proposed MEF method with fuzzy feedback control for sharpness. It can be considered as an improved version of Celebi’s work [18]. As shown in Fig. 1, this method consists of four steps: 1) histogram stretching; 2) formulating a fuzzy rule base; 3) image fusion based on fuzzy weights and GIF; and 4) sharpness evaluation and self-regulation using the feedback structure. The following subsections describe each step.

Fig.1

Overall flowchart of the proposed fuzzy-based MEF method, where abbreviations NE, OE, and UE denote images with normal-, over-, and under-exposures, respectively.

2.1. Local histogram stretching (LHS)

Typically, single image-based MEF requires three exposure-level LDR images: normal-, under-, and over-exposed images. In this paper, we adopt the LHS method proposed in [17] to generate the three LDR images, detailed as follows. In LHS, the input image is transformed from RGB (indicating red, green, and blue) to YUV (indicating luminance, chrominance, and chroma) color space. The input image is assumed to be the normal-exposed image, but with a quite limited dynamic range, and the two other differently exposed LDR images are generated by stretching the Y channel of the input image.

LHS utilizes the weighted histogram separation (WHS) technique, which is based on the concept of data separation units. Let H_l denote the histogram of the Y channel, WHS divides H_l into two subsets by finding an adaptive threshold thr: $thr = \underset{0 \leq t \leq 255}{arg min} | w - \frac{1}{M} \sum_{p = 0}^{t} H_{l} (p) |,$ (1) where w indicates the weighted factor controlling the sizes of two separated subsets (set as 0.35 as default), M indicates the number of pixels in the input image, p indicates the p-th luminance level, and t indicates all possible luminance levels ranging from 0 to 255.

LHS is performed by using thr to divide H_l into two subhistograms (H₀ and H₁), which are defined as $H_{0} (p) = {\begin{matrix} H_{l} (p), & if p \leq thr \\ 0, & otherwise \end{matrix},$ (2a) and $H_{1} (p) = {\begin{matrix} H_{l} (p), & if 255 \geq p > thr \\ 0, & otherwise \end{matrix} .$ (2b)

As shown in (2), the first subhistogram contains information on lower luminance levels (ranging from 0 to thr), and the second subhistogram contains information on higher luminance levels (ranging from thr + 1 to 255). Therefore, LHS generates two subset images by using linear stretching in H₀ and H₁, respectively: ${\hat{S}}_{i} (x, y) = \frac{S_{i} (x, y) - min_{p} H_{i} (p)}{max_{p} H_{i} (p) - min_{p} H_{i} (p)} \times 255,$ (3) where i∈ { 0, 1 } and ${\hat{S}}_{i} (x, y)$ indicates the stretched luminance (Y) level at position (x, y). Using (3), the under- and over-exposed LDR images can be obtained by combining the unchanged UV channels with the stretched Y channel of the subset images H₀ and H₁, respectively.

In Celebi’s work [18], the same WHS technique and same linear stretching process were used. Furthermore, [18] applied an iterative-based RANdom Sample Consensus (RANSAC) method to determine the weighted factor in (1) and used the contrast limited adaptive histogram equalization (CLAHE) method after (3) to partially flatten individual subhistograms through the control of noise amplification. In [18], the quality of under- and over-exposed LDR images was supposedly improved by the addition of RANSAC and CLAHE. However, its fusion stage simply involves inputting the normalized luminance and smoothed local variance as fuzzy inputs and outputting the fuzzy weights.

In contrast, we argue that the main factor of MEF that dominates the visual quality of the final synthesized image, should be the fusion stage; more specifically, determining the appropriate weight level of each pixel when merging different LDR images. Therefore, we simply apply the LHS method of [17] to generate LDR images and improve the design of the fusion process.

2.2. Formulating the fuzzy rules

It is well-known that fuzzy logic provides an intuitive means of modeling complex systems through linguistic variables. In [18], the two fuzzy inputs were apparently overemphasized in the luminance (Y) channel, and chrominance information in the UV channels was underemphasized. We observed that in some cases, the output synthesized image is excessively sharp and thus appears unnatural.

To cope with above disadvantage, this paper proposes that both the gradient in the Y channel and the chrominance information in the UV channels should be considered simultaneously. This is based on our observation that generally, well-exposed regions tend to contain stronger gradients and more vivid colors than under- and over-exposed regions. Two fuzzy inputs are thus proposed. The first input is the local gradient in the Y channel, which is defined as $\begin{matrix} {grad}_{s} (x, y) = max (| Y_{s} (x, y) - Y_{s} (x + 1, y) |, \\ | Y_{s} (x, y) - Y_{s} (x - 1, y) |, \\ | Y_{s} (x, y) - Y_{s} (x, y + 1) |, \\ | Y_{s} (x, y) - Y_{s} (x, y - 1) |), \end{matrix}$ (4) where the subscript s∈ { n, u, o } indicates whether the resulting LDR images are normal-, under-, or over-exposed LDR images, respectively. Typically, gradient value is computed from the directional derivative. However, its computation cost is high. Hence, this paper proposes using the maximum operation in the difference among the four-connected neighbors, as shown in (4). Equation (4) provides the same information as computing an absolute derivative, but with simpler calculations.

The second input is the color vividness in the UV channels, which is defined as $vivid (x, y) = | U (x, y) - 128 | + | V (x, y) - 128 | .$ (5)

During the transformation from RGB to YUV, the values of Y, U, and V all range from 0 to 255: The Y channel indicates the illuminance level, and the chrominance components in the UV channels indicate the color difference signals. If both U and V approach 128, neutral colors are shown such as black, gray, and white, and these colors are usually found in under-exposed, shadow, and over-exposed regions. In such regions, viewers pay more attention to detail contours than to color difference. In other words, for regions with less vivid color, unless the local gradient is high, the weights of the region’s pixels should be low in the construction of the final synthesized image.

Figure 2 specifies the proposed membership functions of the fuzzy inputs and the output weight, and Table 1 constructs the fuzzy rule base for determining the pixel weight in the proposed MEF method. As shown in Table 1, if both gradient and color vividness are high, the pixel is more likely to be well-exposed, and thus this pixel should make a large contribution to the final synthesized image. Combining (4) and (5) with fuzzy logic enables us to model the complicated MEF weighting process in an efficient and straightforward manner.

Fig.2

Membership functions of the proposed fuzzy rule. (a) Normalized input 1 (gradient). (b) Normalized input 2 (vividness). (c) Output weight.

Table 1

Fuzzy rule base of the proposed MEF approach^*

Gradient
UV	Low	Medium	High
Approach 128	L	M-L	M
Normal	M-L	M	M-H
Approach 0 or 255	M	M-H	H

^*L: low, M-L: medium-low, M: medium, M-H: medium-high, H: high.

2.3. Image fusion with GIF-filtered Weight Maps

Edge-preserving denoising is a major concern in the fusion stage because noise tends to be amplified when the Y channel is stretched; by contrast, simple averaging removes not only noise but also the essential edge information of the final image. This paper proposes using the GIF technique [24] to smooth the weight maps (i.e., the outputs of fuzzy logic) of all the LDR images. To save the computation cost, we adopt the fast GIF method [25], which is 10 times faster than traditional GIF but with almost no visible degradation. The filtered weight maps determine the final contribution level of each pixel in the three LDR images. Once the final pixel weights are determined, the final synthesized image is constructed by an arithmetic operation of the input LDR images: $H^{c} (x, y) = \frac{\sum_{s \in {o, n, u}} I_{s}^{c} (x, y) W_{s} (x, y)}{\sum_{s \in {o, n, u}} W_{s} (x, y)},$ (6) where H indicates the final HDR image, I indicates the three input LDR images, and W indicates the corresponding filtered weight map. The superscript c∈ { Y, U, V } indicates each color channel and the subscript s∈ { n, u, o } indicates the exposure levels: normal-, under-, and over-exposed.

It is an important observation of this paper that although edges are mainly perceived according to the Y channel, in order to increase control and consistency when fusing images, GIF is used to smooth the weight maps rather than the stretched histograms or the gradient maps in the Y channel, which is different from our baseline method [18]. This is mainly because in the final step, the fusion of three LDR images is based on a linear relationship among the corresponding weight maps, as shown in (6). However, because fuzzy logic is nonlinear, GIF should be performed after fuzzy logic to achieve more favorable visual quality.

2.4. Sharpness evaluation and self-regulation using feedback structure

Edge enhancement is a common challenge in MEF methods because merging images often blurs edges, significantly degrading visual quality. Many MEF methods try to solve this problem by increasing the importance of edge pixels when determining the pixel weights. However, this tends to unnaturally alter the overall sharpness of the entire image.

By contrast, the proposed method can allow users to dynamically control the sharpness level and appropriately redesign the pixel weights by automatically regulating the magnitude of the fuzzy input through a feedback structure. To do this, we propose a simple sharpness measure for MEF, called edge-map overlapping rate (EOR), which is defined as $EOR = \frac{\sum_{(x, y) \in 1} C_{input} (x, y) \cap C_{fused} (x, y)}{\sum_{(x, y) \in 1} C_{input} (x, y)} .$ (7)

EOR value ranges from 0 to 1 and represents the measure of image sharpness-preserving (compared to the input image), where C_input and C_fused indicate the canny-based edge maps of the input LDR image and the current fused image, respectively. In this paper, we employ Canny edge detector to determine edge maps because it is efficient and commonly used in real-time image processing applications. The underlying concept of the proposed EOR is the preservation of the crucial edge pixels of the input LDR image.

With the proposed EOR, users can control the sharpness level of the output image by manipulating their desired EOR. Once the fused image is obtained using (6), its overall sharpness is checked according to whether $α \in [A - τ, A + τ],$ (8) where A is the desired EOR, α is the EOR of the current fused image, and τ is the tolerance value (set as 0.05 in this study).

When α is larger than A + τ, the current fused image is implied to be excessively sharp and requires less weight in the edge pixels (i.e., pixels with high gradient values). Conversely, when α is smaller than A - τ, insufficient sharpness is implied. Therefore, this work uses the difference between A and α as a reference to adjust the first fuzzy input (i.e., the local gradient) as shown in (4), which can be expressed by ${grad}_{s}^{'} (x, y) = λ \times {grad}_{s} (x, y), s \in {o, n, u},$ (9) where ${grad}_{s}^{'} (x, y)$ indicates the adjusted gradient value, the subscript s∈ { n, u, o } indicates the exposure levels, and the parameter λ (≥0) is adaptively determined through consideration of the EOR difference between A and α, which is defined as $λ = exp (\frac{A - α}{A}) .$ (10)

As shown in (10), the proposed adjustment scheme is adaptive. For example, if α is much smaller than A - τ (i.e., implying insufficient sharpness and blurs in edges), the initial gradient ${grad}_{s}^{'} (x, y)$ will be multiplied by a much larger λ (far greater than 1). Using (10) helps the fuzzy loop to converge within a limited number of loops. The adjusted gradient is then inputted into the fuzzy rule base to determine the updated fuzzy weights.

The adjustment using (9) is repeated until is within the desired range shown in (8). In some cases, the resulting HDR images after the first fuzzy fusion are of highly favorable quality and suitable sharpness, which require no further loop. Based on our experiments, most images converge within three loops, and two loops are required on average.

3. Experimental results and discussion

This section evaluates the performance of the proposed method in various aspects. There are eight test images used for verifying algorithms, and most of them are randomly selected from the online database HDR Photographic Survey. Figure 3 shows thumbnails of all test images. In addition to [18], we also compare the proposed method with a single image-based MEF method [17] and a single image-based HDR method [20]. All four methods were implemented using MATLAB R2013a on an Intel Core i5 machine with 3.3 GHz CPU and 4 GB RAM.

Fig.3

Test images. (Images 1 to 4) First row from left to right: Lounge, Mt. Rushmore, Exploratorium, and Sunset Point. (Images 5 to 7) Second row from left to right: Devil’s Golf Course, Las Vegas, and Niagara Falls. (Image 8) Right side: Chapel.

3.1. Discussion on the performance of this work

This subsection presents the effects of two important steps used in the proposed MEF framework: 1) GIF-based denoising and 2) self-regulation using the feedback structure. First, it is proposed in this paper that before the LDR images are fused, applying GIF in fuzzy weight maps to enhance details and reduce noise. Figure 4 provides an example demonstrating the effect of using the GIF method.

Fig.4

Contribution of GIF: local edge-enhancing denoising (using the portions of Chapel). (a) and (c): Noisy HDR image without GIF. (b) and (d): HDR image after GIF has been performed in weight maps.

As shown in Fig. 4(a), without the application of GIF, the contour information near the window arch is blurred and the shapes of the bricks on the ceiling can barely be distinguished, whereas in Fig. 4(b), the aforementioned problems have been solved using GIF. In addition, as shown in Fig. 4(c), there is noise (spots with inconsistent colors) within the red rectangle, whereas in Fig. 4(d), GIF has successfully removed this noise and enhanced the local edge.

Because the filtered output of GIF is locally a linear transform of the guidance image (in this study, the weight map itself), local structures can be utilized effectively. The advantage of this (i.e., denoising while preserving important structural edge pixels) can be more clearly observed from the Canny-based edge map of the fused image, as shown in Fig. 5.

Fig.5

Contribution of GIF: denoising while preserving essential structural edge pixels (using the edge map of Chapel). (a) Without GIF. (b) With GIF.

Second, the effect of using feedback structure to self-regulate the overall sharpness is presented as follows. Fine-tuning the pixel weights used to fuse the three LDR images is highly challenging. When pixels with high gradient intensity are given weights that are too high to enhance the output details, the fused image might be excessively sharp and the noise might be amplified, degrading the visual quality. Conversely, when the structural edge pixels are not given enough weights, the fused image might be blurred in details, which degrades the visual quality, too. In addition, other factors such as color vividness and consistency should be taken into account.

To deal with this, this work allows the users to dynamically control their preferable sharpness level, and then, the appropriate pixel weights are all adaptively adjusted by the proposed fuzzy rules and feedback structure. The EOR difference between (desired sharpness level) and (current sharpness level) are used as a reference, and thus it avoids the case of over-adjusting the pixel weights. Figure 6 shows an example of executing feedback loop in one and two times: Although the overall sharpness is different between Fig. 6(b) and 6(c), they both have satisfactory visual quality.

Fig.6

Contribution of feedback structure to the fine-tuning of overall sharpness. (a) Original image. (b) After the first fuzzy fusion (EOR = 0.445). (c) After the second fuzzy fusion (EOR = 0.462). Please refer to the barbed wire region in the bottom right to see the slight sharpness difference.

There are two advantages of using the proposed fuzzy feedback loop scheme. First, the proposed method can automatically self-regulate the pixel weights according to the user’s desired sharpness. Second, using feedback loop allows us to tune the pixel weights more precisely by accounting for the EOR difference. Moreover, this method is computationally efficient, because as shown in (9), only the first fuzzy input is adjusted, meaning that the second fuzzy fusion takes less time than the first. Generally, by using fuzzy logic, our baseline method [18] already generates high-quality HDR images that some resulting images of [18] are of highly favorable quality and suitable sharpness, which require no further loop. This work further improves [18] by utilizing the feedback loop to make the fuzzy fusion more self-adaptive and suited to the user’s preference.

As explained in Section 2.4, users are instructed to freely select their desired EOR A within the range between 0.4 and 0.52. Based on our experiments, when the tolerance value (τ) is set to 0.05, an average of two loops are needed for convergence.

3.2. Comparisons between the proposed method and others

To demonstrate the superiority of the proposed method, we compare it to three existing single image-based HDR methods: those of Im [17], Celebi [18] (our baseline method), and Gu [20]. Figure 7 shows the HDR images generated by different methods using the image Mt. Rushmore. For the result of [17] (Fig. 7b), because of the simple fusion process, the local edge is blurred: Please refer to the enlarged version of the fence (bottom middle) and brick regions (bottom right). It is because the fusion process in [17] is almost as simple as an arithmetic averaging of the three input LDR images. For the result of [18] (Fig. 7c), although the detailed edges are preserved using fuzzy logic to determine pixel weights, the dynamic range is not fully stretched and the fence region is therefore slightly dark. Moreover, [18] seems to over-enhance the sharpness of the clouds and some unnatural artifacts are visible in the center of the sky. For the result of [20] (Fig. 7d), the sky region is over-exposed and the detailed contours of the clouds are lost. By contrast, the proposed method demonstrates significant improvement on the aforementioned shortcomings and yields a visually pleasing HDR image, as shown in Fig. 7(e).

Fig.7

Results of HDR methods by using the Mt. Rushmore image: (a) Input image; (b) Im [17]; (c) Celebi [18]; (d) Gu [20]; and (e) Ours.

Figure 8 presents another visual comparison using the image Chapel. For the result of [17] (Fig. 8b), it can be seen that same as the case of Fig. 7(b), it is insufficiently sharp compared with others; so much so that the details of the ceiling (such as the brick texture) are nearly lost. By contrast, although details are preserved in the result of [18] (Fig. 8c), the image is excessively sharp in some regions, as indicated by the unnatural textures visible on the backs of the pews. This occurred because when the fuzzy-based pixel weights are determined in [18], pixels with high gradient or high intensity are given excessive weights, whereas color consistency is ignored. For [20] (Fig. 8d), the output HDR image is somehow over-exposed to the extent that a color shift occurs (which can be seen in the color checker). The result of proposed method (Fig. 8e) provides apparently higher visual quality. Compared to [20], the image produced is brilliant, edges are preserved, and the overall color tone of the input image is completely maintained. The overall sharpness of Fig. 8(e) is controlled to a visually pleasing degree; that is, the local edges are appropriately enhanced without the loss of detail (compared with [17]), neither over-enhancing the noise nor the unnatural textures (compared with [18]).

Fig.8

Results of various HDR methods by using the Chapel image: (a) Input image; (b) Im [17]; (c) Celebi [18]; (d) Gu [20]; and (e) Ours.

For the comparisons involving objective image quality criteria, we adopt the same criteria as were employed in our baseline method [18]: Entropy and Cumulative Probability of Blur Detection (CPBD) [27]. The first criterion (Entropy) measures the uncertainty associated with the richness of color distribution. Higher Entropy value indicates that the image has richer details.

The second criterion (CPBD) measures the sharpness of the image. More specifically, CPBD utilizes the concept of just noticeable blur (JNB) to predict the percentage of edges where blur cannot be detected. Higher CPBD value indicates perceptually better preserving details with enough clarity. Tables 2 and 3 compare the results for Entropy and CPBD, respectively. From them, the proposed method demonstrated better performance (higher Entropy and CPBD) than other methods.

Table 2

Comparisons of Entropy among methods

Method
Image	Input	Ref. [17]	Ref. [18]	Ref. [20]	Ours
Img 1	5.06	5.22	5.20	5.00	5.41
Img 2	4.86	5.28	5.30	5.17	5.33
Img 3	5.12	5.12	5.23	4.85	5.25
Img 4	5.08	5.17	5.23	4.68	5.35
Img 5	5.01	5.22	5.29	4.75	5.30
Img 6	4.88	5.23	5.24	5.14	5.33
Img 7	5.33	5.33	5.39	4.99	5.42
Img 8	4.97	5.27	5.26	5.26	5.35
Avg.	5.04	5.23	5.27	4.98	5.34

Table 3

Comparisons of CPBD among methods

Method
Image	Input	Ref. [17]	Ref. [18]	Ref. [20]	Ours
Img 1	0.65	0.52	0.67	0.69	0.70
Img 2	0.76	0.61	0.77	0.77	0.77
Img 3	0.65	0.51	0.67	0.66	0.69
Img 4	0.69	0.55	0.71	0.70	0.74
Img 5	0.73	0.55	0.74	0.73	0.78
Img 6	0.69	0.55	0.71	0.72	0.75
Img 7	0.48	0.34	0.56	0.60	0.60
Img 8	0.69	0.55	0.67	0.70	0.70
Avg.	0.67	0.52	0.69	0.70	0.71

Table 4 shows the execution times for the generation of HDR images using different methods. The execution time for the image Chapel was about twice as that of other images due to its large size. Compared to our baseline method [18], this work requires less time: Although the proposed method uses a feedback loop to adaptively adjust weights, it saves less time: Although the proposed method uses a feedback loop to adaptively adjust weights, it saves time by omitting the CLAHE (used in [18]) to filter the stretched histograms. For the proposed method (in Table 4), the loop number is set as two according to the average number required for convergence as explained in Section 3.1. The execution times of [17, 20] were shorter than those of [18] and this work. It is because when implementing [18] and the proposed method, we use Matlab fuzzy toolbox to determine fuzzy weights instead of designing a simplified code as used in [18]. (From the result shown in Table 3 of [18], the average execution time of [18] is about twice as that of [17].) In addition, it is important to note that as discussed above, the proposed method achieves apparently higher visual quality than others.

Table 4

Comparisons of execution times among methods (sec)

Method
Image	Ref. [17]	Ref. [18]	Ref. [20]	Ours
Img 1	1.11	8.09	2.29	7.63
Img 2	1.03	8.26	2.248	7.35
Img 3	0.98	8.15	2.266	7.54
Img 4	0.99	8.06	2.179	7.21
Img 5	0.96	7.09	2.006	6.28
Img 6	1.03	7.00	1.97	6.32
Img 7	0.92	6.93	1.94	6.29
Img 8	1.61	18.21	5.02	16.35
Avg.	1.08	8.97	2.49	8.12

4. Conclusion

This paper presents a novel MEF method for HDR imaging. This work is ghost-free because it requires only one input image. Considering various image contents, it is difficult to determine suitable pixel weights without any adjustment; if the output HDR image is slightly too sharp or not sharp enough, its quality will be degraded. Therefore, the proposed method allows users to set their preferred sharpness level, and the pixel weights are automatically and appropriately adjusted by the feedback structure. For the design of fuzzy rules, unlike our baseline method [18], we propose that in addition to the gradient in the Y channel, color vividness in the UV channels should be considered. In addition, GIF is utilized to enhance details and reduce noise. The adaptive adjustment scheme proposed in (10) enables the fuzzy loop to converge within a limited number of loops.

Although this work is single image-based MEF, it actually addresses the fundamental problem of how to appropriately determine and adjust pixel weights when fusing images so as to generate high-quality HDR images, and thus the proposed framework can also be used for multiple image-based MEF. The experimental results of several subjective and objective image quality evaluations demonstrated that the proposed method outperforms other single image-based HDR methods.

Regardless of showing many advantages, there are still some interesting future works of this study. First, a software optimization technique is required. In our baseline [18] and this study, the implementation of fuzzy fusion is based on the Matlab fuzzy toolbox. As shown in Table 4, both methods seem to require much more time than [17]. However, as documented in [18], using a 2D Look-Up Table to simplify fuzzy fusion can greatly decrease the processing time to approximately 1.6 ms for fusing a full HD image through fuzzy fusion. Second, this work proposes EOR as a measure of sharpness because it is simple and compatible with real-time Canny edge detector. However, we found that sometimes, slight differences in overall sharpness cannot be distinguished using EOR. CPBD is an alternative means of sharpness measure, but its computation cost might be a concern in real-time processing. We plan to develop another efficient measure of sharpness and integrate it into our system in the future.

References

P.E.

Debevec and

Malik , Recovering high dynamic range radiance maps from photographs, Proc SIGGRAPH, Los Angeles, CA, USA, 1997, pp. 369–378.

Z.G.

Li and

J.H.

Zheng , Visual-salience-based tone mapping for high dynamic range images, IEEE Trans Ind Electron 61(12) (2014), 7076–7082.

Shen ,

Zhao ,

Jiang and

Suwa , Recovering high dynamic range by multi-exposure Retinex, J Vis Commun Image Represent 20(8) (2009), 521–531.

Gu ,

Li ,

Wong ,

Zhu and

Wang , Gradient field multi-exposure images fusion for high dynamic range image visualization, J Vis Commun Image Represent 23(4) (2012), 604–610.

Shen ,

Zhao ,

Yan and

Li , Exposure fusion using boosting Laplacian pyramid, IEEE Trans Cybern 44(9) (2014), 1579–1590.

Patel ,

Androutsos and

Kyan , Adaptive exposure fusion for high dynamic range imaging, Proc IEEE Int Conf Image Process (ICIP), Quebec, Canada, 2015, pp. 4679–4683.

Z.G.

Li ,

Wei ,

Wen and

Zheng , Detail-enhanced multi-scale exposure fusion, IEEE Trans Image Process 26(3) (2017), 1243–1252.

Hua ,

Wang ,

Rusdi and

Jiang , A novel multi-focus image fusion algorithm based on random walks, J Visual Communication and Image Representation 25(5) (2014), 951–962.

Ma and

Wang , Multi-exposure image fusion: A patch-wise approach, Proc IEEE Int Conf Image Process (ICIP), Quebec, Canada, 2015, pp. 1717–1721.

10.

Hua ,

Wang ,

Yeh ,

Cheng and

Lai , Background extraction using random walk image fusion, IEEE Trans Cybernetics 99 (2016).

11.

Sanchez-Riera ,

Hua ,

Hsiao ,

Lim ,

Hidayati and

Cheng , A comparative study of data fusion for RGB- D based visual recognition, Pattern Recognition Letters 73 (2016), 1–6.

12.

Li ,

Zheng ,

Zhu ,

Yao ,

Wu and

Rahardja , Hybrid patching for a sequence of differently exposed images with moving objects, IEEE Trans Image Process 22(12) (2013), 5190–5201.

13.

Li ,

Zheng ,

Zhu and

Wu , Selectively detail-enhanced fusion of differently exposed images with moving objects, IEEE Trans Image Process 23(10) (2014), 4372–4382.

14.

Bhoskar and

Kakarala , Novel approach to detect HDR scenes and determine suitable frames for image fusion, Proc Digital Photography and Mobile Imaging XII, San Francisco, CA, USA, 2016, pp. 1–8.

15.

Ha ,

Yuan ,

Fierro and

Ha , Ghost removal in exposure fusion by temporal consistency assessment, J Imaging Science and Technology (JIST) 60(4) (2016), 40501:1–40501:12.

16.

Zhang ,

Hu ,

Liu and

Yao , Motion-free exposure fusion based on inter-consistency and intra-consistency, Information Sciences 376 (2017), 190–201.

17.

Im ,

Jeon ,

Hayes and

Paik , Single image-based ghost-free high dynamic range imaging using local histogram stretching and spatially-adaptive denoising, IEEE Trans Image Process 57(4) (2011), 1478–1484.

18.

Celebi ,

Duvar and

Urhan , Fuzzy fusion based high dynamic range imaging using adaptive histogram separation, IEEE Trans Consumer Electronics 61(1) (2015), 119–127.

19.

Raveendranath and

A.K.

Johnson , Single shot high dynamic range imaging using power law transformation and exposure fusion, Proc Int Conf Communication Systems and Networks (ComNet), Thiruvananthapuram, India, 2016, pp. 120–126.

20.

Gu ,

Li ,

Zhu and

Wang , Local edge-preserving multiscale decomposition for HDR image tone mapping, IEEE Trans Image Process 22(1) (2013), 70–79.

21.

Song ,

Wang and

Bai , High dynamic range infrared images detail enhancement based on local edge preserving filter, Infrared Physics & Technology 77 (2016), 464–473.

22.

Kim ,

Park and

Chang , Tone mapping with contrast preservation and lightness correction in high dynamic range imaging, Signal, Image and Video Processing 10(8) (2016), 1425–1432.

23.

Ok and

Lee , HDR tone mapping algorithm based on difference compression with adaptive reference values, J Visual Commun and Image Represent 43 (2017), 61–76.

24.

He ,

Sun and

Tang , Guided image filtering, IEEE Trans Pattern Analysis and Machine Intelligence 35(6) (2013), 1397–1409.

25.

He and

Sun , Fast guided filter, Tech Report (2015).

26.

M.D.

Fairchild , The HDR photographic survey, Proc 15th IS&T/SID Color Imaging Conference, Albuquerque, NM, USA, vol. 15, 2006, pp. 233–1238.

27.

N.D.

Narvekar and

L.J.

Karam , A no-reference image blur metric based on the cumulative probability of blur detection (CPBD), IEEE Trans Image Process 20(9) (2011), 2678–2683.