tceic.com

学霸学习网 这下你爽了

学霸学习网 这下你爽了

- Digital zooming for color filter array-based image sensors
- Exposing digital forgeries in color filter array interpolated images
- A Novel Spatial Interpolation Method Based on the Integrated RBF Neural Network
- Edge Pattern Based Demosaicking Algorithm of Color Filter Array
- Microphone array post-filter based on noise field coherence
- Practical Spectral Imaging Using a Color-Filter Array Digital Camera
- IMPROVED COLOR FILTER ARRAY DEMOSAICKING BY ACCURATE LUMINANCE ESTIMATION
- A rapid stereo matching algorithm based on disparity interpolation
- 3D Fuzzy Vault Based on Cubic Spline Interpolation
- Spatial color histograms for content-based image retrieval

Color Filter Array Interpolation Based on Spatial Adaptivity

Dmitriy Paliya , Radu Bilcub , Vladimir Katkovnika , Markku Vehvil? ainenb

a Institute

of Signal Processing, Tampere University of Technology, P.O.Box 553, FIN-33101 Tampere, Finland. e-mail: rstname.lastname@tut.

b Nokia

Research Center, Tampere, Finland. e-mail: rstname.lastname@nokia.com

ABSTRACT

Conventional approach in single-chip digital cameras is a use of color lter arrays (CFA) in order to sample di erent spectral components. Demosaicing algorithms interpolate these data to complete red, green, and blue values for each image pixel, in order to produce an RGB image. In this paper we propose a novel demosaicing algorithm for the Bayer CFA. For the algorithm design we assume that the initial interpolation estimates of color channels contain two additive components: the true values of color intensities and the errors. The errors are considered as an additive noise, and often called as a demosaicing noise, that has to be removed. This noise is not white and strongly depends on the signal. Usually, the intensity of this noise is higher near edges of image details. We use specially designed signal-adaptive lter to remove the interpolation errors. This lter is based on the local polynomial approximation (LPA) and the paradigm of the intersection of con dence intervals (ICI) applied for selection adaptively varying scales (window sizes) of LPA. The LPA-ICI technique is nonlinear and spatially-adaptive with respect to the smoothness and irregularities of the image. The e ciency of the proposed approach is demonstrated by simulation results. Keywords: Bayer pattern, color lter array interpolation, spatially adaptive denoising

1. INTRODUCTION

The common approach in single-chip digital cameras is a use of CFA to sample di erent spectral components like red, green, and blue. The Bayer CFA patented in 19761 is widely exploited today (see Fig.1). Demosaicing algorithm interpolates sets of complete red, green, and blue values for each pixel, to make an RGB image. Independent interpolation of color channels usually leads to drastic color distortions. The way how produce e ectively a joint color interpolation plays a crucial role for demosaicing. The modern e cient algorithms exploit two main facts. The rst is that for natural images there is a high correlation between the red, green, and blue channels. As a result all three color channels are very likely to have the same texture and edge locations. The second fact is that digital cameras use the CFA in which the luminance (green) channel is sampled at the higher rate than the chrominance (red and blue) channels. Therefore, the green channel is less likely to be aliased, and details are preserved better in the green channel than in the red and blue channels2 :

1.1. Correlation models

There are two basic interplane correlation models in the literature: the color di erence rule4 ; 5 and the color ratio rule6 ; 7 : The rst model asserts that intensity di erences between red, green, and blue channels are slowly varying, that is the di erences between color channels are locally nearly-constant3 ; 4 ; 5 ; 8 ; 9 ; 10 ; 11 ; 12 ; 13 ; 14 : Thus, they contain low-frequency components only, making the interpolation using the color di erences easier10 : The second correlation model is based on the assumption that the ratios between colors are constant over some local regions6 ; 7 : This hypothesis follows from the Lambert's law that if two colors have equal chrominance then the ratios between the intensities of three color components are equal10 : According to this model, the intensity of each color channel is calculated as a projection of the normal vector to the object surface onto the

Figure 1. Bayer color lter array.

light source direction, multiplied by the surface albedo. The albedo captures the characteristics of the object's material, and is di erent for each of the spectral channels. It is assumed that the material, and therefore the albedo, are locally the same within the given object in the image. The rst di erence-based correlation model is found to be more e cient than the ratio-based model and, therefore, exploited more often in practice. Moreover, the color-di erence rule can be implemented with a lower computational cost and better ts linear interpolation models8 :

1.2. Demosaicing methods

Many demosaicing algorithms4 ; 5 ; 6 ; 15 ; 16 ; 18 ; 19 incorporate edge directionality in interpolation. Interpolation along object boundaries is preferable as compared with interpolation across these boundaries for the most image models. We will classify the demosaicing techniques into two categories: noniterative3 ; 4 ; 5 ; 9 ; 11 ; 12 ; 13 ; 19 ; 20 ; 21 ; 22 ; and iterative2 ; 6 ; 7 ; 8 ; 10 : There are also alternative ways of this classi cation, for instance proposed in23 : Noniterative demosaicing techniques basically rely on the idea of edge-directed interpolation. The exploitation of this sort of intraplane correlation typically is done by estimating local gradients under the main assumption that locally the di erence between colors is nearly constant. At each pixel the gradient is estimated, and the color interpolation is carried out directionally, based on this estimated gradient. Directional ltering is the most popular approach for color demosaicing that produces competitive results. The best known directional interpolation scheme is perhaps the method proposed by Hamilton and Adams5 : The authors use the gradients of blue and red channels as the correction terms to interpolate the green channel. Once the green samples are lled, the red and blue samples are interpolated in analogous way11 : Similar idea is exploited e ectively in4 ; 9 ; 13 with the di erence in fusing of vertical and horizontal estimates. In a variety of color demosaicing techniques the gradient estimates analysis plays a central role in reconstructing sharp edges. A new idea in this eld has been proposed and e ectively used in the papers8 ; 11 in 2005, where the authors lter the di erences between the color channels. Since the color channels of a natural image are highly correlated, the di erence signal between the green channel and the red or blue channels constitutes a smoother (low-pass) process. Furthermore, it is observed that this color di erence signal is largely uncorrelated to the interpolation errors of the gradient-guided color demosaicing methods, which are band-pass processes. The authors exploit linear minimum mean square-error estimation (LMMSE) for estimating the color di erence signals. The LMMSE estimates are obtained in both horizontal and vertical directions, and then fused optimally to remove the demosaicing noise. Finally, the fullresolution three color channels are reconstructed from the LMMSE ltered di erence signals11 : In20 the method is proposed, where the green channel is used to determine the pattern at a particular pixel, and then a missing red (blue) pixel value is estimated as a weighted average of the neighboring pixels according to this pattern.

In addition, there are pattern recognition26 ; restoration-based21 ; 27 ; and sampling theory point of view3 ; 12 approaches (see23 for more details). Methods called template matching have been put forth in26 : They estimate edges in the Bayer CFA image and change the interpolation procedure according to edge behavior. This approach works quite well, despite the problem with selection of the threshold for estimation. The algorithm varies with the content of a particular image and can become computationally heavy25 : Methods using the regularization theory28 or the Bayesian approach29 have also been developed. Taubman21 proposed a preconditioned e cient approach of Bayesian demosaicing that is used in some digital cameras today. Alleysson et al.24 ; 25 proposed a model showing that luminance and opponent chromatic signals are well localized in the Fourier spectrum. This spatial Fourier-transform information is used to develop a color demosaicing algorithm by selecting appropriate spatial frequencies corresponding to luminance and opponent chromatic signals. Iterative demosaicing techniques have also been proposed recently2 ; 6 ; 8 ; 10 : The re nement of green pixels and red/blue pixels are mutually dependent and jointly bene cial to each other. It is natural to introduce an iterative strategy to handle such type of problem8 : Kimmel6 in 1999 proposed one of the rst methods where the iterative approach was introduced and described. According to this method green and red/blue channels are iteratively updated by enforcing the color ratio rule. Gunturk et. al. proposed in 2002 an approach using lter-bank for decomposition an image to low- and highfrequency subbands2 that performs undecimated wavelet transform. This is done in both vertical and horizontal directions. The result is four subbands which contain low- and high-frequency components (LL, LH, HL, HH). Because of the fact that the LH, HL, HH components of di erent channels are highly correlated they are replaced by the corresponding components of another channel, but at unknown positions only. Xin Li8 proposed an iterative color-di erence interpolation. This approach successively re nes the estimate of missing data by enforcing the color-di erence rule at each iteration. Three inherent problems often associated with demosaicing algorithms that incorporate directional twodimensional interpolation are misguidance color artifacts, interpolation artifacts, and aliasing. The proposed in10 demosaicing algorithm, which also adopts the directional interpolation approach, addresses these problems explicitly. In the process of estimating the missing pixel components, the aliasing problem is resolved by applying lterbank techniques to directional interpolation. This interpolation procedure produces two images: horizontally interpolated and vertically interpolated ones. The level of misguidance color artifacts present in these images is compared by a color image homogeneity metric. The misguidance color artifacts are minimized by only keeping the pixels interpolated in the direction with fewer artifacts. The interpolation artifacts are reduced using a nonlinear iterative procedure10 : In30 ; 31 authors propose to use directional estimates under assumption that color-di erence is constant. These estimates are fused together by means of weighted average which gives some advantages over usual use of just vertical and horizontal estimates. However, the authors also propose the method of postprocessing based on the assumption that the localized color-ratio is constant7 : It has been observed that iterative demosaicing techniques are capable often of achieving higher quality in the reconstructed images than noniterative ones at the price of increased computational cost8 : This work was inspired by11 where the authors proposed to nd estimates of di erence between luminance and chrominance channels as the result of denoising procedure. In the paper11 the concept of directional "demosaicing noise" was introduced for the interpolation errors. A ltering procedure is exploited to remove this noise. In this paper, in order to remove the interpolation errors we exploit the spatially-adaptive LPA-ICI denoising technique32 ; 33 ; 34 instead of the x-length lter used in11 : The ICI rule applied for selection adaptively varying scales (window sizes) of LPA. The LPA-ICI algorithm is nonlinear and spatially-adaptive with respect to the smoothness and irregularities of the image. The e ciency of introducing this step is shown by simulation results. In many applications the observed data is noisy. We refer reader to35 where similar idea is exploited to design a technique that performs both denoising and interpolation of noisy Bayer data.

The structure of the paper is the following. In Section 2 we introduce the considered Bayer pattern image formation model. The initialization step of the proposed algorithm is described in Section 3. The LPA-ICI ltering is presented in Section 4. In Sections 5-7 the interpolation algorithm of color channels is given. The simulation results are shown in Section 8.

2. IMAGE FORMATION MODEL

We follow the general Bayer mask image formation model: zBayer (i; j ) = where f g is a Bayer sampling operator1 fyRGB (i; j )g; (1)

and zBayer is an output signal of the sensor, yRGB (i; j ) = (R(i; j ); G(i; j ); B (i; j )) is a true color RGB observation scene, X = f(i; j ) : i = 1; :::; 2N; j = 1; :::; 2M g are the spatial coordinates and R (red); G (green); and B (blue) correspond to the color channels. For two available green channels we will use notations G1 (i; j ); such that (i; j ) 2 XG1 = f(i; j ) : i = 1; 3; :::; 2N 1; j = 1; 3; :::; 2M 1g; and G2 (i; j ); such that (i; j ) 2 XG2 = f(i; j ) : i = 2; 4; :::; 2N; j = 2; 4; :::; 2M g. Spatial coordinates for the red R(i; j ) and blue B (i; j ) color channels are denoted XR = f(i; j ) : i = 1; 3; :::; 2N 1; j = 2; 4; :::; 2M g and XB = f(i; j ) : i = 2; 4; :::; 2N; j = 1; 3; :::; 2M 1g, respectively. Demosaicing attempts to invert observations zBayer (i; j ). f g in order to reconstruct R(i; j ); G(i; j ); and B (i; j ) intensities from the

8 G(i; j ); > > < G(i; j ); fyRGB (i; j )g = > R(i; j ); > : B (i; j );

at (i; j ) 2 XG1 ; at (i; j ) 2 XG2 ; at (i; j ) 2 XR at (i; j ) 2 XB

(2)

The algorithm consists of the following steps: initialization, ltering, and interpolation. At the initialization, the approximate color estimates are obtained and directional di erences between G R and G B are calculated. These di erences are considered as degraded by noise and ltered. The modi ed LPA-ICI algorithm32 ; 33 ; 34 is used for this ltering. Finally, the obtained estimates are exploited to calculate missing color values at each pixel.

3. INITIALIZATION

Firstly we calculate the directional (horizontal and vertical) estimates of green channel at every point (i; j ) 2 X following the rules of Hamilton-Adams algorithm5 : Interpolation of G at R positions (i; j ) 2 XR is done as follows: 1 (G(i + 1; j ) + G(i 1; j )) + 2 1 (G(i; j + 1) + G(i; j 1)) + 2 1 ( R(i 2; j ) + 2R(i; j ) 4 1 ( R(i; j 2) + 2R(i; j ) 4

^ h (i; j ) G ^ v (i; j ) G

= =

R(i + 2; j )) ; R(i; j + 2)) :

(3) (4)

Here h and v stay for horizontal and vertical estimates. Similarly to (3)-(4), the initial directional estimates for the red channel R at green positions G ((i; j ) 2 XG1 or (i; j ) 2 XG2 ) are interpolated as: ^ h (i; j ) R ^ v (i; j ) R = = 1 (R(i + 1; j ) + R(i 1; j )) + 2 1 (R(i; j + 1) + R(i; j 1)) + 2 1 ( G(i 2; j ) + 2G(i; j ) 4 1 ( G(i; j 2) + 2G(i; j ) 4 G(i + 2; j )) ; G(i; j + 2)) : (5) (6)

As a result we obtain at the every horizontal line values: ^h G ::: G ^h ::: R R

of R values two sets of true and estimated green and red ^h G R G ^h R ^h G R ::: : :::

Similar calculations are produced for the vertical lines. At every point the di erences between the true values ^ h (i; j ) and G ^ h (i; j ); are calculates as follows: R(i; j ) and G(i; j ); and the directional estimates R ~ h (i; j ) = G(i; j ) g;r and ~ h (i; j ) = G ^ h (i; j ) g;r R(i; j ); (i; j ) 2 XR ; ^ v (i; j ); (i; j ) 2 XG ; R 2 R(i; j ); (i; j ) 2 XR : for the horizontal direction. For the vertical direction the analogous computations are: ~v g;r (i; j ) = G(i; j ) and ~v ^ g;r (i; j ) = Gv (i; j ) ^ h (i; j ); (i; j ) 2 XG ; R 1

We assume for further ltering that these di erences between the intensities of di erent color channels can be presented as the sums of the true values and the random errors: ~h g;r (i; j ) ~v g;r (i; j ) = =

g;r (i; j ) g;r (i; j )

+ "h (i; j ); + "v (i; j );

g;r (i; j )

(7) (8) is the true di erence between

where "h (i; j ) and "v (i; j ) are considered as random demosaicing noise11 ; green and red color channels:

The blue channel B is treated in the same way and as a result we calculate the directional di erences ~ h g;b (i; j ) v ~ and g;b (i; j ).

4. LPA-ICI FILTERING OF DIRECTIONAL DIFFERENCES

~v ~ h (i; j ), ~ v (i; j ) for The LPA-ICI ltering33 ; 34 is used for all noisy estimates ~ h g;r (i; j ), g;r (i; j ) for R, and g;b g;b B . In order to introduce this ltering in the form applicable for any input data assume for a moment that this input noisy data have the form: z (i; j ) = y (i; j ) + n(i; j ); (9) where (i; j ) 2 X; z (i; j ) is a noisy observation, y (i; j ) is a true signal and n(i; j ) is a noise.

The LPA is a general tool for linear lter design, in particular for design of the directional lters of the given orders on the arguments i and j . Let gs; be the impulse response of the 2D directional linear lter designed by the LPA34 ; where is a direction of smoothing and s is a scale parameter (window size of the lter) . A set of the image estimates of di erent scales s and di erent directions are calculated by the convolution y bs; (i; j ) = (z ~ gs; )(i; j ), 2 . (10)

The ICI rule is the algorithm for a proper selection of the scale (close to the optimal value) for every pixel (i; j )34 : In the ICI rule a sequence of con dence intervals is used Ds = y bs; (i; j )

y ^s;

for s 2 S = fs1 ; s2 ; :::; sJ g, where s1 < s2 < ::: < sJ , and

The MATLAB code that implements the LPA-ICI techique is available following the link: http://www.cs.tut.fi/ lasip/.

;y bs; (i; j ) +

y ^s;

; s 2 S;

(11)

where y bs; .

> 0 is a threshold parameter for the ICI, the estimates and

y ^s;

is the standard deviation of the estimate

where the weights are de ned by gs; used in (10). The rotated directional nonsymmetric kernel gs; is used with the angle which de nes the directionality of the lter, and scale s is a length of the kernel support (or a scale parameter of the kernel) in this direction. q ( Note that in the usual form of the ICI34 the standard deviation of the estimate is calculated as

2 y ^s;

In this paper this standard deviation for (11) is calculated as the weighted mean of the squared errors between the estimates and the observations in the directional neighborhood of the pixel (i; j ) : q 2 )(i; j ); y ^s; )2 ~ gs; (12) y ^s; (i; j ) = ((z

(i; j ) =

~

2 gs;

)(i; j ); where

is a given standard deviation of the additive signal-independent observation noise in

the model (9). However, in the considered model (7)-(8) we deal with the data where the noise is a convenient form for modelling of the interpolation errors that are actually nonrandom. Thus, the standard deviation of the estimate y bs; is estimated locally by (12) at every position (i; j ) as an empirical sample mean calculated over the directional local area. The ICI rule de nes the adaptive scale as the largest s+ of those scales in S which estimate does not di er signi cantly from the estimates corresponding to the smaller window sizes. This optimization of s for each of the directional estimates yields the adaptive scales s+ ( ) for each direction . The union of the supports of gs+ ( ); is considered as an approximation of the best local vicinity of (i; j ) in which the estimation model ts the data. The nal estimate is calculated as a linear combination of the obtained adaptive directional estimates y bs+ ; (i; j ) : The nal LPA-ICI estimate y ^(i; j ) combined from the directional ones is computed as the weighted mean y ^(i; j ) = X y ^s+ ; (i; j )w ; w = P

2 y ^ 2 y ^s+ ; 2 2 2 y ^s+ ; 2 y ^s+ ( 1

);

(13)

with the variance

2 y ^

of y ^(i; j ) computed for simplicity as

=

It is convenient to treat this complex LPA-ICI multidirectional algorithm as an adaptive lter with the input z and the output y ^. The input-output equation can be written as y ^ = LI fz g by denoting the calculations imbedded in this algorithm as an LI operator. ~v i; j ) we obtain the corresponding spatially adapApplying the ICI in the form (11), (12) to ~ h g;r (n g;r (i; j ) and o n o h ^ ^ v (i; j ) = LI ~ v (i; j ) : tive estimates. Let us denote these estimates as g;r (i; j ) = LI ~ h ( i; j ) and g;r g;r g;r Combining these vertical and horizontal estimates we arrive to the nal estimate ^ g;r (i; j ) in the form ^ g;r (i; j ) =

2 ^h 2 ^h g;r

g;r

P

2

.

+

2 ^v g;r

^h g;r (i; j ) +

2 ^v 2 ^h g;r

g;r

+

2 ^v g;r

^v g;r (i; j );

(14)

where

^h

g;r

and

^v

g;r

^v are the corresponding standard deviations of ^ h g;r and g;r :

Note that in order to obtain the estimates for ^ h g;r (i; j ) we use only two directions corresponding to the directions of interpolation = f0; g (for horizontal left and right estimates), S = f4; 6; 8; 12g. For vertical estimates ^ v = f =2; 3 =2g g;r (i; j ) we also use two directions corresponding to the directions of interpolation (for vertical up and down estimates): ~ v (i; j ) between G and B color Similar adaptive LPA-ICI ltering is applied for the di erences ~ h g;b (i; j ), g;b channels in order to obtain the estimate ^ g;b (i; j ):

Figure 2. Fragment of the Lighthouse (19) test image (from left to right and from top to bottom): True image; HamiltonAdams interpolation5 PSNR=(36.67 38.34 37.09); Successive Approximation8 PSNR=(38.67 42.07 40.17); Alternating Projections2 PSNR=(38.70 42.12 39.80); DLMMSE based interpolation11 PSNR=(39.83 42.77 40.93); Proposed LPA-ICI based interpolation PSNR=(40.34 43.51 41.56).

5. INTERPOLATION OF G COLOR

The interpolated green color at R ((i; j ) 2 XR ) or B ((i; j ) 2 XB ) positions is calculated as follows: ^ (i; j ) = R(i; j ) + ^ g;r (i; j ); (i; j ) 2 XR ; G ^ (i; j ) = B (i; j ) + ^ g;b (i; j ); (i; j ) 2 XB ; G where ^ g;r is the estimate of G R, and ^ g;b is the estimate of G (11)-(14) described in the previous section. B; obtained using the LPA-ICI technique

6. INTERPOLATION OF R/B COLORS AT B /R POSITIONS

For the interpolation of R=B colors at B=R positions we propose to use a special shift invariant interpolation lter giving the estimates by the standard convolution. This lter has been designed using the LPA for the subsampled grid which corresponds to R/B channel (Fig.1). A variety of polynomial orders and support sizes have been tested. Finally, the second order polynomial interpolation lter grb has been chosen 2 3 0 0 0:0313 0 0:0313 0 0 6 7 0 0 0 0 0 0 0 6 7 6 0:0313 0 0:3125 0 0:3125 0 0:0313 7 6 7 7: 0 0 0 0 0 0 0 grb = 6 (15) 6 7 6 0:0313 0 0:3125 0 0:3125 0 7 0 : 0313 6 7 4 5 0 0 0 0 0 0 0 0 0 0:0313 0 0:0313 0 0

Figure 3. Mean values of PNSR for the Kodak test set of 24 images. The following techniques are compared: HA5 ; LI9 ; SA8 ; HD10 ; AP2 ; CCA15 ; CCA+PP is a demosaicing approach15 with postprocessing7 ; DLMMSE based interpolation11 ; proposed LPA-ICI interpolation; "Oracle " is the LPA-ICI interpolation with the optimal threshold parameter .

Then, the interpolated estimates are computed as follows: ^ (i; j ) = G ^ (i; j ) R ^ (i; j ) = G ^ (i; j ) B where ^ g;r is the estimate of G ( ^ g;r ~ grb )(i; j ); (i; j ) 2 XB , ( ^ g;b ~ grb )(i; j ); (i; j ) 2 XR , B: (16) (17)

R, and ^ g;b is the estimate of G

7. INTERPOLATION OF R/B COLORS AT G POSITIONS

For interpolation of R/B colors at G positions ((i; j ) 2 XG1 [ XG2 ) we use the simplest rst order interpolation kernel g : 3 2 0 0:25 0 0 0:25 5 ; g = 4 0:25 0 0:25 0 because the higher order interpolation does not provide signi cant improvement in performance. Then, the interpolated estimates are computed as follows: ^ (i; j ) = G(i; j ) R ^ (i; j ) = G(i; j ) B (( ^ g;r ~ grb ) ~ g )(i; j ); (( ^ g;b ~ grb ) ~ g )(i; j ):

(18) (19)

8. RESULTS

The proposed LPA-ICI based CFAI is tested on the Kodak set of color test-images. The numerical results are summarized in Tables 1,2 for each of 24 images and ordered in the ascending order of mean PSNR values (the last row). The diagram that illustrates mean values of PSNR for each color channel is shown in Fig. 3 for the better visual perception. The PSNR criterion is calculated excluding 15 border pixels in order to eliminate boundary e ects. The threshold in (11) is an important design parameter of the ICI rule and of the algorithm overall. When is small the ICI selects only the estimates with the smallest scale s; while when is large the ICI selects only the estimates with the largest scale s: The best selection of for each image can be found if the image is known. We call these values of "Oracles". They show the potential of the developed adaptive algorithm provided the best selection of . The corresponding PSNR values are given in the column "Oracle " of Tables 1,2. It is clearly seen that these oracle results are signi cantly better then results for all other methods.

Figure 4. Di erence between initial horizontal G and R estimates (G (G R) (right).

R) (left) and LPA-ICI scales ( =0) of the ltered

We found an empirical formula giving the image dependent with the values close to the oracle ones. Let be standard deviation of high frequency components of G channel calculated as Median Absolute Deviation (MAD).36 Then nearly oracle values of the threshold parameter can be calculated as = 0:05 f + 0:33. The results with this value of are given in the "LPA-ICI" column of Tables 1,2.

f

It is clearly seen that the proposed technique ("LPA-ICI" column) gives about 0.4 dB better mean PSNR value (the last row of Table 2) than DLMMSE11 that shows the best performance among the reviewed CFAI methods. Analyzing the diagram in Fig. 3 it can be seen that this improvement is signi cant. Fig. 4a illustrates the di erences ~ h g;r between horizontal estimates of green and red color channels obtained for Lighthouse test-image (image number 19 in Table 2). The adaptive scales s for this image (direction = 0) are shown in Fig. 4b. It is seen that near vertical details the ltering performed horizontally selects smaller scales allowing to avoid oversmoothing of details. Our study shows that the "demosaicing noise" is not white and strongly localized. At di erent parts of an image the power of noise is di erent. It justi es the use of the local estimates of the variance in (12). As a result, suppression of color distortions becomes much better in terms of both numerical and visual evaluation. Visual comparison of di erent methods is presented in Fig. 2 for the fragment of Lighthouse image. The color artefacts are removed almost completely by the proposed method (Fig. 2 bottom right image) what is signi cantly better than it is done by other methods.

9. ACKNOWLEDGMENTS

This work was supported by the Finnish Funding Agency for Technology and Innovation (Tekes). The authors thank Dr. Lei Zhang for providing the implementation code of the technique11 ; and Dr. Alessandro Foi for useful and practical discussions.

REFERENCES

1. B.E. Bayer, \Color imaging array," U.S. Patent 3 971 065, July 1976. 2. B.K. Gunturk, Y. Altunbasak, R.M. Mersereau, "Color plane interpolation using alternating projections", IEEE Transactions on Image Processing, Volume 11, Issue 9, Page(s):997 - 1013, Sept. 2002. 3. J.E. Adams Jr., \Design of color lter array interpolation algorithms for digital cameras, Part 2," in IEEE Proc. Int. Conf. Image Processing, vol. 1, Oct. 1998, pp. 488{492. 4. C.A. Laroche and M.A. Prescott, "Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients", U.S. Patent 5 373 322, Dec. 1994.

HA Red 01 Green Blue Red 02 Green Blue Red 03 Green Blue Red 04 Green Blue Red 05 Green Blue Red 06 Green Blue Red 07 Green Blue Red 08 Green Blue Red 09 Green Blue Red 10 Green Blue Red 11 Green Blue Red 12 Green Blue

LI

HD

SA

AP

CCA

CCA+PP

DLMMSE

LPA-ICI

Oracle

33.17 34.65 33.29 37.38 40.94 39.60 40.21 42.19 39.79 36.58 40.66 39.64 34.65 35.81 34.27 34.66 35.93 34.24 40.64 42.17 40.25 31.57 33.37 31.55 39.38 41.39 40.23 39.12 41.33 39.49 35.50 37.03 35.87 39.89 42.27 40.21

30.87 35.61 30.98 36.54 41.35 37.32 39.22 43.15 38.37 36.73 42.32 38.91 32.60 36.75 32.25 32.49 36.91 31.98 38.77 42.16 38.51 28.05 32.96 27.83 36.78 41.40 37.39 37.64 42.27 37.69 33.71 37.90 33.65 37.44 42.41 37.74

34.51 36.13 34.74 36.83 41.61 40.44 40.39 43.56 40.03 35.85 41.47 40.74 34.95 37.27 34.27 37.35 38.88 36.49 39.99 42.64 39.37 33.06 35.19 33.11 40.12 42.83 40.45 39.43 42.80 39.89 36.44 38.76 37.25 40.45 43.82 41.11

36.99 40.76 38.77 35.50 40.57 40.05 39.19 41.00 38.84 35.25 41.63 41.91 34.60 36.78 34.26 39.02 42.22 38.00 39.25 41.22 38.75 34.55 38.45 35.28 39.72 42.02 40.90 40.25 43.91 41.45 37.43 41.20 39.03 40.89 44.38 42.21

36.69 40.42 37.26 37.29 42.46 40.84 40.94 43.52 40.34 36.87 43.81 42.33 36.87 39.69 36.06 38.22 41.48 37.35 41.25 43.96 40.69 34.56 38.55 34.67 40.64 43.42 41.90 40.78 44.38 41.41 37.96 41.64 38.92 41.41 45.22 42.09

36.22 39.20 36.72 37.76 43.81 41.35 41.58 45.18 41.20 37.40 44.55 42.46 37.12 40.02 36.42 36.86 39.86 36.37 41.87 45.31 41.32 33.34 36.93 33.46 40.99 43.94 40.92 40.83 44.42 40.96 37.71 41.25 38.70 40.97 44.74 41.31

37.25 41.31 38.29 36.52 43.27 41.06 39.90 43.78 40.30 35.92 44.26 42.32 36.10 39.61 35.72 38.12 41.68 37.43 40.07 43.97 39.91 34.44 38.63 34.82 40.29 43.90 40.91 39.94 44.41 40.67 37.69 42.15 39.32 40.45 44.82 41.46

37.58 40.22 38.02 38.19 44.31 42.54 41.95 45.80 41.40 37.21 44.68 43.61 37.62 40.88 36.71 40.13 42.33 38.80 41.83 45.27 41.01 35.08 38.53 35.23 41.69 45.14 43.00 41.19 45.37 42.08 38.75 42.03 39.87 42.09 46.30 42.98

39.65 42.80 40.15 38.59 44.59 42.78 42.92 46.00 42.29 37.37 44.09 43.19 36.26 39.26 35.67 40.90 43.71 39.33 42.59 45.48 41.74 36.12 40.07 36.34 42.27 45.36 43.22 41.45 45.34 41.99 39.03 42.81 40.19 42.82 46.62 43.38

39.88 43.08 40.48 38.72 44.64 42.81 42.91 46.03 42.30 37.87 44.55 43.38 38.06 41.05 37.17 40.93 43.76 39.33 42.68 45.52 41.86 36.08 40.07 36.32 42.30 45.42 43.31 41.76 45.53 42.19 38.97 42.81 40.16 42.87 46.64 43.41

...

...

...

...

...

...

...

...

...

...

...

...

...

Table 1. PSNR comparison for di erent demosaicing methods computed within 15 pixels border: HA5 ; LI9 ; HD10 ; SA8 ; AP2 ; CCA15 ; CCA+PP is a demosaicing approach15 with postprocessing7 ; DLMMSE based interpolation11 ; proposed LPA-ICI based interpolationin; "Oracle " is the LPA-ICI based interpolation with the optimal threshold parameter .

5. J.F. Hamilton Jr. and J.E. Adams, "Adaptive Color plane Interpolation in single color electronic camera", U.S. Patent 5 629 734, May 1997. 6. R. Kimmel, "Demosaicing: image reconstruction from color CCD samples", IEEE Transactions on Image Processing, Volume 8, Issue 9, Page(s):1221 - 1228, Sept. 1999. 7. R. Lukac, Martin K., Plataniotis K.N., "Demosaicked Image Postprocessing Using Local Color Ratios", IEEE Transactions on Circuits and Systems for Video Technology, Vol. 14, No. 6, pp. 914-920, June 2004. 8. Xin Li, "Demosaicing by successive approximation", IEEE Transactions on Image Processing, Volume 14, Issue 3, Page(s):370 - 379, March 2005.

HA Red 13 Green Blue Red 14 Green Blue Red 15 Green Blue Red 16 Green Blue Red 17 Green Blue Red 18 Green Blue Red 19 Green Blue Red 20 Green Blue Red 21 Green Blue Red 22 Green Blue Red 23 Green Blue Red 24 Green Blue Red Mean PSNR Green Blue

LI

HD

SA

AP

CCA

CCA+PP

DLMMSE

LPA-ICI

Oracle

29.53 30.58 29.10 34.81 37.14 35.02 36.08 39.51 37.96 38.09 39.55 37.88 38.41 39.19 37.74 33.91 35.24 33.75 36.67 38.34 37.09 38.75 39.82 37.31 35.04 36.29 34.48 35.80 37.72 35.60 41.06 43.30 41.54 32.63 33.59 30.76 36.40 38.25 36.53

28.86 32.53 28.36 33.74 37.72 33.31 36.50 41.56 37.57 35.44 40.01 35.30 36.95 40.38 36.67 33.40 36.92 33.00 32.61 37.24 32.64 36.98 40.57 35.72 33.40 37.47 32.74 34.90 38.44 33.84 39.83 43.71 40.27 32.24 35.61 30.47 35.82 39.06 34.69

31.32 32.18 30.43 33.82 37.73 34.62 35.69 40.62 38.82 41.20 42.66 40.41 38.93 40.36 38.32 34.21 36.12 34.26 37.60 39.51 37.86 39.29 40.85 37.55 36.46 37.76 35.41 35.41 38.22 35.48 40.31 43.92 41.15 32.70 35.15 31.90 36.93 39.58 37.26

36.00 38.38 34.05 31.59 34.98 32.64 35.68 40.59 39.78 42.11 45.46 41.08 40.88 43.17 40.52 35.32 38.36 36.41 38.67 42.07 40.17 40.54 42.79 38.12 38.86 41.92 37.50 36.53 38.69 36.18 38.50 41.44 39.51 34.70 37.38 33.03 37.58 40.81 38.27

34.04 36.83 32.86 34.57 38.04 35.11 36.79 42.29 40.22 41.56 44.82 40.85 40.79 43.03 40.31 36.23 39.46 36.86 38.70 42.12 39.80 40.96 43.50 38.71 38.47 41.57 37.19 37.03 39.72 36.71 40.76 44.03 41.61 34.94 37.49 32.93 38.26 41.73 38.63

34.07 36.02 32.97 35.68 40.25 36.42 36.95 42.79 40.59 39.73 42.87 39.49 40.77 42.94 39.92 36.56 39.42 36.71 37.90 41.08 38.16 41.17 43.76 39.24 38.31 40.90 37.16 37.22 40.58 37.13 41.48 45.60 42.40 34.50 37.45 32.95 38.21 41.79 38.51

35.84 38.13 34.00 33.77 39.00 35.11 36.10 42.32 40.62 41.14 44.60 40.49 40.06 43.22 39.80 35.90 39.73 36.88 38.41 42.25 38.85 40.66 43.87 38.98 39.14 42.32 37.70 36.06 39.88 36.55 39.25 44.03 41.02 33.53 37.41 32.82 37.77 42.02 38.54

34.98 36.09 33.56 35.53 40.28 36.25 37.22 43.23 41.24 43.60 45.75 42.49 41.38 43.15 40.83 36.69 39.41 37.27 39.83 42.77 40.93 41.80 43.86 39.27 39.14 41.22 37.65 37.60 40.86 37.42 41.78 46.24 42.78 35.94 38.01 33.74 39.11 42.57 39.53

36.39 38.13 34.27 34.87 38.98 35.70 37.55 42.94 41.10 44.06 46.51 42.86 41.28 43.40 40.72 36.27 39.30 36.90 40.34 43.51 41.56 41.94 44.10 39.67 39.93 42.31 38.12 37.41 40.53 37.27 42.69 46.28 43.25 35.38 38.05 33.55 39.51 42.93 39.80

36.48 38.14 34.37 36.53 40.55 37.03 37.74 43.13 41.18 44.23 46.83 42.84 41.38 43.42 40.81 36.75 39.54 37.23 40.32 43.53 41.57 41.97 44.27 39.68 40.00 42.45 38.09 37.73 40.91 37.67 42.78 46.36 43.39 35.67 38.09 33.65 39.77 43.18 40.01

Table 2. PSNR comparison for di erent demosaicing methods computed within 15 pixels border: HA5 ; LI9 ; HD10 ; SA8 ; AP2 ; CCA15 ; CCA+PP is a demosaicing approach15 with postprocessing7 ; DLMMSE based interpolation11 ; proposed LPA-ICI based interpolationin; "Oracle " is the LPA-ICI based interpolation with the optimal threshold parameter .

9. H.S. Malvar, Li-wei He, and R. Cutler, "High-quality linear interpolation for demosaicing of Bayer-patterned color images", IEEE International Conference (ICASSP '04), Proceedings on Acoustics, Speech, and Signal Processing, Volume 3, Page(s):iii - 485-8, 17-21 May 2004. 10. K. Hirakawa, T.W. Parks, "Adaptive homogeneity-directed demosaicing algorithm", IEEE Transactions on Image Processing, Volume 14, Issue 3, Page(s):360 - 369, March 2005. 11. L. Zhang, X. Wu, "Color Demosaicking Via Directional Linear Minimum Mean Square-Error Estimation",

IEEE Transactions on Image Processing, Vol. 14, No. 12, pp. 2167-2178, December 2005. 12. J.W. Glotzbach, R.W. Schafer, and K. Illgner, \A method of color lter array interpolation with alias cancellation properties," in Proc. Int. Conf. Image Processing, vol. 1, Oct. 2001, pp. 141{144. 13. X. Wu, N. Zhang, "Primary-consistent soft-decision color demosaicking for digital cameras (patent pending)", IEEE Transactions on Image Processing, Volume 13, Issue 9, Page(s):1263 - 1274, Sept. 2004. 14. D.D. Muresan, T.W. Parks, "Demosaicing Using Optimal Recovery", IEEE Transactions on Image Processing, Vol. 14, No. 2, pp. 267-278, February, 2005. 15. R. Lukac, Plataniotis K.N., Hatzinakos D., Aleksic M., "A Novel Cost E ective Demosaicing Approach", IEEE Transactions on Consumer Electronics, Vol. 50, No. 1, pp. 256-261, February 2004. 16. E. Chang, S. Cheung and D. Y. Pan, "Color lter array recovery using a threshold-based variable number of gradients", Proceedings of SPIE, vol. 3650, pp. 36-43, 1999. 17. R. Lukac, K.N. Plataniotis, "Digital Camera Zooming on Colour Filter Array", Electronics Letters, Vol. 39, No. 25, December 2003. 18. Xiaomeng Wang, Weisi Lin, Ping Xue, "Edge-Adaptive Color Reconstruciton for Single-Sensor Digital Cameras", ICICS-PCM 2003 Singapore, pp. 272-276, December, 2003. 19. Wenmiao Lu, Yap-Peng Tan, "Color Filter Array Demosaicking: New Method and Performance Measures", IEEE Transactions on Image Processing, Vol. 12, No. 10, pp. 1194-1210, October, 2003. 20. X. Wu, W. K. Choi, and P. Bao, "Color restoration from digital camera data by pattern matching", Proc. SPIE, vol. 3018, pp. 12{17, 1997. 21. D. Taubman, "Generalized Weiner reconstruction of images from colour sensor data using a scale invariant prior", Proc Int. Conf. Image Proc., pp. 801-804, 2000. 22. S.-C. Pei and I.-K. Tam, "E ective color interpolation in CCD color lter arrays using signal correlation", IEEE Trans. Circuits Systems Video Technol., vol. 13, no. 6, pp. 503{513, Jun. 2003. 23. Gunturk B.K., Glotzbach J., Altunbasak Y., Schafer R., Mersereau R.M., "Demosaicking: Color Filter Array Interpolation", IEEE Signal Processing Magazine, pp. 44-54, January 2005. 24. D. Alleysson, S. Susstrunk, J. Marguier, "Linear Demosaicing Inspired by the Human Visual System", IEEE Transactions on Image Procesing, Vol. 14, No. 4, April 2005. 25. D. Alleysson, S. Susstrunk, J. Herault, "Color demosaicing by estimating luminance and opponent chromatic signals in the Fourier domain", in Proc. IS&T/SID 10th Color Imaging Conf., pp. 331{336, Nov. 2002. 26. D. R. Cok, "Reconstruction of CCD Images Using Template Matching", IS&T's 47th Annual Conference/ICPS, pp380-385, 1994. 27. H.J. Trussel, E. Saber, M. Vrhel, "Color Image Processing: Vector Filtering for Color Imaging", IEEE Signal Processing Magazine, pp. 14-22, January 2005. 28. D. Keren, M. Osadchy, "Restoring Subsampled Color Images", Machine Vision and Applications, pp. 197202, 1999. 29. D.H. Brainard, "Bayesian method for reconstructing color images from trichromatic samples", ISn&T's 47th Annual Conference/ICSP, pp 375-380, 1994. 30. R. Lukac, K.N. Plataniotis, "An E cient CFA Interpolation Solution, 46th International Symposium Electronics in Marine", ELMAR-2004, Zadar, Croatioa, June, 2004. 31. R. Lukac, B. Smolka, K. Martin, K.N. Plataniotis, A.N. Venetsanopulos, "Vector Filtering for Color Imaging", IEEE Signal Processing Magazine, pp. 74-86, January 2005. 32. V. Katkovnik, "A new method for varying adaptive bandwidth selection", IEEE Trans. on Signal Proc., vol. 47, no. 9, pp. 2567-2571, 1999. 33. V. Katkovnik, K. Egiazarian, and J. Astola, "Adaptive window size image de-noising based on intersection of con dence intervals (ICI) rule", J. of Math. Imaging and Vision, vol. 16, no. 3, pp. 223-235, 2002. 34. V. Katkovnik, K. Egiazarian, and J. Astola, Local Approximation Techniques in Signal and Image Processing, SPIE Press, Monograph Vol. PM157, Hardcover, 576 pages, ISBN 0-8194-6092-3, September 2006. 35. Paliy D., M. Trimeche, V. Katkovnik, S. Alenius, "Demosaicing of Noisy Data: Spatially Adaptive Approach", Proc. SPIE Electronic Imaging 2007, Computational Imaging IV, 6497-20, San Jose, CA, January 2007. 36. D.L. Donoho, "De-noising by soft-thresholding," IEEE Tans. Inform. Theory, vol. 41, no. 3, pp. 613-627, May 1995.