PROJECTION BASED IMAGE AND VIDEO INPAINTING USING WAVELETS Kedar A.

Patwardhan and Guillermo Sapiro Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN-55455, USA. kedar@ece.umn.edu guille@ece.umn.edu
ABSTRACT In this paper, we present a technique for automatic color image inpainting, the art of modifying an image-region in a non-detectable form. The main algorithm is based on the theory of projections onto convex sets (POCS). The image and its wavelet transform are projected onto each other after applying suitable constraints in each domain. This technique exploits the frequency-spatial representation provided by wavelets and utilizes the correlation between the damaged area in the image and its neighborhood. The resulting restored area is homogeneous with its surrounding and preserves the aesthetics of the image. The same technique is used for simple video restoration problems. Video frames are stacked and treated as a 3-D volume, making a natural use of inter-frame correlation. 1. INTRODUCTION In recent years there has been a great amount of use of the visual media for commercial and entertainment purposes. This has lead to an increasing emphasis on restoration of old lm archives and photographs. With wireless networking and data exchange becoming the in-thing today, the need for an automatic and fast technique to restore image blocks lost during transmission, is becoming more and more hard-felt. Film post-production, also involves a lot of image inpainting for removal of artifacts and to generate stunning visual effects. This task is conventionally performed manually, requiring time and skill. These problems have motivated us to search for automatic techniques for still-image and video inpainting. Many useful techniques to address the afore mentioned scenarios have been proposed in recent years, e.g., [1, 2, 3, 4, 5, 6, 11, 12, 13, 14, 15, 17]. The success of these techniques varies according to the type of image and size and characteristics of the lost data. These methods, with the exceptions of [5, 17] and texture synthesis algorithms such as [8, 7, 10, 18, 19], run into trouble when employed on textured images, and only [5] explicitly deals with parts of textured images containing prominent edges. The main inspiration behind our paper has been the technique employed by Hirani and Totsuka [11] for removal of wires and scratches from still images. It is a hierarchical and prototype based interactive application of the POCS theory [20]. The disadvantage of this technique, though, is the fact that it requires a human operator to select the prototype window, which is demanding in terms of time and skill required. Till now this algorithm has had demonstrable success in case of thin and elongated wires and scratches on still-images. In this work, we combine the POCS ideas of [11] with those in the previously mentioned inpainting and texture synthesis works, and instead of the user identifying the prototype window, we get the required information from nearby pixels. All this is done in the Wavelet domain and not in Fourier domain as in [11], thereby improving the handling of texture.

2. ALGORITHM In this paper, we try to exploit the multi-resolution property of wavelets, whereby, we can dispense with the human operator required in case of [11]. This algorithm also works for larger lost blocks. In the technique proposed by Hirani and Totsuka [11], a human operator takes a look at the neighborhood of the lost block and decides upon the best matching neighborhood which later acts as the prototype window. In our algorithm, wavelets perform this very task, automatically and considerably fast. The basic assumption when restoring any lost image block is that the lost block contained a continuation of data that surrounds it. Wavelets, make it easy to determine the frequency content of the neighborhood. We put frequency domain constraints on the lost block, based upon the frequency content of its neighborhood in the wavelet domain. The spatial domain constraints help us to get a valid image from the inverse wavelet transform. In addition, we also make sure that there is no discontinuity at the border of the newly inpainted area. c IEEE 2003

To appear in Proc. ICIP2003, Sept 14-17, 2003, Barcelona, Spain

Our algorithm is based on Projection Onto Convex Sets (POCS). POCS theory was introduced by Papoulis and Gerchberg [9, 16] to reconstruct band-limited signals by alternate projections onto transform and signal domains and applying constraints in each domain. In a similar fashion, our algorithm iteratively converges to an image which closely resembles the original and preserves its aesthetical characteristics. The new pixels that are generated using the POCS technique are: (a) as sharp as the surrounding area, (b) maintain continuity of prominent edges running across the scratch and, (c) have a texture matching the surrounding texture. The process of projection onto image and wavelet domains and consequent application of constraints can be summarized as : In+1 = Ci Pwi Cw Piw (In ) where, In+1 : Inpainted window after n + 1 iterations. In : Current repair window. Piw : Projection from image domain onto wavelet domain (the wavelet transform). Cw : Constraints in the wavelet domain. Pwi : Projection from the wavelet domain onto image domain (inverse wavelet transform). Ci : Constraints in the image domain to get pixel values between 0 and 1. The resolution of the image dictates the number of wavelet decomposition levels that can be used. The wavelet approximation in the last level of the decomposition is considered to be the DC level and is left unaltered, in order to maintain the overall brightness of the image. For every wavelet coefcient corresponding to the lost or damaged pixel, we nd the maximum and minimum of the wavelet coefcients in the known neighborhood. The coefcient corresponding to the lost or damaged pixel is then constrained to be between these maximum and minimum values. We call this the constraint Cw . This constrained wavelet decomposition is then projected onto the image domain (Pwi ). The known pixels in the neighborhood are restored to their original value and the newly grown pixels in the region of inpainting are constrained (Ci ) to get a valid image (In+1 ). The constraints in this algorithm preserve known pixel values and enforce band-limits in the wavelet domain, based upon the coefcients in the neighborhood. This can be thought of as making the least possible change to the image in the signal and transform domain such that it satises the above mentioned criteria (a),(b) and (c). 2 (1)

3. IMPLEMENTATION AND RESULTS The results shown, have been generated using Wavelet Toolbox in MATLAB. The algorithm is applied separately on each color channel and the results are combined to get a color image. It should be noted that this does not result in appearance of previously non-existent colors in the image. The algorithm usually converges in less than 25 iterations. On a Pentium II, 400 MHz machine, it takes less than 5 minutes to get the inpainted color image. We have used the simple Daubechies One (db1) wavelet for our implementation. Note that in order to improve the results on textured areas, other decompositions can be used, e.g., the steerable pyramid [10]. The image on the left in Fig.1, shows some objects placed on a pool table, while the image on the right shows the inpainted image with the pouring arm of the teapot and the red ball removed. Notice how the curve in the shape of the tea-pot is preserved, and the texture generated in place of the red ball is homogeneous with its surrounding. In Fig.2, the occupants of the boat and the boy on the shore are removed. Fig.3 illustrates the behavior of our algorithm on random textures. For inpainting a video sequence, we select the object or artifact to be removed in each frame and then stack the frames as a 3-D volume. The damaged areas in each frame are then considered to be a hole in the 3-D volume. We then apply our algorithm, and use the 3-D Wavelet Transform and its inverse for the projections onto transform and image domains respectively. The results for video inpainting can be viewed at : www.ece.umn.edu/users/kedar/icip2003/icip results.htm.

4. CONCLUSION AND FUTURE SCOPE We have demonstrated the effectiveness of using POCS theory in conjunction with wavelets for image restoration. A particular wavelet basis and a projection constraint has been used to illustrate the ideas. The results can be signicantly improved using, for example, steerable pyramids and vectorial constraints. This algorithm is automatic and considerably fast. It works well with color images and on relatively large areas of data loss. This algorithm can be used for structure as well as texture inpainting, provided, both can be characterized in the wavelet domain. The speed of this algorithm makes it a possible candidate for restoring images corrupted due to transmission over fading channels. An important application of this algorithm can be in lm post production for removing artifacts. It can also be used for restoring old lms and photographs. We have got encouraging results when implementing the same algorithm for video inpainting using 3-D wavelets. In conjunction with optical ow data, our algorithm could be a promising technique for video inpainting.

Acknowledgments
This work was partially supported by the Ofce of Naval Research, the Presidential Early Career Awards for Scientists and Engineers (PECASE), and a National Science Foundation CAREER Award. GS thanks Prof. Marcelo Bertalmio and Prof. Vicent Caselles, with whom all this work on image inpainting started. We also thank Shantanu Rane for his comments on this paper. 5. REFERENCES [1] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, Filling-in by joint interpolation of vector elds and grey levels, IEEE Trans. Image Processing 10, pp. 1200-1211, August 2001. [2] C. Ballester,V. Caselles, and J. Verdera, Dissoclusion by joint interpolation of vector elds and gray levels, pre-print, 2002. [3] M. Bertalmio, A. L. Bertozzi, and G. Sapiro, NavierStokes, uid dynamics, and image and video inpainting, Proc. IEEE Computer Vision and Pattern Recognition (CVPR), Hawaii, December 2001. [4] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, Image inpainting, Computer Graphics (SIGGRAPH 2000), pp. 417424, July 2000. [5] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher, Simultaneous texture and structure image inpainting, pre-print, July 2002 (mountains.ece.umn.edu/ guille/inpainting.htm). [6] T. Chan and J. Shen, Local inpainting models and TV inpainting, SIAM J. Appl. Math. 62:3, pp. 1019-1043, 2001. [7] J. S. De Bonet, Multiresolution sampling procedure for analysis and synthesis of texture images, Proceedings of ACM SIGGRAPH, July 1997. [8] A. A. Efros and T. K. Leung, Texture synthesis by non-parametric sampling, IEEE International Conference on Computer Vision, Corfu, Greece, pp. 1033 1038, Sept. 1999. [9] R. W. Gerchberg, Super-resolution through error energy reduction, Opt. Acta, 21, No.9, 709-720, 1974. [10] D. Heeger and J. Bergen, Pyramid based texture analysis/synthesis, Computer Graphics (SIGGRAPH 1995), pp. 229238, July 1995. 3

[11] A. Hirani and T. Totsuka, Combining frequency and spatial domain information for fast interactive image noise removal, Computer Graphics, pp. 269-276, SIGGRAPH 96, 1996. [12] A.C. Kokaram, R.D. Morris, W.J. Fitzgerald, and P.J.W. Rayner, Interpolation of missing data in image sequences, IEEE Transactions on Image Processing 11:4, pp. 1509-1519, 1995. [13] S. Masnou and J. Morel, Level-lines based disocclusion, IEEE Int. Conf. Image Processing, October 1998. [14] M. Nitzberg, D. Mumford, and T. Shiota, Filtering, Segmentation, and Depth, Springer-Verlag, Berlin, 1993. [15] J. M. Ogden, E. H. Adelson, J. R. Bergen, and P. J. Burt, Pyramid-based computer graphics, RCA Engineer 30(5), pp. 4-15, 1985 [16] A. Papoulis, A new algorithm in spectral analysis and bandlimited extrapolation, IEEE Trans. Cir. and Sys. 22:9, 1975. [17] S. Rane, G. Sapiro, and M. Bertalmio, Structure and texture lling-in of missing image blocks in wireless transmission and compression applications, IEEE Trans. Image Processing, to appear. [18] E. Simoncelli and J. Portilla, Texture characterization via joint statistics of wavelet coefcient magnitudes, Proc. 5th IEEE Intl Conf. on Image Processing, 1998. [19] L. Y. Wei and M. Levoy, Fast texture synthesis using tree-structured vector quantization, Computer Graphics (SIGGRAPH 2000), July 2000. [20] D. C. Youla and H. Webb, Image restoration by the method of convex projections - Part 1: Theory, IEEE Trans. Med. Imag. 1:2, pp. 81-94, 1982.

Fig. 1. Pouring arm of the tea-pot and the red ball from the image on the left have been removed using the POCS algorithm.

Fig. 2. The occupants of the boat and the boy on the shore have disappeared.

Fig. 3. The lost area in the grass-like texture is inpainted.