SICE-ICASE International Joint Conference 2006 Oct.

18-21, 2006 in Bexco, Busan, Korea

Importance of Developing Image Restoration Techniques for Security Cameras under Severe Conditions
Yusaku Fujii1, Tadashi Ito2, Naoya Ohta3, Saburou Saitoh4, Tsutomu Matsuura5 and Takao Yamamoto6
Department of Electronic Engineering, Faculty of Engineering, Gunma University (Email: fujii@el.gunma-u.ac.jp ) 2 Department of Electronic Engineering, Faculty of Engineering, Gunma University (Email: tad@el.gunma-u.ac.jp ) 3 Department of Computer Science, Faculty of Engineering, Gunma University (Email: ohta@cs.gunma-u.ac.jp ) 4 Department of Mathematics, Faculty of Engineering, Gunma University (Email: ssaitoh@math.sci.gunma-u.ac.jp ) 5 Department of Mechanical System Engineering, Faculty of Engineering, Gunma University (Email: matsuura@me.gunma-u.ac.jp ) 6 Department of Physics, Faculty of Engineering, Gunma University (Email: yamamoto@phys.sci.gunma-u.ac.jp ) Abstract: A concept, which was proposed and has been pursued by the authors for restoring the images of a suspected person taken by a security camera, is reviewed. In the concept, the peculiar facts concerning the security camera system that all the things in the image except the suspected person itself are usually preserved and that can be used for investigations are to be used as fully as possible. The authors have launched a project in which techniques based on the concept are developed for the police. In the project, some methods of restoring an image of a persons figure taken by a camera with dirty lens have been developed. The importance of pursuing the proposed concept is discussed from the viewpoints of not only the novelty in image processing technology but also the viewpoints of the efficiency in criminal investigation and social security. Keywords: criminal investigation, social security, image processing, security camera, image restoration, blur, Wiener filter.
1

1. INTRODUCTION
There is the peculiar problem concerning the security camera systems that quality of the taken images is sometimes realized to be very poor due to the lack of the maintenance when the pictures become to be necessary for the investigation by the police. Security camera systems are sometimes operated without appropriate maintenance and this fact causes a serious problem at the time of the police investigation. The first and third authors are now promoting a new concept the e-JIKEI Network for community security using widely available information technology with altruism [1,2], in which some residents keep watching on what happens around their houses using their own home computers, inexpensive cameras, and free software. In such case, the lack of maintenance should be much more serious than in the usual case. Many research activities have been made aiming at restoring poor quality images including images taken by security cameras (eg. [3]). Most of them deal with the problems of image blurs. A blur is modeled by a PSF (Point Spread Function) and the blurred images are restored by deconvolution. However, image blurs are sometimes not the only one problem; image patterns generated by oblique illumination, as discussed mainly in this report, can be a problem. In addition, when dealing with image blurs, most of the theories assume linearity of pixel values and a space invariant PSF,

which are doubtful for images taken by low quality security cameras. The methods proposed up to now do not seem to use all the information that could be potentially used under the condition when the crime is serious and it is very important to identify the criminal. The security camera system has the peculiar characteristics; that is, all the things in the image except the suspected person itself are usually preserved and can be used in the criminal investigations by the police. There is a study in which the PSF due to the dirty lens is measured by using the camera itself [4]. However, this study did neither consider factors other than the image blur, nor the non-linearity of pixel values. There is much other information that is available but is not used. For example, the effect of the light source out of the sight can be considered. The authors has launched a project for developing the image restoration techniques for the police, in which the peculiar facts concerning the security camera system that all the things in the image except the suspected person itself are usually preserved and that can be used for investigations are to be fully used. As the first trial in our project, a method for restoring an image of a persons face taken by a camera with dirty lens under strong oblique illumination was proposed and its significant effect was demonstrated experimentally [5, 6]. An improved method has been developed; in which the space variant point spread function (PSF) is determined using a bright spot on a liquid crystal

display [7]. Fig. 1 shows the procedure for restoring images taken by a camera with a dirty lens under strong oblique illumination [5,6]. A usual security camera was used for the experiment. The lens of the camera is covered by a semitransparent plastic sheet with silicon oil, which is used as the stain on the lens or the cover glass. A light source is placed outside the view of the camera, which is used as the source of the oblique illumination. The procedure is divided into two parts: Stage-A and Stage-B, as follows.

taken by a monochrome CCD camera. In addition, the importance of pursuing the proposed concept is discussed from the viewpoints of not only the novelty in image processing technology but also the efficiency in criminal investigation and social security.

2. COLOR ESTIMATION TECHNIQUE

Fig. 2 shows the method for estimating the color information of the suspected persons figure from images taken by a monochrome CCD camera. The method is based on the fact that the stain on the lens of the security camera sometimes functions as a space-variant color filter. In the following analysis, the brightness Br of the image of the point with the color, Color=[R, G, B] is supposed to be expressed as follows, Br = R + G + B = Color[, , ], where R, G and B are the original intensity of the color components of red, green and blue, respectively. The coefficients of , and are the transfer coefficients of the red color, green color and blue color, respectively. They are space-variant coefficients due to many causes, such as the stain on the lens, which functions as a space-variant color filter, the illumination conditions and the surface conditions of the materials.

Stage-A Stage-A is simulated the scene of the crime in which the image of the suspected person is taken. A person acts as the criminals for Stage-A. One picture is taken: Image-A. Stage-B Stage-B is simulated the scene of the criminal investigation made by the police, in which all the objects related the images taken by the camera are preserved and can be used for the investigation. A plate in black color is put around where the criminal is located in Stage-A. One picture is taken: Image-B. The technique has been proved very effective. The influence of the oblique illumination on the image can be removed by subtracting Image-B from Image-A [5,6]. In this paper, another technique based on our concept is proposed. Using the newly developed technique, there are cases, in which the color information of the suspected persons figure can be estimated from images

The procedure of the color estimation is divided into two parts: Stage-A and Stage-B, as follows. Stage-A: simulated scene with the suspected person Stage-A is simulated the scene of the crime in which the images of the suspected person are taken by a

monochrome CCD camera. A person acts as the criminal and walks in the whole area of the camera view. Some pictures are taken: Images-A. From the images, the frame-series (time-series) of the brightness and position in the images of the specific point of the suspected persons figure, AP(i)=(Brp,i, xp,i, yp,i), is measured, where i is the frame number, Brp,i is the brightness of the specific point in the specific frame, xp,i is the horizontal position of the specific point in the specific frame and yp,i is the vertical position of the specific point in the specific frame. Stage-B: simulated scene of the police investigation Stage-B is simulated the scene of the criminal investigation made by the police, in which all the objects related the images taken by the camera in Stage-A are preserved and can be used for the investigation. The three color samples, R(red), G(green), B(blue) plates, are moved as the suspected person moves in Stage-A. Then some pictures are taken by the same camera under the same condition as Stage-A: Images-B. In Stage-B, the brightness in the image of the three color plates, Brred(x, y), Brgreen(x, y), and Brblue(x, y) are measured. = ColorR[(x, y), (x, y), (x, y)] Brred(x, y) Brgreen(x, y) = ColorG[(x, y), (x, y), (x, y)] Brblue(x, y) = ColorB[(x, y), (x, y), (x, y)] where ColorR, ColorG and ColorB are the colors of the R(red) plate, G(green) plate and B(blue) plate, respectively. For example, when the color of the R(red) plate is pure red, then ColorR is expressed using the R G B components as follows, ColorR = [1, 0, 0]. Then the space-variant brightness transfer function for the three primary colors, red, green and blue, (x, y), (x, y) and (x, y) are calculated. Stage-C: estimation of the color Using the results obtained from Stage-A and Stage-B, the following objective function, C, is to be minimized, C= [ Bp,i (a Brred (xp,i, yp,i) + b Brgreen(xp,i, yp,i) + c Brblue(xp,i, yp,i))]2 Then the color of the specific point of the suspected persons figure, Colorp, is estimated to be as follows, Colorp = a ColorR + b ColorG + c ColorR

inserted in its left side. The distributions of the brightness along the three vertical sections, Section-Left, Section-Center and Section-Right are used for the analysis. From the figure, the brightness of Section-Left is darker than those of Section-Center and Section-Right. This is thought to be the effect of the blue glass inserted.

Fig. 5 shows the distributions of the brightness along the three vertical sections, Section-Left, Section-Center and Section-Right.

3. EXPERIMENT
To demonstrate the method, a simple experiment is conducted. Fig. 3 shows the setup for the proof-of-concept experiment. A monochrome CCD camera used in the experiment. A RGB color scale is put in the field of view of the camera and a blue glass plate, which functions as a blue-pass filter, is inserted in the left side of the field of view of the camera. The blue glass is used as a representative of a stain on the lens of the camera. Fig. 4 shows the image of the color-scale captured by the black-and-white CCD camera with blue glass

The results indicate that a monochrome security camera with blue glass partially inserted in its view has the ability to distinguish the component of blue color from the components of red color and green color.

4. DISCUSSIONS
The color-estimation technique proposed in this paper depends on the condition of the stain on the lens of the camera. There might be small possibility in which this technique can work effectively, since the stain on the lens must functions as a space-variant color filter in the proposed technique. However, the proposed technique is important in the viewpoint that there is a possibility in which the technique can provide the key information for the police investigation against the serious crime in some cases. All materials including stains on lens have unique transmittance spectrum to a greater or lesser extent and they can be used for the proposed technique. In Stage-B, it is more appropriate if not only the position but also the material, shape and the attitude of the color plate are close to those of the suspected persons cloth. It is very important that Stage-B should be conducted under the same illumination condition as Stage-A as far as possible. The color-estimation technique proposed in this paper will be improved as a component of computer software in the near future by the authors. The software will simultaneously evaluate and show the uncertainty of the estimation. The authors believe that there is much space for many novel inventions based on our new concept that the peculiar facts concerning the security camera system are fully used. The peculiar facts concerning the security camera system are that all the things in the image except the suspected person itself are usually preserved and that can be used for investigations. In addition, it is important from the viewpoint of social security that user-friendly software for image restoration and its operation manuals are developed and provided to the police even if there software is only based on the old-fashioned image-processing technology. In the investigations made by the police, surprisingly enough, even widely available technology is not usually used. This sometimes causes the un-necessary delay or even failure of the police investigation. This is also the matter of the strength of the social security and the matter of saving a life. We are now commencing the project in which the image restoration techniques for security cameras are developed and provided to the police. In the project, some new technology will be developed including the technique proposed here. In addition, the user-friendly software for police investigation including the widely available image restoration technology will also be developed and provided to the police.

taken by a security camera was proposed base on our proposed concept. In the concept, the peculiar facts concerning the security camera system that all the things in the image except the suspected person itself are usually preserved and that can be used for investigations are to be used as fully as possible. The importance of pursuing the proposed concept was discussed from the viewpoints of not only the novelty in image processing technology but also the viewpoints of the efficiency in criminal investigation and social security.

ACKNOWLEDGMENTS
This study was supported by the research aid fund of the Research Foundation for Safe Society. The authors thank Prof. N. Yoshiura at Saitama University, Prof. M. Ito at Gunma University and Mr. M. Tomaru at Gunma Police Headquarter for their fruitful participation in our discussions.

REFERENCES
[1] Y. Fujii, N. Yoshiura and N. Ohta, "Creating a world wide community security structure using individually maintained home computers: The e-JIKEI Network Project", Social Science Computer Review, Vol.23, No.2, pp.250-258, 2005. [2] Y. Fujii, N. Yoshiura and N. Ohta, " Community security by widely available information technology", Journal of Community Informatics, Vol.2, No.1, 2005. [3] Th. Costello and W. Mikhael, Efficient restoration of space-variant blurs from physical optics by sectioning with modified Wiener, Digital Signal Processing, Vol.13, No.1, pp.1-22, 2003. [4] S. Chitale and W. Padgett, Blur identification and correction for a given imaging system, Proc. IEEE SOUTHEASTCON, pp.268-273 (Lexington, KY) March 1999. [5] Y. Fujii, N. Ohta, T. Ito, S. Saitoh, T. Matsuura and T. Yamamoto, "Image restoration for security cameras with dirty lens under oblique illumination", Proc. IEEE IST 2006, pp.100-103 (Minori, Italy) April 2006. [6] Y. Fujii, N. Ohta, T. Ito, S. Saitoh, T. Matsuura and T. Yamamoto, Image Restoration for Security Cameras with Dirty Lens under Oblique Illumination, IEEE Trans. Instrum. Meas., (submitted. revised version of [5].) [7] T. Ito, Y. Fujii, N. Ohta, S. Saitoh, T. Matsuura and T. Yamamoto, Measurement of Space Variant PSF for Restoring Degraded Images by Security Cameras, Proc. SICE-ICCAS (Busan, Korea) Oct. 2006. (to be published)

5. CONCLUSIONS
A method for restoring the images of a suspected person