Autocalibration of an Ad Hoc Construction of Multi-Projector Displays

Takayuki Okatani and Koichiro Deguchi Graduate School of Information Sciences, Tohoku University 6-6-01 Aramaki Aza Aoba, 980-8579 Sendai, Japan {okatani,kodeg}@fractal.is.tohoku.ac.jp

Abstract
In this paper, we present a method for geometric calibration of a multi-projector display system. It enables easy calibration of the system such that the user needs only to take a picture of the projected images on the planar screen with a hand-held camera to accomplish entire calibration. Using the calibration method, one can realize a large highresolution image display by placing multiple projectors on a desk etc. in an arbitrary manner. The calibration requires four or more projectors and includes not only alignment of the images but overall rectication of the stitched image. The problem to be solved is to recover the Euclidean structure of the system (at least partially); its diculties arise from the fact that the intrinsics of the projectors, especially the focal lengths, need to be estimated along with other parameters. For the problem, we show uniqueness of solutions and several critical congurations, which were unclear in previous studies, and then present an algorithm. We present several experimental results that demonstrate the feasibility of the proposed method.

Introduction

Systems of using multiple projectors to display a seamless, large high-resolution image on a planar surface (e.g. a screen) have been studied and already put into use [2, 8, 10, 7, 11]. An extensive survey is given in [1]. One of the most fundamental problems in the systems to realize high-quality images is geometric calibration. It can be divided into two subproblems. One is to make the projectors share a common coordinate system so that the projected images are precisely stitched to produce a seamless image. The other is to rectify the stitched image so that it has a rectangular shape with correct aspect ratio. For the purpose of the former calibration, cameras are used to take images on the screen of the projected images and to obtain necessary information. This calibration needs to be highly accurate, e.g. with subpixel accuracy. For the purpose of the latter calibration, ducial markers on the screen [13] or physical sensors measuring the world coordinate such as tilt sensors

[9] are often used. Although it need not be so accurate as the former calibration, this calibration is necessary, too. In this paper, we present an easy calibration method for the multi-projector display system. It is such that, for example, even when arbitrarily placing projectors on a desk etc. toward a planar screen, the above two calibrations can be performed by just taking a picture of the projected images on the screen with a still camera. The multi-projector display systems are usually designed and constructed as dedicated systems. The proposed method enables exible, ad hoc construction of a multi-projector display system. As long as a sucient number (at least four) of projectors and a device for distributing video signals to those projectors are available, the system can be constructed anywhere. From an application point of view, the study [9] by Raskar and Beardsley and the one [12] by Steele et al. are the closest to this. However, the method in [9] uses tilt sensors. The method in [12] assumes a calibrated stereo pair of a camera and a projector. We use only ordinary data projectors and a camera, and the geometry among the camera and projectors is assumed to be unknown. The problem to be solved here is formulated as autocalibration of a projector-camera system. More specically, it is to reconstruct (at least partially) the Euclidean structure of the system from only an image taken by the camera, which needs the determination of the intrinsics of the projectors. The intrinsics other than the focal lengths are usually constant and can be determined in advance, while the focal lengths will vary whenever the projectors are (re)set, and therefore, they need to be estimated along with other parameters such as the extrinsics. In [6], Raij and Pollefeys deal with this very problem. Using a dedicated system of multi-projector display, they demonstrate that the autocalibration is possible. They assume a fully calibrated camera and partially calibrated projectors; to be specic, for the projectors, their focal lengths are unknown and the vertical component of the principal points is also unknown but the same for all the projectors. Then they present an algorithm based on nonlinear minimization to estimate those parameters. However, uniqueness of solutions and critical congurations are not shown, although uniqueness is implied by a plot of the objective

function that they minimize. In [5], we deal with a similar autocalibration problem for a projector camera system, where calibrated projectors are assumed. In this paper, we consider a dierent setting from [6], in which the camera is fully uncalibrated and projectors are partially calibrated, to conform to several requirements of the mentioned exible system. For this setting, we discuss uniqueness of solutions and then present a few critical congurations, which could actually happen and needs to be carefully considered when implementing numerical algorithms. These are shown in Section 3. In Section 2, we formulate the problem of calibrating the multi-projector display system. A numerical algorithm is presented in Section 4 and several experimental results are shown in Section 5. Throughout this paper, we use the vector-matrix notation used in the textbook [3]; vectors are in bold face (e.g. v) and matrices in the Courier font (e.g. M).

representation of the constraints in terms of the raw, physical parameters, and then examine it analytically, as will be done in the next section.

2.2 A critical conguration

There are several critical congurations for the autocalibration problem considered here, as will be shown in detail later. One of them could pose a problem for solutions based on Eq.(1). It is the case where a projector has the optical axis perpendicular to the screen plane. In this case, the focal length of the projector and the distance from the projector to the screen plane are mutually coupled and can never be decoupled. This is intuitively reasonable. When the projector has the optical axis perpendicular to the screen, it follows that the projector projects an already correctly rectied image. If so, even when the projector arbitrarily moves along its optical axis, there should always exist a focal length value such that the projected image remains unchanged. Algebraically, this is explained as follows. Suppose that only focal length f is unknown among the intrinsics. Then, K p can be represented as a diagonal matrix by applying appropriate normalization to the image coordinates. Thus, the homography between the screen and the projector is given as r11 r12 t1 f r11 f r12 f t1 (2) H sp K p r21 r22 t2 = f r21 f r22 f t2 , r31 r32 t3 r31 r32 t3 where ri j and ti are components of the rotation matrix and the translation vector representing the projector pose relative to a coordinate frame aligned to the screen plane. When the projector has an optical axis perpendicular to the screen plane, r31 = r32 = 0. Then the homography becomes ( f /t3 )r11 ( f /t3 )r12 ( f /t3 )t1 (3) H sp ( f /t3 )r21 ( f /t3 )r22 ( f /t3 )t2 . 0 0 1 Thus, f and t3 are coupled and it becomes impossible to decouple them on the homography. Suppose an algorithm in which f is rst determined and then other parameters are computed using the estimate of f . (This is the case with solutions based on Eqs.(1).) Although it might work in generic cases, the algorithm will encounter numerical problems in this critical case, due to the degeneracy. Note that considering our original purpose, which is to obtain correctly rectied images, the above degeneracy is only formal. That is, when the projector has the perpendicular axis to the screen, as far as that projector is concerned, the image rectication is not necessary. If our goal is to determine the physical parameters including the focal lengths and the projector positions, the conguration needs to be avoided. However our goal is the rectication of the

2
2.1

Representation of autocalibration constraints

Raij-Pollefeys method

The Raij-Pollefeys method [6] is based on the formulation given by Triggs [14] for the problem of autocalibration of cameras from planar scenes. It can be summarized as follows. Let Kc and K p (p = 1, . . . , n) be intrinsics matrices of the camera and the projectors, respectively. Denoting the homography between the camera and a projector by Hcp (p = 1, . . . , n), the autocalibration constraints can be represented as Xc (K K1 )Xc = 0, c c (Hcp Xc ) (K p K1 )(Hcp Xc ) p = 0, p = 1, . . . , n, (1a) (1b)

1 where Xc = 2 (xc iyc ) is called circular points, which are complex vectors with four degrees of freedom. These equations are nonlinear with respect to the unknowns to be determined. Thus, they minimize the algebraic errors of the above equations to compute the intrinsics of the projectors and the circular points. They then calculate calibrated versions of Hcp , from which the extrinsics of the projectors etc. are determined and the Euclidean structure is recovered. Since each of the above equations gives a constraint of two degrees of freedom, there are 2n + 2 constraints. Then, it follows from a counting argument that up to 2n + 2 parameters could be estimated. However, only representing the available constraints as above does not prove uniqueness of solutions or yield details of critical congurations. As in [14], there are multiple representations of the constraints and Eq.(1) is merely one of them. In order to discuss uniqueness or critical congurations, we need to use a direct

overall image. As will be discussed later, by using a dierent representation of constraints and employing appropriate numerical algorithms, this critical conguration need not be avoided.

This equation gives a dierent representation from Eqs.(1) of the same autocalibration constraint; the homography H pc should be factorized as above, where T p and C p should have the above forms and R p should be a rotation matrix.

2.3

A fundamental representation of the autocalibration constraints

3 Analysis of uniqueness and critical congurations

3.1 Problem formulation
Among the homographies, H pc (p = 1, . . . , n) can be computed from image correspondences between the projector image and the camera image. Thus, from H pc (p = 1, . . . , n), we determine unknowns using the constraint that H pc should be factorized as above. Among many unknowns, the most important one is the homography H sc between the screen and the camera, since once this is determined, H ps can be recovered by H ps H1 H pc , using which the projected images sc can be arbitrarily manipulated. Furthermore, it can be seen that even if the factorization of the remaining part T p R p C p is not unique, as in the case of the described critical conguration, our goal of correcting the projected images is achieved. Thus, we need only to consider uniqueness of H sc , and the problem is stated as follows. Problem 3.1. Under the above assumptions, given H pc (p = 1, . . . , n), nd H sc such that the factorization of Eq.(7) is possible for any p. We will show the next in what follows. Proposition 3.1. Except for a few critical congurations, H sc can be uniquely determined up to a matrix transformation s(X; S0 ) = XS1 with any matrix S0 given by 0 cos sin s13 (8) S0 = sin cos s23 , 0 0 s33 where , s13 , s23 , and s33 are arbitrary numbers and is either +1 or 1. The arbitrariness of S0 corresponds to that of dening the coordinate frame on the screen plane. This means that after the alignment and the rectication of the projected images are performed, the overall image still has freedom of scaling, two-dimensional translation and rotation.

In order to thoroughly investigate uniqueness of solutions and critical congurations, we represent the autocalibration constraints in terms of raw, physical parameters. In what follows, we assume the camera to be fully uncalibrated and projectors to be partially calibrated; only their focal lengths are unknown. These assumptions reect the requirement of the ad hoc construction of multi-projector displays. We suppose that the projectors comprising the system can be dierent products with dierent specications. (Note that, in [6], the y component of the principal point is also estimated along with the focal lengths, assuming that the projectors are the same product and their principal points are the same.) For each projector, other parameters than the focal length is expected to be constant for dierent zoom/focus setting, and thus we may use their specication values from a database of projector products. As for the camera, there is a much wider choice than for projectors, and thus we assume it to be uncalibrated. (On Eqs.(1), this is equivalent to ignoring the constraint (1a).) Assuming a coordinate frame aligned to the screen plane (such that its xy plane coincides with the screen plane), the homography H ps between the projector and the screen can be represented as: H ps T p R p C p , where z p Tp = 0 0 0 zp 0 xp yp , 1 (4)

(5)

where [x p , y p , z p ] and R p are translation and rotational components of the projector pose, respectively, and C p K1 . p The only unknown intrinsic is the focal length, and thus we may write C p as a diagonal matrix by applying an appropriate coordinate normalization: 1/ f p 0 0 1/ f p 0 . (6) Cp = 0 0 0 1 Let H sc be the homography between the screen and the camera. Among H sc , H ps , and the homography H pc between a projector and the camera, there is a relation H pc H sc H ps . From this relation and Eq.(4), we have H pc H sc T p R p C p . (7)

3.2 Uniqueness of solutions

Whether H sc can be uniquely determined or not depends on the existence of a 3 3 matrix S that enables the following refactorization: HTRC HS1 STRC (HS1 )(STRC) H T R C

Thus, what needs to be checked is whether S exists or not such that in an appropriate setting, STRC is factorized as: STRC T R C . (9)

In what follows, we call a matrix V TR-factorizable when V can be factorized as V TR, and a matrix V TRCfactorizable when V can be factorized as V TRC, where T is any matrix of the form of Eq.(5), R is any orthogonal matrix, and C is any matrix of the form of Eq.(6). Lemma 3.2. Let V be a real 3 3 matrix: V = [v1 , v2 , v3 ] . Dene w1 v2 v3 and w2 v1 v3 . Then, V is TRCfactorizable, if and only if (w2 1 w2 )w13 w23 2 = (w2 13 w2 )w1 w2 , 23 (10) =

Proof. Since STRC T R C is equivalent to STR T R C C1 and C C1 has the form (6), if STRC is TRCfactorizable, STR is TRC-factorizable and vice versa. Therefore, from Lemma 3.2, a necessary and sucient condition that STRC be TRC-factorizable is that V STR satises Eq.(10). We want to rewrite Eq.(10) into an equation expressed with the components of S by substituting V = STR into Eq.(10). To do this, we write s11 s12 s13 a 0 b S = s21 s22 s23 and T = 0 a c . s31 s32 s33 0 0 1 Because of the orthogonality of R, only the third row vector of R appears in the resulting equation. We denote this vector by r3 [r1 , r2 , r3 ] . Then, Eq.(10) can be expressed as the form of g(a, b, c, r3 , S) = 0. (12) From the assumption of this lemma, this equation should hold for arbitrary a, b, c, and r3 , which gives constraints on S. We rst consider the identity relations obtained from the arbitrariness of a, b, and c, which are related to the assumption that the projector positions are generically dierent from each other. The function g is a polynomial function of a, b, and c, and each polynomial term ai b j ck should have a zero coefcient. Among many terms available, we choose the term a3 , b3 , c3 , bc2 , and b2 c1 . At least their coecients should always be 0: 6r3 (r2 s31 r1 s32 )g1 (S) = 0, 6r1 r2 s31 g1 (S) = 0, 6r1 r2 s32 g1 (S) = 0,
2 2 2{r1 r2 s31 + (r1 r2 )s32 }g1 (S) = 0, 2 2{(r2

where w13 and w23 are the components of w1 [w11 , w12 , w13 ] and w2 = [w21 , w22 , w23 ] .

Proof. If V is TRC-factorizable, there should exist a matrix C of the form (6) such that V VC1 is TR-factorizable, and vice versa. It can be assumed without loss of generality that C1 1 = 0 0 0 1 0 0 0 , d

where d is a real positive number. Thus, V is TRCfactorizable if and only if d exists such that V VC1 is TR-factorizable. Let V = [v1 , v2 , v3 ] . It can be shown [4] that V is TR-factorizable if and only if (v2 v3 ) (v1 v3 ) = 0 and |v2 v3 | = |v1 v3 |.

(13a) (13b) (13c) (13d) (13e)

Using w1 and w2 dened, we can rewrite the vector products of the above equations as v2 v3 = [dw11 , dw12 , w13 ] and v1 v3 = [dw21 , dw22 , w23 ] . Then the above two equations become d (w11 w21 + w12 w22 ) + w13 w23 = 0, d
2 2

2 r1 )s31

+ r1 r2 s32 }g1 (S) = 0,

where g1 (S) = {(s2 s1 ) s3 } {(s12 s31 s11 s32 )2 + (s22 s31 s21 s32 )2 }, where si (i = 1, 2, 3) is the ith row vector of S. From these, we derive explicit expressions for constraints on S. We rst examine the case of g1 (S) = 0. There are two possible cases, (s2 s1 ) s3 = 0 or (s12 s31 s11 s32 )2 + (s22 s31 s21 s32 )2 = 0. The rst case is impossible since it directly means det S = 0. The second case is impossible, either, unless s31 = s32 = 0, since it means det S = 0 unless s31 = s32 = 0. Next we examine the possibility that other terms in the above coefcients will be 0. It is obvious that there are two possible cases, s31 = s32 = 0 or r1 = r2 = 0.
1 We

(11a) + w2 , 23 (11b)

(w2 11

w2 ) 12

w2 13

=d

(w2 21

w2 ) 22

respectively. A necessary and sucient condition that d exists satisfying Eqs.(11) is given as (w2 +w2 w2 w2 )w13 w23 = (w11 w21 +w12 w22 )(w2 w2 ). 11 12 21 22 13 23 By adding (w2 w2 )w13 w23 to both sides Eq.(10) is de13 23 rived. Lemma 3.3. Let S be a 33 matrix. For any matrix T of the form (5), any orthogonal matrix R, and any diagonal matrix C of the form (6), multiplication STRC is TRC-factorizable, if and only if S is represented as S0 of Eq.(8).

used Mathematica to calculate these.

Substituting s31 = s32 = 0 to g = 0, we have

2 2 s4 (s12 s21 s11 s22 )(r1 (s11 s12 + s21 s22 )r2 (s11 s12 + s21 s22 ) 33

+ r1 r2 (s2 + s2 s2 s2 )) = 0. 12 22 11 21 Neither of s33 = 0 or s12 s21 s11 s22 = 0 is possible, since each results in det S = 0. Thus, the following should hold:
2 2 (r1 r2 )(s11 s12 +s21 s22 )+r1 r2 (s2 +s2 s2 s2 ) = 0. (14) 12 22 11 21

Assuming that r1 and r2 diers for each p, the following equations can be derived: s11 s12 + s21 s22 = s2 + s2 s2 s2 = 0. 12 22 11 21 These mean that S is represented as S0 of Eq.(8). Lemma 3.3 is merely a dierent expression of Proposition 3.1, and thus we have shown Proposition 3.1.

determine H sc . However, in that case, we can use the projector causing the degeneracy to perform the rectication of the overall image. This is done by simply using the projected image by the projector as a key frame and manipulating the other projectors so that their projected images are aligned to the key frame. As for the case (2), there is not an eective solution. Thus, we need to make sure that when placing the projectors, they have dierent orientations from each other. Considering the real world construction of the system, this requirement does not seem so dicult to be satised, since the projectors are not likely to have an identical orientation by accident, except for the orientation perpendicular to the screen plane.

4 Algorithm
Although the proposition 3.1 guarantees that H sc can actually be determined from H pc (p = 1, . . . , n), it does not show how to compute H sc . Because of the nonlinear nature of the computation, we use direct minimization to nd a solution, by assuming that good estimates of the focal lengths of the projectors are somehow available. Assuming temporarily the estimates to be correct (which means the projectors are calibrated), we apply the closed form algorithm of [5] that is designed for the case of calibrated projectors. To do this, we have the projectors project calibration patterns such as a checkerboard pattern and take an image (or images) of them. From this image, H pc s are computed, and other parameters are computed. Using these as initial values, we minimize the sum of the re-projection errors to compute the parameters. The overall algorithm is summarized in Fig.1, where fp is the focal length estimate for the p-th projector. A natural question is to what extent the focal length estimate fp needs to be accurate to make the minimization converge to a global minimum. As will be shown in the next section, they need not be so accurate. The range of the initial values of the focal lengths that makes the algorithm converge to the global minimum is indeed comparable to a typical zooming range of data projectors. The minimum number of projectors for performing the computation is four (n 4). The number is derived from a counting argument as follows. On Eq.(7), one projector pose gives a constraint of one degree of freedom on H sc , since H pc has eight degrees of freedom, whereas the unknowns, T p , R p , and C p have three, three, and one degrees of freedom, respectively, and thus degrees of freedom available for determining H sc is 87 = 1. H sc has eight degrees of freedom, from which we can determine only four degrees of freedom, due to the ambiguity of four degrees of freedom associated with S0 . It can be seen from this that the minimum number of projectors is four. This number four is convenient in practice, since an ideal image with the same

3.3

Critical congurations

In the above proof, we rst use arbitrariness of the projector position [b, c, a] or T, and then use arbitrariness of the projector orientation R (specically, r1 and r2 , the rst two components of the third row vector r3 of R). Critical congurations are found for the cases where the arbitrariness is not available. As for the projector positions, there might be cases where the projectors (rigorously, their projection centers) are exactly on some curve, for which some of the terms of g(a, b, c, r3 , S) are coupled, so that H sc becomes not unique. However, there are many terms of several orders, and it does not seem necessary to seriously consider such cases. It seems more important to consider critical congurations due to degenerate orientations of the projectors; they are more likely to occur, considering possible constructions of the system. It can be seen from the above derivation that there are two cases: 1. The case where r1 = r2 = 0 for any projector (every projector has the optical axis perpendicular to the screen), s31 = s32 = 0 is the only available constraint on S. When this holds for some of the projectors, information available from them is correspondingly limited. 2. The case where r1 and r2 are identical for all or some of the projectors. This means that the projectors share an identical optical axis. In this case, only the upper left 2 2 submatrix of S is constrained by Eq.(14). The case (1) is the conguration we discussed in Section 2.2. If one of the projectors has the perpendicular orientation to the screen, then the full constraints on H sc are not available from the projector. When the number of remaining projectors is not sucient, we might not be able to uniquely

1. Set C p to be: C(0) p fp 0 0 s fp fp 0 1 up vp 1

25 20 15 10 5 0

4 8 12 16 20 Mean reprojection error (x 0.0001)

25 20 15 10 5 0

4 8 12 16 20 Mean reprojection error (x 0.0001)

2. Compute H pc from the correspondences between the camera image and the projector image. 3. Apply the closed form algorithm of [5] and compute H(0) enabling the following factorization: sc H pc C(0) 1 H(0) T(0) R(0) p sc p p 4. Using the estimated H(0) , T(0) , R(0) , and also C(0) as inisc p p p tial values, minimize the sum of the reprojection error with respect to H sc , T p , R p , and C p : J=
p,i

Figure 2: Histograms of minimization residues (mean of reprojection errors) of 100 trials in the case of n = 10 and = 0.001 for two cases of choosing initial values of the focal lengths.Initial values are randomly selected from [0.5 f0 , f0 /0.5] (left) and [0.3 f0 , f0 /0.3] (right). Table 1: The number of trials in which the algorithm converges to the global minimum (in percentage).
n=4 10 20 (a) = 0.001 = 0.8 0.65 100 100 100 100 100 100 0.5 100 100 100 0.5 94 100 99 0.3 93 98 99 0.3 91 97 97

|x pi x pi | + |y pi y pi | ,

where x pi and y pi are the measured coordinates of the ith feature point on the pth projector image, and x pi and y pi are their estimates given as [ x pi , y pi , 1] H sc T p R p C p m pi , where m pi is the homogeneous coordinate of the feature point on the original projector image. The minimization is performed by a standard iterative algorithm such as the Levenberg-Marquardt method.

n=4 10 20

(b) = 0.005 = 0.8 0.65 100 100 100 100 100 100

Figure 1: The proposed algorithm. aspect ratio as a single projector image can be generated from exactly four projector images. As for the freedom of S 0 of Eq.(8) that is still left undetermined, which represents the scaling, two-dimensional translation and rotation of the overall images. Among these, the scaling and translation can be uniquely determined so that, for example, the nal image is maximized within the possible display area depending on the layout and congurations of the projectors. The rotation of the images is still left undetermined, which could be resolved, too, by introducing further assumption (e.g. some information about the geometry among the camera and the projectors). We prefer to leave this to the users manual adjustment.

way. Firstly, the poses of n projectors are generated in a random manner. To be specic, their positions are randomly chosen within a unit cube. Selecting a particular side of the cube as the screen, the orientations of the projectors are chosen so that they are oriented toward a random point on the screen. For every projector, the image size and the focal length is set to 1.0 and 2.0, respectively. Generating 10 regularly-spaced feature points in the image of each projector, the correspondences between these points and their projections on the camera image are used for computing the homography H pc between the projector and the camera. When computing the projections of the points on the camera image, Gaussian noise with mean 0 and variance 2 is added to their x and y coordinates. Convergence performance As described, the proposed algorithm requires initial values for the focal lengths of the projectors. In order to examine dependency of convergence performance on these initial values, we run the algorithm with randomly initial values. For this purpose, an interval [ f0 : f0 /] is used, where f0 is the true focal length and is a parameter for changing the width of the interval. A uniform random value is chosen from this interval and used as the initial value fp . We checked if the algorithm converged to the correct solution for dierent values of . Figure 2 shows examples for = 0.5 and 0.3; other parameters are set as n = 10 and = 0.001. It can be seen that, in the case of = 0.3, there were a few trials in which the algorithm

5 Experimental results
5.1 Synthetic data
We conducted experiments using synthetic data to examine convergence performance of the algorithm and accuracy of the estimation. The data are synthesized in the following

MSE rot. (x 1e-5)

30 25 20 15 10 5 0

MSE rot. (x 1e-3)

10 7.5 5 2.5 0 4 6 8 10 12 14 16 18 20 Number of projectors

4 6 8 10 12 14 16 18 20 Number of projectors

Figure 3: MSE (mean squared error) of the estimates of the projector orientation over 100 trials. Left: = 0.001 (1% of the image width). Right: = 0.005 (5%) did not converge to the correct solution. More detailed results are shown in Table 1. In the experiment, the algorithm converged to the correct solution whenever is lower than 0.65. The true focal length f0 is set to 2.0, while the image size is 1.0, which is equivalent to a projector with a diagonal projecting angle of about 20 degrees. When converted into a diagonal projecting angle, the interval given by = 0.65 corresponds to as wide a range as from 26 to 57 degrees. This assures that the range of the initial value for which the algorithm successfully converges to a global minimum is fairly wide, and therefore very rough estimates of the focal lengths can be used. Performance w.r.t. the number of projectors As mentioned earlier, the calibration requires four or more projectors. It is easily anticipated that the number of projectors will aect the accuracy of the calibration. In order to examine this, we conducted experiments by varying the number of projectors. The results are shown in Fig.3. Although H sc is the most important among the parameters, it is not straightforward to measure its estimation accuracy. Therefore, the orientations of the projectors are recovered using the estimate of H sc and then their accuracy is measured. In the gure, MSE over 100 trials with = 0.001 and 0.005 are shown. It can be seen from the results that the estimation is actually possible from four projectors and the accuracy of the estimate gradually increases with the number of projectors as expected.

Figure 4: Two views of the experimental system of integrating four projector images to generate a single seamless high-resolution image. The four projectors are placed in an arbitrary manner on a desk. A white wall of the room is used as the screen.

5.2

Real data

We also conducted experiments using real data to examine the applicability of the proposed method to real systems. Figure 4 shows the experimental setup. The system consists of four data projectors (three Epson and one NEC projectors) with 1024 768 pixels, a digital camera (NIKON D1) with 2000 1312 pixels, and several notebook PCs supplying the projectors with images. The white wall of the room is used for the screen. The projectors are placed in a random manner, at least as long as their projection areas on the wall form a single closed area of approximately rectangular shape.

The proposed method requires known intrinsics other than the focal length of the projectors, and they were specied in the following way. Firstly, the skew and the aspect ratio are assumed to be 0 and 1, respectively, for every projector. Then the principal point of each projector is roughly estimated by visual inspection using its zoom function. By varying its zoom while having the projector project a test pattern, the magnication center of the images is identied, which is expected to coincide with the principal point. Although the resulting estimates of the principal points can be somewhat inaccurate, we may assume that it will not cause fatal errors in the nal estimation, as in the problem of structure from motion. (Errors in the principal point are expected to be absorbed in the estimation of the projector orientation, since the principal points and the orientation are highly correlated and dicult to separate.) Furthermore, the errors affect only accuracy of the overall image rectication, which need not be so highly accurate as the image alignment, after all. The calibration is performed using checkerboard patterns projected by the projectors. On the image taken by the camera, each pattern belonging to each projector is identied and then the corner points are detected with subpixel accuracy by calculating a crossing point of the two lines locally tted to the corners. As for the initial values of the focal lengths of the projectors, a rough estimate, 2000[pixels], is used for every projector. The proposed algorithm was applied to these data and then it converged in about 30 iterations. The residue of the sum of the reprojection errors after the convergence was about 0.6 pixels per point. Although this number seems larger than expected from the accurate corner detection, it might probably be because of the lens distortion of the projector lenses and imperfect atness of the wall used as the screen. Then, using the homography H sc thus computed, a single image is generated. Several results are shown in Fig.5. It can be seen that the images are accurately stitched and that the synthesized image is of correct rectangular

data show that the initial values need not be so accurate to make the minimization converge to a global minimum. They need only to be within a typical range of focal lengths corresponding to zoom ranges of usual projectors. We have also conrmed through experiments using real data that the method works for a real system.

References
[1] M. Brown, A. Majumder, and R. Yang. Camera-based calibration techniques for seamless multiprojector displays. IEEE Transactions on Visualization and Computer Graphics, 11(2):193206, 2005. [2] Y. Chen, D. W. Clark, A. Finkelstein, T. Housel, and K. Li. Automatic alignment of high-resolution multi-projector displays using an un-calibrated camera. In IEEE Visualization, pages 125130, 2000. [3] R. Hartley and A. Zisserman. Multi-View Geometry in Computer Vision. Cambridge University Press, 2000. o [4] A. Heyden and K. Astr m. Minimal conditions on intrinsic parameters for Euclidean reconstruction. In Proceedings of Asian Conference on Computer Vision, pages 169176, 1998. [5] T. Okatani and K. Deguchi. Autocalibration of a projectorcamera system. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(12):18451855, 2005. [6] A. Raij and M. Pollefeys. Auto-calibration of multi-projector display walls. In Proceedings of International Conference on Pattern Recognition, pages 1417, 2004. [7] J. Raskar, R. van Baar, P. Beardsley, T. Willwacher, S. Rao, and C. Forlines. iLamps: Geometrically aware and selfconguring projectors. In Proceedings of ACM SIGGRAPH, pages 809818, 2003. [8] R. Raskar. Immersive planar display using roughly aligned projector. In Proceedings of IEEE Virtual Reality, pages 109116, 2000. [9] R. Raskar and P. Beardsley. A self correcting projector. In CVPR, pages 626631, 2001. [10] R. Raskar, M. S. Brown, R. Yang, Towles H. Chen, W. C., B. Seales, and H. Fuchs. Mutli-projector displays using camera based registration. In Proceedings of IEEE Visualization, pages 161168, 1999. [11] J. M. Rehg, M. Flagg, T. J. Cham, R. Sukthankar, and G. Sukthankar. Projected light displays using visual feedback. In Proceedings of International Conference on Control, Automation, Robotics and Vision, 2002. [12] R. M. Steele, S. Webb, and C. Jaynes. Monitoring and correction of geometric distortion in projected displays. Journal of the Winter School of Computer Graphics, 10(2):429440, 2002. [13] R. Sukthankar, R. G. Stockton, and M. D. Mullin. Smarter presentations: Exploting homography in camera-projector systems. In Proceedings of IEEE International Conference on Computer Vision, pages 247253, 2001. [14] B. Triggs. Autocalibration from planar scenes. In Proceedings of European Conference on Computer Vision, pages 89 105, 1998.

Figure 5: A few results. Top-left: Overview of an stitched image. Top-right: Geometry of the images. Bottom-left and right: Stitched images taken from a direction perpendicular to the wall. They are of exact rectangular shape, showing that they are correctly rectied. shape.

Summary

We have shown a method for calibration of an ad hoc construction of multi-projector displays. Assuming that the camera is uncalibrated and the projectors are partially calibrated (only focal lengths are unknown), we have examined uniqueness of solutions and critical congurations for the corresponding calibration problem. Four or more projectors are required for the calibration to be performed. The critical congurations shown depend on the orientations of the optical axes of the projectors. Firstly, when there is a projector that has an optical axis perpendicular to the screen plane, the focal length and position of that projector cannot be uniquely determined. Secondly, when there are a pair of projectors whose optical axes coincide, no information can be derived from the pair. The rst critical conguration need not be avoided, since it means that the image of the projector is already correctly rectied, and therefore the overall image can be rectied by aligning the images of other projectors to the images of the projector. However, there is no solution to the second critical conguration and we need to make sure that this will not occur. Since the minimum number of projectors is four, it is necessary to make at least four of the projectors not share an identical orientation. Also, we have presented an algorithm based on nonlinear minimization, in which initial values of focal lengths need to be specied. The experimental results using synthetic