Symmetry Breaking Operators For Strongly Spherical Reductive Pairs and The Gross-Prasad Conjecture For Complex Orthogonal Groups

SYMMETRY BREAKING OPERATORS FOR STRONGLY SPHERICAL
REDUCTIVE PAIRS AND THE GROSSPRASAD CONJECTURE FOR

COMPLEX ORTHOGONAL GROUPS

JAN MOLLERS
arXiv:1705.06109v1 [math.RT] 17 May 2017
Abstract. A real reductive pair (G, H) is called strongly spherical if the homogeneous
space (G H)/ diag(H) is real spherical. This geometric condition is equivalent to the
representation theoretic property that dim HomH (|H , ) < for all smooth admissible
representations of G and of H. In this paper we explicitly construct for all strongly
spherical pairs (G, H) intertwining operators in HomH (|H , ) for and spherical principal
series representations of G and H. These so-called symmetry breaking operators depend
holomorphically on the induction parameters and we further show that they generically
span the space HomH (|H , ). In the special case of multiplicity one pairs we extend our
construction to vector-valued principal series representations and obtain generic formulas for
the multiplicities between arbitrary principal series.
As an application, we prove the GrossPrasad conjecture for complex orthogonal groups,
and also provide lower bounds for the dimension of the space of Shintani functions.
Contents
Introduction 1
1. The structure of strongly spherical reductive pairs 6
2. Spherical matrix coefficients 18
3. Lower multiplicity bounds - Construction of symmetry breaking operators 30
4. Upper multiplicity bounds - Invariant distributions 35
5. Application to Shintani functions 39
6. Multiplicity one pairs 41
7. An example: (G, H) = (GL(n + 1, R), GL(n, R)) 53
Appendix A. Finite-dimensional branching rules for strong Gelfand pairs 54
References 55
Introduction
One major question in the representation theory of real reductive groups is how an irre-
ducible representation of a group G decomposes if restricted to a subgroup H. In the context
of infinite-dimensional representations of non-compact groups this leads to the study of the
multiplicities
dim HomH (|H , ) N {},
2010 Mathematics Subject Classification. Primary 22E46; Secondary 11F70, 53C30.

Key words and phrases. Symmetry breaking operators, real reductive groups, strongly spherical reductive
pair, finite multiplicities, multiplicity one pairs, GrossPrasad conjecture, Shintani functions.
1
2
JAN MOLLERS
where and are irreducible representations of G and H, usually assumed to be smooth

admissible Frechet representations of moderate growth. In general these multiplicities might
be infinite, so to find a good setting for studying them one is interested in pairs (G, H) of
real reductive groups where dim HomH (|H , ) is always finite. Elements of HomH (|H , )
are also referred to as symmetry breaking operators, a term coined by Kobayashi [16].
Following [14] we call a pair (G, H) consisting of a real reductive group G and a reductive
subgroup H strongly spherical provided the homogeneous space (G H)/ diag(H) is real
spherical, i.e. a minimal parabolic subgroup PG PH of G H has an open orbit. We
note that this is equivalent to the double coset space PH \G/PG being finite. The interest in
strongly spherical pairs in the context of representation theory is due to the following result
by KobayashiOshima [19]:
Fact I (see [19, Theorem C]). If (G, H) is strongly spherical then
dim HomH (|H , ) <
for all smooth admissible representations of G and of H. If additionally G and H are
defined algebraically over R, then also the converse statement holds.
Among the strongly spherical pairs, KobayashiOshima [19] also characterized those with
uniformly bounded multiplicities. The corresponding pairs of Lie algebras essentially form
five families, and choosing the right Lie groups yields five families of groups (G, H) whose
multiplicities are uniformly bounded by one. This multiplicity one property is due to Sun
Zhu [39]:
Fact II (see [39, Theorem B]). If (G, H) is one of the pairs
(GL(n + 1, C), GL(n, C)), (GL(n + 1, R), GL(n, R)), (U(p, q + 1), U(p, q)),
(SO(n + 1, C), SO(n, C)), (SO(p, q + 1), SO(p, q)),
then
dim HomH (|H , ) 1
for all irreducible smooth admissible Frechet representations of G and of H of moderate
growth.
Both Fact I and II lead to the natural problem of determining dim HomH (|H , ) for given
irreducible representations and of G and H, as advocated by Kobayashi [16] in his ABC
program as Stage B and C. For the multiplicity one pairs of general linear groups, this question
is linked to the famous RankinSelberg integrals, and for the pairs of orthogonal and unitary
groups the GrossPrasad and GanGrossPrasad conjectures make predictions about when
the multiplicities are non-zero.
Since every irreducible smooth admissible Frechet representation of moderate growth is by
the Harish-Chandra Subquotient Theorem a subquotient of a principal series representation,
it is reasonable to study this problem for principal series representations and . In this pa-
per, we construct for all strongly spherical reductive pairs (G, H) explicit families of symmetry
breaking operators between principal series representations that depend holomorphically on
the principal series parameters. This establishes lower bounds for the multiplicities in ques-
tion. Together with upper bounds (obtained using Bruhats theory of invariant distributions)
we are able to compute dim HomH (|H , ), at least generically, for all strongly spherical pairs
(G, H) and spherical principal series representations, and for the multiplicity one pairs also for
non-spherical principal series representations. In the case (G, H) = (SO(n + 1, C), SO(n, C))
this proves the local GrossPrasad conjecture at the Archimedean place k = C.
SYMMETRY BREAKING OPERATORS FOR STRONGLY SPHERICAL REDUCTIVE PAIRS 3
Statement of the main results. Let G be a real reductive group in the Harish-Chandra
class and H G a reductive subgroup such that the pair (G, H) is strongly spherical. Let g
and h denote the corresponding Lie algebras and write gn and hn for the sums of all simple
non-compact ideals. Even if we assume the reductive pair (g, h) to be indecomposable, it can
happen that g has arbitrarily many simple non-compact factors (see Example 1.3), and we
therefore assume that (g, h) is strictly indecomposable, i.e. (gn , hn ) is indecomposable. This
notion is due to KnopKrotzPecherSchlichtkrull [14]. A classification of all such pairs (g, h)
was established by KobayashiMatsuki [18] and KnopKrotzPecherSchlichtkrull [13, 14],
and we summarize the result in Theorem 1.6. We further assume that (g, h) is non-trivial,
i.e. g 6= h.
Let PG = MG AG NG G and PH = MH AH NH H be minimal parabolic subgroups
and write aG and aH for the Lie algebras of AG and AH . For irreducible finite-dimensional
representations of MG , of MH and a
G,C , aH,C we consider the principal series
representations (smooth normalized parabolic induction)
, = IndG
PG ( e 1), , = IndH
PH ( e 1).
In case = 1 and = 1 are the trivial representations of MG and MH , we abbreviate = 1,

and = 1, . Our first main result relates dim HomH ( |H , ) to #(PH \G/PG )open , the
number of open double cosets in PH \G/PG .
Theorem A (see Corollary 3.4 and 4.3). Assume that (G, H) is a strongly spherical reductive
pair such that (g, h) is non-trivial and strongly indecomposable. Then for all (, ) a
G,C

aH,C we have the lower multiplicity bound
dim HomH ( |H , ) #(PH \G/PG )open ,

and for generic (, ) a
G,C aH,C (see (4.1) for the precise condition) we have
dim HomH ( |H , ) = #(PH \G/PG )open .
Let us briefly explain the method of proof. First of all, the generic upper multiplicity
bound #(PH \G/PG )open is established by identifying intertwining operators with their
distribution kernels which are certain (PH PG )-invariant distributions on G (see Section 4.2).
To bound the dimension of the space of invariant distributions we use Bruhats theory. Here
the non-open (PH PG )-orbits in G are particularly important, and we prove a new structural
result about them (see Theorem 1.4) which is the necessary technical ingredient in the proof
of generic upper bounds (see Theorem 4.1) The lower multiplicity bounds are established by
explicitly constructing a non-trivial holomorphic family A, HomH ( |H , ) of symmetry
breaking operators for every open double coset in PH \G/PG . The construction of this family
of operators is in terms of their distribution kernels which turn out to be products of complex
powers of matrix coefficients belonging to finite-dimensional spherical representations of G.
The technical ingredients in this part are the existence of enough such matrix coefficients
(see Theorem 2.6) and the meromorphic/holomorphic extension of the complex powers (see
Theorem 3.3). Then the lower bounds are established by regularizing the families A, in the
holomorphic parameters (, ).
For the multiplicity one pairs in Fact II we also consider non-spherical principal series
representations and show the following result:
4
JAN MOLLERS
Theorem B (see Theorem 6.5). Assume that (G, H) is a multiplicity one pair. Then for all
cG M
(, ) M cH and (, ) a a we have the lower multiplicity bound
G,C H,C
dim HomH (, |H , , ) 1 whenever HomM (|M , |M ) 6= {0},

G,C aH,C we have
(
1 for HomM (|M , |M ) 6= {0},
dim HomH (, |H , , ) =
0 for HomM (|M , |M ) = {0}.
Here M MG MH denotes the stabilizer of the unique open PH -orbit in G/PG .

The passage from spherical principal series as treated in Theorem A to general principal
series , uses a variant of the JantzenZuckerman translation principle. Here both
and are tensored with certain finite-dimensional representations of G and H, and we
show the existence of such representations case-by-case (see Theorem 6.4). We illustrate the
construction of symmetry breaking operators for the multiplicity one pair (G, H) = (GL(n +
1, R), GL(n, R)) by providing explicit formulas for the distribution kernels (see Section 7).
Applications. Let us present two interesting applications of the main results. Combining
Theorem B with Fact II we immediately obtain:
Corollary C (see Corollary 6.8). Assume that (G, H) is a multiplicity one pair, where we
additionally assume p = q or p = q + 1 in the case of indefinite orthogonal or unitary groups.
Then, if both , and , are irreducible we have
dim HomH (, |H , , ) = 1.
For (G, H) = (SO(n + 1, C), SO(n, C)) this proves the local GrossPrasad conjecture [8,
Conjecture 11.5] at the Archimedean place k = C (see Conjecture 6.7). We expect that
our results also provide some information towards the local GrossPrasad conjecture at the
Archimedean place k = R where one considers the pairs (G, H) = (SO(p, q + 1), SO(p, q)).
However, for real groups it will be necessary to also consider principal series representations
induced from more general cuspidal parabolic subgroups. We hope to return to this topic in
a future work.
Another application concerns the study of Shintani functions for the pair (G, H) which was
recently taken up by Kobayashi [15]. For (, ) a
G,C aH,C we write Sh(, ) for the space
of Shintani functions for (G, H) of type (, ) and Shmod (, ) for its subspace of Shintani
functions of moderate growth (see Section 5 for the precise definitions). Kobayashi [15] showed
that dim Sh(, ) < for all (, ) a
G,C aH,C if and only if (G, H) is strongly spherical.
Combining results from [15] with Theorem A shows:
Corollary D (see Theorem 5.3). Assume that (G, H) is a strongly spherical reductive pair
such that (g, h) is non-trivial and strongly indecomposable. Then for all (, ) a
G,C aH,C
we have
dim Sh(, ) dim Shmod (, ) #(PH \G/PG )open ,
G,C aH,C we have
dim Shmod (, ) = #(PH \G/PG )open .

Relation to other work. Let us first mention previous works in which holomorphic families
of intertwining operators A, HomH ( |H , ) appear. The construction of operators A,
for the pairs (g, h) = (so(1, n) + so(1, n), diag so(1, n)) can be found in the work of Oksak [34]
for n = 3 (the case of so(1, 3) sl(2, C)), BernsteinReznikov [1] for n = 2 (the case of
so(1, 2) sl(2, R)) and ClercKobayashirstedPevzner [4] for arbitrary n 2. Later
Clerc [2, 3] gave a complete description of the space HomH ( |H , ) for all parameters (, ).
For (g, h) = (so(1, n + 1), so(1, n)) KobayashiSpeh [20] obtained a full classification of all
symmetry breaking operators in terms of the holomorphic family A, . In fact, some of the
analytic arguments we use can also be found in [20, 21] (see e.g. [20, Lemma 11.10] which
deduces lower multiplicity bounds from meromorphic families of intertwining operators). For
some symmetric pairs of low rank, in particular for all symmetric pairs (g, h) with g of
rank one, the work of MollersrstedOshima [27] yields holomorphic families of symmetry
breaking operators. We also note that for (g, h) = (gl(n + 1, R), gl(n, R)) the kernel functions
given in Section 7 can be found in the work of MuraseSugano [32] in the context of p-adic
groups, and in a slightly different form also in the recent work of Neretin [33] in the context
of finite-dimensional representations.
We remark that some of our statements can also be proven differently using the recent work
of GourevitchSahiSayag [7] on the extension of invariant distributions. In fact, their work
uses a similar idea, namely the extension of an invariant distribution to a meromorphic family
of distributions. However, their meromorphic families only depend on one complex parameter
whereas our constructed families depend on (, ) and hence contain more information.
Although our holomorphic families of symmetry breaking operators generically span the
space of all intertwining operators, it is much more difficult to determine HomH (, |H , , )
for singular parameters (, ) a
G,C aH,C . Theorem A and B provide lower bounds for
the multiplicities, but it turns out that for singular parameters the multiplicities can be
larger. A systematic study of multiplicities and symmetry breaking operators for (g, h) =
(so(1, n + 1), so(1, n)) was initiated by Kobayashi, and we refer the reader to the relevant
articles by KobayashiSpeh [20, 21] and KobayashiKuboPevzner [17] (see also the work of
FischmannJuhlSomberg [5]).
Outlook. The principal series representations considered in this paper are all induced from
a minimal parabolic subgroup. However, by the Langlands classification every smooth ad-
missible Frechet representation of moderate growth is the unique irreducible quotient of a
generalized principal series representation, induced from an arbitrary cuspidal parabolic sub-
group. At least for the multiplicity one pairs a generalization of our construction to cuspidal
parabolic subgroups is desirable.
Our holomorphic families of symmetry breaking operators also allow an interpretation
as invariant distribution vectors. More precisely, one can view the distribution kernels as
diag(H)-invariant distribution vectors on principal series representations of G H. As such,
they are expected to contribute to the most continuous part of the Plancherel formula for the
real spherical homogeneous spaces (G H)/ diag(H). It would be interesting to investigate
this topic further, especially in connection with the recent advances in the harmonic analysis
on real spherical spaces (see e.g. [23] and references therein).
Another possible application of symmetry breaking operators is the explicit construction
of branching laws for unitary representations. This was successfully carried out for the pair
6
JAN MOLLERS
(G, H) = (O(1, n+1), O(1, m+1)O(nm)) in the case of unitary principal series and comple-
mentary series representations by MollersOshima [29]. For (G, H) = (GL(n+1, C), GL(n, C))
a similar suggestion was made by Neretin [33, Section 5].
Let us also mention connections to boundary value problems [28] and automorphic forms [1,
25] that were established for (G, H) = (O(1, n + 1), O(1, n)) and might be of interest also for
more general strongly spherical pairs.
Structure of the paper. In Section 1 we recall some structure theory and the classification
of strongly spherical real reductive pairs. The main new result here is a characterization of the
open double cosets in PH \G/PG (see Theorem 1.4). The construction of (PH PG )-equivariant
matrix coefficients on G is the content of Section 2, and Theorem 2.6 ensures the existence of
enough such matrix coefficients. Section 3 deals with the explicit construction of symmetry
breaking operators between spherical principal series representations of strongly spherical
pairs (see Theorem 3.3). This construction uses the results of Section 1 and 2 in a crucial way,
and implies the claimed lower bounds for multiplicities. The upper bounds are established in
Section 4 using Bruhats theory of invariant distributions (see Theorem 4.1. The application
of this technique depends heavily on the results of Section 1. In Section 5 the previous results
are applied to obtain bounds for the space of Shintani functions (see Theorem 5.3), following
Kobayashis recent approach via symmetry breaking operators. The topic of Section 6 is the
construction of symmetry breaking operators between general principal series from symmetry
breaking operators between spherical principal series for multiplicity one pairs. To prove the
main statement Theorem 6.5 we employ a variant of the translation principle which we apply
case-by-case to all multiplicity one pairs. Finally, Section 7 illustrates symmetry breaking
operators between principal series for the pair (G, H) = (GL(n + 1, R), GL(n, R)) by explicit
formulas.
Acknowledgements. We thank Yoshiki Oshima for helpful and inspiring conversations on
the topic of this paper.
Notation. N = {0, 1, 2, . . . , }, V = HomC (V, C).
1. The structure of strongly spherical reductive pairs

We discuss strongly spherical reductive pairs (G, H) and their structure theory following
[14, 18]. First, using results from [14], we reduce the study of general strongly spherical
reductive pairs to that of strongly spherical symmetric pairs. For the latter we recall some
structure theory as developed in [18, Section 3]. This is used to derive some new results about
the double coset space PH \G/PG for PG G and PH H minimal parabolic subgroups (see
Theorem 1.4). These results are used both in Section 3 for the construction of symmetry
breaking operators and in Section 4 for their uniqueness.
1.1. Strongly spherical reductive pairs. Consider a real reductive pair (g, h) of Lie al-
gebras, i.e. g is a reductive Lie algebra and h a reductive subalgebra of g. A pair of Lie
groups (G, H) with H a closed subgroup of G is called real reductive if the underlying pair
(g, h) of Lie algebras is real reductive. In this paper we will additionally assume that G is of
Harish-Chandra class (see e.g. [12, Chapter VII.2] for the precise definition).
Definition 1.1. A real reductive pair (g, h) of Lie algebras is called strongly spherical if there
exist minimal parabolic subalgebras pG g and pH h such that
g = pG + pH .
A real reductive pair (G, H) of Lie groups is called strongly spherical if the corresponding
pair (g, h) of Lie algebras is strongly spherical.
This property for a real reductive pair (G, H) was introduced by KobayashiOshima [19]
as property (PP). In this paper we use the notion strongly spherical, following [14]. In view of
Fact I strongly spherical reductive pairs are sometimes also referred to as finite-multiplicity
pairs (see e.g. [18, 19]). The following characterization of strongly spherical pairs follows e.g.
from [19, Lemma 5.3 (1)] and [22, Theorem 1.1]:
Proposition 1.2. Let (G, H) be a real reductive pair of Lie groups and let PG G and
PH H be minimal parabolic subalgebras. Then the following statements are equivalent:
(1) (G, H) is strongly spherical,
(2) The homogeneous space (G H)/ diag(H) is real spherical,
(3) There exists an open double coset in PH \G/PG ,
(4) #(PH \G/PG ) < .
A reductive pair (g, h) is called non-trivial if g 6= h. We call (g, h) indecomposable if
there does not exist a non-trivial decomposition g = g1 g2 with ideals gi g such that
h = (h g1 ) (h g2 ). Every reductive pair (g, h) of Lie algebras can be written as a direct
sum of indecomposable pairs, and we therefore assume that (g, h) is indecomposable.
Example 1.3. As observed in [14] there exist indecomposable strongly spherical pairs (g, h)
such that g has arbitrarily many non-compact simple factors. For instance we have the
indecomposable strongly spherical reductive pair
(g, h) = (sp(p, q + 1) + + sp(p, q + 1), sp(p, q) + + sp(p, q) + sp(1)),
where each of the k factors sp(p, q) of h is embedded into the corresponding factor sp(p, q + 1)
of g in the standard way, and sp(1) is embedded diagonally into g as the centralizer of sp(p, q)
in each factor sp(p, q + 1).
To avoid exotic situations as in Example 1.3 we make a similar assumption as in [14]. For
this write
g = gn gel , h = hn hel ,
where gn resp. hn is the direct sum of all simple non-compact ideals and gel resp. hel
the sum of all simple compact and all abelian ideals. Following [14] we call a pair (g, h)
strictly indecomposable if both (g, h) and (gn , hn ) are indecomposable. Note that hn gn is
automatic from the definition of gn and hn . This definition excludes exotic reductive pairs as
in Example 1.3 but still allows certain central extensions of g and h as for the multiplicity
one pairs (g, h) = (gl(n + 1, F), gl(n, F)), F = R, C, and (u(p, q + 1), u(p, q)).
The main result of this section is a statement about non-open double cosets in PH \G/PG .
Note that for PH gPG PH \G/PG the stabilizer of the PH -orbit through gPG G/PG is given
by PH gPG g1 . Let PG = MG AG NG and PH = MH AH NH be Langlands decompositions
of PG and PH and write mG , aG , nG and mH , aH , nH for the Lie algebras of MG , AG , NG
and MH , AH , NH .
Theorem 1.4. Assume that (G, H) is a strongly spherical reductive pair such that (g, h) is
non-trivial and strictly indecomposable. Then for a double coset PH gPG PH \G/PG the
following are equivalent:
(1) PH gPG is open in G,
(2) The projection of pH Ad(g)pG to aH is trivial.
8
JAN MOLLERS
In this section we prove the implication (2)(1) by showing that for every non-open double
coset PH gPG there exists Z = ZM + ZA + ZN pH Ad(g)pG with ZA 6= 0. The remaining
implication (1)(2) is proved in Section 2.3 (see Corollary 2.7). Note that Theorem 1.4 is
not used in Section 2.
1.2. Classification of strongly spherical reductive pairs. For a reductive Lie algebra g
we denote by p : g gn the projection map onto the ideal gn g consisting of all simple
non-compact ideals of g.
Lemma 1.5. Let (g, h) be an indecomposable reductive pair.
(1) ker p|h does not contain any non-compact abelian ideals of h.
(2) If (g, h) is strongly spherical then (gn , p(h)) is strongly spherical.
Proof. (1) Let a ker p|h be a non-compact abelian ideal, then a gel and since a is
non-compact it has to be an ideal in g. Write g = a g as sum of ideals, then
h = a (g h) so that the indecomposability of (g, h) forces a = 0 or (g, h) = (a, a).
Since (g, h) is assumed to be non-trivial, the latter case cannot occur.
(2) We have pG + pH = g for some choice of minimal parabolic subalgebras pG g and
pH h. Since gel pG we have pG = (pG gn ) gel and pG gn gn is a minimal
parabolic subalgebra. This implies p(pH ) + (pG gn ) = gn . Clearly p(pH ) is a minimal
parabolic subalgebra of p(h) so the claim follows.
By the previous lemma, the study of strictly indecomposable reductive pairs (g, h) can
be reduced to the case where g is semisimple. In this case, a classification was obtained by
KobayashiMatsuki [18] for symmetric pairs and by KnopKrotzPecherSchlichtkrull [13, 14]
for arbitrary reductive pairs:
Theorem 1.6 (see [13, 14, 18]). Let (g, h) be a strictly indecomposable reductive pair with
g semisimple. Then (g, h) is strongly spherical if and only if it is isomorphic to one of the
following pairs:
A) Trivial case: g = h.
C) Compact case: g is the Lie algebra of a compact simple Lie group.
D) Compact subgroup case: h = k is the Lie algebra of a maximal compact subgroup K of
a non-compact simple Lie group G with Lie algebra g, or (g, h) is one of the following
subpairs of (g, k):
(so(1, 2n), su(n) + f) (n 2) with f u(1).
(so(1, 4n), sp(n) + f) (n 1) with f sp(1).
(so(1, 16), spin(9)).
(so(p, 7), so(p) + g2 ) (p = 1, 2).
(so(p, 8), so(p) + spin(7)) (p = 1, 2, 3).
(su(1, 2n), sp(n) + f) (n 1) with f u(1).
(sp(1, n), sp(n) + f) (n 1) with f sp(1).
(su(p, q), su(p) + su(q)) (p, q 1, p 6= q).
(so (2n), su(n)) (n 3 odd).
(e6(14) , so(10)).
E) Split rank one case (rankR g = 1):
E1) (so(1, p + q), so(1, p) + so(q)) (p, q 1) or one of the following subpairs:
(so(1, p + 2q), so(1, p) + su(q) + f) (p 1, q 2) with f u(1).
(so(1, p + 4q), so(1, p) + sp(q) + f) (p 1, q 2) with f sp(1).
(so(1, p + 7), so(1, p) + g2 ) (p 0).

(so(1, p + 8), so(1, p) + spin(7)) (p 0).
(so(1, p + 16), so(1, p) + spin(9)) (p 0).
E2) (su(1, p + q), su(1, p) + su(q) + u(1)) (p, q 1) or one of the following subpairs:
(su(1, p + q), su(1, p) + su(q)) (p, q 1, p + q 3).
(su(1, p + 2q), su(1, p) + sp(q) + f) (p, q 1) with f u(1).
E3) (sp(1, p + q), sp(1, p) + sp(q)) (p, q 1).
E4) (f4(20) , so(8, 1)).
F) Strong Gelfand pairs and their real forms:
F1) (sl(n + 1, C), gl(n, C)) (n 2).
F2) (so(n + 1, C), so(n, C)) (n 2).
F3) (sl(n + 1, R), gl(n, R)) (n 1).
F4) (su(p, q + 1), su(p, q) + f) (p, q + 1 1) with f u(1) and f = u(1) for p = q, q + 1.
F5) (so(p, q + 1), so(p, q)) (p + q 2).
G) Group case: (g, h) = (g + g , diag g )
G1) g is the Lie algebra of a compact simple Lie group.
G2) g = so(1, n) (n 2).
H) Other cases:
H1) (so(2, 2n), su(1, n) + f) (n 1) with f u(1).
H2) (su (2n + 2), su (2n) + R + f) (n 2) with f su(2).
H3) (so (2n + 2), so (2n) + f) (n 1) with f so(2).
H4) (sp(p, q + 1), sp(p, q) + f) (p, q + 1 1) with f sp(1).
H5) (e6(26) , so(9, 1) + R).
The list and its enumeration is a copy of the list in [18, Theorem 1.3] to which we added
the non-symmetric cases obtained in [14, Table 9] and the cases with h compact which are
listed in [13]. This is the reason why B) is missing since it was listed in [18] as the abelian
case (g, h) = (R, 0) which we exclude by assuming that g is semisimple.
The classification immediately implies the following statement (see also [13, Lemma 1.4]):
Corollary 1.7. Let (g, h) be a non-trivial strictly indecomposable strongly spherical reductive
pair with g semisimple. Then there exists a non-trivial involution of g such that h g ,
hn = (g )n , and hel and (g )el differ only in compact factors.
It is clear that in this case (g, g ) is a strongly spherical symmetric pair. This observation
will be used to reduce several statements to the case of symmetric pairs.
1.3. Structure of strongly spherical reductive pairs. We adapt the structure theory
developed in [18] for strongly spherical symmetric pairs to the case of strongly spherical
reductive pairs. For the rest of this section let (G, H) be a strongly spherical reductive pair
with g semisimple such that (g, h) is non-trivial and strictly indecomposable. By Corollary 1.7
there exists an involution of G such that h g , hn = (g )n , and hel and (g )el differ only
in compact factors. We first choose minimal parabolic subgroups PG G and PH H in a
compatible way.
There exists a Cartan involution of G which commutes with and leaves H invariant,
and hence
K = G G and H K = H H
are maximal compact subgroups of G and H. Fix a maximal abelian subspace aH h =
g, and extend it to a maximal abelian subspace aG in g . Then aG is -stable and
10
JAN MOLLERS
aG = aH a
G with aH = aG . We put
AG = exp(aG ), AH = exp(aH ).
For a
G and a
H we write
g(aG ; ) = {X g : [H, X] = (H)X H aG },
g(aH ; ) = {X g : [H, X] = (H)X H aH }.
for the corresponding weight spaces. Let (g, aG ) and (g, aH ) denote the respective non-zero
weights with non-trivial weight spaces, then both sets form root systems. Denote by = |aH
the restriction of a root (g, aG ) to aH , then (g, aH ) {0}. We choose compatible
positive systems + (g, aG ) and + (g, aH ) in the sense that
+ (g, aH ) {0} + (g, aG ).
As usual, for (g, aG ) we write > 0 if + (g, aG ) and < 0 if + (g, aG ).
Further, define the nilpotent subalgebras
M M M
nG = g(aG ; ), n= g(aG ; ) = g(aH ; ).
+ (g,aG ) + (g,aG ) + (g,aH )
6=0
Then n is -stable and therefore we have a direct sum decomposition

n = n n .
Put nH = n and
NG = exp(nG ), N = exp(n), NH = exp(nH ).
Finally, we define
MG = ZK (aG ), L = ZG (aH ), MH = ZHK (aH ).
Then PG = MG AG NG is a minimal parabolic subgroup of G, Q = LN is another parabolic
subgroup of G, and PH = MH AH NH is a minimal parabolic subgroup of H such that
PG Q PH .
1.4. The double coset space PH \G/PG . To study the PH -orbits in G/PG we use the
Bruhat decomposition of G with respect to the parabolic subgroups PG and Q. Let W =
W (aG ) = NK (AG )/AG denote the Weyl group of (g, aG ) and pick a representative w
NK (AG ) for every w W . Then, since G is of Harish-Chandra class, we have the Bruhat
decomposition (see e.g. [44, Proposition 1.2.1.10])
[
G= QwP
G,
wWQ \W
where
WQ = ZW (AH ) = {w W : w L K}.
Since PH Q, every PH -orbit PH gPG is contained in a Bruhat cell Q wP
G G/P . As
homogeneous spaces we have
Q wP
G Q/(Q wP 1 )
Gw
whence we are led to study the PH -orbits in Q/(Q wP Gw1 ) for w W . The following
result is shown in [18, Lemma 3.5 (2)] for strongly spherical symmetric pairs, and we extend
it to the context of strongly spherical reductive pairs:
Lemma 1.8. The extension of aH to aG can be chosen such that

l k = Zk (aH ) = mH + mG and L K = ZK (aH ) = MH MG .
Proof. Let PG = ZK (aH )AH NH , a minimal parabolic subgroup of G . Since (G, H) is
strongly spherical, there exists g G such that PH gPG is open. Now PH PG and hence
PH gPG is contained in an open double coset in PG \G/PG . By [18, Lemma 3.7] such open
double cosets have representatives in exp(n )w e0 so that PH gPG PG exp(X)w
e0 PG for

some X n . Hence, g p exp(X)w e0 PG with p PG so that
e0 PG .
PH gPG = PH p exp(X)w
Now PH p = PH m with m ZK (aH ) and (m1 PH m) exp(X)w
e0 PG G is open. This implies
Ad(exp(X)) Ad(m1 )pH + pG = g,
e0 )pG denotes the opposite parabolic subalgebra. Note that for Z = ZM +
where pG = Ad(w
ZA + ZN Ad(m)1 mH + aH + nH = Ad(m1 )pH we have
Ad(exp(X))Z = Z +Z + (1 Ad(exp(X)))(ZM + ZA ) + Ad(exp(X))ZN
| M {z A} | {z }
Ad(m1 )mH +aH l n
and further
pG = (l pG ) nG .
| {z } |{z}
l n
From the decomposition g = nln it now follows that Ad(m1 )mH +(lpG ) = l. Intersecting
with k we obtain Ad(m1 )mH + mG = l k, or equivalently mH + Ad(m)mG = l k. Replacing
aG by Ad(m)aG changes mG to Ad(m)mG and hence we may assume mH + mG = l k. The
identity L K = MH MG then follows since MG AG L is the Levi factor of a minimal
parabolic subgroup of L and hence MG meets every connected component of L.
For the rest of this section we choose the extension aG of aH as in Lemma 1.8. Then we
obtain the following generalization of [18, Lemma 3.7] to not necessarily symmetric (G, H):
Lemma 1.9. For every w W we have
Q = MH NH exp(n )(L wP 1 ).
Gw
In particular, every PH -orbit in Q/(Q wP 1 ) has a representative of the form exp(X)
Gw

for some X n , or equivalently every PH -orbit in G/PG has a representative of the form
exp(X)w for some X n , w W .
Proof. First note that wP
Gw 1 is a minimal parabolic subgroup of G. Since L is a reductive
subgroup of G, the intersection L wP
Gw1 is parabolic in L. By the Iwasawa decomposition
for L we find that L = (L K)(L wP Gw 1 ). Now, L K = MH MG by Lemma 1.8, whence
L = MH MG (L wP 1 ) = MH (L wP
Gw 1 ).
Gw
Inserting this into the Langlands decomposition for Q we find
Q = N L = N MH (L wP 1 ) = MH N (L wP
Gw 1 ).
Gw
Finally, N = NH exp(n ) by [18, Lemma 3.6] and the proof is complete.
12
JAN MOLLERS
1.5. PH -orbits in the open Bruhat cell. Let w0 W denote the longest Weyl group
element. Then the Bruhat cell Qw 0 PG is open and dense in G. Hence, QwP G is the unique
open Bruhat cell.
Now let us consider the double cosets PH gPG which are contained in the open Bruhat cell
Qw 0 PH . These can be identified with the orbits of the adjoint action of (MG MH )AH on
n :
Lemma 1.10. The natural inclusion
exp
n N Q
induces a bijection
n /(MG MH )AH PH \Q/(Q w 01 ).
0 PG w (1.1)
Proof. The proof of [18, Lemma 3.7] for the case of symmetric pairs translates literally to our
situation.
Since aH preserves n we can write
M
n = g (aH ; ), (1.2)
(n )
where g (aH ; ) = g(aH ; ) g and

(n ) = { + (g, aH ) : g(aH ; ) n 6= {0}}.
Clearly MH preserves this decomposition. We can therefore endow n with an MH -invariant
inner product such that the decomposition (1.2) is orthogonal. For each (n ) denote
by S g (aH ; ) the unit sphere with respect to this inner product. We then have the
following version of [18, Proposition 3.11]:
Lemma 1.11. If the pair (G, H) is strongly spherical, then:
Q
(1) The orbits of (MG MH ) on (n ) S are unions of connected components, in
particular open and compact,
(2) (n ) is a basis of a
H.
Proof. The proof of [18, Proposition 3.11] implies (1) and linear independence of (n )
in (2). Assume that (n ) does not span a H , then there exists 0 6= H aH such that
(H) = 0 for all (n ), or equivalently ad(H)|n = 0. But this implies H pH
Ad(eX w 0 )pG = pH Ad(eX )pG for all X n . Since by Lemma 1.9 every open double
coset in PH gPG PH \G/PG has a representative of the form g = eX w 0 , this shows that the
projection of pH Ad(g)pG to aH is non-trivial, which contradicts Theorem 1.4.
1.6. Proof of Theorem 1.4. Let PH gPG be a non-open PH -orbit. By Lemma 1.9 there ex-
ists p PH such that pgPG = exp(X)wP G for some X n and w W . Then the stabilizers
of gPG and exp(X)wP G are conjugate in PH via p. Since the projection of Ad(p)aH pH to
aH is equal to aH , we may without loss of generality assume that g = exp(X)w.

Write
S(X, w)
= PH (exp(X)wP 1 exp(X)) = {p PH : p exp(X) exp(X)(wP
Gw 1 )}
Gw
for the stabilizer of exp(X)wP
G in PH and
s(X, w)
= {Z pH : exp(RZ) exp(X) exp(X)(wP 1 )} = {Z pH : e ad(X) Z w pG }
Gw
for its Lie algebra. Note that the Lie algebra w pG of wP

Gw 1 is given by
M
w pG = mG aG g(aG ; ).
(g,aG )
w 1 >0
In view of the decomposition (1.2) we can write
X
X= X
(n )
with X g (aH ; ).
1.6.1. Reduction to generic X. Assume first that X = 0 for some (n ). It follows
from Lemma 1.11 (2) that the intersection
\
ker aH
(n )\{}
is non-trivial, so there exists ZA aH \ {0} with (ZA ) = 0 for all 6= . Then [ZA , X] = 0
since X = 0 and [ZA , X ] = 0 for 6= , and hence etZA eX = eX etZA for all t R. Since
ZA aH w pG this implies ZA s(X, w).
We may therefore assume X 6= 0 for all (n ) for the rest of the proof. By
Lemma 1.11 this guarantees that the (MG MH )AH -orbit of X in n is open.
1.6.2. Double cosets in the open Bruhat cell. Now suppose QwP G = Qw 0 PG , i.e. the double
coset PH exp(X)wP G is contained in the open Bruhat cell Qw
0 PG . Without loss of generality
we may assume w = w0 . If the double coset PH exp(X)w0 PG is not open, then by Lemma 1.10
the (MG MH )AH -orbit of X in n is not open. By Lemma 1.11 this implies that one of
the X must vanish, the case we already treated in Section 1.6.1.
1.6.3. Double cosets in the non-open Bruhat cells. Now assume that QwP
G 6= Qw
0 PG .
Lemma 1.12. Let w W . If QwP 0 PG , then there exists + (g, aG )w+ (g, aG )
G 6= Qw
with 6= 0.
Proof. Assume w1 < 0 for all > 0 with 6= 0. The double coset QwP G is the orbit of
w under the action of Q PG on G given by (q, p) g = qgp1 . Then the stabilizer of w in
Q PG is given by
1 q w)
{(q, w : q Q wP
Gw1 }
and hence its Lie algebra is isomorphic to q w pG . Since aG q w pG we can decompose
this Lie algebra into root spaces with respect to aG :
M M M
q w pG = mG aG g(aG ; ) g(aG ; ) = mG aG g(aG ; ).
=0 6=0 =0
w 1 >0 >0, w 1 >0 w 1 >0
Now, if = 0 then either w1 > 0 or w1 () = w1 > 0, and therefore

X
dim(q w pG ) = dim mG + dim aG + dim g(aG ; ) = dim pG dim n.
>0
=0
Hence,
dim(QwP
G ) = dim q + dim pG dim(q w pG ) = dim q + dim n = dim g
14
JAN MOLLERS
so that QwP
G must be open, whence equal to the unique open Bruhat cell Qw
0 PG . This
contradicts the assumption QwP
G 6= Qw
0 PG and the proof is complete.
Choose any root > 0 with 6= 0 and w1 > 0, then g(aG ; ) q w pG . Let us fix
Y g(aG ; ) for now; later we will specify an appropriate choice of Y . Then etY N and
also eX etY N for all t R. By [18, Lemma 3.6] we have N = N exp(n ) so that we can
uniquely write
eX etY = nt eXt
with nt N and Xt n depending differentiably on t R. Recall that we may assume
X 6= 0 for all (n ), so that X is contained in an open (MG MH )AH -orbit in
n . Then there exists an interval (, ) such that Xt belongs to the open orbit Ad((MG
MH )AH )X, so there exists mt at (MG MH )AH such that Ad(mt at )X = Xt . Clearly, mt at
can be chosen to depend differentiably on t with m0 = a0 = 1. Summarizing, we have
eX etY = nt eAd(mt at )X , t (, ). (1.3)
Denoting pt = nt at mt NH AH (MG MH ) PG PH we have
pt eX = eX etY mt at eX (wP 1 ),
Gw
whence pt S(X, w).
Now put

d
Z = pt s(X, w)

dt t=0
and write Z = ZM + ZA + ZN (mG mH ) aH nH . It remains to show that Y g(aG ; )
can be chosen such that ZA 6= 0. For this we use the following identity which follows by
taking the left logarithmic derivative of (1.3):
Y = e ad(X) Z (ZM + ZA ) = (e ad(X) 1)(ZM + ZA ) + e ad(X) ZN . (1.4)
Now, write X
ZN = ZN,
(n )
with ZN, g (aH ; ).
Lemma 1.13. If Y g(aG ; ) for > 0 with = 6= 0 and w1 > 0, and Z =
ZM + ZA + ZN (mG mH ) aH nH satisfies (1.4), then
Y = ad(ZM + ZA )X + ZN, .
Proof. We can enumerate the positive aH -roots as + (g, aH ) = {1 , . . . , p } so that j 6< i
whenever i j. Form the nilpotent subalgebras
p
M
ni = g(aH ; k ) n,
k=i
then n1 = n and [n, ni ] ni+1 . Since > 0 with 6= 0 there exists 1 i p with = i .
We first prove by induction that
ad(ZM + ZA )Xj = 0 = ZN,j 1 j < i.
For j = 1 < p, the 1 -component of Y is trivial, and therefore, taking the 1 -component of
(1.4) yields
0 = ad(ZM + ZA )X1 + ZN,1 .
Since ad(ZM + ZA )X1 n and ZN,1 n this implies ad(ZM + ZA )X1 = 0 = ZN,1 .
For the induction step assume that ad(ZM + ZA )Xk = 0 = ZN,k for 1 k j 1. If j < i
then the j -component of Y is trivial, and we can again take the j -component of (1.4) and
find
0 = ad(ZM + ZA )Xj + ZN,j .
The same argument as above shows ad(ZM + ZA )Xj = 0 = ZN,j . Finally, taking the
i -component in (1.4) gives the desired identity.
To choose Y g(aG ; ) such that ZA 6= 0 we need to relate g(aG ; ) and g(aH ; ).
Lemma 1.14. Let + (g, aG ) with 6= 0. Then precisely one of the following three
statements holds:
(1) 6= and g (aH ; ) = {Y Y : Y g(aG ; )} =
6 {0},
(2) = and g (aH ; ) = g(aG ; ) n 6= {0},
(3) = and g(aG ; ) nH .
Proof. Assume first that 6= , then (g(aG ; )) = g(aG ; ) and the map
g(aG ; ) g (aH ; ), Y 7 Y (Y )
is non-trivial and (MG MH )-equivariant. Since g (aH ; ) is irreducible under the action
of MG MH by Lemma 1.11 (1), the map must be a linear isomorphism, so (1) holds.
Now assume, = , then g(aG ; ) is -stable and we have a decomposition into eigenspaces
g(aG ; ) = (g(aG ; ) nH ) (g(aG ; ) n ).
If g(aG ; ) n 6= {0}, then this is a non-trivial (MG MH )-invariant subspace of g (aH ; )
and the latter is irreducible under the action of MG MH . This implies (2). The remaining
possibility is g(aG ; ) n = {0} which clearly implies (3).
Before we can finish the proof we need to study case (3) in Lemma 1.14 in more detail. For
this, the following two results will be used:
Lemma 1.15 ([35, Lemma 2.9]). Let 1 , 2 (g, aG ) and assume that (1 , 2 ) < 0. Then
for any X1 g(aG ; 1 ) and X2 g(aG ; 2 ), X1 , X2 6= 0, we have [X1 , X2 ] 6= 0.
Lemma 1.16. Let w W , then for any + (g, aG ) w+ (g, aG ) there exists a sequence
= 1 , . . . , r + (g, aG ) w+ (g, aG ) such that hi , i+1 i 6= 0 for i = 1, . . . , r 1 and r
is simple in + (g, aG ).
This result and its proof were communicated to us by Yoshiki Oshima.
Proof. Let ww0 = s1 sn be a reduced expression for ww0 W , i.e. 1 , . . . , n are simple
roots. It is known that

+ (g, aG ) w+ (g, aG ) = + (g, aG ) ww0 + (g, aG ) = {s1 sk1 k : 1 k n},
and we can write 1 = = s1 sk1 k . If 1 is not a simple root, there exists 1 i < k
such that si+1 sk1 k is simple, but si sk1 k is not. In particular, si+1 sk1 k 6=
si sk1 k and hence hi , si+1 sk1 k i 6= 0, which implies hs1 si1 i , s1 sk1 k i =
6
0. Put 2 := s1 si1 i , then h1 , 2 i 6= 0. Repeating this arguments yields the desired
sequence.
The next lemma provides more information about case (3) in Lemma 1.14:
16
JAN MOLLERS
Lemma 1.17. Let w W with QwP G= 6 Qw0 PG . If for all + (g, aG ) w+ (g, aG ) with
6= 0 we have g(aG ; ) nH , then for one of those we must have [g(aG ; ), n ] 6= {0}.
Proof. Let w W with QwP G 6= Qw 0 PG and assume that g(aG ; ) nH for all
(g, aG )w (g, aG ) with 6= 0. We explicitly construct a root + (g, aG )w+ (g, aG )
+ +
with 6= 0 such that [g(aG ; ), n ] 6= {0}.

Step 1. If QwP G 6= Qw 0 PG , then there exists + (g, aG ) w+ (g, aG ) with 6= 0
by Lemma 1.12. We claim that there exists a simple root with this property. In fact, by
Lemma 1.16 there exists a sequence = 1 , . . . , r + (g, aG )w+ (g, aG ) with hi , i+1 i =
6
0 for i = 1, . . . , r 1 and r simple. By our assumption, every + (g, aG ) w+ (g, aG )
satisfies either = 0 or g(aG ; ) nH . By Lemma 1.14 this implies = or = .
These two types of roots are obviously orthogonal to each other. Hence, = implies
i = i for all roots i in the sequence. In particular, g(aG ; r ) nH by Lemma 1.14, and
we can replace by the simple root r .
Step 2. Next, we claim that there exists a simple root + (g, aG ) with 6= 0 and
g(aG ; ) 6 nH . Assume that such a simple root does not exist. Then for every simple root
+ (g, aG ) we have either = 0 or g(aG ; ) nH . This implies that either = or
= , so that the set of simple roots is the disjoint union of the two mutually orthogonal
subsets { + (g, aG ) simple, = }. First note that this cannot occur in the case
(g, h) = (g + g , diag g ), so that we may assume g to be simple. Then the Dynkin diagram of
(g, aG ) is connected and we must have = for all simple roots, whence g(aG ; ) nH
for all simple roots. This implies g(aG ; ) nH for all positive roots and therefore nG = nH
and also nG = nH . But nG and nG generate g, whence g = h which contradicts our assumption
that (g, h) is non-trivial.
Step 3. By Step 1 we find a simple root + (g, aG ) w+ (g, aG ) with 6= 0 and
hence g(aG ; ) nH . We claim that [g(aG ; ), n ] 6= {0}. To see this, we use Step 2
to find another simple root + (g, aG ) with 6= 0 and g(aH ; ) 6 nH . Connect-
ing and in the Dynkin diagram for (g, aG ), we obtain a sequence of simple roots
= 1 , 2 , . . . , p1 , p = such that (i , j ) 6= 0 if and only if |i j| 1. By pos-
sibly replacing p by one of the i we may assume that for all 1 i p 1 we either
have i = 0 or g(aG ; i ) nH . We now construct a root = n2 2 + + np p , ni 1,
such that 6= 0 and g(aG ; ) 6 nH , then [g(aG ; ), g(aG ; )] 6= {0} by Lemma 1.15 since
(, ) = n2 (1 , 2 ) < 0. By Lemma 1.14 this implies [g(aG ; ), n ] 6= {0}.
We inductively construct a root k = nk k + + np p (2 k p) with k 6= 0 and
g(aG ; k ) 6 nH . Note that for k of the above form we always have k = nk k + +np p 6= 0
since i is either = 0 or a positive root, and by assumption p 6= 0. For k = p we can choose
the simple root p = p . Now assume k+1 = nk+1 k+1 + + np p has been constructed
with g(aG ; k+1 ) 6 nH . Then by Lemma 1.14 there are four possibilities for k+1 and k .
(1) k+1 = k+1 and k 6= 0. Then g(aG ; k+1 ) n 6= {0} and g(aG ; k ) nH , hence
by Lemma 1.15
6 [g(aG ; k ), g(aG ; k+1 ) n ] g(aG ; k + k+1 ) n ,

{0} =
so that k = k + k+1 satisfies g(aG ; k ) 6 nH .

(2) k+1 = k+1 and k = 0. Then g(aG ; k+1 ) n 6= {0} and k = k . Hence, we
have for any X g(aG ; k+1 ) n and Y g(aG ; k ), X, Y 6= 0, that 0 6= [X, Y ]
g(aG ; k + k+1 ) by Lemma 1.15 and [X, Y ] = [X, Y ] g(aG ; k + k+1 ).
Therefore, [X, Y ] 6= [X, Y ] and hence g(aG ; k + k+1 ) 6 nH , so that we can choose
k = k + k+1 .
(3) k+1 6= k+1 and k 6= 0. Then {X X : X g(aG ; k+1 )} n and g(aG ; k )
nH . Hence, we have for any X g(aG ; k+1 ) and Y g(aG ; k ), X, Y 6= 0, that 0 6=
[X, Y ] g(aG ; k + k+1 ) by Lemma 1.15 and [X, Y ] = [X, Y ] g(aG ; k + k+1 ).
Therefore, [X, Y ] 6= [X, Y ] and hence g(aG ; k + k+1 ) 6 nH , so that we can choose
k = k + k+1 .
(4) k+1 6= k+1 and k = 0. Then {X X : X g(aG ; k+1 )} n and k = k .
Let X g(aG ; k+1 ) and Y g(aG ; k ), X, Y 6= 0, then by Lemma 1.15 we have
[X, Y ] 6= 0.
(a) If [X, Y ] 6= [X, Y ] then we can choose k = k + k+1 as in (1), (2) and (3).
(b) If [X, Y ] = [X, Y ] then (k + k+1 ) = k + k+1 so that k+1 = 2k + k+1 .
In this case we choose k = k+1 = 2k + nk+1 k+1 + + npp which is clearly
a positive root. Further, g(aG ; k ) 6 nH since k = k+1 6= k .
Inductively, for k = 2 this produces the desired root = 2 .
We can finally finish the proof of Theorem 1.4 by choosing Y g(aG ; ) according to the
three cases in Lemma 1.14. Write = .
(1) If 6= we can write X = Y Y for some Y g(aG ; ). Using this Y in the
above construction, we have by Lemma 1.13
Y = ad(ZM + ZA )X + ZN, .
Both sides are contained in n = n n , and taking n -components gives 2X =
ad(ZM + ZA )X = ad(ZM )X + (ZA )X . Note that we assume X 6= 0 by the
reduction in Section 1.6.1. Then ad(ZM )X = (2 (ZA ))X so that ZM acts by a
scalar on X . Since ZM mG mH and MG MH is compact, this scalar has to be
imaginary, so that (ZA ) = 2. In particular, ZA 6= 0.
(2) If = and g (aG ; ) = g (aH ; ) we take Y = X . Then Lemma 1.13 implies
X = Y = ad(ZM + ZA )X = ad(ZM )X + (ZA )X .
The same argument as in (1) shows (ZA ) = 1 and in particular ZA 6= 0.
(3) If (1) and (2) do not hold for any such , we have = and g(aG ; ) nH for all
> 0 with w1 > 0, 6= 0. Then Lemma 1.17 implies that there exists an with
[g(aG ; ), n ] 6= {0}. Let 0 6= Y g(aG ; ) and 0 6= Bi g (aH ; i ) (i = 1, 2) with
1 , 2 (n ), such that [Y, B1 ] = B2 . By Lemma 1.11 there exists ma (MG
MH )AH such that Ad(ma)Bi = Xi (i = 1, 2). Hence, [Ad(ma)Y, X1 ] = X2 .
Replacing Y by Ad(ma)Y g(aG ; ) we may therefore assume that [Y, X1 ] = X2 .
Note that
tY
eX etY = etY eAd(e )X NH exp(n ),
and Ad(etY )X = et ad(Y ) X n so that nt = etY and Xt = Ad(mt at )X =
et ad(Y ) X. Hence, ZN = Y and [ZM + ZA , X] = [Y, X]. Taking the 2 -component
gives
[ZM , X2 ] + 2 (ZA )X2 = [Y, X1 ] = X2 .
By the same argument as in (1) and (2) we find 2 (ZA ) = 1 and hence ZA 6= 0.
This finishes the proof of Theorem 1.4.
18
JAN MOLLERS
2. Spherical matrix coefficients

We study matrix coefficients of finite-dimensional representations of G which are equivari-
ant under the action of PH PG by left and right multiplication. Such matrix coefficients
correspond to finite-dimensional spherical representations of G whose restriction to H con-
tains a spherical representation, and we show that there exist enough such representations
(see Theorem 2.6). In Section 3 these matrix coefficients are used to explicitly construct
symmetry breaking operators.
2.1. Reduction to complex connected groups. Since both G and H might be discon-
nected, their finite-dimensional representations are not easily described in terms of highest
weights. To overcome this difficulty we first reduce the construction of matrix coefficients to
the case of complex connected groups. Since G is of Harish-Chandra class, there exist
a complex connected linear reductive group GC with Lie algebra gC ,
an antiholomorphic involution : GC GC such that the derived involution : gC
gC is the conjugation with respect to the real form g,
a homomorphism : G G from G to the real form G = GC of GC with finite kernel
and cokernel.
Note that the Lie algebra of G is equal to g. We denote by HC the complex connected subgroup
of GC with Lie algebra hC and by H = (H)0 = (H0 ) HC the connected subgroup of HC
with Lie algebra h. Then the finite-dimensional holomorphic representations of GC and HC
are classified in terms of their highest weights, and via the homomorphism they give rise
to finite-dimensional representations of G and H0 .
The image K = (K) of the maximal compact subgroup of G under is a maximal compact
subgroup of G, and the intersection H K is maximal compact in H. Further, let
MG = (MG ), AG = (AG ) and NG = (NG ),
then PG = (PG ) = MG AG NG is a minimal parabolic subgroup of G.
2.2. The CartanHelgason Theorem. We now recall the classification of irreducible finite-
dimensional spherical representations of G in terms of their highest weights, the so-called
CartanHelgason Theorem. Recall that a representation of G is called spherical if it contains
a non-zero K-invariant vector.
We choose a maximal abelian subalgebra tG in mG , then jG = tG aG is a Cartan subalgebra
of g and jG,C is a Cartan subalgebra of gC . Roots in (gC , jG,C ) are real on aG and imaginary on
tG . Fix a positive system + (gC , jG,C ) such that the non-zero restrictions to aG are contained
in + (g, aG ). With respect to this data, the irreducible finite-dimensional representations of
G are classified by their highest weights in j G,C .
Recall the following theorem (see e.g. [12, Theorem 8.49]):
Theorem 2.1 (CartanHelgason). For an irreducible finite-dimensional representation of
G the following statements are equivalent:
(1) has a non-zero K-fixed vector.
(2) MG acts by the 1-dimensional trivial representation in the highest restricted weight
space of .
(3) The highest weight of vanishes on tG , and its restriction to aG is contained in the
set
+ (g, aG ) = { a 2 +
G : h, i/|| N (g, aG )}.
Let + +
G (g, aG ) (g, aG ) denote the subset of all for which there exists an irreducible
finite-dimensional G-representation ( , V ) of highest weight . Then spanR + G (g, aG ) =
+
spanR + (g, aG ) = aG , and if G is simply connected even G (g, aG ) = + (g, a ). Theorem 2.1
G
immediately gives the action of PG on the highest weight space of V :
Corollary 2.2. For every + G (g, aG ) the minimal parabolic subgroup PG = MG AG NG
acts on the highest weight space of V by the character 1 e 1.
Here 1 denotes the trivial representation of MG and NG , respectively, and e is the character
of AG = exp(aG ) given by e (eX ) = e(X) , X aG .
Now, for an irreducible finite-dimensional representation (, V ) of G we denote by ( , V )
the contragredient representation on the dual space V = HomC (V, C) given by
h (g), vi = h, (g1 )vi, g G, v V, V .
The following statement is standard:
Lemma 2.3. (1) (, V ) has a non-zero K-fixed vector if and only if ( , V ) has a non-
zero K-fixed vector.
(2) For + +
G (g, aG ) let G (g, aG ) be defined by ( , V ) ( , V ), then the
map
+ +
G (g, aG ) G (g, aG ), 7
is the restriction to +
G (g, aG ) of a linear map aG aG .
2.3. Matrix coefficients. As for G, we denote by +

H (h, aH ) aH the set of all highest
weights of irreducible finite-dimensional representations ( , W ) of H which have a one-
dimensional PH -invariant subspace isomorphic to 1 e 1, where PH = MH AH NH is the
corresponding minimal parabolic subgroup of H
Lemma 2.4. Let + +
G (g, aG ) and H (h, aH ) and pick non-zero highest weight vectors
v0 V and 0 W . Then for every 0 6= HomH ( |H , ) the function
f : G C, f (g) = h0 , ( (g)v0 )i
is non-zero, real-analytic and satisfies

f (m a n gman) = a (a ) f (g)
for g G, man PG and m a n PH .
Proof. It is clear that f is real analytic as a matrix coefficient of a finite-dimensional represen-
tation. Further, f is non-zero since ( , V ) is irreducible. By Corollary 2.2 and Lemma 2.3

we have (man)v0 = a v0 and (m a n )0 = (a ) 0 and the claim follows.
Note that since ( , V ) and ( , W ) extend to holomorphic representations of the complex

connected groups GC and HC , we have HomH ( |H , ) = Homh ( |h , ). Abusing notation,
we also write ( , V ) and ( , W ) for the Lie algebra representations of g and h for arbitrary
(, ) + (g, aH ) and + (h, aH ).
To obtain matrix coefficients with the same properties as in Lemma 2.4, but for the pair
(G, H) instead of (G, H), we use the homomorphism : G G.
20
JAN MOLLERS
Proposition 2.5. Assume that (G, H) is a strongly spherical reductive pair. Then for each
pair (, ) + (g, aG ) + (h, aH ) with Homh ( |h , ) 6= {0} there exists k 1 and a
non-zero real-analytic function F : G R, F 0, satisfying

F (m a n gman) = ak (a )k F (g) (2.1)
for g G, man PG and m a n PH .
Proof. First note that there exists k 1 such that k + +
G (g, aG ) and k H (h, aH ). Then
also Homh (k |h , k ) 6= {0} and by Lemma 2.4 there exists a non-zero real-analytic function
f : G C with

f (m a n gman) = ak (a )k f (g)
for g G, man PG and m a n PH . Replacing f by |f |2 we may further assume that
f : G R and f 0. We consider the non-zero real-analytic function f : G R
which satisfies (2.1) at least for m MH,0 1 (MH ) MH . Since the component group
MH /MH,0 of MH is finite, we can form the finite sum
X
F (g) = f ((mg)),
mMH,0 MH /MH,0
and this clearly defines a real-analytic function F : G R, F 0, with the equivariance

property (2.1). Finally, F is non-zero since f 0 and f 6= 0.
The main result of this section asserts that for all strongly spherical reductive pairs (G, H)
there exist enough pairs (, ) a
G aH with Homh ( |h , ) 6= {0}:
non-trivial and strictly indecomposable. Then the set of pairs (, ) + (g, aG ) + (h, aH )
such that Homh ( |h , ) 6= {0} spans a
G aH .
Before we come to the proof of this result, let us see how it can be used to show the
implication (1)(2) in Theorem 1.4:
Corollary 2.7. If the double coset PH gPG is open, then the projection of pH Ad(g)pG to
aH is trivial.
Proof. Let PH gPG be an open double coset and Z = ZM + ZA + ZN pH Ad(g)pG . Put
X = Ad(g)1 Z = XM + XA + XN pG . For any (, ) + (g, aG ) + (h, aH ) with
Homh ( |h , ) 6= {0} let F be as in Proposition 2.5. Since F is non-zero and real-analytic,
it is non-zero on the open set PH gPG . In particular, F (g) 6= 0 and for all t R we have
(Z
et A)
F (g) = F (etZ g) = F (getX ) = et(XA ) F (g),
so that (XA ) + (ZA ) = 0. Since the pairs (, ) + (g, aG ) + (h, aH ) satisfying
Homh ( |h , ) 6= {0} span a
G aH this implies XA = 0 and ZA = 0 and the proof is
complete.
Note that the statement in Theorem 2.6 only depends on the pair of Lie algebras (g, h). If
we define
(g, h) = {(, ) + (g, aG ) + (h, aH ) : Homh ( |h , ) 6= {0}},
then we have to show that (g, h) spans a
G aH . Note that (g, h) is a subsemigroup of
+ +
(g, aG ) (h, aH ). We first reduce Theorem 2.6 to the case of g semisimple and (g, h)
symmetric and then use the classification in Theorem 1.6 to show the statement case-by-case.
Lemma 2.8. Assume that Theorem 2.6 holds for g semisimple, then it holds for g reductive.
Proof. As in Section 1.2 we write g = gn gel and p : g gn for the canonical projection.
Then aG = aG,n aG,el with aG,n = aG gn and aG,el = aG gel , and by Lemma 1.5 (1)
the restriction p|aH : aH aG,n is injective. Further, by Lemma 1.5 (2) the pair (gn , p(h)) is
also strongly spherical, so by assumption (gn , p(h)) spans a
G,n p(aH ) . Now for every pair
(0 , 0 ) (gn , p(h)) the pair (, ) a
G aH with |aG,n = 0 , |aG,el = 0 and = 0 p|aH
is contained in (g, h). Hence, the span of (g, h) contains at least a
G,n aH . Further, for
every 1 a G,el the representation is one-dimensional and hence its restriction is an h k-
spherical representation of h with highest weight 1 a H , so that (1 , 1 ) (g, h). This
shows that (g, h) indeed spans a G a .
H
Lemma 2.9. Assume that Theorem 2.6 holds for (g, h) symmetric, then it holds for (g, h)
reductive.
Proof. By the previous lemma we may assume that g is semisimple. Then, thanks to Corol-
lary 1.7, there exists a non-trivial involution of g such that h g , and h and g differ only
in compact factors. Hence, + (h, aH ) = + (g , aH ) and (g, g ) (g, h).
2.4. Finite-dimensional branching. We now prove Theorem 2.6 case-by-case for all sym-
metric pairs in the classification in Section 1.2. For this we first fix some notation.
Let jH h be a Cartan subalgebra of h and extend it to a Cartan subalgebra jH jG g
of g. Note that we do no longer assume that aG jG and aH jH . Choose a system of
positive roots + (gC , jG,C ) for g such that
+ (hC , jH,C ) = {|jH,C : + (gC , jG,C )} (hC , jH,C )
is a system of positive roots for h. Denote by 1 , . . . , s the fundamental weights for g
with respect to + (gC , jG,C ) and by 1 , . . . , t the fundamental weights for h with respect to
+ (hC , jH,C ). Then any dominant integral weight of g with respect to + (gC , jG,C ) is of the
form = k1 1 + + ks s , k1 , . . . , ks N, and we write F g () for the corresponding
finite-dimensional representation of g. The analogous notation is used for h and ideals of h.
To simplify some statements we further put 0 := 0 so that F h (0 ) is the trivial representation
of h.
We make use of the Satake diagrams for g and h (see e.g. [9, Chapter X, Appendix F] for
details). From the Satake diagram the highest weights belonging to spherical representations
can be read off. In fact, for every simple root i whose vertex in the Satake diagram is white
and not linked to any other vertex by an arrow, the representations F g (2ki ) (k N) are
spherical. If the vertices of two simple roots i and j are white and linked by an arrow, then
2k(i + j ) (k N) are highest weights of spherical representations. Moreover, if F g () and
F g ( ) are spherical, then F g ( + ) is spherical. In many cases we compute the explicit
branching for F g (i ) resp. F g (i + j ) and then use the semigroup property of (g, h) to
conclude that the spherical representation F g (2i ) resp. F g (2(i + j )) contains a certain
spherical h-representation.
The following reduction from complex Lie algebras to split real forms allows to minimize
the number of different cases:
Lemma 2.10. Let (g, h) be a reductive pair with g and h split. If the statement in Theorem 2.6
holds for (g, h), then it also holds for (gC , hC ) viewed as real Lie algebras.
22
JAN MOLLERS
Proof. Since g and h are split, we can choose jG = aG and jH = aH . Let u = k + ip gC ,

then u is maximally compact in gC with complement u = ik + p. Further, aG,C is a (real)
Cartan subalgebra of gC which splits into aG,C = iaG + aG with iaG u and aG u .
By the CartanHelgason Theorem, the highest weights of u-spherical representations of gC
vanish on iaG and their restrictions to aG are contained in + (gC , aG ) = + (g, aG ). If ( , V )
denotes a k-spherical representation of g with highest weight + (g, aG ), then a u-spherical
representation of gC with highest weight 2 is given by V V where gC acts by
X(v v ) = (Xv) v + v (Xv ), X gC , v, v V .
Now let (, ) + (g, aG ) + (h, aH ). Then for any A Homh (V , W ) we clearly have
A A HomhC (V V , W W ). Hence, (, ) (g, h) implies (2, 2) (gC , hC ) and
the claim follows.
Finally, we prove Theorem 2.6 case-by-case for all strongly spherical symmetric pairs in
the classification of Theorem 1.6:
A) Trivial case. This case g = h is by assumption excluded.
C) Compact case. Let g be the Lie algebra of a compact simple Lie group, then also h is the
Lie algebra of a compact group and aG = aH = {0} so that (g, h) = a
G aH = {0} {0}
holds trivially.
D) Compact subgroup case. Let h = k be the Lie algebra of a maximal compact subgroup
K of a non-compact simple Lie group G with Lie algebra g. Then aH = {0} and the only
spherical representation of h is the trivial representation W0 = C. By definition, for every
+ (g, aG ) the representation V contains a k-fixed vector, hence also V contains a k-
fixed vector by Lemma 2.3. But k-fixed vectors in V = HomC (V , C) are simply k-equivariant
homomorphisms from V to the trivial representation W0 of k. Thus, (g, h) = + (g, aG ){0}
spans a
G {0} = aG aH .
E1) (g, h) = (so(1, p + q), so(1, p) + so(q)). The Satake diagrams of g and so(1, p) h are
s1

1 2 s1 s 1 2 s2
+3
g:

s
(p + q = 2s) (p + q = 2s 1)
t1
1 2 t1 t 1 2 t2
+3
so(1, p) :

t
(p = 2t) (p = 2t 1)
We realize the root system of g as {ei ej : 1 i < j s} and additionally {ei : 1
i s} if p + q is even. Choose the simple roots i = ei ei+1 (1 i s 1) and s = es for
p + q even and s = es1 + es for p + q odd. We distinguish two cases:
Assume first that p + q is even or p is odd, then jG = jH . If we choose the simple roots
for so(1, p) h as i = i (1 i t 1) and t = et for p even and t = et1 + et
for p odd, then i = i (t + 1 i s 1) and s = es1 + es for q even and s = es
for q odd are the simple roots for so(q) h.
Assume now that p + q is odd and p is even, then we can choose jH jG such that
es |jH,C = 0 and the simple roots for so(1, p) h are i = i |jH,C (1 i t 1) plus
t = et |jH,C , and the simple roots for so(q) h are i = i |jH,C (t + 1 i s 1).
Consider the fundamental weight 1 = e1 . Clearly e1 |jH,C = 1 is also a highest weight for
so(1, p) h and hence (21 , 21 ) (g, h). Further, from the Satake diagram for the real
form so(p + 1, q) of gC so(p + q + 1, C) it follows that F g (21 ) is h-spherical and hence also
(21 , 0) (g, h). Clearly (21 , 21 ) and (21 , 0) span a
G aH which is 2-dimensional.
E2) (g, h) = (su(1, p + q), s(u(1, p) + u(q))). The Satake diagrams of g and h are
1 2 p+q1 p+q
g: g 7
1 2 p1 p p+2 p+q
h: g 7
We realize the root system of g as {(ei ej ) : 1 i < j p + q + 1} in the vector

space {x Rp+q+1 : x1 + + xp+q+1 = 0} and choose the simple roots i = ei ei+1
(1 i p + q). If we choose the simple roots i = i (1 i p) for su(1, p) h, then
i = i (p + 2 i p + q) are simple roots for su(q) h and the fundamental weight p+1
describes a character of u(1) h.
Consider the dominant integral weight 1 + p+q = e1 ep+q+1 . In the Weyl group
orbit of e1 ep+q+1 the weight e1 ep+1 = 1 + p is a highest weights for h and hence
(2(1 + p+q ), 2(1 + p )) (g, h). Further, from the Satake diagram for the real form
su(p + 1, q) of gC sl(p + q + 1, C) it follows that F g (2(1 + p+q )) is h-spherical and hence
also (2(1 + p+q ), 0) (g, h). Clearly (2(1 + p+q ), 2(1 + p )) and (2(1 + p+q ), 0)
span a
E3) (g, h) = (sp(1, p + q), sp(1, p) + sp(q))). The Satake diagrams of g and h are
1 2 3 p+q p+q+1
g: ks
1 2 3 p p+1 p+2 p+q p+q+1

h: ks ks
We realize the root system of g as {ei ej : 1 i < j p+q+1}{2ei : 1 i p+q+1}
and choose the simple roots i = ei ei+1 (1 i p + q) and p+q+1 = 2ep+q+1 . If we
choose the simple roots i = i (1 i p) and p+1 = 2ep+1 for sp(1, p) h, then i = i
(p + 2 i p + q + 1) are simple roots for sp(q) h.
Consider the fundamental weight 2 = e1 + e2 . Clearly e1 + e2 = 2 is also a highest weight
for sp(1, p) h and hence (22 , 22 ) (g, h). Further, from the Satake diagram for the real
form sp(p + 1, q) of gC sp(p + q + 1, C) it follows that F g (22 ) is h-spherical and hence also
(22 , 0) (g, h). Clearly (22 , 22 ) and (22 , 0) span a
24
JAN MOLLERS
E4) (g, h) = (f4(20) , so(8, 1)). The Satake diagrams of g and h are
1 2 3 4
g: +3
1 2 3 4
h: +3
By [24] we have
F g (4 )|h F h (1 ) F h (4 ) F h (0)
and hence (24 , 21 ), (24 , 0) (g, h) span a
F1) (g, h) = (sl(n + 1, C), gl(n, C)). This follows from F3) with Lemma 2.10.
F2) (g, h) = (so(n + 1, C), so(n, C)). This follows from F5) with Lemma 2.10.
F3) (g, h) = (sl(n + 1, R), gl(n, R)). The Satake diagrams of g and h are
1 2 n1 n
g:
1 2 n1
h:
We choose i = i (1 i n 1) as simple roots for sl(n, R) h, then n := n defines a
non-trivial character of R h. By Appendix A.1 the pairs (2i , 2i + 2 ni n ) (1 i n 1),
(2i , 2i1 2 ni+1
n n ) (2 i n) and (21 , 2n ), (2n , 2n ) are contained in (g, h) and
they clearly span a
G aH which is 2n-dimensional.
F4) (g, h) = (su(p, q + 1), u(p, q)). Set s = min(p, q + 1) and t = min(p, q), then the Satake
diagrams of g and h are

1 s s+1 p+qs p+qs+1 p+q

k k
3 3 for p 6= q + 1,

g:

1 s1 s s+1 p+q

j g 8 4 for p = q + 1,

1 t t+1 p+qt1 p+qs p+q1

k k 3 3 for p 6= q,

h:

1 t1 t t+1 p+q1

j g 8 4 for p = q.
We choose the simple roots i = i (1 i p + q 1) for su(p, q) h, then p+q := p+q

is trivial on su(p, q) and describes a character of u(1) h. We use Appendix A.1 in what
follows:
(p q) Then s = t = p and the pairs (2(i + p+qi+1 ), 2(i + p+qi )), (2(i +
p+qi+1 ), 2(i1 + p+qi+1 )) (1 i p) are contained in (g, h) and span a
G aH
which is of dimension s + t = 2p.
(p q + 1) Then s = q + 1 and t = q, so that the pairs (2(i + p+qi+1 ), 2(i +

p+qi )), (2(i + p+qi+1 ), 2(i1 + p+qi+1 )) (1 i q) and (2(q+1 + p ), 2(q +
p )) are contained in (g, h) and span a
G aH which is of dimension s + t = 2q + 1.
F5) (g, h) = (so(p, q + 1), so(p, q)). For p + q = 2m even and s = min(p, q + 1), t = min(p, q)
the Satake diagrams of g and h are
1 s s+1 m1 m
g: +3

m1

1 t t+1 m2

for t m 2,

m

V m1

1 m2
h: for t = m 1,

m

m1

1 m2

for t = m,

m
We use Appendix A.2 in what follows:

(p + 2 < q) Then s = t = p m 2 and the pairs (2i , 2i ), (2i , 2i1 ) (1 i p)
are contained in (g, h) and span a
G aH which is of dimension s + t = 2p.
(p + 2 = q) Again s = t = p, but now t = m 1. The pairs (2i , 2i ), (2i , 2i1 )
(1 i p 1) and (2p , 2(p + p+1 )), (2p , 2p1 ) are contained in (g, h) and span
a
(p = q) Then s = t = m = p and the pairs (2i , 2i ), (2i , 2i1 ) (1 i p 2) and
(2p1 , 2p2 ), (2p1 , 2(p1 + p )), (2p , 2p ), (2p , 2p1 ) are contained in (g, h)
and span a
(p = q + 2) Then s = q + 1, t = q and t < s = m. The pairs (2i , 2i ), (2i , 2i1 )
(1 i q1) and (2q , 2q1 ), (2q , 2(q +q+1 )), (4q+1 , 2(q +q+1 )) are contained
in (g, h) and span a
G aH which is of dimension s + t = 2q + 1.
(p > q+2) Again s = q+1, t = q, but now t < s < m. The pairs (2i , 2i ), (2i , 2i1 )
(1 i q) and (2q+1 , 2q ) are contained in (g, h) and span a
G aH which is of
dimension s + t = 2q + 1.
The case p + q = 2m 1 odd is treated similarly.
G1) (g, h) = (g + g , diag g ) with g simple compact. This is the same situation as in C).
26
JAN MOLLERS
G2) (g, h) = (so(1, n) + so(1, n), diag so(1, n)). The Satake diagram of so(1, n) is

s
1 2 s1 s 1 2 s1
+3
so(1, n) :

s+1
(n = 2s) (n = 2s + 1)
The spherical representations of so(1, n) are of the form F so(1,n) (2k1 ), k N, and hence
the spherical representations of g are of the form F so(1,n) (2k1 1 )F so(n,1) (2k2 1 ), k1 , k2 N.
The representation F so(1,n) (1 ) is self-dual, and hence the trivial representation F so(1,n) (0) is
contained in the tensor product F so(1,n) (1 ) F so(1,n) (1 ). Further, clearly
F so(1,n) (1 ) F so(1,n) (0) F so(1,n) (0) F so(1,n) (1 ) F so(1,n) (1 ).
Hence, ((21 , 0), 21 ), ((0, 21 ), 21 ), ((21 , 21 ), 0) (g, h) span a
G aH which is 3-
dimensional.
H1) (g, h) = (so(2, 2n), u(1, n)). The Satake diagrams of g and h are

n
n1
1 2 3
g:

n+1
1 2 n1 n
h: g 7
We realize the root system of g as {ei ej : 1 i < j n + 1} with simple roots

i = ei ei+1 , i = 1, . . . , n and n+1 = en + en+1 . Choose the simple roots i = i
(i = 1, . . . , n) for su(1, n) h, then n+1 := n+1 = 21 (e1 + + en+1 ) defines a non-trivial
character of u(1) h.
First consider the fundamental weight 2 = e1 + e2 , then the Weyl group orbit of 2 is
equal to the set of roots {ei ej : 1 i < j n + 1}. Hence, the weights e1 + e2 , en en+1
and e1 en+1 of F g (2 ) are highest weights for h so that the representations F h (e1 + e2 ),
F h (en en+1 ) and F h (e1 en+1 ) are contained in F g (2 )|h . Note that e1 en+1 = 1 + n .
Using the Weyl dimension formula we find
dim F g (2 ) = dim F h (e1 + e2 ) + dim F h (en en+1 ) + dim F h (e1 en+1 ) + 1.
Using the Kostant Branching Formula it is further easy to see that the remaining one-
dimensional representation in F g (2 )|h is the trivial representation so that (22 , 2(1 +
n )), (22 , 0) (g, h).
Next consider the highest weight 21 = 2e1 of g. Then similar considerations as above
show that F g (21 )|h contains F h (1 + n ) so that (41 , 2(1 + n )) (g, h).
Together the three pairs (41 , 2(1 + n )), (22 , 2(1 + n )), (22 , 0) (g, h) span a
G aH
which is 3-dimensional.
H2) (g, h) = (su (2n + 2), su (2n) + R + su(2)). The Satake diagrams of g and h are
1 2 3 2n 2n+1
g:
1 2 3 2n2 2n1
su (2n) :
We realize the root system of g as {(ei ej ) : 1 i < j 2n + 2} in the vector space
V = {x R2n+2 : x1 + + x2n+2 = 0}. To simplify notation, denote by (x) the orthogonal
projection of x R2n+2 to V . We choose the simple roots i = ei ei+1 for g and the
simple roots i = i (i = 1, . . . , 2n 1) for su (2n). Then 2n := 2n describes a character
of R h and 2n+1 is the non-trivial root for su(2) h. Let 2n+1 := 12 2n+1 denote the
fundamental weight for su(2). Then a general irreducible representation of h takes the form

F su (2n) (1 1 + +2n1 2n1 )F R (2n 2n )F su(2) (2n+1 2n+1 ) with 1 , . . . , 2n1 , 2n+1 N
and 2n C.
Consider the fundamental weight i = (e1 + + ei ) = 2ni+2 i
2n+2 (e1 + + ei ) 2n+2 (ei+1 +
+ e2n+2 ) of g. In the Weyl group orbit of i the three weights (e1 + + ei ), (e1 + +
ei1 + e2n+1 ) and (e1 + + ei2 + e2n+1 + e2n+2 ) are highest weights for h. Moreover,
k
k = k + 2n 2n (1 k 2n 1), 2n = 2n
and
ni+1
(e1 + + ei1 + e2n+1 ) = i1 2n 2n + 2n+1 ,
2ni+2
(e1 + + ei2 + e2n+1 + e2n+2 ) = i2 2n 2n ,
so that the representations

(2n)
k
F su (i ) F R ( 2n 2n ) F su(2) (0),
(2n)
F su (i1 ) F R ( ni+1
2n 2n ) F
su(2)
(2n+1 ),
(2n)
F su (i2 ) F R ( 2ni+2
2n 2n ) F
su(2)
(0)
are contained in F g (i )|h . Using the Weyl Dimension Formula we find

F g (i )|h F su (2n) (i ) F R ( 2n
k
2n ) F su(2) (0)

F su (2n) (i1 ) F R ( ni+1 2n 2n ) F su(2)
( 2n+1 )

su (2n)
F (i2 ) F R ( 2ni+2
2n 2n ) F
su(2)
(0) .
This implies that

(22 , 22 + n2 2n ), (22 , 22n )
2i 2(ni+1)
(22i , 22i + n 2n ), (22i , 22i2 n 2n ) (2 i n 1),
2
(22n , 22n ), (22n , 22n2 n 2n )
are contained in (g, h), and since a
G aH is 2n-dimensional, they form a basis.
28
JAN MOLLERS
H3) (g, h) = (so (2n + 2), so (2n) + so(2)). The Satake diagrams of g and h are

n V
n
1 2 3 n1 1 2 3 n1
g:

n+1 n+1
V n1 n1
1 2 3 n2 1 2 3 n2

h:

n n
(n odd) (n even)
We realize the root system of g as {ei ej : 1 i < j n + 1} with simple roots
i = ei ei+1 (1 i n) and n+1 = en + en+1 . Then we may choose the simple roots for h
as i = i+1 (1 i n). The fundamental weight n+1 = 1 = e1 is trivial on so (2n) h
and hence defines a non-trivial character of so(2) h.
Consider the fundamental weight i = e1 + +ei , i = 1, . . . , n1. By branching in stages,
first from gC = so(2n + 2, C) to so(2n + 1, C), then from so(2n + 1, C) to so(2n, C) = so (2n)C ,
it follows that
(2n) (2n) (2n)
F g (i )|so (2n) F so (i ) F so (i1 )2 F so (i2 ).
To find the action of the so(2)-factor we consider the weights of F g (i ) which are given by
{ek1 ekm : 1 k1 < . . . < km 2n + 2, i m 2N} {0}.
Among these clearly e2 + +ei+1 = i , e1 +e2 + +ei = n+1 +i1 and e2 + +ei1 =
i2 are highest weights for h and hence

F g (i )|h F so (2n) (i ) F so(2) (0) F so (2n) (i1 ) F so(2) (n+1 )

F so (2n) (i1 ) F so(2) (n+1 ) F so (2n) (i2 ) F so(2) (0) .
Similarly, for the highest weight n + n+1 = e1 + + en one obtains

F g (n + n+1 )|h F so (2n) (2n1 ) F so(2) (0) F so (2n) (2n ) F so(2) (0)

F so (2n) (n1 + n ) F so(2) (n+1 ) F so(2) (n+1 )

F so (2n) (n2 ) F so(2) (0) ,
and for the highest weight 2n+1 = e1 + + en+1 accordingly

F g (2n+1 )|h F so (2n) (2n ) F so(2) (n+1 ) F so (2n) (2n1 ) F so(2) (n+1 )

F so (2n) (n1 + n ) F so(2) (0) .
Now, if n = 2s 1 is odd, then

(22i , 22i ), (22i , 22i2 ) (1 i s 1), (42s , 2(2s2 + 2s1 ))

G aH which is of dimension s + (s 1) = 2s 1. If n = 2s
is even, then
(22i , 22i ), (22i , 22i2 ) (1 i s1), (2(2s +2s+1 ), 42s ), (2(2s +2s+1 ), 22s2 )
G aH which is of dimension s + s = 2s.
H4) (g, h) = (sp(p, q + 1), sp(p, q) + sp(1)). Let s = min(p, q + 1) and t = min(p, q), then the
Satake diagrams of g and sp(p, q) are
1 2 3 2s 2s+1 p+q p+q+1
g: ks
1 2 3 2t 2t+1 p+q1 p+q

sp(p, q) : ks
We realize the root system of g as {ei ej : 1 i < j p + q + 1} {2ei : 1
i p + q + 1} and choose i = ei ei+1 (1 i p + q) and p+q+1 = 2ep+q+1 as simple
roots. We further choose i = i (1 i p + q 1) and p+q = 2ep+q as simple roots for
sp(p, q) h, then p+q+1 = p+q+1 is a simple root for sp(1) and p+q+1 = p+q+1 = ep+q+1
the corresponding fundamental weight. A general irreducible representation of h takes the
form F sp(p,q) (1 1 + + p+q p+q ) F sp(1) (p+q+1 p+q+1 ) with 1 , . . . , p+q+1 N.
We consider the fundamental weight i = e1 + + ei . Then by [43, Theorem 3.3] we have

F g (i )|h F sp(p,q) (i ) F sp(1) (0) F sp(p,q) (i1 ) F sp(1) (p+q+1 )

F sp(p,q) (i2 ) F sp(1) (0)
for i = 1, . . . , p + q, and

F g (p+q+1 )|h F sp(p,q) (p+q ) F sp(1) (p+q+1 ) F sp(p,q) (p+q1 ) F sp(1) (0) .
Hence, the following pairs in (g, h) span a
G aH :
(p q) Then s = t = p, so that (22i , 22i ), (22i , 22i2 ) (1 i p) are contained
in (g, h) and span a
G aH .
(p q + 1) Then s = q + 1 and t = q, so that (22i , 22i ), (22i , 22i2 ) (1 i q)
and (22q+2 , 22q ) are contained in (g, h) and span a
G aH .
H5) (g, h) = (e6(26) , so(9, 1) + R). The Satake diagrams of g and so(9, 1) h are
2

g:

1 3 4 5 6
4
1 2 3
so(9, 1) :

5
30
JAN MOLLERS
We choose the simple roots for g such that 1 = 1 , 2 = 4 , 3 = 2 , 4 = 3 , 5 = 5 ,

then 6 := 6 defines a non-trivial character of R h. A general irreducible representation
of h is of the form F so(9,1) (1 1 + + 5 5 ) F R (6 6 ) for 1 , . . . , 5 N and 6 C.
By writing i as a linear combination of the simple roots j (see e.g. [12, Appendix C.2],
and writing i as a linear combination of the simple roots j , we find that
1 = 1 12 6 , 2 = 3 6 , 3 = 4 32 6 , 4 = 2 43 6 , 5 = 5 54 6 .
The spherical representations of g have highest weights 2k1 1 + 2k2 6 , k1 , k2 N, and the
spherical representations of h have highest weights 21 1 + 2 6 , 1 N and 2 C.
By [24] we have
F g (1 )|so(9,1) F so(9,1) (1 ) F so(9,1) (4 ) F so(9,1) (0).
To determine the action of the center of h on each summand we employ the weight space
decomposition (using e.g. LiE) and find that the weights of F g (1 ) which are highest weights
for so(9, 1) are
1 = 1 + 12 6 , 2 6 = 4 14 6 , and 6 = 6 ,
so that

F g (1 )|h F so(9,1) (1 ) F R ( 21 6 ) F so(9,1) (4 ) F R ( 14 6 ) F so(9,1) (0) F R (6 ) .
Taking contragredient representations further gives

F g (6 )|h F so(9,1) (1 ) F R ( 12 6 ) F so(9,1) (5 ) F R ( 14 6 ) F so(9,1) (0) F R (6 ) .
Hence,
(21 , 21 + 6 ), (26 , 21 6 ), (21 , 26 ), (26 , 26 ) (g, h)
and they clearly generate a
G aH which is of dimension 4.
3. Lower multiplicity bounds - Construction of symmetry breaking operators

We use the results of Section 1 and 2 to construct for every strongly spherical reductive
pair (G, H) intertwining operators between spherical principal series representations of G and
H in terms of their distribution kernels (see Theorem 3.3).
3.1. Principal series representations and symmetry breaking operators. For (, V )
cG and a we define the principal series representation
M G,C
, = IndG
PG ( e 1),
realized as the left-regular representation on the space C (G/P, V, ) of smooth sections of the
vector bundle V, = GPG V, G/PG associated to the representation V, = e+nG 1.
Here nG = 12 tr adnG a
G , the half sum of all positive roots. Denote by V, = V , the
dual bundle of V, .
Similarly, we define for (, W ) M cH and a principal series representations of H
H,C
by
, = IndH
PH ( e 1)
and write W, = e+nH 1.
Consider the space
HomH (, |H , , )
of intertwining operators between principal series of G and H, also referred to as symmetry

breaking operators by Kobayashi [16]. By the Schwartz Kernel Theorem, symmetry breaking
operators can be studied in terms of their distribution kernels.
Proposition 3.1 ([20, Proposition 3.2]). The natural map
PH
HomH (, |H , , ) D (G/PG , V,

) W, , A 7 K A ,
which is characterized by A(h) = hK A , (h )i, is an isomorphism.
As in [20] we use generalized functions rather than distributions, so that D (G/PG , V,
) can
1
be identified with the dual space of Cc (G/PG , V, ) and L (G/PG , V, ) D (G/PG , V, ).
cG and M
Let us first consider the case where both M cH are the trivial representation.
We use the notation
= 1, and = 1, .
To be able to apply the theory of BernsteinSato identities, we first translate line bundle
valued distributions on G/PG to scalar-valued distributions on the maximal compact sub-
group K G. Write : G K and H : G aG for the Iwasawa projections which are
characterized by
g (g)eH(g) NG KAG NG = G.
We define a smooth G-action on K by
g k = (gk), k K, g G.
Using this action and the isomorphism G/PG K/MG , the double cosets in PH \G/PG corre-
spond to the double cosets in PH \K/MG with respect to the action on the left. Moreover,
every double coset PH xPG has a representative x K, and
(PH xPG ) K = PH xMG = {p xm : p PH , m MG }.
The following result is immediate:
Lemma 3.2. The composition K G G/PG induces an isomorphism
P
D (G/PG , V1,

) W1, H D (K), ,
where
n
D (K), = K D (K) : K(m a n km) = e(nG )H(a n k) (a )+nH K(k)
o
k K, m a n PH , m MG .
Clearly every restriction to K of a smooth function f on G satisfying f (m a n gman) =

(nG )
a (a )+nH f (g) is contained in D (K), . Further, the support of u D (K), is
invariant under the right action of MG and the left action of PH and hence a union of
double cosets PH xMG , x K.
For the statement of the main result of this section we denote by a
G,+ and aH,+ the positive
Weyl chambers corresponding to + (g, aG ) and + (h, aH ).
non-trivial and strictly indecomposable. Then for every open double coset = PH xMG
K there exists a family of distributions K, D (K), unique up to scalar multiples and
depending holomorphically on (, ) a
G,C aH,C , such that
32
JAN MOLLERS
(1) K, D (K), and supp K, for all (, ) a

G,C aH,C ,
(2) K, L (K) and supp K, = for generic (, ) aG,C a
1
H,C , in particular for

Re( nG ) aG,+ and Re( + nH ) aH,+ .
By holomorphic dependence on the parameters (, ) we mean that for every C (K)
the map (, ) 7 hK, , i is a holomorphic function on a
G,C aH,C .
The proof of Theorem 3.3 is carried out in Section 3.3.
3.2. Lower bounds for multiplicities. Before we come to the proof, let us observe that
Theorem 3.3 implies lower bounds for multiplicities of intertwining operators. For this denote
by (PH \G/PG )open PH \G/PG the collection of open double cosets and by #(PH \G/PG )open
its cardinality (which is finite by Proposition 1.2).
Corollary 3.4. Assume that (G, H) is a strongly spherical reductive pair such that (g, h) is
non-trivial and strictly indecomposable. Then for all (, ) a
G,C aH,C we have
dim HomH ( |H , ) #(PH \G/PG )open .
Proof. Let x1 , . . . , xn K be representatives for the open double cosets in PH \K/MG
PH \G/PG , then by Theorem 3.3 there exist holomorphic families of distributions K, i , 1
i n, such that supp K, i = P x M for Re( ) a

H i G nG G,+ and Re( +nH ) aH,+ . Let
+ aG,+ and + aH,+ , then for fixed (0 , 0 ) aG,C aH,C we have Re(0 + z+ ) a

G,+
and Re(0 z+ ) a H,+ for z C, Re(z) 0. This implies that the holomorphic functions
fi : C D (K), fi (z) = Ki 0 +z+ ,0 z+ ,
satisfy supp fi (z) = PH xi MG whenever Re(z) 0. Hence, f1 (z), . . . , fn (z) are generically
linearly independent and the claim follows from the next lemma combined with Proposition 3.1
and Lemma 3.2.
Lemma 3.5. Let f1 , . . . , fn : C V be holomorphic functions with values in a complete
locally convex topological vector space V . If f1 (z), . . . , fn (z) are generically linear independent,
then there exists A = (aij ) GL(n, C) and m1 , . . . , mn 0 such that the functions
n
X
gi (z) = aij z mi fi (z)
j=1
are holomorphic in z = 0 and g1 (0), . . . , gn (0) are linearly independent.

Proof. Rearrange f1 , . . . , fn such that f1 (0), . . . , fp (0) are linearly independent and the re-
maining fp+1 (0), . . . , fn (0) are linear combinations of f1 (0), . . . , fp (0). We now show that
fp+1 (z) can be replaced by a renormalized linear combination of f1 (z), . . . , fp+1 (z) such that
f1 (0), . . . , fp+1 (0) are linearly independent. Applying this argument recursively to fp+1 , . . . , fn
shows the statement.
So assume that
p
X
fp+1 (0) = 0i fi (0).
i=1
1 (z) = f
Pp 0 1 (0) = 0 and renormal-
We form the new function fp+1 p+1 (z) i=1 i fi (z), then fp+1
izing
1 (z)
fp+1
fp+1
1
(z) =
z
gives a holomorphic function fp+1

1 (z) such that f (z), . . . , f (z), f1 (z) are still generically
1 p p+1
linearly independent. If now f1 (0), . . . , fp (0), fp+1
1 (0) are linearly independent, we are done,
otherwise we have
X p
1
fp+1 (0) = 1i fi (0).
i=1
Pp
As before, we form 2 (z)
fp+1 = fp+1
1 (z) 1 2
i=1 i fi (z), then fp+1 (0) = 0 and renormalizing
2 (z)
fp+1
fp+1
2
(z) =
z
gives a holomorphic function fp+1 2 (z) such that f (z), . . . , f (z), f2 (z) are still generically
1 p p+1
linearly independent. We repeat this procedure as long as possible. If the procedure does not
terminate, we obtain 0i , 1i , 2i , . . . C, 1 i p, and holomorphic functions fp+1k (z) with
p
X
z fp+1
k+1
(z) = fp+1
k
(z) ki fi (z),
i=1
where we put fp+1

0 = fp+1 . Form the holomorphic functions (the convergence of the sums
follows by applying continuous linear functionals to the above identities)

X
i : C C, i (z) = ki z k ,
k=0
then
X p p
X
X
X
i (z)fi (z) = ki fi (z)z k = fp+1
k
(z) z fp+1
k+1
(z) z k = fp+1
0
(z) = fp+1 (z)
i=1 k=0 i=1 k=0
for all z C. But this implies that f1 (z), . . . , fp (z), fp+1 (z) are linearly dependent for all
z C, contradicting the assumption. Hence, the procedure has to terminate at some stage,
which implies that f1 (0), . . . , fp (0), fp+1
k (0) are linearly independent for some k proving our
claim.
3.3. BernsteinSato identities and meromorphic extension. Write
r = rankR (G) + rankR (H) = dim aG + dim aH .
By Proposition 2.5 and Theorem 2.6 there exists a basis {(j , j )}j=1,...,r + (g, a)
+ (h, aH ) of a
G aH and non-zero real-analytic functions Fj : G R, Fj 0, such that

Fj (m a n gman) = aj (a )j Fj (g) (3.1)
for g G, man PG and m a n PH . Note that (3.1) implies

Fj (m a n km) = ej H(a n k) (a )j Fj (k) k K, m MG , m a n PH . (3.2)
Now, for (s1 , . . . , sr ) Cr with Re(s1 ), . . . , Re(sr ) 0 we define
Ks1 ,...,sr (g) = F1 (g)s1 Fr (g)sm , g G.
Let PH \K/MG be an open double coset and write : K {0, 1} for its characteristic
function. Then it remains to show that Ks1 ,...,sr defines a family of distributions on K which
extends meromorphically in (s1 , . . . , sr ) Cr . For this we use BernsteinSato identities.
34
JAN MOLLERS
By [36, 37] there exists a differential operator Ds1 ,...,sr on K depending polynomially on
(s1 , . . . , sr ) Cr such that
Ds1 ,...,sr Ks1 ,...,sr = b(s1 , . . . , sr )Ks1 1,...,sr 1 Re(s1 ), . . . , Re(sr ) 0, (3.3)
where
m
Y
b(s1 , . . . , sr ) = c ai,1 s1 + + ai,r sr + bi
i=1
with (ai,1 , . . . , ai,r ) Cr \ {0}, bi C and c 6= 0. This identity immediately shows the mero-
morphic extension of the family Ks1 ,...,sr D (K). However, if we want to meromorphically
extend Ks1 ,...,sr , we have to control Ks1 ,...,sr at the boundary of .
Lemma 3.6. For every double coset there exists 1 j r such that
Fj | = 0.
Proof. Let g , then by the equivariance property (3.1) it suffices to show that Fj (g) = 0
for some 1 j r. Since , the double coset PH gPG is not open. Using Theorem 1.4
we find an element Z = ZM + ZA + ZN pH with ZA 6= 0 such that X = Ad(g1 )Z =
XM + XA + XN pG . This implies

et(s1 1 ++sr r )(ZA ) Ks1 ,...,sr (g) = Ks1 ,...,sr (etZ g) = Ks1 ,...,sr (getX )
= et(s1 1 ++sr r )(XA ) Ks1 ,...,sr (g)
for all t R. Now assume that Fj (g) 6= 0 for all 1 j r, then Ks1 ,...,sr (g) 6= 0 for all
(s1 , . . . , sr ) Cr with Re(s1 ), . . . , Re(sr ) 0. Hence
(s1 1 + + sr r )(XA ) = (s1 1 + + sr r )(ZA ),
or equivalently
s1 (1 (XA ) + 1 (ZA )) + + sr (r (XA ) + r (ZA )) = 0
for (s1 , . . . , sr ) Cr with Re(s1 ), . . . , Re(sr ) 0. This implies j (XA ) + j (ZA ) = 0 for all
1 j r. But since the pairs (j , j ) form a basis of a
G aH , the pairs (j , j ) also form a
basis and hence ZA = 0 which gives a contradiction.
By the previous lemma we have Ks1 ,...,sr (g) = 0 whenever g and Re(s1 ), . . . , Re(sr ) >
0. This implies a regularity statement for the functions Ks1 ,...,sr :
Lemma 3.7. Let M be a smooth manifold, M open and C (M ) with 0,
| = 0. Then s C k (R) whenever Re s > k.
The proof is an easy calculus exercise. We can now proceed to prove Theorem 3.3.
Proof of Theorem 3.3. Let PH \K/MG be an open double coset, then Ks1 ,...,sr | = 0 for
Re(s1 ), . . . , Re(sr ) > 0 by Lemma 3.6. By Lemma 3.7 this implies that Ks1 ,...,sr C k (K)
for Re(s1 ), . . . , Re(sr ) > k. Since the BernsteinSato operator Ds1 ,...,sr is of finite order, say
k 0, this implies that Ds1 ,...,sr [ Ks1 ,...,sr ] is a continuous function for Re(s1 ), . . . , Re(sr ) >
k. Replacing Ks1 ,...,sr in the BernsteinSato identity (3.3) by Ks1 ,...,sr yields
Ds1 ,...,sr [ Ks1 ,...,sr ] = b(s1 , . . . , sr ) Ks1 1,...,sr 1 (3.4)
which holds on by (3.3), and trivially also on K \ since both sides vanish. As ob-
served above, for Re(s1 ), . . . , Re(sr ) > k both sides are continuous functions, and hence
(3.4) holds everywhere. This shows the meromorphic continuation of Ks1 ,...,sr D (K) to
(s1 , . . . , sr ) Cr , or more precisely the holomorphic continuation of
m
Y
ai,1 s1 + + ai,r sr + bi Ks1 ,...,sr .
i=1
Finally, the change of variables
Cr a
G,C aH,C , (s1 , . . . , sr ) 7 (, ) = s1 (1 , 1 ) + + sr (r , r ) + (nG , nH )
constructs the desired family K, . The equivariance property which ensures that K, is
contained in D (K), follows from (3.2) for Re(s1 ), . . . , Re(sr ) > k and extends by analytic
continuation.
4. Upper multiplicity bounds - Invariant distributions

In this section we establish upper bounds for the multiplicities dim HomH (, |H , , )
using Bruhats theory of invariant distributions. In particular, we obtain that for and
the trivial representations, the intertwining operators constructed in Section 3 form a basis
of HomH ( |H , ) for generic (, ) a
G,C aH,C .
4.1. Upper bounds for multiplicities. Let M MG MH denote the stabilizer of a

generic element X n . More precisely,
M = (MG MH )X = {g MG MH : Ad(g)X = X},
P
where X n with X = (n ) X , X g (aH ; ) \ {0}. Since for every (n ),
the group MG MH acts either transitively on the unit sphere S g (aH ; ) or with
possibly two orbits if dim g (aH ; ) = 1, it follows that for generic X, Y n the stabilizers
(MG MH )X and (MG MH )Y are conjugate in MG MH .
The main statement of this section is:
non-trivial and strictly indecomposable. Then for (, ) a
G,C aH,C satisfying the generic
condition

/ w N- span + (g, aG ) |aH
w(Re nG )|aH (Re nH ) w W (aG ), (4.1)
we have
dim HomH (, |H , , ) #(PH \G/PG )open dim HomM ( w0 |M , |M ).
In particular, for and the trivial representations, we have generically
dim HomH ( |H , ) #(PH \G/PG )open .
Here g denotes the representation of gMG g1 given by g (m) = (g1 mg).
Remark 4.2. If mG mH = {0} then the condition (4.1) in Theorem 4.1 can be replaced by
the weaker condition

/ w N- span + (g, aG ) |aH
w( nG )|aH ( nH ) w W, (4.2)
This follows from the proof in Section 4.2.2.
Before we prove the theorem, note that combined with Corollary 3.4 it implies the following
formula for the generic multiplicities:
36
JAN MOLLERS
Corollary 4.3. Assume that (G, H) is a strongly spherical reductive pair such that (g, h) is
non-trivial and strictly indecomposable. Then for (, ) a
G,C aH,C satisfying the generic
condition (4.1) we have
dim HomH ( |H , ) = #(PH \G/PG )open .
The rest of this section is devoted to the proof of Theorem 4.1.
as well as W
4.2. Invariant distributions. Recall the notation V, , V, , V, , from Sec-
tion 3.1. By Proposition 3.1 we have
dim HomH (, |H , , ) = dim(D (G/PG , V,

) W, )PH .
Consider the surjective linear map
Z
: Cc (G)
V, C (G/PG , V, ), (g) = ( e+nG 1)(p)(gp) dp,
PG
where dp denotes a left-invariant measure on PG . Then its transpose : D (G/PG , V,

)
D (G) V,
is injective and embeds (D (G/P , V ) W
G , , )
PH into
D (G)(,),(,) = {u D (G) HomC (V , W ) :

u(pH gpG ) = ( e+nH 1)(pH ) u(g) ( enG 1)(pG )}.
Upper bounds for spaces of invariant distributions such as D (G)(,),(,) are provided by
Bruhats theory of invariant distributions (see e.g. [44, Chapter 5.2] for a detailed account;
see also [27, Section 3.4] for an application in a similar setting). This method applies to our
situation since the group PH PG that acts on G has an open orbit. In this case, Bruhats
theory implies the following upper bound::
X
X g
dim D (G)(,),(,) dim HomPH gPG g1 enG 1 ,
PH gPG m=0
PH \G/PG
P gP g1
e+nH 1 H G S m (V (g)) ,
PH gPG g1
where for a Lie group S we denote by S (x) = | det Ad(x1 )|, x S, its modular function,
and V (g) = g/(Ad(g)pG + pH ). Clearly PG (man) = a2nG and PH (man) = a2nH . We
estimate the contributions from open and non-open orbits separately.
4.2.1. Open orbits. If PH gPG is an open orbit, we may choose g = eX w 0 with X n

1 nG g w
generic. Then PH gPG g = M and ( e 1) |M = |M . Further, since M is
0
compact PH = gPG g1 = PH gPG g1 = 1 on M . Finally, S m (V (g)) = {0} for m > 0,

because V (g) = {0} in this case, and S 0 (V (g)) = C is the trivial representation. This implies
that

X g P gP g1
dim HomPH gPG g1 enG 1 , e+nH 1 H G S m (V (g))
PH gPG g1
m=0
= dim HomM ( w0 |M , |M ).
4.2.2. Non-open orbits. In this case it is more convenient to work with

g1 PG g1 PH g
HomPG g1 PH g enG 1 , e+nH 1 S m (W (g))
PG g1 PH g
g1 PG g1 PH g PG g1 PH g
enG 1 e+nH 1 S m (W (g)) , (4.3)
PG g1 PH g
where W (g) = g/(pG + Ad(g1 )pH ). If PH gPG is a non-open orbit, then by Theorem 1.4
the stabilizer PH gPG g 1 of gPG in PH contains a one-parameter subgroup exp(RZ), where
Z = ZM + ZA + ZN (mG mH ) + aH + nH with ZA 6= 0. Note that
X = Ad(g1 )Z = XM + XA + XN pG Ad(g1 )pH
1 )ZM mG and XA = Ad(w
with XM = Ad(w 1 )ZA aG , XA 6= 0. Then

enG 1 (etX ) = et(nG )(XA ) (etXM ),
g1 tX
e+nH 1 (e ) = et(+nH )(ZA ) (etZM ).
Further, PG (etX ) = e2tnG (XA ) and g1 PH g (etX ) = e2tnH (ZA ) .
P
Lemma 4.4. PG g1 PH g (etX ) = exp(t + (g,aG ) n (XA )) for some integers 0 n
dim g(aG ; ).
Proof. We compute PG g1 PH g (eX ) for X = XM + XA + XN mG + aG + nG using the
formula
PG g1 PH g (eX ) = exp( tr ad(X)|pG Ad(g1 )pH ).
Choose a basis (Y ) (Y ) (Y ) of pG Ad(g1 )pH with the following properties:
(1) Y = Y,M + Y,A + Y,N mG + aG + nG such that (Y,M ) is an orthogonal basis of
the projection of pG Ad(g1 )pH to mG ,
(2) Y = Y,A + Y,N aG + nG such that (Y,A ) is linearly independent,
(3) Y = Y,N nG such that (Y,N ) is a basis of nG Ad(g1 )pH .
Here orthogonal bases are taken with respect to any mG -invariant inner product on g. We
now study the action of ad(X) on Y , Y and Y . First,
ad(X)Y = [XM , Y,M ] + [XN , Y,M + Y,A ] + [X, Y,N ] [XM , Y,M ] + nG .
Note that [XM , Y,M ] Y,M since the inner product is mG -invariant. Hence, the coefficient
of Y in the expression of ad(X)Y as linear combination of the basis elements is zero. This
implies that the contribution of the basis elements Y to the trace is trivial. Next, we have
ad(X)Y = [XN , Y,A ] + [X, Y,N ] nG
so that ad(X)Y is a linear combination of the basis elements Y . Therefore, also the basis
elements Y do not contribute to the trace and we have
tr ad(X)|pG Ad(g1 )pH = tr ad(X)|nG Ad(g1 )pH .
We now specify a basis (Y ) of nG Ad(g 1 +
Ln )pH . Order the positive roots (g, aG ) =
{1 , . . . , n } such that [nG , g(aG ; j )] i=j+1 g(aG ; i ). Then we can choose the basis
{Y = Yj,k : 1 j n, 1 k nj } such that
P Ln
Yj,k = ni=j Yj,k i
i=j g(aG ; i ) for all 1 j n, 1 k nj ,
38
JAN MOLLERS
j
(Yj,k )k=1,...,nj are mutually orthogonal with respect to a mG -invariant inner product
on g(aG ; j ).
We have
Mn
j
ad(X)Yj,k = [XM + XA , Yj,k ] + [XN , Yj,k ] [XM + XA , Yj,k ]+ g(aG ; i ).
i=j+1
j j
Since [XM , Yj,k ] Yj,k we obtain
M n
M
ad(X)Yj,k j (XA )Yj,k + CYj, + g(aG ; i ),
6=k i=j+1
so that
n
X
tr ad(X)|nG Ad(g1 )pH = nj j (XA ).
j=1
m
P Let P S (W (g)), P =
Lemma 4.5. 6 0, with ad(X)P = P for some C, then we have
Re = + (g,aG ) m (XA ) for some integers m 0. If additionally XM = 0 then
P
= + (g,aG ) m (XA ).
Proof. First note that W (g) = g/(pG +Ad(g)1 pH ) nG /(nG (pG +Ad(g) 1
Ln pH )). We order
+
the positive roots (g, aG ) = {1 , . . . , n } such that [nG , g(aG ; j )] i=j+1 g(aG ; i ).
Then we can choose a basis {Yj,k : 1 j n, 1 k nj } of nG (pG + Ad(g)1 pH ) such
that
P Ln
Yj,k = ni=j Yj,k
i
i=j g(aG ; i ) for all 1 j n, 1 k nj ,
j
(Yj,k )k=1,...,nj g(aG ; j ) are linearly independent for every 1 j n.
j j
For every 1 j n we extend (Yj,k )k=1,...,nj to a C-basis (Yj,k )k=1,...,nj (Zj,k )k=1,...,mj
of g(aG ; j )C , then the equivalence classes Z j,k = Zj,k + (nG (pG + Ad(g)1 pH )), 1
j n, 1 k mj , form a basis of nG /(nG (pG + Ad(g)1 pH )). Since XM mG is
contained in a maximal torus in mG we may assume that every Zj,k is an eigenvector of
ad(XM ) with imaginary eigenvalue: ad(XM )Zj,k = 1j,k Zj,k . Further, by definition we
have ad(XA )Zj,k = j (XA )Zj,k , so that
n
M

ad(X)Zj,k (j (XA ) + 1j,k )Zj,k + g(aG ; i )C .
i=j+1
Now assume P S m (W (g)), P 6= 0, with ad(X)P = P . Write P as linear combination of

the basis elements Z j1 ,k1 Z jm ,km of S m (W (g)), then there exist 1 j1 . . . jm n
and 1 ki nji such that the coefficient of Z j1 ,k1 Z jm ,km in P is non-zero. Choose such
(j1 , k1 ), . . . , (jm , km ) with the property that j1 , . . . , jm are minimal. Considering only the
coefficient of Z j1 ,k1 Z jm ,km in ad(X)P = P we obtain

= j1 (XA ) + + jm (XA ) + 1j1 ,k1 + 1jm ,km .
which implies the claim.
Now assume that V W S m (W (g)), 6= 0, is contained in the space of invariants
(4.3). Since XM resp. ZM is contained in a maximal torus in mG resp. mH , there exist bases

(vi )i of V and (wj )j of W such that (XM )vi = 1ai vi and (ZM )wj = 1bj wj with
ai , bj R. Write in terms of this basis as
X
= vi wj Pi,j
i,j
with Pi,j S m (W (g)), then X = 0 implies, using Lemma 4.4:

ad(X)Pi,j = ai 1 ( nG )(XA ) + bj 1 + ( + nH )(wXA )
X
2nG (XA ) 2nH (wXA ) + n (XA ) Pi,j .
+ (g,aG )
At least one Pi,j is non-trivial and hence it follows from Lemma 4.5 that
w( nG )(ZA ) ( nH )(ZA )
X
= m + (dim g(aG ; ) n ) w(ZA ) + (ai + bj ) 1.
| {z }
+ (g,aG ) 0
Hence, the space (4.3) of invariants must be trivial if (4.1) is satisfied. This completes the
proof of Theorem 4.1.
5. Application to Shintani functions

In [15] Kobayashi established a connection between symmetry breaking operators and Shin-
tani functions of a pair (G, H) of real reductive groups. Combining his results with Corol-
lary 3.4 we prove lower bounds for the dimension of the space of Shintani functions (see
Theorem 5.3.
5.1. Shintani functions for real reductive groups. Let tG mG and tH mH be Cartan
subalgebras of mG and mH , then jG = tG + aG g and jH = tH + aH h are maximally split
Cartan subalgebras of g and h. We identify j
G,C tG,C aG,C and jH,C tH,C aH,C . Let us
choose positive systems + (gC , jG,C ) (gC , jG,C ) and + (hC , jH,C ) (hC , jH,C ) such that
the restriction of a positive root to aG resp. aH is either zero or contained in + (g, aG ) resp.
+ (h, aH ). Write g resp. h for the half sum of all roots in + (gC , jG,C ) resp. + (hC , jH,C ).
Then g = mG + nG with mG = g |tG , and similarly h = mH + nH . Further, let W (jG,C )
and W (jH,C ) denote the Weyl groups of (gC , jG,C ) and (hC , jH,C ).
The Harish-Chandra isomorphism provides a natural identification

j
G,C /W (jG,C ) HomCalg (Z(gC ), C), 7 ,
where Z(gC ) denotes the center of the universal enveloping algebra U (gC ) of gC . We use the
same notation HomCalg (Z(hC ), C) for j H,C . The left resp. right action of U (gC )
on C (G) will be denoted by Lu resp. Ru , u U (gC ).
Definition 5.1 (see [15, Definition 1.1], [32]). A function f C (G) is called Shintani
function of type (, ) j
G,C jH,C if f satisfies the following three properties:
(1) f (k gk) = f (g) for any k H K, k K.
(2) Ru f = (u)f for any u Z(gC ).
(3) Lv f = (v)f for any v Z(hC ).
40
JAN MOLLERS
The space of Shintani functions of type (, ) is denoted by Sh(, ). We write Shmod (, )

for the subspace of Shintani functions of moderate growth.
For the definition of moderate growth see e.g. [15, Definition 3.3]. The following result is
due to Kobayashi:
Fact 5.2 (see [15, Theorem 1.2 and 8.1]). Let (G, H) be a pair of real reductive algebraic
groups.
(1) dim Sh(, ) < for all (, ) j
G,C jH,C if and only if (G, H) is strongly spherical.
(2) dim Sh(, ) 6= 0 implies W (jG,C )(mG + a
G,C ) and W (jH,C )(mH + aH,C ).
In view of this statement we abuse notation and write
Sh(, ) = Sh(mG + , mH + ), (, ) a
G,C aH,C .
5.2. Lower bounds for dim Sh(, ). The following statement gives more detailed informa-
tion about dim Sh(, ):
non-trivial and strictly indecomposable. Then for all (, ) a
G,C aH,C we have
dim Sh(, ) dim Shmod (, ) #(PH \G/PG )open ,

G,C aH,C we have
dim Shmod (, ) = #(PH \G/PG )open .
Proof. Let + W (aG ) and W (aH )() such that Reh+ , i 0 for all + (g, aG )
and Reh , i 0 for all + (h, aH ). Then by [15, Theorem 8.2 and Remark 8.3] there
exists a natural embedding
HomH (+ |H , ) Shmod (, )
which is an isomorphism for generic (, ). Now the statement follows from Corollary 3.4 and
4.3.
Remark 5.4. According to Kobayashi [15, Remark 10.2 (4)] it is plausible that Shmod (, ) =
Sh(, ) for all (, ) j
G,C jH,C if the pair (G, H) is strongly spherical. In this case,
Theorem 5.3 would imply
dim Sh(, ) = #(PH \G/PG )open
for generic (, ).
Remark 5.5. For some strongly spherical pairs (G, H) of low rank the space of Shintani
functions was studied in more detail, and in some cases its dimension was computed for
all parameters (, ) a
G,C aH,C . The pairs (GL(2, F), GL(1, F) GL(1, F)), F = R, C,
were studied by Hirano [10, 11], the pairs (O(1, n + 1), O(1, n)) by Kobayashi [15], the pairs
(U(1, n + 1), U(1, n) U(1)) by Tsuzuki [40, 41, 42], the pairs (Sp(2, R), Sp(1, R) Sp(1, R))
and (Sp(2, R), SL(2, C)) by Moriyama [30, 31]. and the pair (SL(3, R), GL(2, R)) by Sono [38].
6. Multiplicity one pairs

For the pairs
(GL(n + 1, C), GL(n, C)), (GL(n + 1, R), GL(n, R)), (U(p, q + 1), U(p, q)),
(6.1)
(SO(n + 1, C), SO(n, C)), (SO(p, q + 1), SO(p, q))
it was proven by SunZhu [39] that dim HomH (|H , ) 1 for all irreducible smooth ad-
missible Frechet representations of G and of H of moderate growth. In this section we
determine for which principal series representations = , and = , the multiplicities
are = 1, at least for generic (, ) a
G,C aH,C . The main tool to pass from spherical prin-
cipal series and to general vector-valued principal series , and , is a variant of the
JantzenZuckerman translation principle. This method was previously used in the construc-
tion of symmetry breaking operators in [26] and we briefly recall the main steps, adapted to
our setting.
6.1. The translation principle. Let (, E) be an irreducible finite-dimensional represen-
tation of G, then the space
E := E nG = {v E : (X)v = 0 X nG }
of nG -fixed vectors is an irreducible representation of MG AG , and we obtain a P G -equivariant
embedding E E. Similarly, we let (, F ) be an irreducible finite-dimensional representa-
tion of H and consider the irreducible MH AH -representation F = F nH . Then the projection
F F onto the highest restricted weight space is PH -equivariant, where we let NH act
trivially on F . We assume that HomH (E|H , F ) 6= {0}.
Now suppose we are given an H-intertwining operator
A : IndG H
PG (V ) IndPH (W )
for V and W finite-dimensional representations of PG and PH . Tensoring with a non-trivial
H-intertwiner : E F gives an intertwiner
A : IndG H
PG (V ) E IndPH (W ) F, (6.2)
from which we want to construct an intertwiner between certain vector-valued principal series.
For this, we first consider the representation IndG
PG (V ) E. Since w 01 = P G , the
0 PG w
01 ) defines an isomorphism
map f 7 f ( w
IndG
P
(V w0 ) IndG
PG (V ).
G
Next, we make use of the isomorphism

: IndPG (V w0 E|P G ) IndG
P
(V w0 ) E, f (g) = (idV (g))f (g),
G G
where we view both sides as (V E)-valued smooth functions on G. Now, the P G -equivariant
embedding E E induces an embedding
IndG
P
(V w0 E ) IndG
P
(V w0 E|P G )
G G
which we compose with the isomorphism

1
IndG w
PG (V (E )
0
) IndPG (V w0 E ), f 7 f ( w
0 )
G
to obtain an embedding
1
IndG w
PG (V (E )
0
) IndG
PG (V ) E. (6.3)
42
JAN MOLLERS
Similarly, without having to invoke w

0 , we have a surjection

IndH H H
PH (W ) F IndPH (W F |PH ) IndPH (W F ), (6.4)
where the second map is induced by the PH -equivariant projection E F .
Finally, composing (6.2) with the embedding (6.3) and the surjection (6.4) defines an H-
intertwining operator
1
(A) : IndG w
PG (V (E )
0
) IndH
PH (W F ).
1
Now let V = 1 e 1 and W = 1 e 1, and write (E )w0 e0 1 and
cG , M
F = e0 1 for M
cH and (0 , 0 ) a a . Then A 7 (A) defines a map
G H
: HomH ( |H , ) HomH (,+0 |H , ,+0 ).

Recall that by Proposition 3.1 every intertwining operator A HomH (, |H , , ) is given by
a distribution kernel KA (D (G/PG , V,
)W
, )
PH . To describe how the distribution kernel
of (A) arises from the distribution kernel of A we denote by i : E E the P G -equivariant

1
embedding and by p : F F the PH -equivariant quotient. Further let E = G PG (E )w0
1
and denote by (E ) = G PG ((E )w0 ) the contragredient bundle.
Theorem 6.1 ([26, Section 2]). The map
: HomH ( |H , ) HomH (,+0 |H , ,+0 )
has the property that the distribution kernel K(A) (D (G/PG , V,+

0
) W,+0 )PH is
given in terms of the distribution kernel KA (D (G/PG , V ) W )PH as
K(A) = KA ,
where (C (G/PG , (E ) ) F )PH is given by

(g) = p (gw0 ) i HomC (E , F ) (E ) F , g G.
Note that in the statement we identify V (E ) V,+

0
and W F W,+0 .
Corollary 6.2. Let M cG and M cH and assume that there exist irreducible finite-
dimensional representations E of G and F of H such that w0 E nG |MG , F nH |MH and
HomH (E|H , F ) 6= {0}. Then there exist (0 , 0 ) a
G aH and a linear map
: HomH ( |H , ) HomH (,+0 |H , ,+0 )

which is on the level of distribution kernels given by tensoring with a fixed non-trivial real-
analytic section. In particular, supp(K(A) ) = G/PG whenever supp KA = G/PG .
Proof. Replacing E and F by their twists with the Cartan involution of G we may assume
that w0 E nG |MG , F nH |MH and HomH (E w0 |H , F ) 6= {0}. Then E = E nG and F = F nG
1
satisfy (E )w0 |MG and F |MH , so that the statement follows from Theorem 6.1. It
remains to show that is a non-trivial real-analytic section. That is non-trivial is a
consequence of the irreducibility of E. Further, as a matrix coefficient of a finite-dimensional
representation it is clearly real-analytic.
6.2. Iwasawa decompositions. To construct finite-dimensional representations E of G and

F of H for the translation principle, we first give the explicit Iwasawa decompositions for all
pairs (G, H) in (6.1) (following the notation of Section 1.3), compute the generic stabilizer
M MG MH of an element of n defined in Section 4.1, and prove in particular the
following statement:
Lemma 6.3. Let (G, H) be one of the pairs in (6.1), then #(PH \G/PG )open = 1.
6.2.1. (G, H) = (GL(n + 1, C), GL(n, C)). Let G = GL(n + 1, C), n 2, and define a Cartan
involution on G by (g) = (g )1 , then K = G = U(n + 1). We embed H = GL(n, C) in the
upper left corner of G, then leaves H invariant and H K = H = U(n). We choose the
maximal abelian subalgebra
aH = {diag(t1 , . . . , tn , 0) : ti R}
of h and extend it to the maximal abelian subalgebra
aG = {diag(t1 , . . . , tn , tn+1 ) : ti R}
of g . Then
MG = ZK (aG ) = {diag(z1 , . . . , zn , zn+1 ) : |zi | = 1} U(1)n+1 ,
MH = ZHK (aH ) = {diag(z1 , . . . , zn , 1) : |zi | = 1} U(1)n .
Further,

0n z n
n = :zC
0 0
and such an element of n is contained in an open (MG MH )AH -orbit if and only if
z1 , . . . , zn 6= 0. Hence M = {1} is trivial.
6.2.2. (G, H) = (GL(n + 1, R), GL(n, R)). Let G = GL(n + 1, R), n 2, and define a Cartan
involution on G by (g) = (g )1 , then K = G = O(n + 1). We embed H = GL(n, R) in
the upper left corner of H, then leaves H invariant and H K = H = O(n). We choose
the maximal abelian subalgebra
aH = {diag(t1 , . . . , tn , 0) : ti R}
of h and extend it to the maximal abelian subalgebra
aG = {diag(t1 , . . . , tn , tn+1 ) : ti R}
of g . Then
MG = ZK (aG ) = {diag(x1 , . . . , xn , xn+1 ) : xi = 1} O(1)n+1 ,
MH = ZHK (aH ) = {diag(x1 , . . . , xn , 1) : xi = 1} O(1)n .
Further,

0n x n
n = :xR
0 0
and such an element of n is contained in an open (MG MH )AH -orbit if and only if
x1 , . . . , xn 6= 0. Hence M = {1} is trivial.
44
JAN MOLLERS
6.2.3. (G, H) = (U(p, q + 1), U(p, q)). Let G = U(p, q + 1), p, q 1, and define a Cartan
involution on G by (g) = (g )1 , then K = G = U(p) U(q + 1). We embed H = U(p, q)
in the upper left corner of G, then leaves H invariant and H K = H = U(p) U(q).
Assume that p q + 1, the case p q is handled similarly (see Section 6.2.5 for a related
computation). We choose the maximal abelian subalgebra

0q+1 D
aG = 0pq1 : D = diag(t1 , . . . , tq+1 ), t1 , . . . , tq+1 R

D 0q+1
of g , then aH = aG h is maximal abelian in h . Write ei aG for the functional mapping
a matrix of the above form to ti . Then
(
{ei ej : 1 i < j q + 1} {2ei : 1 i q + 1} for p = q + 1,
(g, aG ) =
{ei ej : 1 i < j q + 1} {ei , 2ei : 1 i q + 1} for p > q + 1,
with root spaces given by
n
g(aG ; (ei + ej )) = z(Ei,j Ei,p+j Ep+i,j Ep+i,p+j )
o
z(Ej,i Ej,p+i Ep+j,i Ep+j,p+i) : z C ,
n
g(aG ; (ei ej )) = z(Ei,j Ei,p+j Ep+i,j + Ep+i,p+j )
o
z(Ej,i Ej,p+i Ep+j,i + Ep+j,p+i) : z C ,

g(aG ; 2ei ) = R 1(Ei,i Ei,p+i Ep+i,i Ep+i,p+i),
and if p > q + 1 additionally
n pq1
X
g(aG ; ei ) = zk (Ei,q+k+1 Ep+i,q+k+1 ) zk (Eq+k+1,i Eq+k+1,p+i ) :
k=1
o
zk C .
We choose the positive system

(
{ei ej : 1 i < j q + 1} {2ei : 1 i q + 1} for p = q + 1,
+ (g, aG ) =
{ei ej : 1 i < j q + 1} {ei , 2ei : 1 i q + 1} for p > q + 1,
then
MG = {diag(z1 , . . . , zq+1 , k, z1 , . . . , zq+1 ) : |zi | = 1, k U(p q 1)}
U(1)q+1 U(p q 1),
MH = {diag(z1 , . . . , zq , k, z1 , . . . , zq , 1) : |zi | = 1, k U(p q)}
U(1)q U(p q),
and
n q
X o

n = X(z) = zi (Ei,p+q+1 + Ep+i,p+q+1) + zi (Ep+q+1,i Ep+q+1,p+i ) : zi C .
i=1
The action of MG MH U(1)q U(p q 1) is given by the natural action of U(1)q on

z Cq , and the second factor U(p q 1) acts trivially. Hence, the element X(t) is contained
in an open (MG MH )AH -orbit if and only if t1 , . . . , tq 6= 0 and then M = U(p q 1).
Further, U(1)q MG MH acts transitively on the product (S 1 )q Cq of the unit spheres
S 1 C and therefore there is a unique open orbit.
6.2.4. (G, H) = (SO(n + 1, C), SO(n, C)). Let G = SO(n + 1, C), n 2, and define a Cartan
involution on G by (g) = (g )1 = g, then K = G = SO(n + 1). We embed H = SO(n, C)
in the upper left corner of G, then leaves H invariant and H K = H = SO(n). Write
n = 2m if n is even and n = 2m 1 if it is odd. We choose the maximal abelian subalgebra
(
diag(D( 1t1 ), . . . , D( 1tm )) : t1 , . . . , tm R for n odd,
aG =
diag(D( 1t1 ), . . . , D( 1tm ), 0) : t1 , . . . , tm R for n even,
of g , where

0 t
D(t) = .
t 0
Then aH = aG h is maximal abelian in h . Write ei a G for the functional mapping a
matrix of the above form to ti . Then
(
{ei ej : 1 i < j m} for n odd,
(g, aG ) =
{ei ej : 1 i < j m} {ei : 1 i m} for n even,

g(aG ; (ei + ej )) = C(E2i1,2j1 1E2i1,2j 1E2i,2j1 + E2i,2j

E2j1,2i1 1E2j,2i1 1E2j1,2i E2j,2i ) (1 i < j m),

g(aG ; (ei ej )) = C(E2i1,2j1 1E2i1,2j 1E2i,2j1 E2i,2j

E2j1,2i1 1E2j,2i1 1E2j1,2i + E2j,2i ) (1 i < j m),
and if n is even additionally

g(aG ; ei ) = C(E2i1,n+1 1E2i,n+1 En+1,2i1 1En+1,2i ) (1 i m).
We choose the positive system

(
{ei ej : 1 i < j m} for n odd,
+ (g, aG ) =
{ei ej : 1 i < j m} {ei : 1 i m} for n even,
then
(
{exp(diag(D(t1 ), . . . , D(tm ))) : ti R} SO(2)m for n odd,
MG =
{exp(diag(D(t1 ), . . . , D(tm ), 0)) : ti R} SO(2)m for n even,
(
{exp(diag(D(t1 ), . . . , D(tm1 ), 0)) : ti R} SO(2)m1 for n odd,
MH =
{exp(diag(D(t1 ), . . . , D(tm ))) : ti R} SO(2)m for n even,
46
JAN MOLLERS
and
P

{X(t) = m1
i=1 ti (E2i1,n+1 1E2i,n+1 En+1,2i1 + 1En+1,2i ) : ti C}

for n odd,
n = Pm

{X(t) = i=1 ti (E2i1,n+1 1E2i,n+1 En+1,2i1 + 1En+1,2i ) : ti C}

for n even.
The action of (
SO(2)m1 for n odd,
MG MH
SO(2)m for n even,
is given by the natural action of SO(2) U(1) on ti C by rotation. Hence, the element
X(t) is contained in an open (MG MH )AH -orbit if and only if t1 , . . . , tm1 6= 0 resp.
t1 , . . . , tm 6= 0 and then M = {1}. Further, SO(2) acts transitively on the unit sphere in C
and hence MG MH acts transitively on the product of the unit spheres in n , so there is
a unique open orbit.
6.2.5. (G, H) = (SO(p, q + 1), SO(p, q)). Let G = SO(p, q + 1), p, q 1, and define a Cartan
involution on G by (g) = (g )1 , then K = G = S(O(p)O(q+1)). We embed H = SO(p, q)
in the upper left corner of G, then leaves H invariant and H K = H = S(O(p) O(q)).
Assume that p q, the case p q + 1 is handled similarly (see Section 6.2.3 for a related
computation). We choose the maximal abelian subalgebra

0p D
aG = aH = D 0p : D = diag(t1 , . . . , tp ), t1 , . . . , tp R

0qp+1
of g and h . Write ei a
G for the functional mapping a matrix of the above form to ti .
Then
(g, aG ) = {ei ej : 1 i < j p} {ei : 1 i p}
g(aG ; (ei + ej )) = R(Ei,j Ej,i Ei,j+p Ej,i+p
Ei+p,j Ej+p,i Ei+p,j+p Ej+p,i+p) (1 i < j p),
g(aG ; (ei ej )) = R(Ei,j Ej,i Ei,j+p Ej,i+p
Ei+p,j Ej+p,i Ei+p,j+p Ej+p,i+p) (1 i < j p),
qp+1
M
g(aG ; ei ) = R(Ei,2p+k Ei+p,2p+k + E2p+k,i E2p+k,i+p ) (1 i p).
k=1
W choose the positive system

+ (g, aG ) = {ei ej : 1 i < j p} {ei : 1 i p},
then
MG = {diag(x1 , . . . , xp , x1 , . . . , xp , k) : xi = 1, k SO(q p + 1)} O(1)p SO(q p + 1),
MH = {diag(x1 , . . . , xp , x1 , . . . , xp , k, 1) : xi = 1, k SO(q p)} O(1)p SO(q p),
and
n p
X o

n = X(t) = ti (Ei,p+q+1 + Ei+p,p+q+1 + Ep+q+1,i Ep+q+1,i+p ) : ti R .
i=1
The action of MG MH O(1)p SO(q p) is given by the natural action of O(1)p on t Rp

by sign changes of the coordinates, and the second factor SO(q p) acts trivially. Hence, the
element X(t) is contained in an open (MG MH )AH -orbit if and only if t1 , . . . , tp 6= 0 and
then M = SO(q p). Further, O(1)p MG MH acts transitively on the product {1}p Rp
of the unit spheres {1} R and therefore there is a unique open orbit.
6.3. Finite-dimensional branching. Using case-by-case arguments, we prove in this sub-

section the following statement:
cG M
Proposition 6.4. Let (G, H) be one of the pairs in (6.1) and (, ) M cH . Then
(1) dim HomM ( w0 |M , |M ) = dim HomM (|M , |M ) 1.
(2) Whenever HomM (|M , |M ) 6= {0}, there exist finite-dimensional representations E
of G and F of H such that E nG |MG , F nH |MH and HomH (E|H , F ) 6= {0}.
In all five cases G and H have connected complexifications GC and HC , and we can param-
etrize irreducible finite-dimensional representations of GC and HC by their highest weights
j
G,C and jH,C for jG g and jH h Cartan subalgebras. Denote the restrictions of
the corresponding representations to G and H by F G () and F H ().
In each case, we use the notation introduced in Section 6.2.
6.3.1. (G, H) = (GL(n + 1, C), GL(n, C)). We extend aG to the Cartan subalgebra jG =
tG aG with

tG = mG = 1aG = { 1 diag(t1 , . . . , tn+1 ) : ti R}.
Let fi t
G,C (1 i n + 1) be the linear functionals mapping a diagonal matrix as above

to 1ti . Then the root system (gC , jG,C ) is of the form
{(ei ej ) (fi fj ) : 1 i < j n + 1}
and putting
i = ei + fi and i = ei fi
we have (gC , jG,C ) = {(i j ), (i j ) : 1 i < j n + 1}. Further, the positive
system + (gC , jG,C ) = {i j , i j : 1 i < j n + 1} is compatible with the positive
system + (g, aG ).
We choose the complexification GC = GL(n+1, C)GL(n+1, C) with embedding G GC
given by g 7 (g, g), then jG,C is a Cartan subalgebra of gC = gl(n+1, C)+gl(n+1, C) and the
irreducible finite-dimensional representations of GC are parametrized by their highest weights
= 1 1 + + n+1 n+1 + 1 1 + + n+1 n+1 , 1 . . . n+1 , 1 . . . n+1 ,

i , i Z. Clearly, an element
g = exp( 1
diag(t1 , . . . , tn+1 )) MG acts on the highest
G 1( )t 1( )t
weight space of F () by e 1 1 1 e m m m.
In the same way we parametrize irreducible finite-dimensional representations of HC =

GL(n, C) GL(n, C) by highest weight = 1 1 + + n n + 1 1 + + n n , 1
. . . n , 1 . . . n , i , i Z. On its highest weight space an element of the form

h = exp( 1 diag(t1 , . . . , tn , 0)) MH acts by e 1(1 1 )t1 e 1(n n )tn .
48
JAN MOLLERS
Now, MG = U(1)n+1 , MH = U(1)n and M = {1}. Hence, irreducible representations of

MG and MH are one-dimensional and dim HomM ( w0 |M , |M ) = dim HomM (M , |M ) = 1

for all (, ) McG M cH . Every M cG has the form (exp( 1 diag(t1 , . . . , tn+1 ))) =

e 11 t1 e 1n+1 tn+1 with 1 , . .
. , n+1 Z. Similarly, every M cH has the form

(exp( 1 diag(t1 , . . . , tn , 0))) = e 11 t1 e 1n tn with 1 , . . . , n Z. By the above
observations F G ()nG |MG if and only if i i = i (1 i n + 1). Further,
F H ()nH |MH if and only if i i = i (1 i n). Moreover, we have HomH (F G (), F H ()) 6=
{0} if and only if
1 1 2 . . . n n n+1 and 1 1 2 . . . n n n+1 .
We first choose (n+1 , n+1 ) Z Z with n+1 n+1 = n+1 . Next we choose (n , n )
N N with n n+1 , n n+1 and n n = n . Iterating this procedure constructs (not
necessarily unique) highest weights and with the desired properties.
6.3.2. (G, H) = (GL(n + 1, R), GL(n, R)). We choose the natural complexification GC =
GL(n + 1, C), then aG,C is a maximal torus in gC = gl(n + 1, C) and the irreducible finite-
dimensional representations of GC are parametrized by their highest weights = 1 e1 + +
n+1 en+1 , 1 . . . n+1 , i Z. For H we use the similar notation = 1 e1 + + n en ,
1 . . . , n , i Z.
Note that an element g = diag((1)k1 , . . . , (1)kn+1 ) MG acts on the highest weight
space F G ()nG by (1)k1 1 ++kn+1 n+1 . Similarly, h = diag((1)1 , . . . , (1)n , 1) MH
acts on F H ()nH by (1)1 1 ++n n . By the classical branching laws, the restriction of
F G () to H contains all representations F H () with
1 1 2 . . . n n n+1 . (6.5)
Now, MG = O(1)n+1 , MH = O(1)n and M = {1}. Hence, irreducible representations of
MG and MH are one-dimensional and dim HomM ( w0 |M , |M ) = dim HomM (|M , |M ) = 1
for all (, ) M cG McH . Every M cG has the form (diag((1)k1 , . . . , (1)kn+1 )) =
k
(1) 1 1 ++k
n+1 n+1 with Z/2Z. Similarly, every representation M
i
cH has the form

(diag((1) , . . . , (1) , 1)) = (1)
1 n 1 1 ++ n n
with i Z/2Z. By the above observations
F G ()nG |MG if and only if i = i + 2Z. Further, F H ()nH |MH if and only if
i = i + 2Z. It is clear that for fixed 1 , . . . , n+1 , 1 , . . . , n Z/2Z there always exist
integers 1 , . . . , n+1 and 1 , . . . , n satisfying the interlacing condition (6.5) and i = i + 2Z,
i = i + 2Z.
6.3.3. (G, H) = (U(p, q + 1), U(p, q)). Assume that p q + 1, the case p q is handled
similarly. We extend aG to the Cartan subalgebra jG = tG aG with

tG = { 1 diag(t1 , . . . , tq+1 , s1 , . . . , spq1 , t1 , . . . , tq+1 ) : si , ti R} mG .
Let fi tG,C (1 i q + 1) be the linear functionals mapping
a diagonal matrix as above to

1ti , and gi tG,C (1 i p q 1) the ones mapping to 1si . Then the root system
(gC , jG,C ) is of the form
{ei ej (fi fj ) : 1 i < j q + 1} {2ei : 1 i q + 1}
{ei (fi gj ) : 1 i q + 1, 1 j p q 1} {(gi gj ) : 1 i < j p q 1}
and putting
(1 , . . . , p+q+1 ) = (f1 + e1 , . . . , fq+1 + eq+1 , g1 , . . . , gpq1 , fq+1 eq+1 , . . . , f1 e1 )
we have (gC , jG,C ) = {(i j ) : 1 i < j p + q + 1}. Further, the positive system
+ (gC , jG,C ) = {i j : 1 i < j p + q + 1} is compatible with the positive system
+ (g, aG ).
We choose the complexification GC = GL(p + q + 1, C), then jG,C is a Cartan subalgebra
of gC = gl(p + q + 1, C) and the irreducible finite-dimensional representations of GC are
parametrized by their highest weights = 1 1 + + p+q+1 p+q+1 , 1 . . . p+q+1 ,
i Z.
An element g = diag(z1 , . . . , zq+1 , 1pq1 , z1 , . . . , zq+1 ) MG acts on the highest restricted
+ q+1 +p+1
weight space of F G () by z1 1 p+q+1 zq+1 . Further, U(p q 1) MG has roots
(i j ) (q + 2 i < j p) and therefore its action on the highest restricted weight space
is given by F U(pq1) (q+2 q+2 + + p p ).
GL(p + q, C) by their highest weights = 1 1 + + p+q p+q . An element of the form
h = diag(z1 , . . . , zq , 1pq , z1 , . . . , zq ) MH acts on the highest restricted weight space by by
+ +
z1 1 p+q zq q p+1 and U(p q) MH acts by F U(pq) (q+1 q+1 + + p p ).
Now, MG = U(1)q+1 U(p q 1), MH = U(1)q U(p q) and M = U(p q 1). An
irreducible representation M cG is of the form = with

(diag(z1 , . . . , zq+1 , 1pq1 , z1 , . . . , zq+1 )) = z11 zq+1
q+1
,

1 , . . . , q+1
Z, and = F U(pq1) (1 q+2 + + pq1
p ), 1 . . . pq1 . Similarly,
every M cH has the form = with

(diag(z1 , . . . , zq , 1pq , z1 , . . . , zq )) = z1 1 zq q ,
), . . . .
1 , . . . , q Z, and = F U(pq) (1 q+1 + + pq p 1 pq
This implies that we have to put
(q+2 , . . . , p ) = (1 , . . . , pq1

) and (q+1 , . . . , p ) = (1 , . . . , pq

).
0 = diag(1p , 1q+1 ) commutes with MG we have w0 = , so that HomM ( w0 |M , |M ) =
Since w
HomM (|M , |M ), and the condition HomM (|M , |M ) 6= {0} is equivalent to the condition
HomU(pq1) ( , |U(pq1) ) 6= {0} which is in turn equivalent to
1 1 2 . . . pq1

pq .
Therefore, the already chosen i s and j s satisfy the necessary interlacing condition for
HomH (F G (), F H ()) 6= {0}. It remains to show that one can choose the remaining i s and
j s such that the interlacing condition still holds and additionally i + p+qi+2 = i and
j + p+qj+1 = j , which is an easy exercise.
6.3.4. (G, H) = (SO(n + 1, C), SO(n, C)). Assume that n = 2m is even, the case of odd n is
treated similarly. We extend aG to the Cartan subalgebra jG = tG aG with

tG = mG = 1aG = { 1 diag(D(t1 ), . . . , D(tm )) : ti R}.
Let fi t
G,C (1 i m) be the linear functionals mapping a diagonal matrix as above to
1ti . Then the root system (gC , jG,C ) is of the form

{(ei + fi ) (ej + fj ) : 1 i < j m} {(ei + fi ) : 1 i m}
{(ei fi ) (ej fj ) : 1 i < j m} {(ei fi ) : 1 i m}
50
JAN MOLLERS
and putting
i = ei + fi and i = ei fi
we have (gC , jG,C ) = {i j , i j : 1 i < j m} {i , i : 1 i m}. Further,
the positive system + (gC , jG,C ) = {i j , i j : 1 i < j m} {i , i : 1 i m} is
compatible with the positive system + (g, aG ).
We choose the complexification GC = SO(n+1, C)SO(n+1, C) with embedding G GC
given by g 7 (g, g), then jG,C is a Cartan subalgebra of gC = so(n+1, C)+so(n+1, C) and the
irreducible finite-dimensional representations of GC are parametrized by their highest weights
= 1 1 + + m m + 1 1 + + m m , 1 . . . m1 |m |, 1 . . . m1 |m |,

i , i Z. An element g = exp( 1 diag(D(t 1 ), . . . , D(tm ))) MG acts on the highest

weight space of F G () by e 1(1 1 )t1 e 1(m m )tm .
SO(n, C)SO(n, C) by their highest weights = 1 1 + +m1
m1 +1 1 + +m1 m1 ,

1 . . . m1 , 1 . . . m1 , i , i Z. On its highest weight space an element h =

exp( 1 diag(D(t1 ), . . . , D(tm1 ), 0)) MH acts by e 1(1 1 )t1 e 1(m1 m1 )tm1 .
Now, MG = SO(2)m , MH = SO(2)m1 and M = {1}. Hence, irreducible representations
of MG and MH are one-dimensional and dim HomM ( w0 |M , |M ) = dim HomM (|M , |M ) = 1
for all (, ) M cG M cH . Every M cG has the form

(exp( 1 diag(D(t1 ), . . . , D(tm )))) = e 11 t1 e 1m tm
with 1 , . . . , m Z. Similarly, every M cH has the form

(exp( 1 diag(D(t1 ), . . . , D(tm1 ), 0))) = e 11 t1 e 1m1 tm1
with 1 , . . . , m1 Z. By the above observations F G ()nG |MG if and only if i i = i
(1 i m). Further, F H ()nH |MH if and only if i i = i (1 i m 1). Moreover,
we have HomH (F G (), F H ()) 6= {0} if and only if
1 1 2 . . . m1 m1

|m | and
1 1 2 ... m1
m1 |m |.
We first choose (m , m ) ZZ with m m = m . Next we choose (m1

, m1 ) NN

with m1 |m |, m1 |m | and m1 m1 = m1 . Iterating this procedure shows
the claim.
6.3.5. (G, H) = (SO(p, q + 1), SO(p, q)). Assume that p q, the case p q + 1 is handled
similarly. We further assume that q p = 2m is even, leaving the odd case to the reader. We
extend aG to the Cartan subalgebra jG = tG aG with
tG = {diag(02p , D(t1 ), . . . , D(tm ), 0) : ti R} mG .

Let ep+i t
(1 i m) be the linear functional mapping a matrix as above to
G,C 1ti .
Then the root system (gC , jG,C ) is of the form
{ei ej : 1 i < j p + m} {ei : 1 i p + m}
and the positive system
+ (gC , jG,C ) = {ei ej : 1 i < j p + m} {ei : 1 i p + m}
is compatible with the positive system + (g, aG ).
We choose the complexification GC = SO(p + q + 1, C), then jG,C is a Cartan subalgebra

of gC = so(p + q + 1, C) and the irreducible finite-dimensional representations of GC are
parametrized by their highest weights = 1 e1 + +p+m ep+m , where 1 . . . p+m 0,
i Z. An element g = diag((1)k1 , . . . , (1)kp , (1)k1 , . . . , (1)kp , 1qp1 ) MG acts on
the highest restricted weight space of F G () by (1)k1 1 ++kp p . Further, SO(qp+1) MG
acts on the highest restricted weight space by F SO(qp+1) (p+1 ep+1 + + p+m ep+m ).
SO(p + q, C) by their highest weights = 1 e1 + + p+m ep+m ). An element h =

diag(x1 , . . . , xp , x1 , . . . , xp , 1qp+1 ) MH acts by x11 xpp and SO(q p) MH acts by
F SO(qp) (p+1 ep+1 + + p+m ep+m ).
Now, MG = O(1)p SO(q p + 1), MH = O(1)p SO(q p) and M = SO(q p). An
irreducible representation M cG is of the form = with

(x1 , . . . , xp , x1 , . . . , xp , 1qp+1 )) = x11 xpp ,
e
1 , . . . , p Z/2Z, and = F SO(qp+1) (1 ep+1 + + m
p+m ), 1 . . . m 0. Similarly,
every M cH has the form = with

(diag(x1 , . . . , xp , x1 , . . . , xp , 1qp+1 )) = x11 xpp ,
1 , . . . , p Z/2Z, and = F SO(qp) (1 ep+1 + + m
e
p+m ), 1 . . . m1 |m |.
This implies that we have to put
(p+1 , . . . , p+m ) = (1 , . . . , m

) and (p+1 , . . . , p+m ) = (1 , . . . , m

).
0 = diag(1p , 1q+1 ) commutes with MG we have w0 = , so that HomM ( w0 |M , |M ) =
Since w
HomM (|M , |M ), and the condition HomM (|M , |M ) 6= {0} is equivalent to the condition
HomSO(qp) ( |SO(qp) , ) 6= {0} which is in turn equivalent to
1 1 2 . . . m1

m
|m |.
Therefore, the already chosen i s and j s satisfy the necessary interlacing condition for
HomH (F G (), F H ()) 6= {0}. It remains to show that one can choose the remaining i s and
j s such that the interlacing condition holds and additionally i = i + 2Z and j = j + 2Z,
which is an easy exercise.
6.4. Generic multiplicities. Using the intertwining operators between spherical principal
series constructed in Section 3 and the upper multiplicity bounds obtained in Section 4 we
prove the following generic multiplicity formula for general principal series representations of
multiplicity one pairs:
Theorem 6.5. Assume that (G, H) is one of the multiplicity one pairs in (6.1). Then for
cG M
all (, ) M cH and (, ) a a we have the lower multiplicity bound
G,C H,C
dim HomH (, |H , , ) 1 whenever HomM (|M , |M ) 6= {0},

and for (, ) a
G,C aH,C satisfying the generic condition (4.1) we have
(
1 for HomM (|M , |M ) 6= {0},
dim HomH (, |H , , ) =
0 for HomM (|M , |M ) = {0}.
52
JAN MOLLERS
Proof. By Lemma 6.3 we have #(PH \G/PG )open = 1. Hence, it follows from Theorem 4.1
and Proposition 6.4 (1) that
dim HomH (, |H , , ) dim HomM ( w0 |M , |M ) = dim HomM (|M , |M ) 1
for (, ) a
G,C aH,C satisfying (4.1). It therefore suffices to show that for HomM (|M , |M ) 6=
{0} we have dim HomH (, |H , , ) 1 for all (, ) a
G,C aH,C .
By Proposition 6.4 (2) and Corollary 6.2 there exist (0 , 0 ) a
G aH and a linear map
: HomH ( |H , ) HomH (,+0 |H , ,+0 )
which is on the level of distribution kernels given by tensoring with a fixed non-trivial real-
analytic section. We apply this map to the holomorphic family of intertwining operators
obtained in Theorem 3.3. More precisely by Theorem 3.3 there exists a family K, of
distribution kernels of intertwining operators A, HomH ( |H , ), depending holomor-
phically on (, ) a
G,C aH,C , such that generically supp K, = G/PG . Then the dis-
tribution kernels of (A, ) depend holomorphically on (, ) a
G,C aH,C since they are
given by tensoring the holomorphic family K, with a fixed smooth section. Further, by
Corollary 6.2 they are generically supported on G/PG and hence the holomorphic family
(A, ) HomH (,+0 |H , ,+0 ) is non-trivial. Now the desired lower multiplicity bound
follows from Lemma 3.5.
Combining Theorem 6.5 with the multiplicity one statement in Fact II we immediately
obtain:
Corollary 6.6. Let (G, H) be one of the pairs in (6.1) and assume that , and , are
irreducible. Then, if HomM (|M , |M ) 6= {0} we have
dim HomH (, |H , , ) = 1.
6.5. The GrossPrasad conjecture for complex orthogonal groups. In 1992 B. Gross
and D. Prasad [8] formulated a conjecture about the multiplicities dimH (|H , ) for the re-
ductive pair (G, H) = (SO(n + 1), SO(n)) over local and global fields. For the field k = C the
local conjecture takes the following form:
Conjecture 6.7 ([8, Conjecture 11.5]). Let (G, H) = (SO(n + 1, C), SO(n, C)) and assume
that , and , are irreducible, then dim HomH (, |H , , ) = 1.
Using our results from the previous section we can prove this conjecture. It follows from
the following more general statement:
Corollary 6.8. Let (G, H) be one of the pairs in (6.1) and assume that p = q or p = q + 1
in the case of indefinite unitary or orthogonal groups. Then, if the representations , and
, are irreducible, we have
dim HomH (, |H , , ) = 1.
cG and
Proof. In all cases we have M = {1} so that HomM (|M , |M ) 6= {0} for all M
M cH . Then the statement follows from Corollary 6.6.
7. An example: (G, H) = (GL(n + 1, R), GL(n, R))

For the multiplicity one pair (G, H) = (GL(n + 1, R), GL(n, R)) we describe the meromor-
phic families of intertwining operators between principal series as constructed in Section 6 ex-
plicitly. Over p-adic fields such operators were previously constructed by MuraseSugano [32]
(see also the recent work by Neretin [33] in the context of finite-dimensional representations)
and the formulas for the distribution kernels turn out to be formally the same in the real case.
7.1. Distribution kernels. For 1 p n + 1 and 1 q n we define the following
polynomial functions on M ((n + 1) (n + 1), R):
p (x) = det((xij )1i,jp ), q (x) = det((xij )2iq+1,1jq ).
With the representative
1
.
w
0 = ..
1
for the longest Weyl group element w0 W (aG ) we then consider the functions g 7
k (w
0 g), k (w
0 g) on G. For d = diag(d1 , . . . , dn+1 ) MG AG and n NG we have
0 gdn) = d1 dk k (w
k (w 0 g), k (w
0 gdn) = d1 dk k (w
0 g),
and for d = diag(d1 , . . . , dn , 1) MH AH and n NH additionally
k (w
0 dng) = dnk+2 dn k (w
0 g), k (w
0 dng) = dnk+1 dn k (w
0 g).
cG (Z/2Z)n+1 by mapping = (1 , . . . , n+1 ) (Z/2Z)n+1 to the character
We identify M
MG {1}, diag(x1 , . . . , xn+1 ) 7 sgn(x1 )1 sgn(xn+1 )n+1 .
cH (Z/2Z)n . Further, we identify a Cn+1 by 7 ((E1,1 ), . . . , (En+1,n+1 ))
Similarly M G,C
cG , = (1 , . . . , n )
and similarly aH,C C . Then for = (1 , . . . , n+1 ) (Z/2Z)n+1 M
n
n
(Z/2Z) M cH and C n+1
a ,C a n we put
G,C H,C
0 g)s1 ,1 n+1 (w
K(,),(,) (g) = 1 (w 0 g)sn+1 ,n+1 1 (w
0 g)t1 ,1 n (w
0 g)tn ,n , g G,
where
si = i ni+1 12 (1 i n), sn+1 = n+1 + n2 , tj = nj+1 j+1 21 (1 j n)
and
i = i ni+1 (1 i n), n+1 = n+1 , j = nj+1 j+1 (1 j n).
Here we have used the notation
xs, = sgn(x) |x|s , x R , s C, Z/2Z.
Then K(,),(,) satisfies
K(,),(,) (m a n gman) = (m)anG (m )(a )+nH K(,),(,) (g)
for g G, man PG , m a n PH . Hence, the functions K(,),(,) define a meromorphic
family of intertwining operators A(,),(,) : , |H , by
Z
A(,),(,) f (h) = K(,),(,) (k, h)f (k) dk, h H.
K
54
JAN MOLLERS
Remark 7.1. It is easy to see that the functions g 7 k (g), k (g) are matrix coefficients for
V
the irreducible finite-dimensional representation of GL(n + 1, R) on k Rn+1 .
Appendix A. Finite-dimensional branching rules for strong Gelfand pairs

We list the explicit branching rules for some small representations of the strong Gelfand
pairs (sl(n + 1, C), gl(n, C)) and so(n + 1, C), so(n, C)). The classical branching rules for these
pairs can e.g. be found in [6, Chapter 8].
A.1. (sl(n + 1, C), gl(n, C)). We label the Dynkin diagrams of sl(n + 1, C) and sl(n, C) as
usual:
1 2 n1 n
sl(n + 1, C) :
1 2 n1
sl(n, C) :
Realize the root system of sl(n + 1, C) as {(ei ej ) : 1 i < j n + 1} in the vector space
V = {x Rn+1 : x1 + + xn+1 = 0}. To simplify notation, denote by (x) the orthogonal
projection of x Rn+1 to V . We choose the simple roots i = ei ei+1 (1 i n) for
sl(n + 1, C) and the simple roots i = i (i = 1, . . . , n 1) for sl(n, C). Denote by 1 , . . . , n
the corresponding fundamental weights for sl(n + 1, C) and by 1 , . . . , n1 the fundamental
weights for sl(n, C). Further put n := n , then n describes a character of z(gl(n, C)) C.
Consider the fundamental weight i = (e1 + + ei ). From the classical branching laws
we know that F g (i )|h decomposes into the direct sum of the two h-representations with
highest weights (e1 + + ei ) and (e1 + + ei1 + en+1 ). Using
i = i + ni n , 1 i n 1,
it follows that

sl(n,C) C 1 sl(n,C) C

F (1 ) F ( n n ) F (0) F (n ) for i = 1,

sl(n,C) C i sl(n,C) C ni+1

F ( i ) F (
n n ) F ( i1 ) F ( n n )
F sl(n+1,C) (i )|gl(n,C) for 2 i n 1,

F

sl(n,C) C
(0) F (n ) F sl(n,C) (n1 ) F ( n1 n )
C

for i = n.
Now consider the fundamental weight i + ni+1 = (e1 + + ei ) (eni+2 + + en+1 )
(1 i n2 ). From the classical branching laws we know that F g (i + ni+1 )|h decomposes
into the direct sum of the four h-representations with highest weights
(e1 + + ei ) (eni+1 + + en ), (e1 + + ei ) (eni+2 + + en+1 ),
(e1 + + ei1 ) (eni+1 + + en ) + en+1 , (e1 + + ei1 ) (eni+2 + + en ),
so that
F sl(n+1,C) (i + ni+1 )|gl(n,C)

F sl(n,C) (i + ni ) F C (0) F sl(n,C) (i + ni+1 ) F C ( n+1n n )

F sl(n,C) (i1 + ni ) F C ( n+1
n n ) F sl(n,C)
( i1 + ni+1 ) F C
(0) .
Note that for i = n2 with n = 2m even the formula still holds and we have i + ni = 2m .
Similarly one obtains for i = n+1 2 with n = 2m 1 odd:

F sl(n+1,C) (2m )|gl(n,C) F sl(n,C) (2m ) F C ( n+1
n n ) F sl(n,C)
(2 m1 ) F C
( n+1
2 n )

F sl(n,C) (m1 + m ) F C (0) .
A.2. (so(n + 1, C), so(n, C)). We label the Dynkin diagrams of so(n + 1, C) and so(n, C) as
usual:
m1
m1 m
1 2 1 2 m2
so(n + 1, C) : +3

m
m1
m2 m1
1 2 m2 1 2
so(n, C) : +3

m
(n = 2m) (n = 2m 1)
From the classical branching rules it follows that, for n = 2m even:
F so(n+1,C) (i )|so(n,C) F so(n,C) (i ) F so(n,C) (i1 ) (1 i m 2),
so(n+1,C) so(n,C) so(n,C)
F (m1 )|so(n,C) F (m1 + m ) F (m2 ),
F so(n+1,C) (m )|so(n,C) F so(n,C) (m ) F so(n,C) (m1 ),
F so(n+1,C) (2m )|so(n,C) F so(n,C) (2m ) F so(n,C) (2m1 ) F so(n,C) (m1 + m ),
and for n = 2m 1 odd:
F so(n+1,C) (i )|so(n,C) F so(n,C) (i ) F so(n,C) (i1 ) (1 i m 2),
so(n+1,C) so(n+1,C) so(n,C)
F (m1 )|so(n,C) F (m )|so(n,C) F (m1 ),
F so(n+1,C) (m1 + m )|so(n,C) F so(n,C) (2m1 ) F so(n,C) (m2 ).
References
[1] Joseph Bernstein and Andre Reznikov, Estimates of automorphic functions, Mosc. Math. J. 4 (2004),
no. 1, 1937, 310.
[2] Jean-Louis Clerc, Singular conformally invariant trilinear forms, I: The multiplicity one theorem, Trans-
form. Groups 21 (2016), no. 3, 619652.
[3] , Singular conformally invariant trilinear forms, II: The higher multiplicity case, (2016), to appear
in Transform. Groups, published online first at doi:10.1007/s00031-016-9404-7.
[4] Jean-Louis Clerc, Toshiyuki Kobayashi, Bent rsted, and Michael Pevzner, Generalized Bernstein
Reznikov integrals, Math. Ann. 349 (2011), no. 2, 395431.
[5] Matthias Fischmann, Andreas Juhl, and Petr Somberg, Conformal symmetry breaking differential opera-
tors on differential forms, (2016), preprint, available at arXiv:1605.04517.
[6] Roe Goodman and Nolan R. Wallach, Symmetry, representations, and invariants, Graduate Texts in
Mathematics, vol. 255, Springer, Dordrecht, 2009.
56
JAN MOLLERS
[7] Dmitry Gourevitch, Siddhartha Sahi, and Eitan Sayag, Analytic continuation of equivariant distributions,
(2016), preprint, available at arXiv:1608.03442.
[8] Benedict H. Gross and Dipendra Prasad, On the decomposition of a representation of SOn when restricted
to SOn1 , Canad. J. Math. 44 (1992), no. 5, 9741002.
[9] Sigurdur Helgason, Differential geometry, Lie groups, and symmetric spaces, Pure and Applied Mathe-
matics, vol. 80, Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York-London, 1978.
[10] Miki Hirano, Shintani functions on GL(2, R), Trans. Amer. Math. Soc. 352 (2000), no. 4, 17091721.
[11] , Shintani functions on GL(2, C), Trans. Amer. Math. Soc. 353 (2001), no. 4, 15351550.
[12] Anthony W. Knapp, Lie groups beyond an introduction, second ed., Progress in Mathematics, vol. 140,
Birkhauser Boston, Inc., Boston, MA, 2002.
[13] Friedrich Knop, Bernhard Kr otz, Tobias Pecher, and Henrik Schlichtkrull, Classification of reductive real
spherical pairs I. The simple case, (2016), preprint, available at arXiv:1609.00963.
[14] , Classification of reductive real spherical pairs II. The semisimple case, (2017), preprint, available
at arXiv:1703.08048.
[15] Toshiyuki Kobayashi, Shintani functions, real spherical manifolds, and symmetry breaking operators, De-
velopments and retrospectives in Lie theory, Dev. Math., vol. 37, Springer, Cham, 2014, pp. 127159.
[16] , A program for branching problems in the representation theory of real reductive groups, Repre-
sentations of reductive groups, Progr. Math., vol. 312, Birkh auser/Springer, Cham, 2015, pp. 277322.
[17] Toshiyuki Kobayashi, Toshihisa Kubo, and Michael Pevzner, Conformal symmetry breaking operators for
differential forms on spheres, Lecture Notes in Mathematics, vol. 2170, Springer, Singapore, 2016.
[18] Toshiyuki Kobayashi and Toshihiko Matsuki, Classification of finite-multiplicity symmetric pairs, Trans-
form. Groups 19 (2014), no. 2, 457493.
[19] Toshiyuki Kobayashi and Toshio Oshima, Finite multiplicity theorems for induction and restriction, Adv.
Math. 248 (2013), 921944.
[20] Toshiyuki Kobayashi and Birgit Speh, Symmetry breaking for representations of rank one orthogonal
groups, Mem. Amer. Math. Soc. 238 (2015), no. 1126.
[21] , Symmetry breaking for orthogonal groups and a conjecture by B. Gross and D. Prasad, (2017),
preprint, available at arXiv:1702.00263.
[22] Bernhard Kr otz and Henrik Schlichtkrull, Finite orbit decomposition of real flag manifolds, J. Eur. Math.
Soc. (JEMS) 18 (2016), no. 6, 13911403.
[23] , Harmonic analysis for real spherical spaces, (2016), to appear in Acta Math. Sinica, available at
arXiv:1612.01841.
[24] W. G. McKay and Jir Patera, Tables of dimensions, indices, and branching rules for representations of
simple Lie algebras, Lecture Notes in Pure and Applied Mathematics, vol. 69, Marcel Dekker, Inc., New
York, 1981.
[25] Jan M ollers and Bent rsted, Estimates for the restriction of automorphic forms on hyperbolic manifolds
to compact geodesic cycles, (2014), to appear in Int. Math. Res. Not. IMRN, published online first at
doi:10.1093/imrn/rnw119.
[26] , KnappStein type intertwining operators for symmetric pairs II. The translation principle and
intertwining operators for spinors, (2017), preprint, available at arXiv:1702.02326.
[27] Jan M ollers, Bent rsted, and Yoshiki Oshima, KnappStein type intertwining operators for symmetric
pairs, Adv. Math. 294 (2016), 256306.
[28] Jan M ollers, Bent rsted, and Genkai Zhang, On boundary value problems for some conformally invariant
differential operators, Comm. Partial Differential Equations 41 (2016), no. 4, 609643.
[29] Jan M ollers and Yoshiki Oshima, Restriction of most degenerate representations of O(1, N ) with respect
to symmetric pairs, J. Math. Sci. Univ. Tokyo 22 (2015), no. 1, 279338.
[30] Tomonori Moriyama, Spherical functions with respect to the semisimple symmetric pair
(Sp(2, R), SL(2, R) SL(2, R)), J. Math. Sci. Univ. Tokyo 6 (1999), no. 1, 127179.
[31] , Spherical functions for the semisimple symmetry pair (Sp(2, R), SL(2, C)), Canad. J. Math. 54
(2002), no. 4, 828865.
[32] Atsushi Murase and Takashi Sugano, Shintani functions and automorphic L-functions for GL(n), Tohoku
Math. J. (2) 48 (1996), no. 2, 165202.
[33] Yury A. Neretin, Restriction of representations of GL(n + 1, C) to GL(n, C) and action of the Lie overal-
gebra, (2015), preprint, available at arXiv:1510.03611.
[34] Anatolii Ivanovich Oksak, Trilinear Lorentz invariant forms, Comm. Math. Phys. 29 (1973), 189217.

[35] Toshio Oshima and Jir o Sekiguchi, The restricted root system of a semisimple symmetric pair, Group
representations and systems of differential equations (Tokyo, 1982), Adv. Stud. Pure Math., vol. 4, North-
Holland, Amsterdam, 1984, pp. 433497. MR 810638
[36] Claude Sabbah, Proximite evanescente. I. La structure polaire dun D-module, Compositio Math. 62
(1987), no. 3, 283328.
[37]
, Proximite evanescente. II. Equations fonctionnelles pour plusieurs fonctions analytiques, Com-
positio Math. 64 (1987), no. 2, 213241.
[38] Keiju Sono, Shintani functions on SL(3, R), Int. J. Math. Math. Sci. (2011), Art. ID 842806, 33.
[39] Binyong Sun and Chen-Bo Zhu, Multiplicity one theorems: the Archimedean case, Ann. of Math. (2) 175
(2012), no. 1, 2344.
[40] Masao Tsuzuki, Real Shintani functions on U(n, 1), J. Math. Sci. Univ. Tokyo 8 (2001), no. 4, 609688.
[41] , Real Shintani functions on U(n, 1). II. Computation of zeta integrals, J. Math. Sci. Univ. Tokyo
8 (2001), no. 4, 689719.
[42] , Real Shintani functions on U(n, 1). III. Construction of intertwining operators, J. Math. Sci.
Univ. Tokyo 9 (2002), no. 1, 165215.
[43] Nolan Wallach and Oded Yacobi, A multiplicity formula for tensor products of SL2 modules and an explicit
Sp2n to Sp2n2 Sp2 branching formula, Symmetry in mathematics and physics, Contemp. Math., vol.
490, Amer. Math. Soc., Providence, RI, 2009, pp. 151155.
[44] Garth Warner, Harmonic analysis on semi-simple Lie groups. I, Springer-Verlag, New York-Heidelberg,
1972, Die Grundlehren der mathematischen Wissenschaften, Band 188.
rnberg, Cauerstr. 11, 91058 Erlangen, Germany

Department Mathematik, FAU Erlangen-Nu
E-mail address: moellers@math.fau.de

Symmetry Breaking Operators For Strongly Spherical Reductive Pairs and The Gross-Prasad Conjecture For Complex Orthogonal Groups

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Symmetry Breaking Operators For Strongly Spherical Reductive Pairs and The Gross-Prasad Conjecture For Complex Orthogonal Groups

Uploaded by

Copyright:

Available Formats

SYMMETRY BREAKING OPERATORS FOR STRONGLY SPHERICAL

REDUCTIVE PAIRS AND THE GROSSPRASAD CONJECTURE FOR

2010 Mathematics Subject Classification. Primary 22E46; Secondary 11F70, 53C30.

where and are irreducible representations of G and H, usually assumed to be smooth

In case = 1 and = 1 are the trivial representations of MG and MH , we abbreviate = 1,

dim HomH ( |H , ) #(PH \G/PG )open ,

dim HomH ( |H , ) = #(PH \G/PG )open .

dim HomH (, |H , , ) 1 whenever HomM (|M , |M ) 6= {0},

Here M MG MH denotes the stabilizer of the unique open PH -orbit in G/PG .

dim Shmod (, ) = #(PH \G/PG )open .

1. The structure of strongly spherical reductive pairs

(so(1, p + 7), so(1, p) + g2 ) (p 0).

Then n is -stable and therefore we have a direct sum decomposition

Lemma 1.8. The extension of aH to aG can be chosen such that

where g (aH ; ) = g(aH ; ) g and

for its Lie algebra. Note that the Lie algebra w pG of wP

Now, if = 0 then either w1 > 0 or w1 () = w1 > 0, and therefore

with 6= 0 such that [g(aG ; ), n ] 6= {0}.

6 [g(aG ; k ), g(aG ; k+1 ) n ] g(aG ; k + k+1 ) n ,

so that k = k + k+1 satisfies g(aG ; k ) 6 nH .

2. Spherical matrix coefficients

2.3. Matrix coefficients. As for G, we denote by +

Note that since ( , V ) and ( , W ) extend to holomorphic representations of the complex

and this clearly defines a real-analytic function F : G R, F 0, with the equivariance

Proof. Since g and h are split, we can choose jG = aG and jH = aH . Let u = k + ip gC ,

A) Trivial case. This case g = h is by assumption excluded.

We realize the root system of g as {(ei ej ) : 1 i < j p + q + 1} in the vector

1 2 3 p p+1 p+2 p+q p+q+1

We choose the simple roots i = i (1 i p + q 1) for su(p, q) h, then p+q := p+q

(p q + 1) Then s = q + 1 and t = q, so that the pairs (2(i + p+qi+1 ), 2(i +

We use Appendix A.2 in what follows:

We realize the root system of g as {ei ej : 1 i < j n + 1} with simple roots

so that the representations

This implies that

Similarly, for the highest weight n + n+1 = e1 + + en one obtains

Now, if n = 2s 1 is odd, then

are contained in (g, h) and span a

1 2 3 2t 2t+1 p+q1 p+q

We choose the simple roots for g such that 1 = 1 , 2 = 4 , 3 = 2 , 4 = 3 , 5 = 5 ,

3. Lower multiplicity bounds - Construction of symmetry breaking operators

of intertwining operators between principal series of G and H, also referred to as symmetry

Clearly every restriction to K of a smooth function f on G satisfying f (m a n gman) =

(1) K, D (K), and supp K, for all (, ) a

i n, such that supp K, i = P x M for Re( ) a

are holomorphic in z = 0 and g1 (0), . . . , gn (0) are linearly independent.

gives a holomorphic function fp+1

where we put fp+1

4. Upper multiplicity bounds - Invariant distributions

4.1. Upper bounds for multiplicities. Let M MG MH denote the stabilizer of a

where dp denotes a left-invariant measure on PG . Then its transpose : D (G/PG , V,

D (G)(,),(,) = {u D (G) HomC (V , W ) :

4.2.1. Open orbits. If PH gPG is an open orbit, we may choose g = eX w 0 with X n

compact PH = gPG g1 = PH gPG g1 = 1 on M . Finally, S m (V (g)) = {0} for m > 0,

4.2.2. Non-open orbits. In this case it is more convenient to work with

Now assume P S m (W (g)), P 6= 0, with ad(X)P = P . Write P as linear combination of

with Pi,j S m (W (g)), then X = 0 implies, using Lemma 4.4:

5. Application to Shintani functions

The space of Shintani functions of type (, ) is denoted by Sh(, ). We write Shmod (, )

In view of this statement we abuse notation and write

dim Sh(, ) dim Shmod (, ) #(PH \G/PG )open ,

and for generic (, ) a

dim Shmod (, ) = #(PH \G/PG )open .

dim Sh(, ) = #(PH \G/PG )open