WIXLIB

PREFACExiplaced on the book. It is also clear that many such results will be obsoleteby publication time and we have ruled out their inclusion for this reason.A major area in the theory of smooth actions is occupied by the welldeveloped cobordism theory of group actions. We do not consider thistheory here because of the prior existence of an excellent introduction toit in the pioneering monograph of Conner and Floyd [8].The background demands on the reader are gradually increased throughout the book. This is especially true of the last four sections of ChapterVI and of parts of Chapter VII.

This Page Intentionally Left Blank

ACKNOWLEDGMENTSI would like to acknowledge my indebtedness to the book of Montgomery and Zippin [4], which provided my own introduction to the subject.The major part of that book was devoted to the solution (by Gleason,Montgomery, and Zippin) of Hilbert’s fifth problem on the structure oflocally compact groups, a subject which we do not consider here. For thisreason, and because of the numerous advances in transformation groupssince its publication, there is very little overlap between it and this book.Many mathematicians owe a great deal to the friendship, encouragement,and help extended to them over the years by Professor Montgomery, andI am not the least of them.I also wish to extend my thanks to Otto Morphy, who typed the entiremanuscript and corrected the errors. All remaining mistakes, mathematicaland otherwise, are his sole responsibility.Xlll

This Page Intentionally Left Blank

CHAPTER 0BACKGROUND ON TOPOLOGICAL GROUPS AND LIE GROUPSIn this chapter, we present some background material on topologicalgroups, the classical groups, and compact Lie groups. Much of the materialin this chapter is used only rarely in later chapters, so that it is not at allnecessary for the reader to be familiar with all of the results presentedhere. In fact, readers with a scant background would probably be wise toread the first three sections of this chapter and then pass directly to ChapterI, returning to this chapter only when needed. Several of the proofs areomitted, when giving them would take us too far afield, but all the resultsare readily available in a number of standard sources on Lie groups, suchas Chevalley [l], Hochschild [l], and Pontriagin [l].1. ELEMENTARY PROPERTIES OF TOPOLOGICAL GROUPSA topological group is a Hausdorff space G together with a continuousmultiplication G x G G [usually denoted by juxtaposition (g, h ) Hgh]which makes G into a group, and such that the map g - g - l of G Gis continuous. The identity element of G will usually be denoted by e.For g E G there is the left translation map L,: G G defined by Lo@) gh. Clearly L,h L , o Lh and LS-' (L,)-l and it follows that eachL , is a homeomorphism of G onto itself. Similarly, right translation R,:G defined by R,(h) hg-l satisfies R , o Rh Rghand R,-I (RU)-l,Gand each R , is a homeomorphism.For subsets A and B of G, we denote by AB, the subset {ab I a E A , b E B }of G ; that is, the image of A x B under multiplication G x G G. Simia, I aiE A } .larly, A-l {a-l I a E A } , and An {alasSince L , is a homeomorphism we see that gU is a neighborhood of g iffU is a neighborhood of e. Moreover, if U is a neighborhood of e, then soare U-l and U n U-I. A neighborhood U of e is called symmetric if U U F , and it follows that the symmetric neighborhoods of e form a neighborhood basis at e and this basis completely describes the topology of G.--f. .1

20. BACKGROUNDON TOPOLOGICAL GROUPS AND LIE GROUPSFrom continuity of the multiplication map G x G x G G at thepoint (e, g, e), we obtain the following lemma.1.I.Lemma Given g E G and a neighborhood U of g, there exists a neighborhood V of e with VgV c U. !Similarly, from multiplication G xlemma.- - . x G G we have the following1.2. Lemma Given a neighborhood U of e and an integer n there existsa neighborhood V of e with V n c U.1.3. Proposition If H is a subgroup of a topological group G, thenis also a subgroup. fl H is normal, then so is R.ProofLet p : Gx G G be p(g, h ) gh-l. Thenp(RXR) p ( H x H ) c p ( H x H ) Rso that B is a subgroup. If H is normal, then, since L,R,: Gh F ghg-l is a homeomorphism, we havegRg-'so thatR L,R,(R) L,R,(H) ---fG takingBR is normal.1.4. Proposition Let H be a closed subgroup of G. Then the space GIHof left cosets gH of W in G, with the quotient topology induced by the canonical map p : G G/H, is a Hausdorff space and p is continuous and open.Proof The map p is continuous by definition and it is open since q-lq(U) UH UIEHUh is open for all open U c G. To show that G / H is Hausdorff, suppose that g,H # g,H; that is gy1g2 @ H. Since H is closed, 1.1implies that there is a symmetric neighborhood U of e with (Ug;lgZU) n H 0.Thus g;'g,U n UH 0 which implies that g,U n g,UH 0and hence that g,UH n g,UH 0.However, p(giU) giUH are thendisjoint open sets in GIH containing g,H and g,H.1.5. PropositionI f H is a closed normal subgroup of a topological groupG , then GIH is a topological group.

1. ELEMENTARY PROPERTIES OF TOPOLOGICAL GROUPS3Proof Consider the commutative diagramGxGdG I9( G / H ) x (GIH) 5G/Hwhere y p X p, p(g,H2 g,H) g;lgzH, and q(gl, gz) g;lg,. It suffices to show that p is continuous. However, for W c GIH open, we havep-y W ) yy-'p-1( W ) yq-1p-1( W )which is open since p and q are continuous and y px p is open.I1.6. Proposition If G is a topological group and Go is the component ofG containing the identity e, then Go is a closed normal subgroup of G.Proof Since components are closed it suffices to show that Go is a subgroup and is normal. However, Go x Go is connected and hence G0G;'(its continuous image) is connected and contains e. Thus G0GG1 c Gowhich means that Go is a subgroup of G. If cc): G - G is any automorphismof G which is also a homeomorphism, then cc)(Go) is a component of G andcontains e, whence w(Go) Go. In particular, this holds for the innerautomorphisms L,R,, and hence Go is normal. IIf G and G' are topological groups, then by a homomorphism p: G G'we mean a homomorphism of groups which is also continuous. The readermay supply the easy details of the following result.1.7. Proposition If p: G H is a surjective homomorphism of topological groups, then K ker p is a closed normal subgroup of G and the induced map p': G/K H is continuous and bijective. r f G is compact, thenp' is open, and hence is an isomorphism of topological groups.IWe now turn to some special facts for compact groups.1.8. Proposition Let G be a compact group and let g{g" 1 n 0, I , 2, . . .}. Then A is a subgroup of G.EG. Let A

40. BACKGROUND ON TOPOLOGICAL GROUPS ANDLIE GROUPSProof Put B {gn I n E Z} which is a subgroup of G. ThenB is also asubgroup of G by 1.3. If e is isolated in B, then B is compact and discreteand hence finite and we must then have g" e for some n 0. Thus suppose that e is not isolated in B. Then for any symmetric neighborhoodU of e in G there is an integer n # 0 with gn E U. Since U is symmetricwe may suppose that n 0. Then gn-l E (g-'U) n A. Since the g-lUform a neighborhood basis at g-l it follows that g-1 E A and this clearlyimplies that A B. IIf H is a subgroup of G, then the normalizer N ( H ) of H in G is the subsetof G consisting of those elements g E G such that gHg-l H. This isclearly a subgroup of G and is closed when H is closed.1.9. Proposition I f G is a compact group and H c G is a closed subgroup, then gHg-l H [i.e., g E N ( H ) ] zj- gHg-l c H.Let v: G x G- G be q(g, k ) gkg-'. Suppose that g E G issuch that gHg-' c H and let A {g" I n 0, 1,2, . . . } as in 1.8. Thenv ( A x H ) c H and continuity o f 1 implies that p(A x H ) c H. By 1.8we have that g-1 E A and hence the latter inclusion implies that g-IHg c H.Thus H c gHg-'.Proof1.lo. Proposition If G is a compact group, then every neighborhood Uof e in G contains a neighborhood V of e which is invariant under conjugation.Proof Let 9: G x G - G be y(g, k ) gkg-I. If U is an open neighborhood of e, then G - U is compact and q ( G x (G - U ) ) is a compactset whose complement V c U is open, invariant, and contains e. IWe remark that 1.9 and 1.10 would be false in general without the compactness assumption (or some other assumption).1.11. Proposition If N is a totally disconnected normal subgroup of aconnected topological group G, then N is central in G.Proof For k E N the map yk: G - N , defined by yk(g) gkg-', is continuous. Since G is connected, y k ( G )is connected. Since N is totally disconnected, y k ( G ) must consist of exactly one point {k}. Thus gkg-' k forall g E G and k E N. I

2. THECLASSICAL GROUPS5The following fact is an elementary exercise that we leave to the reader.1.12. Proposition Let G be a topological group and H c G a closedsubgroup. If H and GIH are connected, then so is G. I2. THE CLASSICAL GROUPSLet K stand for either the field R of real numbers or the field C of complex numbers. For an n x n matrix A over K we denote by A‘ its transpose,by A its complex conjugate, and let A* A’. Let M J K ) denote the algebra of all n x n matrices over K and note that this is a K-vector spaceof dimension n2. The group, under multiplication, of nonsingular n x nmatrices over K is called the general linear group Gl(n, K ) and the subgroup of matrices of determinant 1 is called the special linear group Sl(n, K).The orthogonal group O(n, K ) consists of those A E Gl(n, K ) with AA’ Iand SO(n, K ) O(n, K ) n Sl(n, K). We put O(n) O(n, R) and SO(n) SO(n, R). The unitary group U(n) consists of those A E Gl(n, C ) withAA* I and SU(n) U(n) n Sl(n, C). Since A’, A*, and det A arecontinuous functions of A and multiplication K ( K ) x M&C)- M&C)is continuous, it is clear that these are all closed subgroups of Gl(n,K).Since Gl(n, K ) is defined by the inequality det A # 0, it is an open subsetof M#). The formula for A-l in terms of the coefficients of A showsthat A HA-l is continuous on Gl(n, K ) ; and in fact that it is analytic.(This also follows from the Implicit Function Theorem since multiplicationW ( K ) x m(K)-tm(K)is analytic.) It follows that all of the abovegroups are topological groups with the relative topology from M J K ) .Since A* is a continuous function of A the equation AA* I definingU(n) shows that U(n) is closed in W(C) Cna.It also shows that thecoefficients of A are bounded by one in absolute value, so that U(n) isbounded in m(C).Thus U(n) is compact. Since SU(n), O(n), and SO@)are closed subgroups of U(n), they are also compact groups.Let A be any n x n matrix over K. If the absolute values of the coefficients of A are bounded by c, then those of Ak are bounded by (nc)k as iseasily seen by induction on k. Thus the series of coefficients of the matrixseries-

60. BACKGROUNDON TOPOLOGICAL GROUPS AND LIE GROUPSare all absolutely dominated by the convergent series1 n c - ( -n ) ( n ) 2!3!Thus this series of matrices converges to a matrixeA Z AA2A3 - - 2!3!e-0and the convergence is uniform in any compact subset of &(K). Thusthe coefficients of eA are analytic functions of the coefficients of A . That is,the map exp: &(K) M J K ) , taking A to exp(A) eA, is analytic. Notethat e0 I. To compute the Jacobian of exp at 0 we use the usual matrixcoordinates x i j and note that, for X [xij],ex [xij] 21![x . . [d, 4-xij higher degree terms]. IThus the Jacobian is the n2 x n2 matrix with entries xaj higher degree terms)dXk IIx oif ( i , j ) ( k , I),otherwise,1 (0and hence is the identity n2 x n2 matrix. From the Implicit Function Theorem it follows that exp is one-one on some neighborhood of 0 E m(K)to some neighborhood of I E Gl(n, K ) and has an analytic inverse there.Note that B(Z AA2 2! .)B-l BeAB-l.Also exp over R is just the restriction of exp over C to M%(R)c Ik&(C).For any A E m(C)there is a B E Gl(n, C ) such that BAB-I has superdiagonal formBAB-'

2.where thethatTHE CLASSICAL GROUPS7Li are the eigenvalues of A . From this it is trivial to calculateand thus the eLi are the eigenvalues of eA with the correct multiplicities.In particular, their product shows thatdet eA etrace(A) [where trace(A) ll- . An]. In particular, det eA # 0 for all A ,and hence the image of exp is in Gl(n, K ) .2.1. LemmaIf Aand B commute, then eA B eAeB.For A and B fixed, let s and t be real variables and consider theanalytic functions esA tBand esAetBof s and t . We haveProofwhere the last product is to be regarded as a formal product of matricesof power series in s and t and the last equality means an equality of matricesof formal power series in s and t . However, the last term is just the powerseries expansion of eSAetB,as is seen by computation of the partial derivaand esAetBare analytic functions of s and ttives in s and t . Sinceit follows that they coincide for all s and t. 1It follows that for any given AK-Em(K)the mapGl(n, K )given byt HetA

0.8BACKGROUND ON TOPOLOGICAL GROUPS AND LIE GROUPSis a homomorphism of topological groups (with the additive group structureon K ) . In particular, e-A (d)".Since Gl(n, K) is open in the K-vector space I&(K), the tangent vectorto a curve in Gl(n, K), or in I&@), can be regarded as an element ofI&(K). In particular, for the curve t w e t A of R Gl(n, K) [for fixedA E w ( K ) ] we compute that its tangent vector at t 0 islimt O etA - I- lim ( Att O t -A22! - a * ) A.Thus the real tangent space at Z in Gl(n, K) coincides with the space of tangent vectors at t 0 to the real one-parameter groups t H etA.In the same way, for the operation of Gl(n, K ) on Kn,the tangent vectorto the curve t w etA v in Kn, where A E M J K ) and v E Kn are fixed, islimt O-etA - v - vt Av.t OIf etA v v for all t, then it follows that Av 0. Conversely, if Av 0,then etA v ( I tA)v v. Thus we have the following proposition.- --2.2. Proposition For v E Kn and A E W ( K ) , the tangent vector to thecurve t H eta v at t 0 is 0 zfletA v v for all t. I-2.3. Theorem For A E I&(C) we have the following facts:(1) Zf A( 2 ) Zf A( 3 ) Zf A(4) If Ais real, then eA is real.is skew symmetric (A' - A ) , then eA E O(n, C ) .is skew Hermitian (A* - A ) , then eA E U(n).has trace 0, then eA E Sl(n, C).Moreover,for A in a suitable neighborhood of 0 E w(C),the reverse implications all hold.For example, suppose that A' -A. Then (eA)' eA' e-A (d)-lso that eA E O(n, C). Conversely, if (&)' (eA)-l,then eA' e-Awhich implies that A' - A provided that A is sufficiently near 0, sinceexp is one-one near 0. The proof of the other implications are all similarto this and will be omitted. IProof

2. THE CLASSICAL GROUPS9Note the various combinations of the conditions in 2.3. It is also importantto note that the conditions of 2.3 are all real linear and define real vectorsubspaces of m(C)whose dimensions are easy t o calculate. Thus we havethe following easy consequence of 2.3.2.4. Corollary Let G c Gl(n, C)be one of the groups Gl(n, C), Gl(n,R),Sl(n, C ,Sl(n, R), U(n ,SU(n), O(n, C), SO(n, C), O(n, R), or SO(n, R).Then there is a real vector subspace T G of w(C) such that exp: w(C) . Gl(n, C)maps a neighborhood of 0 in T G homeomorphically onto a neighborhood of Z in G. The dimensions of these subspaces are, respectively,2n2, n2, 2n2 - 2, n2 - 1, n2, n2 - 1, n(n - I), n(n - l), &n(n- I), andBn(n-- 1).Let U c Gl(n, C) be a symmetric open neighborhood of Z on which theinverse, log: U m(C),of exp is defined and analytic. Let V be a symmetric open neighborhood of Z with Y 2c U and put W log(V). Thenthe compositionexp x exp9is real analytic, where p(gh) gh-I, and takesThis is just the expression of the multiplication map p in terms of the coordinates about Z in G given by log. Thus each of the above groups hascoordinates about Z in which multiplication and inversion are expressedas real analytic functions of the coordinates.We now turn to the discussion of one family of classical groups that wehave thus far omitted, the symplectic groups Sp(n). The group Sp(n) isdefined to be the group of all n x n quaternionic matrices A with AA* Z.Here A* A',where A is the quaternionic conjugate of A ; that is A [Zij] when A [aij] and where q abi cj dk has conjugate4 a - bi - cj - dk. For a quaternionic n x n matrix P we can write P A Bj,where A, B E m(C).Note that j B Bj and thus multiplication is givenby( A Bj)(C D j ) ( A C - BD) ( A D B c ) j.

10For P0.BACKGROUND ON TOPOLOGICAL GROUPS AND LIE GROUPS A Bj we defineA short calculation shows that pl is an isomorphism of the algebra of alln x n quaternionic matrices into that of 2n x 2n complex matrices, andthat pl(P*) p(P)*. Thus P is symplectic (PP* I ) iff I q(Z) pl(PP*) p(P)pl(P*) p(P)p(P)* which holds iff p(P) is unitary. It is easy tocalculate that a 2n x 2n complex matrix Q has the formiffJQJ-'where Q,o rJ [-I01.For Q unitary, this equation becomesQ'JQ J.Thus the symplectic group Sp(n) is isomorphic via pl to the subgroup ofU(2n) consisting of those unitary matrices preserving the bilinear formJ (i.e., Q'JQ J ) .As in 2.3 one sees that if A E M,,(C) and if A* - A and J AA'J 0,then eA E p(Sp(n)), and the converse holds for A sufficiently close to 0.Thus 2.4 also holds for G p(Sp(n)) e Sp(n) and the dimension of TGcan be computed to be 2n2 n.We conclude this section by defining the notion of a representation. IfG is a topological group, then a real representation of G is just a (continuous) homomorphism of G into Gl(n, R). This can be thought of as anaction of G on R" by linear transformations. Similarly, a complex representation of G is a homomorphism G Gl(n, C). Since Gl(n, R) c Gl(n, C)each real representation gives rise

In topology, one studies such objects as topological spaces, topological manifolds, differentiable manifolds, polyhedra, and so on. In the theory of transformation groups, one studies the symmetries of such objects, or generally subgroups of the full group of symmetries. Usually, the group of