WIXLIB

xiiCONTENTS6.4 Hypothesis testing and confidence intervals6.5 Asymptotic testsExercises2462502557 Prediction7.1 Bayesian prediction7.2 Classical prediction7.3 Prediction in the Normal model7.3.1 Bayesian approach7.3.2 Classical approach7.4 Linear predictionExercises2632642682712712732742778 Introduction to linear models8.1 The linear model8.2 Classical estimation of linear models8.3 Bayesian estimation of linear models8.4 Hierarchical linear models8.5 Dynamic linear models8.5.1 Definition and examples8.5.2 Evolution and updating equations8.5.3 Special aspects8.5.4 Hyperparameter estimation8.6 Linear models with 309314Sketched solutions to selected exercises319List of distributions331References337Index343

Preface to the Second EditionIt has been more than a decade since the 1st edition was released. Over thisperiod the need for a more balanced account of the main schools of thoughtto statistical inference became clear. The main purpose of the book has always been a balanced and integrated presentation of these approaches to thesubject. But the first edition was written only by the first two authors of thisedition; both were assumedly Bayesian and mentioned that in the Preface forthat edition.Experience gathered in the meantime indicated to us some ingredients toget back to our original goal. Certainly a more detailed and comprehensivepresentation of the pros and cons of each approach had to be pursued. Theinclusion of a co-author who was more involved with the frequentist schoolwas also recommended to achieve a more balanced presentation. We werefortunate to add both ingredients and present the current version of our workafter these additions.We have tried to eliminate (or reduce, more realistically) the typos presentin the 1st edition. Additionally, we have included more explanation, more illustration and more exercises for the concepts already present here. Moreimportantly, new material has been included. The main novelties are associated with a new section on empirical Bayes and penalized likelihoods and theirimpact on the regression models in the last chapter. Interestingly, these aretopics that lie closer to the border between the two schools of thought. Thecontent of this new material has also been connected to many other parts ofthe book, throughout its chapters. Additionally, we have expanded material inmany sections, specially on hypothesis testing, method of moments and biascorrection.This has been an interesting journey for the authors that is far from over.Statistics is alive and growing as a discipline with an ever increasing impactof the growth in computational power. We have no illusion of having reacheda complete and final text on the subject, but hope to have reached a stagewhere readers can grasp the main task ahead of them in their search for anintegrated understanding of Statistical Inference.xiii

xivPREFACE TO THE SECOND EDITIONThere are a number of people that we would like to thank and show ourgratitude. We all thank our common friend Luis Raul Pericchi for the suggestion of the 3rd author name and many other suggestions. Thanks are alsodue to our lifelong friend Basilio Pereira for much and relentless advice withreferences and perspective. There is no way to avoid acknowledging the fundamental help of Rob Calver. He has been keen on making this project viablesince the release of the 1st edition. There were many many moments that hewas the only one believing it. His persistence and enthusiasm was crucial tohave us reached this stage. He is complemented by a number of competentcolleagues at CRC who have helped us in many different ways. We also thankcolleagues and students who have helped shape the work in various ways, andour families for their encouragement and support. To all of them, our sincerest gratitude. We hold none of them responsible and take full account for theviews expressed in this book.H.S.M., D.G. and F.L.Rio de Janeiro and São Carlos, May 2014

Preface to the First EditionThis book originated from the lecture notes of a course in Statistical Inferencetaught at the M.Sc. programs in Statistics at UFRJ and IMPA (once). Thesehave been used since 1987. During this period, various modifications have beenintroduced until we arrived at this version, judged as minimally presentable.The motivation to prepare this book came from two different sources.The first and more obvious one for us was the lack of texts in Portuguese,dealing with statistical inference to the desired depth. This motivation led usto prepare the first draft of this book in the Portuguese language in 1993.The second, and perhaps the most attractive as a personal challenge, was theperspective adopted in this text. Although there are various good books in theliterature dealing with this subject, in none of them could we find an integratedpresentation of the two main schools of statistical thought: the frequentist (orclassical) and the Bayesian. This second motivation led to the preparationof this English version. This version has substantial changes with respect tothe Portuguese version of 1993. The most notable one is the inclusion of awhole new chapter dealing with approximation and computationally intensivemethods.Generally, statistical books follow their author’s point of view, presenting at most, and in separate sections, related results from the alternativeapproaches. In this book, our proposal was to show, wherever possible, theparallels existing between the results given by both methodologies. Comparative Statistical Inference by V. D. Barnett (1973) is the book that is closestto this proposal. It does not, however, present many of the basic inference results that should be included in a text proposing a wide study of the subject.Also we wanted to be as comprehensive as possible for our aim of writing atextbook in statistical inference.This book is organized as follows. The first chapter is an introduction, presenting the readers with the way we find most appropriate to think of Statistics: discussing the concept of information. Chapter 2 presents some basicconcepts of statistics such as sufficiency, exponential family, Fisher information, permutability and likelihood functions. Another basic concept specificxv

xviPREFACE TO THE FIRST EDITIONto Bayesian inference is prior distribution; that is separately dealt with inChapter 3.Certain aspects of inference are individually presented in Chapters 4, 6, and7. Chapter 4 deals with parameter estimation where, intentionally, point andinterval estimation are presented as responses to the summarization question,and not as two unrelated procedures. The important results for the Normaldistribution are presented and also serve an illustrative purpose. Chapter 6 isabout hypotheses testing problems under the frequentist approach and alsounder the various possible forms of the Bayesian paradigm.In between them, lies Chapter 5 where all approximation and computationally based results are gathered. The reader will find there at least a shortdescription of the main tools used to approximately solve the relevant statistical problem for situations where an explicit analytic solution is not available.For this reason, asymptotic theory is also included in this chapter.Chapter 7 covers prediction from both the frequentist and Bayesian pointsof view, and includes the linear Bayes method. Finally in Chapter 8, an introduction to Normal linear models is made. Initially the frequentist approachis presented, followed by the Bayesian one. Based upon the latter approach,generalizations are presented leading to the hierarchical and dynamic models.We would like to alert the readers from the onset that the our preferredpoint of view is the Bayesian one. So, a natural emphasis is given to thisapproach. We tried, however, to develop a critical analysis and to present themost important results of both approaches commenting on the positive andnegative aspects of both. As it has already been said, the level of this bookis adequate for an M.Sc. course in Statistics, although we do not rule out thepossibility of its use in an advanced undergraduate course aiming to comparethe two approaches.This book can also be useful for the more mathematically trained professionals from related areas of Science such as Economics, Mathematics, Engineering, Operations Research and Epidemiology. The basic requirements areknowledge of calculus and probability, although basic notions of linear algebraare also used. As this book is intended as a basic text in statistical inference,various exercises are included at the end of each chapter. We have also included sketched solutions to some of the exercises and a list of distributionsat the end of the book, for easy reference.There are many possible uses of this book as a textbook. The first andmost obvious one is to present all the material in the order it appears inthe book and without skipping sections. This may be a heavy workload for aone-semester course. In this case we suggest dropping Chapter 8 for a later

PREFACE TO THE FIRST EDITIONxviicourse. A second option for exclusion in a first course is Chapter 5, althoughwe strongly recommend it for anybody interested in the modern approach tostatistics, geared towards applications. The book can also be used as a text fora course that is more strongly oriented towards one of the schools of thought.For a Bayesian route, follow Chapters 1, 2, 3, Sections 4.1, 4.4.1 and 4.5,Chapter 5, Sections 6.3, 6.4, 6.5, 7.1, 7.3.1, 7.4, 8.1, 8.3, 8.4 and 8.5. For aclassical route, follow Chapter 1, Sections 2.1, 2.2, 2.5, 2.6, 4.2, 4.3, 4.4.2 and4.5, Chapter 5, Sections 6.1, 6.2, 6.4, 6.5, 7.2, 7.3.2, 7.4, 8.1 and 8.2.This book would not have been possible without the cooperation of variouspeople. An initial and very important impulse was the typing of the originallecture notes in TeX by Ricardo Sandes Ehlers. Further help was providedby Ana Beatriz Soares Monteiro, Carolina Gomes, Eliane Amiune Camargo,Monica Magnanini and Otávio Santos Figueiredo. Besides these, many of ourformer students helped with suggestions and criticism. Careful proofreading ofthis manuscript was made by our past M.Sc. students and present colleaguesAlexandra Mello Schmidt, Hedibert Freitas Lopes and Marco Antonio RosaFerreira. Many useful suggestions and comments were provided at this laterstage by Steve Brooks, Eduardo Gutierrez-Peña and Gabriel Huerta. We alsohad the stimulus of several colleagues; in particular, we would like to mentionBası́lio de B. Pereira. I would also like to thank Nicki Dennis for her supportand encouragement throughout all the stages of preparation of this book andfor making us feel at home with Arnold. Our families also played the importantroles of support and understanding, especially in the weekends and late nightsspent trying to meet deadlines! To all of them, our gratitude.Finally, the subject of the book is not new and we are not claiming anyoriginality here. We would like to think that we are presenting the subject ina way that is not favored in many textbooks and that will help the readersto have an integrated view of the subject. In our path to achieve this goal,we have had the influence of many researchers and books. We have triedto acknowledge this influence by referring to these books whenever we feltit provided a worthwhile reading description of a topic. Therefore, for everymajor subject presented in our book we tried to relate it to books that treatedthe subject in a more complete or more interesting way. In line with a textbookcharacter, we opted to favor books rather than research papers as references.We would like to think of our book as a basis for discovery and will feelour task is acomplished whenever readers understand the subject through thebook alone, its references or a combination of both.H.S.M. & D.G.Rio de Janeiro, December 1998

This page intentionally left blank

Chapter 1IntroductionBefore beginning the study of Statistics, it is relevant to characterize the scopeof the area and the main issues involved in this study. We avoid directly defining the subject, which is a hard and polemical task. Some of the componentsinvolved in this area of science will be presented in the hope that at the endthe reader will have a clear notion of the broadness of the subject under consideration. The fundamental problem towards which the study of Statistics isaddressed is that where randomness is present. The statistical methodology todeal with the resulting uncertainty is based on the elaboration of probabilisticmodels in order to summarize the relevant information available.There are many concepts used in the last sentence that deserve a clarifiedexplanation of their meaning in order to ensure a unified understanding. Amodel is a formal collection of coherent rules describing, in a simplified way,some real world problem. The language used to describe precisely a model inwhich uncertainty is present is probability. The meaning of summarization, inthis context, refers to the ability to describe a problem in the most conciseway (under the assumption that there are many possible forms to describe aproblem). The art involved in model building is the desire to balance the needto include as many aspects of reality as possible while keeping it not very complex. A related concept, which is useful to have in mind, is that of parsimony.This means that the model must have an appropriate level of complexity. Avery simple model can be misleading since it is eventually missing relevantaspects of the reality. On the other hand, if the model is highly complex itwill be hard to understand and extract meaningful information from it. Fromthe previous discussion it is not difficult to guess that information is the maininput for Statistics. However, this is a hard concept to define.Among the objectives of our study, we can identify the two main onesas being to understand reality (estimation and hypothesis testing) and tomake a decision (prediction). A strong limitation of many presentations of1

2INTRODUCTIONstatistical inference is being mainly concentrated in estimation and testing. Inthis context, it only deals with quantities that can never be observed in thepresent or in the future. Nevertheless, our view of Statistics is that it mustbe concerned with observable quantities. In this way, one is able to verify themodel adequacy in an irrefutable way. For the sake of completeness, however,we will present the main results available from all of the topics above.1.1InformationAs we have already said, the notion of information is present in all the studiesdeveloped in Statistics. As far as uncertainty is one of the main ingredientsin our models, we need to gather as much information as possible in order toreduce our initial uncertainty. A fundamental question which we are concernedwith is about the type of information that is relevant and must be retained inthe analysis. A possible reply to this question is that all available informationis useful and must be taken in consideration. Another answer is to avoidarbitrariness and take in consideration only objective observation coming froma sampling process. For this all the subjective information must be discarded.These two points of view roughly form the bases for two different forms ofstatistical analysis: the Bayesian (or subjectivist) and the classical (or frequentist) approaches, respectively. As we will see in the next section the divergenceamong these two approaches is much stronger, beginning with the interpretation of the concept of probability. This is always the starting point of astatistical model.An example to illustrate and clarify these points follows.Example 1.1 Consider the situation described in Berger (1985) concerningthe following experiments:1. A fine musician, specialized in classical plays, tell us that he is able todistinguish if Hayden or Mozart composed some classical song. Smallexcerpts of the compositions of both authors are selected at randomand the experiment consists of playing them for identification by themusician. The musician makes 10 correct guesses in exactly 10 trials.2. A drunk man says that he can correctly guess in a coin toss whatface of the coin will fall down. Again, after 10 trials the drunk mancorrectly guesses the outcomes of the 10 throws.3. An old English lady is well known for her ability to distinguish whethera cup of tea is prepared by first pouring the milk or the tea. Thencups filled with tea and milk, well mixed and in a random order, arepresented to her. She correctly identifies all ten cups.

THE CONCEPT OF PROBABILITY3It is not difficult to see that the three experiments provide the same information, and therefore, any test to verify the authenticity of the person’sstatement would result positive for all of them, with the same confidence.This does not make any sense! We have more reason to believe in the authenticity of the statement of the musician than of the old lady and, certainly,much more than of the drunk man. There is no doubt that the experimentaloutcome increases the veracity of the statements made. But we cannot reasonably say that we have the same confidence in the three assertions. By commonsense, there is a long way to go before one accepts this conclusion.1.2The concept of probabilityAlthough the definition of probability is well accepted by almost every statistician (keeping away some technical details), its interpretation or the senseattributed to it varies considerably. We mention here some of the more common interpretations: physical (or classical), frequentist and subjective.Physical or classicalThe probability of any event is the ratio between the number of favorableoutcomes and the total number of possible experimental results. It is implicitlyassumed that all the elem

Elementary Applications of Probability eory, Second Edition H.C. Tuckwell Introduction to Statistical Inference and Its Applications with R M.W. Trosset Understanding Advanced Statistical Methods P.H. Westfall and K.S.S. Henning Statistical Process Control: eory and Practice, ird Edition G.B. Wetherill and D.W. Brown Generalized Additive Models: