WIXLIB

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningComputing expectationsIf you know analytically the probability distribution and cancompute the sums (integrals), no problemIf you know the distribution but cannot compute the sums(integrals), enter the magical realm of approximate inference(fun but out of scope)If you know nothing but have NS samples, then use a sampleapproximationApproximate the probability of an outcome with the frequencyin the samplehf (x)i 'NSX nx1 Xf (x) f (xi )NSNSxi 1(prove the last equality)Guido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningIndependenceTwo random variables x and y are independent if their jointprobability factorises in terms of marginalsp(x, y ) p(x)p(y )Using the product rule, this is equivalent to the conditionalbeing equal to the marginalp(x, y ) p(x)p(y ) p(x y ) p(x)Exercise: if two variables are independent, then theircorrelation is zero. NOT TRUE viceversa (no correlationdoes not imply independence)Guido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningContinuous statesIf the state space Ω is continuous some of the previousdefinitions must be modifiedThe general case is mathematically difficult; we restrictourselves to Ω Rn and to distributions which admit adensity, a functionZp : Ω R s.t. p(x) 0 x andp(x)dx 1ΩIt can be shown that the rules of probability distributions holdalso for probability densitiesNotice that p(x) is NOT the probability of the randomvariable being in state x (that is always zero for boundeddensities); probabilities are only defined as integrals oversubsets of ΩGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningEntropy and divergenceProbability theory is the basis of information theory(interesting, but not the topic of this course).An important quantity is the entropy of a distributionXH[p] pi log2 piiEntropy measures the level of disorder of a distribution; fordiscrete distributions, it is always 0 and 0 only fordeterministic distirbutionsThe relative entropy or Kullback-Leibler (KL) divergencebetween two distributions isXqiKL[qkp] qi logpiiFact: KL is convex and 0Guido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningBasic distributionsDiscrete distribution: a random variable can take N distinctvalues with probability pi , i 1, . . . , N . FormallyY δijp(x i) pjjδij is the Kronecker delta and the pi s are parameters.Poisson distribution: a distribution over non-negative integersµnp (n µ) exp[ µ]n!The parameter µ is often called the rate of the distribution.The Poisson distribution is often used for rare events, e.g.decaying of particles of binding of DNA fragments to a probe(more later!)Exercise: compute mean and variance of a Poisson distributionGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningBasic distributionsMultivariate normal: distribution over vectors x, density 11T 1pp (x µ, Σ) exp (x µ) Σ (x µ)22π Σ µ is the mean and Σ is the covariance matrix. Often useful toparametrise it in terms of the precision matrix Σ 1 . Howmany parameters does a multivariate normal have?Gamma distribution: distribution over positive real numbers,densityx k 1 exp( x/θ)p (x k, θ) θk Γ(θ)with shape parameter k and scale parameter θGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningInteresting exerciseThis exercise illustrates the pitfalls of working in high dimensions.1Curse of dimensionality: Suppose you want to exploreuniformly a region by gridding it. How many grid points doyou need?2Even worse: Suppose you sample from a spherical Gaussiandistribution. Where do the points lie as the dimensionsincrease?Guido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningMixtures: how to build more distributionsMore general distributions can be built via mixtures: e.g.X2p(x µ1.,n , σ1,.,n) πi N (µi , σi2 )iwhere the mixing coefficients πi are discretely distributedYou can interpret this as a two stage hierarchical process:choose one component out of a discrete distribution, thenchoose the distribution for that componentIMPORTANT CONCEPT: this is an example of latentvariable model, with a latent class variable and an observedcontinuous value. The mixture is the marginal distribution forthe observationsThe probability of the latent variables given the observationscan be obtained using Bayes’ theorem: see next weekGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningContinuous mixtures: some cool distributionsNo need for the mixing distribution (latent variable) to bediscreteSuppose you are interested in the means of normallydistributed samples (possibly with different variances/precisions)Marginalising the precision in a Gaussian using a Gammamixing distribution yields a Student t-distributionSuppose you have multiple rare event processes happeningwith slightly different ratesMarginalising the rate in a Poisson distribution using aGamma mixing distribution yields a negative binomialdistributionGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningParameters?Many distributions are written as conditional probabilitiesgiven the parametersOften the values of the parameters are not knownGiven independent and identically distributed (i.i.d.)observations, we can estimate them; e.g., we pick θ bymaximum likelihood"#Yθ̂ argmaxθp(xi θ)iAlternatively, you could have a prior over the parameters p(θ)and take the maximum a posteriori (MAP) estimate"#Yθ̂MAP argmaxθ p(θ)p(xi θ)iGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningJustification for maximum likelihoodGiven a data set {xi },distribution bei 1, . . . , N, let the empiricalN1 Xpemp (x) I(xi )Ni 1with I the indicator function of a setTo find a suitable distribution q to model the data, one maywish to minimize the Kullback-Leibler divergenceKL[pemp kq] H[pemp ] hlog q(x)ipemp 1 Xlog q(xi )NMaximum likelihood is equivalent to minimizing a KLdivergence with the empirical distirbutionGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningExercise: fitting a discrete distributionWe have independent observations x1 , . . . , xN each taking oneof D possible values, giving a likelihoodL NYp (xi p)i 1Compute the Maximum Likelihood estimate of p. What is theintuitive meaning of the result? What happens if one of the Dvalues is not represented in your sample?Guido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningExercise II: fitting a Gaussian distributionWe have independent, real valued observations x1 , . . . , xNFind the parameters of the optimal Gaussian fit by maximumlikelihoodGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningBayesian estimationThe Bayesian approach quantifies uncertainty at every stepThe parameters are treated as additional random variableswith their own prior distribution p(θ)The observation likelihood is combined with the prior toobtain a posterior distribution via Bayes’ theoremp(θ xI ) p(xI θ)p(θ)p(xI )where I is the set indexing the observationsThe distribution of the observable x (predictive distribution) isobtained asZp(x xI ) dθp(x θ)p(θ xI )Guido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningExercise: Bayesian fitting of GaussiansLet data xi i 1, . . . , N be distributed according to aGaussian with mean µ and variance σ 2Let the prior distribution over the mean µ be a Gaussian withmean m and variance v 2Compute the posterior and predictive distributionGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningEstimatorsA procedure to calculate an expectation is called an estimatore.g., fitting a Gaussian to data by maximum likelihoodprovides the M.L. estimator for mean and variance, orBayesian posterior meanAn estimator will be a noisy estimate of the true value, due tofinite sample effectsAn estimator fˆ is unbiased if its expectation (under the jointdistribution of the data set) coincides with the true valueExercise: show that the ML estimator of variance is biasedGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningLearning as fitting distributionsThe world-view of this course is that a model consists of a setof random variables and probabilistic relationships describingtheir interactionsLearning refers to computing conditional distributions w.r.t.some observations of subsets of the modelPredictions are then carried out using the Bayesian predictivedistributionGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningSupervised and unsupervised learningSlightly meaningless terms still heavily usedFocus on models involving more than one random variableIn particular, supervised learning applies when data is in theform of input-output pairsSupervised learning aims at learning the (probabilistic)functional relationship between the output and the inputUnsupervised learning refers to purely learn the structure ofthe probability distribution underlying the dataGuido SanguinettiBayesian Machine Learning - Lecture 1

Some factsPhilosophy and road mapBasics of probability theorySome probability distributionsFitting distributionsBasics of learningGenerative and discriminative modelsSupervised learning can have two flavoursTwo different types of question can be asked:what is the joint probability of input/ output pairs?given a new input, what will be the output?The first question requires a model of the population structureof the inputs, and of the conditional probability of the outputgiven the input generative modellingThe second question is more parsimonious but lessexplanatory discriminative learningNotice that the difference between generative supervisedlearning and unsupervised learning is mootGuido SanguinettiBayesian Machine Learning - Lecture 1

D. Barber, Bayesian Reasoning and Machine Learning, CUP 2010 Some slides also taken from the teaching material attached to the book (thanks David!) Other good books: C.M. Bishop, Pattern Recognition and Machine Learning (Springer 2006); K. Murphy, Machine Learning - a probabilistic perspective (MIT Press 2012).