Transcription
ECE 3800 Probabilistic Methods ofSignal and System AnalysisCourse Topics:1.2.3.4.5.ProbabilityRandom variablesMultiple random variablesRandom processesLinear systemsCourse Objectives:This course seeks to develop a mathematical understanding of basic statistical tools andprocesses as applied to inferential statistics and statistical signal processing.1. convert an English problem description into a precise mathematical probabilisticstatement2. use the general properties of random variables to solve a probabilistic problem3. be able to use a set of standard probability distribution functions suitable for engineeringapplications4. be able to calculate standard statistics from mass, distribution and density functions5. calculate confidence intervals for a population mean6. recognize and interpret a variety of deterministic and nondeterministic random processesthat occur in engineering7. calculate the autocorrelation and spectral density of an arbitrary random process8. understand stochastic phenomena such as white noise9. relate the correlation of and between input and output of autocorrelation and spectraldensity10. understand the mathematical characteristics of standard frequency isolation filters11. understand the signal-to-noise optimization principle as applied to filter design12. design Weiner and matched noise filtersNotes Based on:Probabilistic Methods of Signal and System Analysis (3rd ed.) by George R. Cooper and ClareD. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
Table of Contents1. Introduction to Probability1.1. Engineering Applications of Probability1.2. Random Experiments and Events1.3. Definitions of ProbabilityExperimentPossible OutcomesTrialsEventEqually Likely Events/OutcomesObjectsAttributeSample SpaceWith Replacement and Without Replacement1.4. The Relative-Frequency ApproachNANPr A lim r A r A Where Pr A 1.2.3.4.N is defined as the probability of event A.0 Pr A 1Pr A Pr B Pr C 1 , for mutually exclusive eventsAn impossible event, A, can be represented as Pr A 0 .A certain event, A, can be represented as Pr A 1 .1.5. Elementary Set TheorySetSubsetSpaceNull Set or Empty SetVenn DiagramEqualitySum or UnionProducts or IntersectionMutually Exclusive or Disjoint SetsComplementDifferencesProofs of Set Algebra1.6. The Axiomatic ApproachNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
1.7. Conditional ProbabilityPr A B Pr A B Pr B , for Pr B 0Pr A B Pr A B , for Pr B 0Pr B Joint ProbabilityPr A, B Pr A B Pr A when A follows BPr A, B Pr B, A Pr A B Pr B Pr B A Pr A Marginal ProbabilitiesTotal ProbabilityPr B Pr B A1 Pr A1 Pr B A2 Pr A2 Pr B An Pr An Bayes TheoremPr B Ai Pr Ai Pr Ai B Pr B A1 Pr A1 Pr B A2 Pr A2 Pr B An Pr An 1.8. IndependencePr A, B Pr B, A Pr A Pr B 1.9. Combined Experiments1.10. Bernoulli Trials n Pr A occuring k times in n trials p n k p k q n k k 1.11. Applications of Bernoulli TrialsNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
2. Random Variables2.1. Concept of a Random Variable2.2. Distribution FunctionsProbability Distribution Function (PDF) 0 FX x 1, for x FX 0 and FX 1 FX is non-decreasing as x increases Pr x1 X x2 FX x 2 FX x1 For discrete eventsFor continuous events2.3. Density FunctionsProbability Density Function (pdf) f X x 0, for x f x dx 1X xFX f u duX Pr x1 X x 2 x2 f x dxXx1Probability Mass Function (pmf) f X x 0, for x f X u 1u FX x x f X u u Pr x1 X x 2 x2 f X u u x1Functions of random variablesf Y y f X x dxdyNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
2.4. Mean Values and Moments1st, general, nth Moments X E X x fXor X E X x dxE g X g X Pr X x g X f X x dx or E g X x xX EXnnn x f X x dx or X E Xnnn Pr X x x Central Moments X X n X X n E X X E X X 2 2 X X 2 X X 2 x X nn x X nn f X x dx Pr X x x Variance and Standard Deviation x Pr X x x E X X E X X x X 2 f X x dx2 x X 22 Pr X x x 2.5. The Gaussian Random Variable where x X 2 , for x f X x exp 2 2 2 X is the mean and is the variance1x v X 2 dv exp FX x 22 2 v Unit Normal (Appendix D)1 x x u2 duexp 22 u 1 x 1 x x XFX x x X or FX x 1 Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
The Q-function is the complement of the normal function, : (Appendix E)Q x u2 duexp 2 2 u x1 2.6. Density Functions Related to GaussianRayleigh Distribution (noise contribution for complex comm. Symbols)For a two-dimensional problem (positions in x and y with two independent Gaussian randomvariable noise or offset terms), the distance from a desired point is described as a radial or vectormagnitude, the radial error or offset is described by the Rayleigh Distribution.R X 2 Y 2ForThe probability density function (pdf) is๐๐๐ ๐ ๐๐ฅ๐, ๐ ๐๐0 ๐๐2 ๐The probability distribution function (PDF) can be derived as:๐ฃ๐ฃ๐น ๐ ๐๐ฅ๐ ๐๐ฃ , ๐ ๐๐0 ๐๐2 ๐๐๐น ๐1 ๐๐ฅ๐, ๐ ๐๐0 ๐2 ๐2.7. Other Probability Density FunctionsExponential Distribution1 exp f T ,M M 0, FT 1 exp , M 0,T E T Mfor 0 for 0for 0 for 0 T 2 E T 2 2 M 2 2E T T2 T 2 E T 2 2 M 2 M 2 M 2 Binomial Distributionf B x FB x n n k p k 1 p n k x k k 0n n k p k 1 p n k u x k k 0Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
2.8. Conditional Probability Distribution and Density FunctionsPr A B Pr A B Pr B , for Pr B 0Pr A B Pr A B , for Pr B 0Pr B Pr A B Pr A, B Pr A B , for Pr B 0Pr B Pr B It can be shown that F x M is a valid probability distribution function with all theexpected characteristics: 0 F x M 1, for x F M 0 and F M 1 F x M is non-decreasing as x increases Pr x1 X x2 M F x2 M F x1 M 2.9. Examples and ApplicationsNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
3. Several Random Variables3.1. Two Random VariablesJoint Probability Distribution Function (PDF)F x, y Pr X x, Y y for x and y 0 F x, y 1, F , y F x, F , 0 F , 1 F x, y is non-decreasing as either x or y increases. F x, FX x and F , y FY y Joint Probability Density Function (pdf) 2 FX x f x, y x yfor x and y f x, y 0, f x, y dx dy 1 y F x, y x f u, v du dv f X x f x, y dyand f Y y f x, y dx Pr x1 X x2 , y1 Y y 2 y 2 x2 f x, y dx dyy1 x1Expected ValuesE g X , Y g x, y f x, y dx dy CorrelationE X Y x y f x, y dx dy Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
3.2. Conditional Probability--RevisitedPr X x M F x, y FY y Pr M F x, y 2 F x, y1 FX x y1 Y y 2 FY y 2 FY y1 f x, y FX x Y y fY y f x, y FY y X x f X x f y x f X x f x y fY y f x, y f x Y y f Y y f y X x f X x f y X x f X x f x Y y fY y f x Y y fY y f y X x f X x FX x Y y 3.3. Statistical Independencef x, y f X x f Y y E X Y E X E Y X Y3.4. Correlation between Random Variables E X Y x y f x, y dx dy CovarianceE X E X Y E Y x X y Y f x, y dx dy Correlation coefficient or normalized covariance, X X E X Y Y Y x X X y Y Y f x, y dx dy E x y X Y X YNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
3.5. Density Function of the Sum of Two Random Variables (statistically independent)Z X Yyx f u, v du dvF x, y FZ z fY y f Z z z y f X x dx dy f X x fY z x dx fY y f X z y dy3.6. Probability Density Function of a Function of Two Random Variables x xg z , w f 1 z, w , 2 z , w z w f 1 z , w , 2 z , w J y y z w3.7. The Characteristic Function u E exp j u X u f x exp j u x dx The inverse of the characteristic function is then defined as:1f x 2 u exp j u x du Computing other moments is performed similarly, where:d n u dudu u 0 exp j u x dx j x n f x j x n d n u nn f x dx j n xn f x dx j n E X n Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
4. Elements of Statistics4.1. Introduction4.2. Sampling Theory--The Sample Mean1Xห nSample Meann Xi ,where X i are random variables with a pdf.i 1Variance of the sample mean 2 2 1 n X 2 X 2 XVar Xห X 2 nn2 n n 2 N n Var Xห n N 1 4.3. Sampling Theory--The Sample Variance X Xห n 1E S nN n 1 E S N 1 nS2 1nn2ii 1222unbiased 2 nN E S E S N 1 n 1 nn 1 ES2 E S2 n 1n 1 n ni 1 n221หXi X X i Xหn 1i 1224.4. Sampling Distributions and Confidence IntervalsGaussianXห XZ nStudentโs t distributionXห XXห XT SSn 1nk k X Xห X nnNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
4.5. Hypothesis TestingOne tail or two-tail testingRelates to communication bit/symbol value detection/estimation!4.6. Curve Fitting and Linear RegressionYห R XX Xห R XYYห R XX Xห R XYa 2C XXR XX XหR XY Yห XหCb XY2C XXR XX Xห 4.7. Correlation between Two Sets of Data xi xi 2E X 2 R XX 1 X 2 C XX n1 ni 1ni 12n 2 1xi xi R XX X 2 ni 1 i 1 n 1R XY E X Y n n1 X E X n C XY E X X Y Y n xi y i1 ni 1n xi y i X Yi 1C XY XY X YNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
5. Random Processes5.1. IntroductionEnsembleFor example, assume that there is a known AM signal transmitted:s t 1 b A t sin w t at an undetermined distance the signal is received asy t 1 b A t sin w t ,0 2 The received signal is mixed and low pass filtered x t h t y t cos w t h t 1 b A t sin w t cos w t ,0 2 x t h t y t cos w t h t 1 b A t 0.5 sin 2 w t sin ,0 2 If the filter removes the 2wt term, we have 1 b A t sin ,0 2 x t h t y t cos w t 2Notice that based on the value of the random variable, the output can change significantly! Fromproducing no output signal, ( 0, ), to having the output be positive or negative ( 0 to or to 2 ). P.S. This is not how you perform non-coherent AM demodulation.To perform coherent AM demodulation, all I need to do is measured the value of the randomvariable and use it to ensure that the output is a maximum (i.e. mix withcos w t m , where m t1 .5.2. Continuous and Discrete Random Processes5.3. Deterministic and Nondeterministic Random Processes5.4. Stationary and Nonstationary Random ProcessesThe requirement that all marginal and joint density functions be independent of the choice oftime origin is frequently more stringent (tighter) than is necessary for system analysis. A morerelaxed requirement is called stationary in the wide sense: where the mean value of any randomvariable is independent of the choice of time, t, and that the correlation of two random variablesdepends only upon the time difference between them. That isE X t X X andE X t1 X t 2 E X 0 X t 2 t1 X 0 X R XX for t 2 t1You will typically deal with Wide-Sense Stationary Signals.Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
5.5. Ergodic and Nonergodic Random ProcessesA Process for Determining Stationarity and Ergodicitya) Find the mean and the 2nd moment based on the probabilityb) Find the time sample mean and time sample 2nd moment based on timeaveraging.c) If the means or 2nd moments are functions of time non-stationaryd) If the time average mean and moments are not equal to the probabilistic meanand moments or if it is not stationary, then it is non ergodic.For ergodic processes, all the statistics can be determined from a single function of the process.This may also be stated based on the time averages. For an ergodic process, the time averages(expected values) equal the ensemble averages (expected values). That is to say, Xn 1x n f x dx limT 2T T Xn t dt TNote that ergodicity cannot exist unless the process is stationary!5.6. Measurement of Process Parameters5.7. Smoothing Data with a Moving Window AverageA Process for Determining Stationarity and Ergodicitya) Find the mean and the 2nd moment based on the probabilityb) Find the time sample mean and time sample 2nd moment based on time averaging.c) If the means or 2nd moments are functions of time non-stationaryd) If the time average mean and moments are not equal to the probabilistic mean and momentsor if it is not stationary, then it is non ergodic.Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
6. Correlation Functions6.1. Introduction6.2. Example: Autocorrelation Function of a Binary ProcessR XX t1 , t 2 E X 1 X 2 dx1 dx2 x1x2 f x1, x2 The above function is valid for all processes, stationary and non-stationary.For WSS processes:R XX t1 , t 2 E X t X t R XX If the process is ergodic, the time average is equivalent to the probabilistic expectation, or1 XX limT 2TandT x t x t dt x t x t T XX R XX 6.3. Properties of Autocorrelation Functions 1)R XX 0 E X 2 X 2 or XX 0 x t 2)R XX R XX 2R XX R XX 0 3)4)If X has a DC component, then Rxx has a constant factor.5)If X has a periodic component, then Rxx has a will also have a periodic component of thesame period.6)If X is ergodic and zero mean and has no periodic component, thenlim R XX 0 7)Autocorrelation functions cannot have an arbitrary shape. One way of specifying shapespermissible is in terms of the Fourier transform of the autocorrelation function. That is, if R XX R XX exp jwt dt then the restriction states that R XX 0for all w6.4. Measurement of Autocorrelation Functions6.5. Examples of Autocorrelation FunctionsNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
6.6. Crosscorrelation FunctionsThe cross-correlation is defined as:R XY t1 , t 2 E X 1Y2 RYX t1 , t 2 E Y1 X 2 For jointly WSS processes:and dx1 dy2 x1 y2 f x1, y2 dy1 dx2 y1x2 f y1, x2 R XY t1 , t 2 E X t Y t R XY RYX t1 , t 2 E Y t X t RYX XY R XY YX RYX 6.7. Properties of Cross-correlation Functions1)The properties of the zoreth lag have no particular significance and do not representmean-square values. It is true that the ordered crosscorrelations are equal at 0. .R XY 0 RYX 0 or XY 0 YX 0 2)Crosscorrelation functions are not generally even functions. There is an anti-symmetry tothe ordered crosscorrelations:R XY RYX 3)The crosscorrelation does not necessarily have its maximum at the zeroth lag. This makessense if you are correlating a signal with a timed delayed version of itself. The crosscorrelationshould be a maximum when the lag equals the time delay!4)If X and Y are statistically impendent, then the ordering is not importantR XY E X t Y t E X t E Y t X YandR XY X Y RYX 5)If X is a stationary random process and id differentiable with respect to time, thecrosscorrelation of the signal and itโs derivative is given bydR XX R XX d 6.8. Examples and Applications of Crosscorrelation Functions6.9. Correlation Matrices for Sampled FunctionsNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
7. Spectral Density7.1. IntroductionTherefore, we can define a power spectral density for the ensemble as:S XX w R XX R XX exp iw d S XX w R XX R XX 1 S XX w 1R XX t 2 S XX w exp iwt dw 7.2. Relation of Spectral Density to the Fourier Transform XX X w X w X w 27.3. Properties of Spectral DensityThe power spectral density as a function is always real, positive, and an even function in w.7.4. Spectral Density and the Complex Frequency Plane7.5. Mean-Square Values From Spectral DensityThe mean squared value of a random process is equal to the 0th lag of the autocorrelation EX21 R XX 0 2 E X 2 R XX 0 1S XX w exp iw 0 dw 2 S XX w dw S XX f exp i2 f 0 dw S XX f df Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
As a note, since the PSD is real and symmetric, the integral can be performed as EX21 R XX 0 2 2 S XX w dw0 E X 2 R XX 0 2 S XX f df07.6. Relation of Spectral Density to the Autocorrelation FunctionThe Fourier Transform in f S XX f R XX exp i2 f d R XX t S XX f exp i2 ft df 7.7. White NoiseAs a result, we define โWhite Noiseโ asR XX S 0 t NS XX w S 0 027.8. Cross-Spectral DensityThe Fourier Transform in wS XY w R XY exp iw d 1R XY t 2 and SYX w RYX exp iw d 1S XY w exp iwt dw and RYX t 2 Properties of the functions SYX w exp iwt dw S XY w conj SYX w Since the cross-correlation is real, the real portion of the spectrum is even the imaginary portion of the spectrum is odd7.9. Autocorrelation Function Estimate of Spectral Density7.10. Periodogram Estimate of Spectral Density7.11. Examples and Applications of Spectral DensityNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
8. Response of Linear Systems to Random Inputs8.1. Introduction8.2. Analysis in the Time Domain8.3. Mean and Mean-Square Value of System Output8,4. Autocorrelation Function of System Output8.5. Crosscorrelation between Input and Output8.6. Example of Time-Domain System Analysis8.7. Analysis in the Frequency Domain8.8. Spectral Density at the System Output8.9. Cross-Spectral Densities between Input and Output8.10. Examples of Frequency-Domain Analysis8.11. Numerical Computation of System Output9. Optimum Linear Systems9.1. Introduction9.2. Criteria of Optimality9.3. Restrictions on the Optimum System9.4. Optimization by Parameter Adjustment9.5. Systems That Maximize Signal-to-Noise Ratio9.6. Systems That Minimize Mean-Square ErrorNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
AppendicesA. Mathematical TablesA.1. Trigonometric IdentitiesA.2. Indefinite IntegralsA.3. Definite IntegralsA.4. Fourier Transform OperationsA.5. Fourier TransformsA.6. One-Sided Laplace TransformsB. Frequently Encountered Probability DistributionsB.1. Discrete Probability FunctionsB.2. Continuous DistributionsC. Binomial CoefficientsD. Normal Probability Distribution FunctionE. The Q-FunctionF. Student's t Distribution FunctionG. Computer ComputationsH. Table of Correlation Function--Spectral Density PairsI. Contour IntegrationNotes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and SystemAnalysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9.
Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: -19-512354-9. ECE 3800 Probabilistic Methods of Signal and System Analysis Course Topics: 1. Probability 2. Random variables 3.
