Part I Fundamentals of Probability
1 Probability Models ........................................... 3
1.1 Random Experiments ...................................... 3
1.2 Sample Space ............................................ 5
1.3 Events .................................................. 6
1.4 Probability ............................................. 9
1.5 Conditional Probability and Independence ............... 12
1.5.1 Product Rule .................................... 14
1.5.2 Law of Total Probability and Bayes' Rule ........ 16
1.5.3 Independence .................................... 17
1.6 Problems ............................................... 19
2 Random Variables and Probability Distributions .............. 23
2.1 Random Variables ....................................... 23
2.2 Probability Distribution ............................... 25
2.2.1 Discrete Distributions .......................... 27
2.2.2 Continuous Distributions ........................ 28
2.3 Expectation ............................................ 29
2.4 Transforms ............................................. 33
2.5 Common Discrete Distributions .......................... 36
2.5.1 Bernoulli Distribution .......................... 36
2.5.2 Binomial Distribution ........................... 37
2.5.3 Geometric Distribution .......................... 38
2.5.4 Poisson Distribution ............................ 40
2.6 Common Continuous Distributions ........................ 42
2.6.1 Uniform Distribution ............................ 42
2.6.2 Exponential Distribution ........................ 43
2.6.3 Normal (Gaussian) Distribution .................. 45
2.6.4 Gamma and χ2 Distribution ....................... 48
2.6.5 F Distribution .................................. 49
2.6.6 Student's t Distribution ........................ 50
2.7 Generating Random Variables ............................ 51
2.7.1 Generating Uniform Random Variables ............. 52
2.7.2 Inverse-Transform Method ........................ 53
2.7.3 Acceptance-Rejection Method ..................... 55
2.8 Problems ............................................... 57
3 Joint Distributions ......................................... 63
3.1 Discrete Joint Distributions ........................... 64
3.1.1 Multinomial Distribution ........................ 68
3.2 Continuous Joint Distributions ......................... 69
3.3 Mixed Joint Distributions .............................. 73
3.4 Expectations for Joint Distributions ................... 74
3.5 Functions of Random Variables .......................... 78
3.5.1 Linear Transformations .......................... 79
3.5.2 General Transformations ......................... 81
3.6 Multivariate Normal Distribution ....................... 82
3.7 Limit Theorems ......................................... 89
3.8 Problems ............................................... 93
Part II Statistical Modeling and Classical and Bayesian
Inference
4 Common Statistical Models .................................. 101
4.1 Independent Sampling from a Fixed Distribution ........ 101
4.2 Multiple Independent Samples .......................... 103
4.3 Regression Models ..................................... 104
4.3.1 Simple Linear Regression ....................... 105
4.3.2 Multiple Linear Regression ..................... 106
4.3.3 Regression in General .......................... 108
4.4 Analysis of Variance Models ........................... 111
4.4.1 Single-Factor ANOVA ............................ 111
4.4.2 Two-Factor ANOVA ............................... 113
4.5 Normal Linear Model ................................... 114
4.6 Problems .............................................. 118
5 Statistical Inference ...................................... 121
5.1 Estimation ............................................ 122
5.1.1 Method of Moments .............................. 123
5.1.2 Least-Squares Estimation ....................... 125
5.2 Confidence Intervals .................................. 128
5.2.1 lid Data: Approximate Confidence Interval
for μ .......................................... 130
5.2.2 Normal Data: Confidence Intervals for μ and
σ2 ............................................. 131
5.2.3 Two Normal Samples: Confidence Intervals
for μx — μy and σ2X/σ2Y ........................ 133
5.2.4 Binomial Data: Approximate Confidence
Intervals for Proportions ...................... 135
5.2.5 Confidence Intervals for the Normal Linear
Model .......................................... 137
5.3 Hypothesis Testing .................................... 140
5.3.1 ANOVA for the Normal Linear Model .............. 142
5.4 Cross-Validation ...................................... 146
5.5 Sufficiency and Exponential Families .................. 150
5.6 Problems .............................................. 154
6 Likelihood ................................................. 161
6.1 Log-Likelihood and Score Functions .................... 165
6.2 Fisher Information and Cramer-Rao Inequality .......... 167
6.3 Likelihood Methods for Estimation ..................... 172
6.3.1 Score Intervals ................................ 174
6.3.2 Properties of the ML Estimator ................. 175
6.4 Likelihood Methods in Statistical Tests ............... 178
6.5 Newton-Raphson Method ................................. 180
6.6 Expectation-Maximization (EM) Algorithm ............... 182
6.7 Problems .............................................. 188
7 Monte Carlo Sampling ....................................... 195
7.1 Empirical Cdf ......................................... 196
7.2 Density Estimation .................................... 201
7.3 Resampling and the Bootstrap Method ................... 203
7.4 Markov Chain Monte Carlo .............................. 209
7.5 Metropolis-Hastings Algorithm ......................... 214
7.6 Gibbs Sampler ......................................... 218
7.7 Problems .............................................. 220
8 Bayesian Inference ......................................... 227
8.1 Hierarchical Bayesian Models .......................... 229
8.2 Common Bayesian Models ................................ 233
8.2.1 Normal Model with Unknown μ and σ2 ............. 233
8.2.2 Bayesian Normal Linear Model ................... 237
8.2.3 Bayesian Multinomial Model ..................... 240
8.3 Bayesian Networks ..................................... 244
8.4 Asymptotic Normality of the Posterior Distribution .... 248
8.5 Priors and Conjugacy .................................. 249
8.6 Bayesian Model Comparison ............................. 251
8.7 Problems .............................................. 256
Part III Advanced Models and Inference
9 Generalized Linear Models .................................. 265
9.1 Generalized Linear Models ............................. 265
9.2 Logit and Probit Models ............................... 267
9.2.1 Logit Model .................................... 267
9.2.2 Probit Model ................................... 273
9.2.3 Latent Variable Representation ................. 278
9.3 Poisson Regression .................................... 282
9.4 Problems .............................................. 284
10 Dependent Data Models ...................................... 287
10.1 Autoregressive and Moving Average Models .............. 287
10.1.1 Autoregressive Models .......................... 287
10.1.2 Moving Average Models .......................... 297
10.1.3 Autoregressive-Moving Average Models ........... 303
10.2 Gaussian Models ....................................... 305
10.2.1 Gaussian Graphical Model ....................... 306
10.2.2 Random Effects ................................. 308
10.2.3 Gaussian Linear Mixed Models ................... 315
10.3 Problems .............................................. 320
11 State Space Models ......................................... 323
11.1 Unobserved Components Model ........................... 325
11.1.1 Classical Estimation ........................... 327
11.1.2 Bayesian Estimation ............................ 332
11.2 Time-Varying Parameter Model .......................... 333
11.2.1 Bayesian Estimation ............................ 334
11.3 Stochastic Volatility Model ........................... 339
11.3.1 Auxiliary Mixture Sampling Approach ............ 340
11.4 Problems .............................................. 346
A Matlab Primer .............................................. 349
A.1 Matrices and Matrix Operations ........................ 349
A.2 Some Useful Built-in Functions ........................ 352
A.3 Flow Control .......................................... 354
A.4 Function Handles and Function Files ................... 355
A.5 Graphics .............................................. 356
A.6 Optimization Routines ................................. 360
A.7 Handling Sparse Matrices .............................. 362
A.8 Gamma and Dirichlet Generator ......................... 364
A.9 Cdfs and Inverse Cdfs ................................. 365
A.10 Further Reading and References ........................ 366
В. Mathematical Supplement .................................... 367
B.l Multivariate Differentiation .......................... 367
B.2 Proof of Theorem 2.6 and Corollary 2.2 ................ 369
B.3 Proof of Theorem 2.7 .................................. 370
B.4 Proof of Theorem 3.10 ................................. 371
B.5 Proof of Theorem 5.2 .................................. 371
References ................................................. 173
Solutions .....................................................
Index ......................................................... 395
| 
