IS CS-2020S2-06

题目来源：Problem 6 日期：2024-08-11 题目主题：Math/CS-概率论与统计-正态分布与 EM 算法

解题思路

这道题目涉及到正态分布的基本性质、条件分布以及使用 EM 算法进行参数估计的问题。首先，我们需要计算复合随机变量的期望和方差，然后推导出条件分布，再进一步推导联合概率密度函数，最后，运用 EM 算法对缺失数据进行参数估计。

Solution

Question 1

The random variable $Y$ is defined as $Y=\theta X+Z$, where $X\sim N(\mu ,1)$ and $Z\sim N(0,1)$. Since $X$ and $Z$ are independent, we can calculate the expectation and variance of $Y$ as follows:

1. Expectation of $Y$:

   $$\mathbb{E}[Y]=\mathbb{E}[\theta X+Z]=\theta \mathbb{E}[X]+\mathbb{E}[Z]=\theta \mu +0=\theta \mu$$

2. Variance of $Y$:

   $$\mathbb{V}[Y]=\mathbb{V}[\theta X+Z]={\theta }^2\mathbb{V}[X]+\mathbb{V}[Z]={\theta }^2\cdot 1+1={\theta }^2+1$$

Question 2

To find the conditional distribution of $X$ given $Y$, note that $Y=\theta X+Z$, where $X\sim N(\mu ,1)$ and $Z\sim N(0,1)$. The joint distribution of $(X,Y)$ is bivariate normal, which implies that the conditional distribution $X|Y$ is also normal.

1. Expectation of $X|Y$:

   $$\mathbb{E}[X|Y]=\mu +\frac{\theta }{{\theta }^2+1}(Y-\theta \mu )$$

2. Variance of $X|Y$:

   $$\mathbb{V}[X|Y]=\frac{1}{{\theta }^2+1}$$

This can be derived using the properties of conditional distributions for bivariate normal distributions.

Question 3

The joint probability density function $p_{\mu ,\theta }({\mathbf{x}}^{(n)},{\mathbf{y}}^{(n)})$ for the random variables ${\mathbf{X}}^{(n)}=(X_1,X_2,\dots ,X_n)$ and ${\mathbf{Y}}^{(n)}=(Y_1,Y_2,\dots ,Y_n)$ can be expressed as the product of the marginal distributions of $X_i$ and the conditional distributions of $Y_i$ given $X_i$:

$$p_{\mu ,\theta }({\mathbf{x}}^{(n)},{\mathbf{y}}^{(n)})=\prod _{i=1}^n\left(\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{(x_i-\mu {)}^2}{2}\right)\cdot \frac{1}{\sqrt{2\pi }}\exp \left(-\frac{(y_i-\theta x_i{)}^2}{2}\right)\right)$$

Expanding this, we get:

$$p_{\mu ,\theta }({\mathbf{x}}^{(n)},{\mathbf{y}}^{(n)})=\frac{1}{(2\pi {)}^n}\exp \left(-\sum _{i=1}^n\left[\frac{(x_i-\mu {)}^2}{2}+\frac{(y_i-\theta x_i{)}^2}{2}\right]\right)$$

Question 4

Part (i)

The expectation ${\mathbb{E}}_{X_n\sim N(\bar{\mu },{\bar{\sigma }}^2)}[\log p_{\mu ,\theta }({\mathbf{X}}^{(n)},{\mathbf{Y}}^{(n)})]$ is given by:

$${\mathbb{E}}_{X_n\sim N(\bar{\mu },{\bar{\sigma }}^2)}[\log p_{\mu ,\theta }({\mathbf{X}}^{(n)},{\mathbf{Y}}^{(n)})]={\mathbb{E}}_{X_n\sim N(\bar{\mu },{\bar{\sigma }}^2)}\left[-\sum _{i=1}^{n-1}\left(\frac{(x_i-\mu {)}^2}{2}+\frac{(y_i-\theta x_i{)}^2}{2}\right)-\left(\frac{(X_n-\mu {)}^2}{2}+\frac{(y_n-\theta X_n{)}^2}{2}\right)\right]$$

Simplifying further using the properties of the expectation for a normal distribution:

$${\mathbb{E}}_{X_n\sim N(\bar{\mu },{\bar{\sigma }}^2)}[\log p_{\mu ,\theta }({\mathbf{X}}^{(n)},{\mathbf{Y}}^{(n)})]=-\sum _{i=1}^{n-1}\left(\frac{(x_i-\mu {)}^2}{2}+\frac{(y_i-\theta x_i{)}^2}{2}\right)-\frac{1}{2}\left((\bar{\mu }-\mu {)}^2+{\bar{\sigma }}^2+\frac{(y_n-\theta \bar{\mu }{)}^2}{{\theta }^2+1}\right)$$

Part (ii)

The update rule for $({\mu }_{t+1},{\theta }_{t+1})$ in the EM algorithm is obtained by maximizing the expression found in part (i):

$$({\mu }_{t+1},{\theta }_{t+1})=\underset{(\mu ,\theta )\in {\mathbb{R}}^2}{\mathrm{argmax}}\left[-\sum _{i=1}^{n-1}\left(\frac{(x_i-\mu {)}^2}{2}+\frac{(y_i-\theta x_i{)}^2}{2}\right)-\frac{1}{2}\left((\bar{\mu }-\mu {)}^2+{\bar{\sigma }}^2+\frac{(y_n-\theta \bar{\mu }{)}^2}{{\theta }^2+1}\right)\right]$$

Solving this for $\mu$ and $\theta$, we find:

$${\mu }_{t+1}=\frac{1}{n}\left(\sum _{i=1}^{n-1}{x}_i+\bar{\mu }\right)$$ $${\theta }_{t+1}=\frac{{\sum }_{i=1}^{n-1}{y}_i{x}_i+y_n\bar{\mu }}{{\sum }_{i=1}^{n-1}{x}_i^2+{\bar{\mu }}^2+\frac{1}{{\theta }^2+1}}$$

This update rule depends on the observed data ${\mathbf{X}}^{(n-1)},{\mathbf{Y}}^{(n)}$ and the estimates $\bar{\mu },{\bar{\sigma }}^2$ obtained from the conditional expectation.

知识点

正态分布条件分布数值期望EM算法最大似然估计

难点思路

推导条件分布涉及到二元正态分布的性质，尤其是推导条件期望和方差时，需要对协方差矩阵有深刻理解。EM 算法的难点在于构建对数似然函数的期望，并通过优化找到参数的更新规则。

解题技巧和信息

1. 条件分布：对于二元正态分布，条件分布仍然是正态分布，且其参数可以通过边际分布的参数计算得到。
2. EM 算法：EM 算法通过最大化对数似然函数的期望来迭代更新参数，对于缺失数据的问题尤为有效。
3. 最大似然估计：通常情况下，EM 算法能够保证参数的渐进一致性，即经过多次迭代，参数估计会收敛到真值。

重点词汇

• Expectation-Maximization (EM) Algorithm: 期望最大化算法
• Conditional distribution: 条件分布
• Maximum likelihood estimation: 最大似然估计
• Normal distribution: 正态分布

参考资料

1. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Chapter 9: Mixture Models and EM.
2. Casella, G., & Berger, R. L. (2001). Statistical Inference (2nd ed.). Duxbury. Chapter 7: Estimation.