IS CS-2019S2-02

题目来源：Problem 2 日期：2024-08-02 题目主题：CS-机器学习/统计-正态分布参数估计

解题思路

本题涉及正态分布的参数估计，特别是利用给定的损失函数估计均值 $\mu$ 和方差 $\nu$，以及矩阵变换后变量的性质。将讨论最小化损失函数的估计 $\hat{\mu }$ 和 $\hat{\nu }$，求解其期望值，并讨论 $\hat{\mu }$ 和 $\hat{\nu }$ 的独立性。具体分析包括从给定的损失函数推导出估计量、利用正态分布的性质求期望，以及利用线性变换和矩阵的性质证明估计量之间的独立性。

Solution

Question 1: Estimation of $(\hat{\mu },\hat{\nu })$

To find the estimators $(\hat{\mu },\hat{\nu })$, we minimize the given loss function $L(\mu ,\nu )$:

$$L(\mu ,\nu )=\sum _{i=1}^n\left(\frac{(X_i-\mu {)}^2}{\nu }+\log \nu \right)+\frac{\lambda }{\nu }.$$

Finding $\hat{\mu }$

To find $\hat{\mu }$, we take the partial derivative of $L(\mu ,\nu )$ with respect to $\mu$ and set it to zero:

$$\frac{\partial L(\mu ,\nu )}{\partial \mu }=\sum _{i=1}^n\left(\frac{-2(X_i-\mu )}{\nu }\right)=0.$$ $$\sum _{i=1}^n(X_i-\hat{\mu })=0.$$

Thus,

$$\hat{\mu }=\frac{1}{n}\sum _{i=1}^n{X}_i.$$

Finding $\hat{\nu }$

To find $\hat{\nu }$, we take the partial derivative of $L(\mu ,\nu )$ with respect to $\nu$ and set it to zero:

$$\frac{\partial L(\nu )}{\partial \nu }=\sum _{i=1}^n\left(-\frac{(X_i-\mu {)}^2}{{\nu }^2}+\frac{1}{\nu }\right)-\frac{\lambda }{{\nu }^2}=0.$$

Multiplying through by ${\nu }^2$, we get:

$$\sum _{i=1}^n(X_i-\mu {)}^2-n\nu +\lambda =0.$$

Thus,

$$\hat{\nu }=\frac{1}{n}\sum _{i=1}^n(X_i-\hat{\mu }{)}^2+\frac{\lambda }{n}.$$

Question 2: Expectations of $\hat{\mu }$ and $\hat{\nu }$

Expectation of $\hat{\mu }$

Since $X_i\sim \mathcal{N}(\mu ,\nu )$ independently,

$$\mathbb{E}[\hat{\mu }]=\mathbb{E}\left[\frac{1}{n}\sum _{i=1}^n{X}_i\right]=\frac{1}{n}\sum _{i=1}^n\mathbb{E}[X_i]=\frac{1}{n}\cdot n\mu =\mu .$$

Expectation of $\hat{\nu }$

For the expectation of $\hat{\nu }$, we have:

$$\hat{\nu }=\frac{1}{n}\sum _{i=1}^n(X_i-\hat{\mu }{)}^2+\frac{\lambda }{n}.$$

Since $\mathbb{E}[(X_i-\mu {)}^2]=\nu$ and $\hat{\mu }$ is an unbiased estimator of $\mu$,

$$\mathbb{E}[\hat{\nu }]=\mathbb{E}\left[\frac{1}{n}\sum _{i=1}^n(X_i-\mu {)}^2\right]+\frac{\lambda }{n}.$$

Thus,

$$\mathbb{E}[\hat{\nu }]=\nu +\frac{\lambda }{n}.$$

Question 3: Orthogonal Matrix and $\mathbf{Y}$

Given $n=3$, we have:

$$A=\left(\begin{array}{ccc}\frac{1}{\sqrt{3}}& \frac{1}{\sqrt{3}}& \frac{1}{\sqrt{3}}\\ \frac{1}{\sqrt{2}}& 0& a\\ \frac{1}{\sqrt{6}}& b& c\end{array}\right),\mathbf{Y}=\left(\begin{array}{c}{Y}_1\\ Y_2\\ Y_3\end{array}\right)=A\left(\begin{array}{c}{X}_1\\ X_2\\ X_3\end{array}\right).$$

(i) Values of $a,b,c$

To determine the values of $a,b$, and $c$, we use the fact that matrix $A$ is orthogonal. An orthogonal matrix satisfies the property $A{A}^{\top }=I$, where $I$ is the identity matrix.

We derive the values of $a,b$, and $c$ by ensuring that the rows of $A$ are orthonormal.

• First Row:

  The first row is $\left(\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}}\right)$.

• Second Row:

  The orthogonality condition between the first and second rows requires:

  $$\frac{1}{\sqrt{3}}\cdot \frac{1}{\sqrt{2}}+\frac{1}{\sqrt{3}}\cdot 0+\frac{1}{\sqrt{3}}\cdot a=0$$

  So, we can get:

  $a=-\frac{1}{\sqrt{2}}$

• Third Row:

  The third row is $\left(\frac{1}{\sqrt{6}},b,c\right)$. The orthonormality condition requires:

  $${\left(\frac{1}{\sqrt{6}}\right)}^2+b^2+c^2=1$$ $$\frac{1}{6}+b^2+c^2=1$$ $$b^2+c^2=\frac{5}{6}$$

  The orthogonality condition between the first and third rows requires:

  $$\frac{1}{\sqrt{3}}\cdot \frac{1}{\sqrt{6}}+\frac{1}{\sqrt{3}}\cdot b+\frac{1}{\sqrt{3}}\cdot c=0$$

  Solving, we find:

  $$b+c=-\frac{\sqrt{2}}{3}$$

  Combining the above equations, we find:

  $$b=-\frac{\sqrt{6}}{3},c=\frac{\sqrt{6}}{6}$$

Thus, the values of $a,b$, and $c$ are:

• $a=-\frac{\sqrt{2}}{2}$
• $b=-\frac{\sqrt{6}}{3}$
• $c=\frac{\sqrt{6}}{6}$

(ii) Express ${\sum }_{i=1}^3{X}_i$ and ${\sum }_{i=1}^3{X}_i^2$ using $Y_1,Y_2\text{ and }{Y}_3$

We can use the relationship $\mathbf{Y}=A\mathbf{X}$ and the properties of $A$ to express these sums.

For ${\sum }_{i=1}^3{X}_i$:

$$\begin{array}{rl}{Y}_1& =\frac{1}{\sqrt{3}}(X_1+X_2+X_3)\\ \sum _{i=1}^3{X}_i& =\sqrt{3}{Y}_1\end{array}$$

For ${\sum }_{i=1}^3{X}_i^2$, we can use the fact that for an orthogonal matrix $A$, $|\mathbf{X}{|}^2=|\mathbf{Y}{|}^2$:

$X_1^2+X_2^2+X_3^2=Y_1^2+Y_2^2+Y_3^2$

Therefore:

${\sum }_{i=1}^3{X}_i=\sqrt{3}{Y}_1$

${\sum }_{i=1}^3{X}_i^2={\sum }_{i=1}^3{Y}_i^2$

(iii) Independence of $\hat{\mu }$ and $\hat{\nu }$

To show that $\hat{\nu }$ and $\hat{\mu }$ are independent, we’ll express them in terms of $Y_1$, $Y_2$, and $Y_3$, and then use the properties of the multivariate normal distribution.

First, let’s express $\hat{\mu }$ and $\hat{\nu }$ in terms of $Y_1$, $Y_2$, and $Y_3$:

$\hat{\mu }=\frac{1}{3}{\sum }_{i=1}^3{X}_i=\frac{\sqrt{3}}{3}{Y}_1$

$$\begin{array}{rl}\hat{\nu }& =\frac{1}{3}\sum _{i=1}^3(X_i-\hat{\mu }{)}^2+\frac{\lambda }{3}\\ & =\frac{1}{3}\left(\sum _{i=1}^3{X}_i^2-3{\hat{\mu }}^2\right)+\frac{\lambda }{3}\\ & =\frac{1}{3}(Y_1^2+Y_2^2+Y_3^2-3(\frac{\sqrt{3}}{3}{Y}_1{)}^2)+\frac{\lambda }{3}\\ & =\frac{1}{3}(Y_1^2+Y_2^2+Y_3^2-Y_1^2)+\frac{\lambda }{3}\\ & =\frac{1}{3}(Y_2^2+Y_3^2)+\frac{\lambda }{3}\end{array}$$

Now, we can see that $\hat{\mu }$ only depends on $Y_1$, while $\hat{\nu }$ only depends on $Y_2$ and $Y_3$.

From the given facts, we know that $\mathbf{Y}$ follows a tri-variate normal distribution. For multivariate normal distributions, uncorrelated variables are independent. Therefore, if we can show that $Y_1$ is uncorrelated with $Y_2$ and $Y_3$, we can conclude that $\hat{\mu }$ is independent of $\hat{\nu }$.

The covariance matrix of $\mathbf{Y}$ is given by ${\mathbf{V}}_{\mathbf{Y}}=\nu A{A}^{\top }=\nu I_3$ (since $A$ is orthogonal). This means that $Y_1$, $Y_2$, and $Y_3$ are uncorrelated.

Therefore, $Y_1$ is independent of $Y_2$ and $Y_3$, which implies that $\hat{\mu }$ is independent of $\hat{\nu }$.

知识点

正态分布估计量损失函数正交矩阵矩阵变换参数估计极大似然估计正交矩阵机器学习统计

难点思路

1. 理解损失函数的构造：这个损失函数实际上是对数似然函数的负值加上一个正则化项。理解这一点有助于解释为什么要这样构造损失函数。
2. 正交矩阵的性质：利用正交矩阵的列向量正交且单位长度的性质来求解 $a$, $b$, 和 $c$ 的值。
3. 多元正态分布的性质：利用多元正态分布中 uncorrelated implies independent 的性质来证明 $\hat{\mu }$ 和 $\hat{\nu }$ 的独立性。

解题技巧和信息

1. 参数估计问题中，常用的方法是最小化损失函数或最大化似然函数。本题中的损失函数实际上是负对数似然函数加上正则化项。
2. 在处理涉及矩阵的问题时，充分利用矩阵的性质（如正交矩阵的性质）可以大大简化计算。
3. 在证明随机变量的独立性时，如果已知它们服从多元正态分布，只需证明它们不相关即可。
4. 在计算期望时，利用期望的线性性质可以简化计算过程。

重点词汇

• Normal distribution 正态分布
• Probability density function 概率密度函数
• Loss function 损失函数
• Regularization parameter 正则化参数
• Orthogonal matrix 正交矩阵
• Multivariate normal distribution 多元正态分布
• Covariance matrix 协方差矩阵
• Maximum likelihood estimation 极大似然估计
• Independence 独立性
• Uncorrelated 不相关

参考资料

1. “Statistical Inference” by George Casella and Roger L. Berger: Chapter 7 covers estimation theory and methods for deriving estimators.
2. “The Elements of Statistical Learning” by Trevor Hastie, Robert Tibshirani, and Jerome Friedman: Chapter 3 discusses linear methods for regression and classification, including regularization techniques.
3. “Matrix Analysis” by Roger A. Horn and Charles R. Johnson: Provides a comprehensive treatment of matrix theory, including properties of orthogonal matrices.