矩阵求导

矩阵求导 | Matrix Calculus

简介 | Introduction

矩阵求导是线性代数和微积分的交叉领域，涉及到对矩阵、向量以及标量函数求导的技术。它在机器学习、优化、统计学等多个领域中具有广泛的应用。矩阵求导和标量微分有相似之处，但它具有更多的复杂性，因为它考虑的是多维的结构。

Matrix calculus is an intersection of linear algebra and calculus, dealing with techniques for taking derivatives with respect to matrices, vectors, and scalar functions. It has broad applications in fields such as machine learning, optimization, and statistics. While similar to scalar calculus, matrix calculus involves additional complexity due to its consideration of multidimensional structures.

标量作为变量的矩阵求导 | Scalar as Variable in Matrix Calculus

标量对标量求导 | Scalar Derivative of a Scalar Function

如果函数 $f$ 仅依赖于一个标量变量 $x$，那么其导数与一元微分学中定义的导数相同：

$$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\underset{\Delta x\to 0}{\lim }\frac{f(x+\Delta x)-f(x)}{\Delta x}$$

If a function $f$ depends only on a single scalar variable $x$, its derivative is the same as defined in single-variable calculus:

$$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\underset{\Delta x\to 0}{\lim }\frac{f(x+\Delta x)-f(x)}{\Delta x}$$

标量对向量求导 | Scalar Derivative of a Vector Function

对于一个依赖于向量 $\mathbf{x}$ 的标量函数 $f(\mathbf{x})$，其导数是一个向量，称为梯度 (Gradient)：

$$\frac{\partial f(\mathbf{x})}{\partial \mathbf{x}}={\nabla }_{\mathbf{x}}f(\mathbf{x})=\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ ⋮\\ \frac{\partial f}{\partial x_n}\end{array}\right]$$

For a scalar function $f(\mathbf{x})$ that depends on a vector $\mathbf{x}$, the derivative is a vector, known as the gradient:

$$\frac{\partial f(\mathbf{x})}{\partial \mathbf{x}}={\nabla }_{\mathbf{x}}f(\mathbf{x})=\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ ⋮\\ \frac{\partial f}{\partial x_n}\end{array}\right]$$

标量对矩阵求导 | Scalar Derivative of a Matrix Function

对于一个依赖于矩阵 $\mathbf{X}$ 的标量函数 $f(\mathbf{X})$，其导数是一个矩阵，称为梯度矩阵 (Gradient Matrix)：

$$\frac{\partial f(\mathbf{X})}{\partial \mathbf{X}}=\left[\begin{array}{ccc}\frac{\partial f}{\partial X_{11}}& \cdots & \frac{\partial f}{\partial X_{1n}}\\ ⋮& \ddots & ⋮\\ \frac{\partial f}{\partial X_{m1}}& \cdots & \frac{\partial f}{\partial X_{mn}}\end{array}\right]$$

For a scalar function $f(\mathbf{X})$ that depends on a matrix $\mathbf{X}$, the derivative is a matrix, known as the gradient matrix:

$$\frac{\partial f(\mathbf{X})}{\partial \mathbf{X}}=\left[\begin{array}{ccc}\frac{\partial f}{\partial X_{11}}& \cdots & \frac{\partial f}{\partial X_{1n}}\\ ⋮& \ddots & ⋮\\ \frac{\partial f}{\partial X_{m1}}& \cdots & \frac{\partial f}{\partial X_{mn}}\end{array}\right]$$

向量作为变量的矩阵求导 | Vector as Variable in Matrix Calculus

向量对标量求导 | Vector Derivative of a Scalar Function

对于一个依赖于标量 $x$ 的向量函数 $\mathbf{f}(x)$，其导数是一个向量：

$$\frac{\mathrm{d}\mathbf{f}(x)}{\mathrm{d}x}=\left[\begin{array}{c}\frac{\mathrm{d}{f}_1(x)}{\mathrm{d}x}\\ \frac{\mathrm{d}{f}_2(x)}{\mathrm{d}x}\\ ⋮\\ \frac{\mathrm{d}{f}_n(x)}{\mathrm{d}x}\end{array}\right]$$

For a vector function $\mathbf{f}(x)$ that depends on a scalar $x$, the derivative is a vector:

$$\frac{\mathrm{d}\mathbf{f}(x)}{\mathrm{d}x}=\left[\begin{array}{c}\frac{\mathrm{d}{f}_1(x)}{\mathrm{d}x}\\ \frac{\mathrm{d}{f}_2(x)}{\mathrm{d}x}\\ ⋮\\ \frac{\mathrm{d}{f}_n(x)}{\mathrm{d}x}\end{array}\right]$$

向量对向量求导 | Vector Derivative of a Vector Function

对于一个依赖于向量 $\mathbf{x}$ 的向量函数 $\mathbf{f}(\mathbf{x})$，其导数是一个雅可比矩阵 (Jacobian Matrix)：

$$\frac{\partial \mathbf{f}(\mathbf{x})}{\partial \mathbf{x}}={\mathbf{J}}_{\mathbf{f},\mathbf{x}}=\left[\begin{array}{ccc}\frac{\partial f_1}{\partial x_1}& \cdots & \frac{\partial f_1}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial f_m}{\partial x_1}& \cdots & \frac{\partial f_m}{\partial x_n}\end{array}\right]$$

For a vector function $\mathbf{f}(\mathbf{x})$ that depends on a vector $\mathbf{x}$, the derivative is a Jacobian matrix:

$$\frac{\partial \mathbf{f}(\mathbf{x})}{\partial \mathbf{x}}={\mathbf{J}}_{\mathbf{f},\mathbf{x}}=\left[\begin{array}{ccc}\frac{\partial f_1}{\partial x_1}& \cdots & \frac{\partial f_1}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial f_m}{\partial x_1}& \cdots & \frac{\partial f_m}{\partial x_n}\end{array}\right]$$

向量对矩阵求导 | Vector Derivative of a Matrix Function

对于一个依赖于矩阵 $\mathbf{X}$ 的向量函数 $\mathbf{f}(\mathbf{X})$，其导数是一个张量 (Tensor)，通常不常见，也不易表述。通常我们通过分块矩阵或其他方式来表示这种导数。

For a vector function $\mathbf{f}(\mathbf{X})$ that depends on a matrix $\mathbf{X}$, the derivative is a tensor, which is less common and harder to express. Usually, we represent such derivatives using block matrices or other methods.

矩阵作为变量的矩阵求导 | Matrix as Variable in Matrix Calculus

矩阵对标量求导 | Matrix Derivative of a Scalar Function

对于一个依赖于标量 $x$ 的矩阵函数 $\mathbf{F}(x)$，其导数是一个矩阵：

$$\frac{\mathrm{d}\mathbf{F}(x)}{\mathrm{d}x}=\left[\begin{array}{ccc}\frac{\mathrm{d}{F}_{11}(x)}{\mathrm{d}x}& \cdots & \frac{\mathrm{d}{F}_{1n}(x)}{\mathrm{d}x}\\ ⋮& \ddots & ⋮\\ \frac{\mathrm{d}{F}_{m1}(x)}{\mathrm{d}x}& \cdots & \frac{\mathrm{d}{F}_{mn}(x)}{\mathrm{d}x}\end{array}\right]$$

For a matrix function $\mathbf{F}(x)$ that depends on a scalar $x$, the derivative is a matrix:

$$\frac{\mathrm{d}\mathbf{F}(x)}{\mathrm{d}x}=\left[\begin{array}{ccc}\frac{\mathrm{d}{F}_{11}(x)}{\mathrm{d}x}& \cdots & \frac{\mathrm{d}{F}_{1n}(x)}{\mathrm{d}x}\\ ⋮& \ddots & ⋮\\ \frac{\mathrm{d}{F}_{m1}(x)}{\mathrm{d}x}& \cdots & \frac{\mathrm{d}{F}_{mn}(x)}{\mathrm{d}x}\end{array}\right]$$

矩阵对向量求导 | Matrix Derivative of a Vector Function

对于一个依赖于向量 $\mathbf{x}$ 的矩阵函数 $\mathbf{F}(\mathbf{x})$，其导数是一个三阶张量：

$$\frac{\partial \mathbf{F}(\mathbf{x})}{\partial \mathbf{x}}=\left[\begin{array}{ccc}\frac{\partial F_{11}}{\partial x_1}& \cdots & \frac{\partial F_{11}}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial F_{m1}}{\partial x_1}& \cdots & \frac{\partial F_{m1}}{\partial x_n}\end{array}\right],\cdots ,\left[\begin{array}{ccc}\frac{\partial F_{1n}}{\partial x_1}& \cdots & \frac{\partial F_{1n}}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial F_{mn}}{\partial x_1}& \cdots & \frac{\partial F_{mn}}{\partial x_n}\end{array}\right]$$

For a matrix function $\mathbf{F}(\mathbf{x})$ that depends on a vector $\mathbf{x}$, the derivative is a third-order tensor:

$$\frac{\partial \mathbf{F}(\mathbf{x})}{\partial \mathbf{x}}=\left[\begin{array}{ccc}\frac{\partial F_{11}}{\partial x_1}& \cdots & \frac{\partial F_{11}}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial F_{m1}}{\partial x_1}& \cdots & \frac{\partial F_{m1}}{\partial x_n}\end{array}\right],\cdots ,\left[\begin{array}{ccc}\frac{\partial F_{1n}}{\partial x_1}& \cdots & \frac{\partial F_{1n}}{\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{\partial F_{mn}}{\partial x_1}& \cdots & \frac{\partial F_{mn}}{\partial x_n}\end{array}\right]$$

矩阵对矩阵求导 | Matrix Derivative of a Matrix Function

对于一个依赖于矩阵 $\mathbf{X}$ 的矩阵函数 $\mathbf{F}(\mathbf{X})$，其导数是一个四阶张量。通常，通过 Kronecker 积或者其他特殊的方式来简化表示。

For a matrix function $\mathbf{F}(\mathbf{X})$ that depends on a matrix $\mathbf{X}$, the derivative is a fourth-order tensor. Usually, it is simplified using the Kronecker product or other special methods.

矩阵求导法则 | Rules of Matrix Calculus

链式法则 | Chain Rule

当复合函数的形式存在时，链式法则在矩阵求导中同样适用：

$$\frac{\mathrm{d}f(g(\mathbf{x}))}{\mathrm{d}\mathbf{x}}=\frac{\mathrm{d}f}{\mathrm{d}g}\cdot \frac{\mathrm{d}g}{\mathrm{d}\mathbf{x}}$$

When dealing with composite functions, the chain rule is also applicable in matrix calculus:

$$\frac{\mathrm{d}f(g(\mathbf{x}))}{\mathrm{d}\mathbf{x}}=\frac{\mathrm{d}f}{\mathrm{d}g}\cdot \frac{\mathrm{d}g}{\mathrm{d}\mathbf{x}}$$

积法则 | Product Rule

对矩阵乘积求导时，积法则为：

$$\frac{\mathrm{d}}{\mathrm{d}x}(\mathbf{AB})=\frac{\mathrm{d}\mathbf{A}}{\mathrm{d}x}\mathbf{B}+\mathbf{A}\frac{\mathrm{d}\mathbf{B}}{\mathrm{d}x}$$

For the derivative of a matrix product, the product rule states:

$$\frac{\mathrm{d}}{\mathrm{d}x}(\mathbf{AB})=\frac{\mathrm{d}\mathbf{A}}{\mathrm{d}x}\mathbf{B}+\mathbf{A}\frac{\mathrm{d}\mathbf{B}}{\mathrm{d}x}$$

转置求导法则 | Transpose Rule

对于转置矩阵的求导，转置求导法则为：

$$\frac{\mathrm{d}({\mathbf{A}}^{\top })}{\mathrm{d}x}={\left(\frac{\mathrm{d}\mathbf{A}}{\mathrm{d}x}\right)}^{\top }$$

For the derivative of a transposed matrix, the transpose rule states:

$$\frac{\mathrm{d}({\mathbf{A}}^{\top })}{\mathrm{d}x}={\left(\frac{\mathrm{d}\mathbf{A}}{\mathrm{d}x}\right)}^{\top }$$

逆矩阵求导法则 | Inverse Matrix Rule

对于逆矩阵的求导，逆矩阵求导法则为：

$$\frac{\mathrm{d}({\mathbf{A}}^{-1})}{\mathrm{d}x}=-{\mathbf{A}}^{-1}\frac{\mathrm{d}\mathbf{A}}{\mathrm{d}x}{\mathbf{A}}^{-1}$$

For the derivative of an inverse matrix, the inverse matrix rule states:

$$\frac{\mathrm{d}({\mathbf{A}}^{-1})}{\mathrm{d}x}=-{\mathbf{A}}^{-1}\frac{\mathrm{d}\mathbf{A}}{\mathrm{d}x}{\mathbf{A}}^{-1}$$

矩阵求导常见公式推导 | Derivation of Common Formulas in Matrix Calculus

1. 对迹函数求导 | Derivative of Trace Function

推导过程 | Derivation:

考虑迹函数 $\mathrm{tr}(\mathbf{AX})$，其中 $\mathbf{A}$ 是常矩阵，$\mathbf{X}$ 是变量矩阵。首先，利用迹的性质 $\mathrm{tr}(\mathbf{AB})=\mathrm{tr}(\mathbf{BA})$，可以将其展开：

Consider the trace function $\mathrm{tr}(\mathbf{AX})$, where $\mathbf{A}$ is a constant matrix and $\mathbf{X}$ is the variable matrix. First, using the property of the trace, $\mathrm{tr}(\mathbf{AB})=\mathrm{tr}(\mathbf{BA})$, we can expand it as:

$$\mathrm{tr}(\mathbf{AX})=\sum _{i=1}^n\sum _{j=1}^m{A}_{ij}{X}_{ji}$$

对 $\mathbf{X}$ 的元素 $X_{pq}$ 求导：

Taking the derivative with respect to an element $X_{pq}$ of $\mathbf{X}$:

$$\frac{\partial \mathrm{tr}(\mathbf{AX})}{\partial X_{pq}}=\frac{\partial }{\partial X_{pq}}\sum _{i=1}^n\sum _{j=1}^m{A}_{ij}{X}_{ji}$$

这里，仅当 $i=p$ 且 $j=q$ 时，求和中的项不为零，因此：

Here, the only non-zero term in the summation occurs when $i=p$ and $j=q$, so:

$$\frac{\partial \mathrm{tr}(\mathbf{AX})}{\partial X_{pq}}=A_{qp}$$

将结果组合在一起，可以得出：

Combining the results, we get:

$$\frac{\partial \mathrm{tr}(\mathbf{AX})}{\partial \mathbf{X}}={\mathbf{A}}^{\top }$$

2. 对行列式求导 | Derivative of Determinant

推导过程 | Derivation:

考虑行列式 $\det (\mathbf{X})$，我们利用 $\mathbf{X}$ 的行列式对其元素的导数公式。已知：

Consider the determinant $\det (\mathbf{X})$, we use the formula for the derivative of the determinant with respect to its elements. It is known that:

$$\frac{\partial \det (\mathbf{X})}{\partial X_{ij}}=\det (\mathbf{X})\cdot ({\mathbf{X}}^{-1}{)}_{ij}^{\top }$$

将其扩展到整个矩阵 $\mathbf{X}$，可以得到：

Expanding this to the entire matrix $\mathbf{X}$, we obtain:

$$\frac{\partial \det (\mathbf{X})}{\partial \mathbf{X}}=\det (\mathbf{X})\cdot ({\mathbf{X}}^{-1}{)}^{\top }$$

3. 对二次型求导 | Derivative of Quadratic Form

推导过程 | Derivation:

考虑一个二次型 ${\mathbf{x}}^{\top }\mathbf{Ax}$，其中 $\mathbf{x}$ 是向量，$\mathbf{A}$ 是对称矩阵。我们对 $\mathbf{x}$ 进行求导：

Consider a quadratic form ${\mathbf{x}}^{\top }\mathbf{Ax}$, where $\mathbf{x}$ is a vector and $\mathbf{A}$ is a symmetric matrix. We take the derivative with respect to $\mathbf{x}$:

$$\frac{\partial ({\mathbf{x}}^{\top }\mathbf{Ax})}{\partial \mathbf{x}}=\frac{\partial \left({\sum }_{i=1}^n{\sum }_{j=1}^n{x}_i{A}_{ij}{x}_j\right)}{\partial \mathbf{x}}$$

首先，我们分别对 $x_i$ 和 $x_j$ 求导：

First, we differentiate with respect to $x_i$ and $x_j$:

$$\frac{\partial ({\mathbf{x}}^{\top }\mathbf{Ax})}{\partial x_k}=2\sum _{j=1}^n{A}_{kj}{x}_j=2(\mathbf{Ax}{)}_k$$

将所有分量组合在一起，我们可以得出：

Combining all the components together, we obtain:

$$\frac{\partial ({\mathbf{x}}^{\top }\mathbf{Ax})}{\partial \mathbf{x}}=\mathbf{Ax}+{\mathbf{A}}^{\top }\mathbf{x}$$

对于对称矩阵 $\mathbf{A}$，有 $\mathbf{A}={\mathbf{A}}^{\top }$，所以最终可以简化为：

For a symmetric matrix $\mathbf{A}$, where $\mathbf{A}={\mathbf{A}}^{\top }$, this simplifies to:

$$\frac{\partial ({\mathbf{x}}^{\top }\mathbf{Ax})}{\partial \mathbf{x}}=2\mathbf{Ax}$$

4. 对矩阵乘积求导 | Derivative of a Matrix Product

推导过程 | Derivation:

考虑两个矩阵 $\mathbf{A}$ 和 $\mathbf{X}$ 的乘积 $\mathbf{AX}$，对 $\mathbf{X}$ 求导。我们使用矩阵乘法的求导法则：

Consider the product of two matrices $\mathbf{A}$ and $\mathbf{X}$, i.e., $\mathbf{AX}$, and take the derivative with respect to $\mathbf{X}$. We use the product rule for matrix calculus:

$$\frac{\partial (\mathbf{AX})}{\partial \mathbf{X}}={\mathbf{A}}^{\top }$$

类似地，对于右乘的情况 $\mathbf{XA}$，对 $\mathbf{X}$ 求导，可以得到：

Similarly, for the case of right multiplication $\mathbf{XA}$, the derivative with respect to $\mathbf{X}$ is:

$$\frac{\partial (\mathbf{XA})}{\partial \mathbf{X}}={\mathbf{A}}^{\top }$$

5. 对矩阵 Frobenius 范数的求导 | Derivative of the Frobenius Norm

推导过程 | Derivation:

Frobenius 范数是矩阵元素平方和的平方根，定义为：

The Frobenius norm is the square root of the sum of the squares of the matrix elements, defined as:

$$\Vert \mathbf{X}{\Vert }_F=\sqrt{\sum _{i,j}{X}_{ij}^2}$$

其平方的导数比原范数更常用：

The derivative of its square is more commonly used:

$$\frac{\partial \Vert \mathbf{X}{\Vert }_F^2}{\partial \mathbf{X}}=2\mathbf{X}$$

具体推导如下：

The derivation is as follows:

$$\Vert \mathbf{X}{\Vert }_F^2=\sum _{i,j}{X}_{ij}^2$$

对 $\mathbf{X}$ 的元素 $X_{ij}$ 求导：

Taking the derivative with respect to an element $X_{ij}$ of $\mathbf{X}$:

$$\frac{\partial \Vert \mathbf{X}{\Vert }_F^2}{\partial X_{ij}}=2X_{ij}$$

组合成矩阵形式，得到：

Combining into matrix form, we get:

$$\frac{\partial \Vert \mathbf{X}{\Vert }_F^2}{\partial \mathbf{X}}=2\mathbf{X}$$

6. 对矩阵迹二次型求导 | Derivative of the Trace of a Quadratic Form

推导过程 | Derivation:

考虑二次型 $\mathrm{tr}({\mathbf{X}}^{\top }\mathbf{AX})$，其中 $\mathbf{A}$ 是对称矩阵，$\mathbf{X}$ 是变量矩阵。我们可以通过应用迹函数的导数规则推导其对 $\mathbf{X}$ 的导数：

Consider the quadratic form $\mathrm{tr}({\mathbf{X}}^{\top }\mathbf{AX})$, where $\mathbf{A}$ is a symmetric matrix and $\mathbf{X}$ is the variable matrix. We can derive its derivative with respect to $\mathbf{X}$ by applying the derivative rule for trace functions:

$$\frac{\partial \mathrm{tr}({\mathbf{X}}^{\top }\mathbf{AX})}{\partial \mathbf{X}}=\mathbf{AX}+{\mathbf{A}}^{\top }\mathbf{X}$$

由于 $\mathbf{A}$ 是对称的，我们得到：

Since $\mathbf{A}$ is symmetric, we obtain:

$$\frac{\partial \mathrm{tr}({\mathbf{X}}^{\top }\mathbf{AX})}{\partial \mathbf{X}}=2\mathbf{AX}$$

7. 对矩阵对数行列式求导 | Derivative of the Log-Determinant

推导过程 | Derivation:

考虑对矩阵 $\mathbf{X}$ 的对数行列式 $\log \det (\mathbf{X})$ 的求导。已知：

Consider the derivative of the log-determinant $\log \det (\mathbf{X})$ with respect to the matrix $\mathbf{X}$. It is known that:

$$\frac{\partial \log \det (\mathbf{X})}{\partial \mathbf{X}}={\mathbf{X}}^{-1}$$

推导可以通过以下关系来理解：

This derivation can be understood by the relationship:

$$\log \det (\mathbf{X})=\mathrm{tr}(\log (\mathbf{X}))$$

并结合对迹函数的求导公式：

And applying the derivative formula for the trace function:

$$\frac{\partial \mathrm{tr}(\log (\mathbf{X}))}{\partial \mathbf{X}}={\mathbf{X}}^{-1}$$

8. 对向量内积求导 | Derivative of a

Vector Inner Product

推导过程 | Derivation:

考虑两个向量 $\mathbf{x}$ 和 $\mathbf{y}$ 的内积 ${\mathbf{x}}^{\top }\mathbf{y}$，对 $\mathbf{x}$ 求导得到：

Consider the inner product of two vectors $\mathbf{x}$ and $\mathbf{y}$, ${\mathbf{x}}^{\top }\mathbf{y}$, the derivative with respect to $\mathbf{x}$ is:

$$\frac{\partial ({\mathbf{x}}^{\top }\mathbf{y})}{\partial \mathbf{x}}=\mathbf{y}$$

类似地，对 $\mathbf{y}$ 求导，得到：

Similarly, the derivative with respect to $\mathbf{y}$ is:

$$\frac{\partial ({\mathbf{x}}^{\top }\mathbf{y})}{\partial \mathbf{y}}=\mathbf{x}$$

---

这些推导为常见的矩阵求导公式提供了清晰的理论基础。这些公式在实际应用中，例如机器学习、优化、信号处理等领域，能够有效简化推导过程和计算复杂度。