Source of most content: Gunhee Cho

SVD

\[\text{Let } A \in Mat_{m\times n}(F = \mathbb{R} \text{ or } \mathbb{C}). \text{ Then, } A \text{ can be decomposed as of the form } A = U \Sigma V^H. \nonumber\]

Here, $U$ is an $m\times m$ unitary/orthogonal matrix, $\Sigma$ is an $m\times n$ diagonal matrix with non-negative real number, and $V$ is $n\times n$ unitary/orthogonal matrix.

Proof

Let $rk(A)=r \leq \min(m,n)$ and $F = \mathbb{C}$ for simplicity.

1. Since $A^H A$ is Hermitian matrix, $\forall x\in F^n, \ x^H A^H A x = (Ax)^H Ax = \Vert Ax \Vert^2 \succeq 0$. Thus, $A^H A$ is positive semi-definite, i.e., all eigenvalues are non-negative. Then, we can rewrite $A^H A=V \Lambda V^H$, where $V$ is unitary and $\Lambda=diag(\lambda_1, \cdots, \lambda_r^2, 0, \cdots, 0)$ since $A^\top A$ is symmetric, i.e., normal, by Corollary.
2. $rk(A) = rk(A^H A)= r$, i.e., $Ax=0$ and $A^H A x=0$ have the same solution.
   • $(\Rightarrow)$: trivial.
   • $(\Leftarrow)$: If $A^H A x =0$, then $x^H A^H A x =0$. $\Leftrightarrow (Ax)^H Ax = \Vert Ax \Vert^2 = 0 \Leftrightarrow Ax = 0$.
     In particular, $rk(\Lambda)=r$.
3. Denote $\sigma_i = \sqrt{\lambda_i}$ and $\Sigma=diag(\sigma_1, \cdots, \sigma_r^2, 0, \cdots, 0)$. Since there exists ONB $\beta=\{ v_1, \cdots, v_n \}$ consisting of eigenvectors, $A^H A v_i = \sigma_i^2 v_i$ for all $1 \leq i\leq n$ if we set $\forall j >r, \ \sigma_j =0$. Let $u_i := \frac{Av_i}{\sigma_i}$ for $1\leq i \leq r$. Then, $u_i$ is an unit eigenvector of $AA^H$.
   • (e.v) $AA^H u_i = AA^H \frac{Av_i}{\sigma_i} = A \sigma_i^2 v_i \frac{1}{\sigma_i} = \sigma_i^2 \frac{Av_i}{\sigma_i} = \sigma_i^2 u_i$.
   • (unit) $u_i^H u_i = \frac{v_i^H A^H A v_i}{\sigma_i^2} = \frac{v_i^H \sigma_i^2 v_i}{\sigma_i^2} = v_i^H v_i = \Vert v_i \Vert = 1$. \

From $u_i = \frac{Av_i}{\sigma_i}$, $U = AV\Sigma^{-1} \Leftrightarrow U\Sigma = AV \Leftrightarrow A = U\Sigma V^{-1} \stackrel{\text{unitary} V}{=} U \Sigma V^H$.