Four perspectives on matrix multiplication

The four different perspectives of matrix multiplication.

· 2 min read · #linear-algebra · @mathematics · 한국어

In one of Hyukppen Heim’s YouTube videos, the section introducing four different perspectives on matrix multiplication stuck with me. Being able to move freely between these four views should make linear algebra click much faster.

$A\in\mathbb{R}^{m\times n},\ B\in\mathbb{R}^{n\times p}$

Inner products

Write the $i$-th row of $A$ as $a_i^T$ and the $j$-th column of $B$ as $b_j$:

\[A= \begin{bmatrix} a_1^T\\ a_2^T\\ \vdots\\ a_m^T \end{bmatrix}, \qquad B=[b_1\ b_2\ \cdots\ b_p]\]

Then the entries of $C=AB$ are

\[c_{ij}=a_i^T b_j\]

That is, “$(i,j)$ of $C$” is the inner product of the $i$-th row of $A$ and the $j$-th column of $B$.

A sum of rank-1 matrices

This time expand $A$ into column vectors and $B$ into row vectors:

\[A=[a_1\ a_2\ \cdots\ a_n], \qquad B= \begin{bmatrix} b_1^T\\ b_2^T\\ \vdots\\ b_n^T \end{bmatrix}\]
  • $a_k\in\mathbb{R}^{m}$: the $k$-th column of $A$
  • $b_k^T\in\mathbb{R}^{1\times p}$: the $k$-th row of $B$

The product then becomes

\[AB= a_1 b_1^T + a_2 b_2^T + \cdots + a_n b_n^T= \sum_{k=1}^{n} a_k b_k^T\]

So $AB$ is the sum of these rank-1 matrices over $k=1$ through $n$.

The column space view = $Ax$

Writing $A$ as column vectors,

\[A = [a_1\ a_2\ \cdots\ a_n] \quad (a_k \in \mathbb{R}^m)\]

and taking $x\in\mathbb{R}^n$ to be

\[x=\begin{bmatrix}x_1\\ x_2\\ \vdots\\ x_n \end{bmatrix}\]

we get

\[Ax = [a_1\ a_2\ \cdots\ a_n] \begin{bmatrix}x_1\\ x_2\\ \vdots \\x_n \end{bmatrix} = x_1 a_1 + x_2 a_2 + \cdots + x_n a_n\]

$Ax$ is a linear combination of the column vectors $a_k$ of $A$, so it always lies in $\mathrm{Col}(A)$.

The row space view = $x^TA$

Now write $A$ as row vectors,

\[A = \begin{bmatrix} a_1^T\\ a_2^T\\ \vdots\\ a_m^T \end{bmatrix} \quad (a_i^T \in \mathbb{R}^{1\times n})\]

and take $x\in\mathbb{R}^m$ to be

\[x^T=[x_1\ x_2\ \cdots\ x_m]\]

which gives

\[x^T A= [x_1\ x_2\ \cdots\ x_m] \begin{bmatrix} a_1^T\\ a_2^T\\ \vdots\\ a_m^T \end{bmatrix} = x_1 a_1^T + x_2 a_2^T + \cdots + x_m a_m^T\]

$x^T A$ is a linear combination of the row vectors $a_i^T$ of $A$, so it always lies in $\mathrm{Row}(A)$.