Linear Algebra Basics Chapter 4: Eigenvalues, Eigenvectors, and PCA

In Chapter 3 we covered vector operations, linear transformations, matrix multiplication, and inverses. In this final installment, we'll interpret determinants geometrically, dive deep into eigenvalues and eigenvectors — the crown jewel of linear algebra — and trace exactly how they power PCA (Principal Component Analysis) in machine learning.

This post is based on Week 4 of DeepLearning.AI's Mathematics for Machine Learning and Data Science course.

What You'll Learn in This Chapter#

• Interpreting determinants as area (or volume)
• The product rule and inverse rule for determinants
• The concepts of Basis and Span
• Intuition and computation of eigenvalues and eigenvectors
• PCA — dimensionality reduction using variance, covariance, and eigenvectors
• Markov matrices and discrete dynamical systems

Singularity and Rank of Linear Transformations#

Linear transformations, like matrices, can be classified as singular or non-singular.

A non-singular linear transformation covers every point in the plane. A singular transformation compresses the plane onto a line or a point.

Determinant = Area (Geometric Interpretation)#

A determinant is more than just a number. It represents the scaling factor of area (or volume) — how much a linear transformation stretches or shrinks space.

$A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$

Apply this transformation to the unit square (area = 1), and the resulting parallelogram has area $|\det(A)|$.

Key insight: A singular matrix compresses area down to 0, which is exactly why $\det = 0$.

Determinant of a Product#

$\det(AB) = \det(A) \cdot \det(B)$

The area scaling factor of two composed transformations equals the product of each transformation's scaling factor.

Important consequence:

$A \text{ or } B \text{ is singular} \Rightarrow \det(AB) = 0 \Rightarrow AB \text{ is also singular}$

The product of a singular and a non-singular matrix is always singular.

Determinant of an Inverse#

When $A$ is non-singular (invertible):

$\det(A^{-1}) = \frac{1}{\det(A)}$

Intuition: If $A$ stretches space by a factor of 5, then $A^{-1}$ must shrink it by $\frac{1}{5}$ to undo the transformation.

Examples:

$\det(M) = 0.2 \Rightarrow \det(M^{-1}) = 5$

$\det(N) = 0.125 \Rightarrow \det(N^{-1}) = 8$

Basis#

A basis is the minimal set of vectors that defines a vector space.

$\vec{e_1} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}, \quad \vec{e_2} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$

The defining property of a basis: every point in the space can be expressed as a linear combination of the basis vectors.

$\begin{bmatrix} 3 \\ 5 \end{bmatrix} = 3\vec{e_1} + 5\vec{e_2}$

Requirements to form a basis:

• The vectors must be linearly independent (no vector can be expressed as a linear combination of the others)
• Vectors pointing in the same or opposite directions cannot form a basis

Span#

The span of a set of vectors is the set of all points reachable by taking every possible linear combination of those vectors.

Basis = minimal spanning set: the smallest number of linearly independent vectors needed to span the space.

If you have more vectors than the dimension of the space, there will always be linear dependence among them.

Eigenbasis#

When a linear transformation $A$ is applied, most vectors change direction. But certain special vectors preserve their direction and only change in magnitude (scale). These are called eigenvectors.

$A\vec{v} = \lambda\vec{v}$

• $\vec{v}$: eigenvector (direction preserved)
• $\lambda$: eigenvalue (scaling factor)

Intuition: When you look at data along an eigenvector direction, the matrix multiplication (a complex operation) reduces to a simple scalar multiplication.

Computing Eigenvalues and Eigenvectors#

The Core Formula#

$A\vec{v} = \lambda\vec{v}$ $A\vec{v} - \lambda I\vec{v} = \vec{0}$ $(A - \lambda I)\vec{v} = \vec{0}$

For $\vec{v} \neq \vec{0}$ to exist, $(A - \lambda I)$ must be a singular matrix:

$\det(A - \lambda I) = 0 \quad \text{(characteristic polynomial)}$

2×2 Matrix Example#

$A = \begin{bmatrix} 5 & 1 \\ 3 & 3 \end{bmatrix}$

Step 1 — Characteristic polynomial:

$\det(A - \lambda I) = \det\begin{bmatrix} 5-\lambda & 1 \\ 3 & 3-\lambda \end{bmatrix} = 0$

$(5-\lambda)(3-\lambda) - 3 = 0$

$\lambda^2 - 8\lambda + 12 = 0$

$(\lambda - 6)(\lambda - 2) = 0$

Eigenvalues: $\lambda_1 = 6,\quad \lambda_2 = 2$

Step 2 — Eigenvector for $\lambda_1 = 6$:

$(A - 6I)\vec{v} = \vec{0}$

$\begin{bmatrix} -1 & 1 \\ 3 & -3 \end{bmatrix}\vec{v} = \vec{0}$

$-v_1 + v_2 = 0 \Rightarrow v_1 = v_2 \Rightarrow \vec{v_1} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$

Eigenvector for $\lambda_2 = 2$:

$\begin{bmatrix} 3 & 1 \\ 3 & 1 \end{bmatrix}\vec{v} = \vec{0}$

$3v_1 + v_2 = 0 \Rightarrow \vec{v_2} = \begin{bmatrix} 1 \\ -3 \end{bmatrix}$

Eigenvalues and eigenvectors always come in pairs.

Number of Eigenvectors#

Note: Eigenvalues and eigenvectors can only be computed for square matrices.

Dimensionality Reduction and Projection#

The goal of dimensionality reduction is to map high-dimensional data into a lower-dimensional space while preserving as much information as possible.

Projection Formula#

Projecting a data point $\vec{x}$ onto a unit vector $\hat{u}$:

$\text{projection} = (\vec{x} \cdot \hat{u})$

• $\vec{x} \cdot \hat{u}$: the dot product gives the projection value
• Normalizing $\hat{u}$ to a unit vector ($\|\hat{u}\| = 1$) ensures no stretching

The amount of information preserved depends on the direction you project onto. To minimize information loss, you should choose the direction of maximum variance (spread).

Variance and Covariance#

Understanding PCA requires two statistical concepts.

Variance#

How spread out the data is from its mean.

$\text{Var}(X) = \frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2$

Covariance#

Whether two variables tend to move together in the same direction.

$\text{Cov}(X, Y) = \frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})$

Even if the mean and variance are identical, different covariance means the data patterns are completely different.

Covariance Matrix#

A matrix that captures all variances and covariances among multiple variables.

$\Sigma = \begin{bmatrix} \text{Var}(X) & \text{Cov}(X,Y) \\ \text{Cov}(Y,X) & \text{Var}(Y) \end{bmatrix}$

• Diagonal entries: variance of each variable
• Off-diagonal entries: covariance between pairs of variables
• Symmetric matrix: $\text{Cov}(X,Y) = \text{Cov}(Y,X)$

Matrix notation:

$\Sigma = \frac{1}{n}(X - \mu)^T(X - \mu)$

Here $X - \mu$ is the mean-centered (centered) data matrix.

PCA — Principal Component Analysis#

PCA combines projection + eigenvalues/eigenvectors + covariance matrix to reduce dimensionality while preserving as much information as possible.

Core Idea#

The eigenvectors of the covariance matrix = the directions along which the data spreads the most.

The eigenvector corresponding to the largest eigenvalue = the direction containing the most information (the 1st principal component).

PCA Step by Step#

Step 1 — Center the data

Subtract the mean of each column so every feature has zero mean.

$X_{\text{centered}} = X - \mu$

Step 2 — Compute the covariance matrix

$\Sigma = \frac{1}{n}X_{\text{centered}}^T X_{\text{centered}}$

Step 3 — Compute eigenvalues and eigenvectors

$\Sigma \vec{v} = \lambda \vec{v}$

Sort eigenvalues in descending order: ($\lambda_1 \geq \lambda_2 \geq \cdots$)

Step 4 — Build the projection matrix

Select as many eigenvectors as the target number of dimensions to form the projection matrix $V$. (Normalize each eigenvector to a unit vector.)

Step 5 — Project the data

$X_{\text{PCA}} = X_{\text{centered}} \cdot V$

Why Does It Work?#

The covariance matrix captures the direction and magnitude of the data's spread. Viewed as a linear transformation, the eigenvectors are the directions where the transformation only scales without rotating, and the eigenvalues tell us how much it scales.

Projecting onto the direction of the largest eigenvector → maximum variance (information) is preserved.

In practice: scikit-learn's PCA(n_components=2) performs this entire process automatically. It's widely used for facial recognition, noise reduction, and data visualization.

from sklearn.decomposition import PCA
import numpy as np

X = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9],
              [1.9, 2.2], [3.1, 3.0], [2.3, 2.7]])

pca = PCA(n_components=1)
X_pca = pca.fit_transform(X)

print("Eigenvalues:", pca.explained_variance_)
print("Eigenvectors:", pca.components_)
print("Transformed data:", X_pca)

Markov Matrices and Discrete Dynamical Systems#

One practical application of eigenvectors is Markov matrices.

Markov Matrix#

• All entries are non-negative
• Each column sums to 1 — represents state transition probabilities

$M = \begin{bmatrix} 0.9 & 0.2 \\ 0.1 & 0.8 \end{bmatrix}$

(Example: sunny day → stays sunny with 90% probability, sunny day → becomes cloudy with 10% probability)

Updating the State Vector#

From the current state $\vec{s}_t$, the next state is:

$\vec{s}_{t+1} = M \cdot \vec{s}_t$

Repeated application causes the state vector to converge (reach equilibrium).

$\vec{s}_\infty = \text{equilibrium vector}$

Equilibrium vector = eigenvector corresponding to eigenvalue 1

Markov matrices always have eigenvalue 1, so the long-run steady state can be found as an eigenvector.

In practice: Google's PageRank algorithm uses a Markov matrix to rank web pages.

Full Summary#

Quiz#

Q1. Given $\det(A) = 4$ and $\det(B) = -3$, find $\det(AB)$ and $\det(A^{-1})$.

Q2. Can the set $\{(1, 2),\ (4, 8)\}$ form a basis for $\mathbb{R}^2$?

Q3. Find the eigenvalues of the following matrix.

$A = \begin{bmatrix} 3 & 1 \\ 0 & 2 \end{bmatrix}$

Q4. Why do we choose the eigenvector corresponding to the largest eigenvalue in PCA?

Q5. The equilibrium vector of a Markov matrix $M$ is the eigenvector of what?

This concludes the Linear Algebra Basics series. Here's a recap of the journey from Chapter 1 to Chapter 4: