Linear Algebra in Practice: Problems You'll Actually Face in ML

A rank of 30 means there are effectively 30 linearly independent features.

• Of the 50 features, 20 can be expressed as linear combinations of the others → redundant information
• The true dimensionality of the information in this dataset is 30, not 50

Practical implication: Applying PCA can reduce the data from 50 to 30 dimensions with zero information loss. In practice, the top few principal components typically explain most of the variance, so you can often reduce even further below 30.

Since $x_3 = 2x_1 + x_2$, the columns of the data matrix are linearly dependent.

This makes $X^T X$ a singular matrix, so:

$\det(X^T X) = 0 \Rightarrow (X^T X)^{-1} \text{ does not exist}$

The closed-form solution for linear regression, $\hat{w} = (X^T X)^{-1} X^T y$, cannot be computed.

Solutions:

• Remove the redundant feature
• L2 regularization (Ridge): $(X^T X + \lambda I)^{-1}$ — the inverse always exists
• Use PCA to extract only independent components

For a batch matrix $X$ (rows = samples, columns = features):

Step 1 — Mean centering:

$X_{\text{centered}} = X - \mathbf{1}\mu^T$

where $\mu$ is the mean vector of each feature and $\mathbf{1}$ is a ones vector

Step 2 — Divide by standard deviation:

$X_{\text{norm}} = X_{\text{centered}} \oslash \sigma$

This is the same operation as the data centering step in PCA. From a linear algebra perspective, it is a preparatory step for aligning the data along the eigenvector directions of the covariance matrix.

$\det(W) \to 0$ for a weight matrix $W$ means the matrix is approaching singularity.

Problems that follow:

1. Vanishing Gradient: During backpropagation, gradients are propagated via matrix multiplication — multiplying by a near-singular matrix drives the gradient toward 0
2. Loss of representational capacity: As the rank drops, the space the neural network can represent shrinks
3. Numerically unstable inverse: Computing $W^{-1}$ becomes numerically unreliable

Solutions: Batch normalization, residual connections (ResNet), proper weight initialization (Xavier, He initialization)

If $W$ is $4 \times 3$ (4 rows, 3 columns):

• Input dimension: 3 (a 3-dimensional vector $x$)
• Output dimension: 4 (a 4-dimensional vector $y$)

This layer performs a linear transformation from 3-dimensional space → 4-dimensional space.

No inverse: Since $W$ is not square, its inverse does not exist. However, the pseudoinverse $W^+ = (W^T W)^{-1} W^T$ can be used for an approximate inverse transformation.

Intuition: Lifting 3-dimensional data into 4 dimensions is analogous to embedding. The encoder/decoder in an autoencoder uses this kind of non-square transformation.

SVD decomposes a matrix in order of information importance. Larger singular values = directions carrying more information.

Original matrix: $W = \sum_{i=1}^{r} \sigma_i \vec{u}_i \vec{v}_i^T$

Keeping only $k$ terms: $\hat{W} = \sum_{i=1}^{k} \sigma_i \vec{u}_i \vec{v}_i^T$

Compression gain:

• Original storage: $m \times n$ parameters
• Compressed storage: $k(m + n + 1)$ parameters
• When $k \ll \min(m, n)$, the compression ratio is significant

Real-world application: LoRA (Low-Rank Adaptation) uses this principle to fine-tune LLMs efficiently, approximating large weight matrices with two low-rank matrices.

When to choose which:

• Interpretability matters → Feature selection
• Information retention matters → PCA
• Multicollinearity is a problem → PCA is the solution

$\text{Explained variance ratio} = \frac{\lambda_1 + \lambda_2}{\sum_i \lambda_i} = \frac{8.5 + 5.2}{25} = \frac{13.7}{25} = 54.8\%$

Interpretation: The first two principal components alone explain roughly 55% of the total variance in the data.

Practical decision guidelines:

• Typically, choose enough principal components to retain 80–95%+ of explained variance
• On a Scree Plot (eigenvalues plotted in descending order), select the point where the slope sharply flattens (the "elbow")

Use cosine similarity (dot product divided by the product of L2-norms):

$\cos\theta = \frac{\vec{a} \cdot \vec{b}}{\|\vec{a}\|_2 \|\vec{b}\|_2}$

$\vec{a} \cdot \vec{b} = 3 + 5 + 8 + 6 + 5 = 27$

$\|\vec{a}\|_2 = \sqrt{9+1+16+1+25} = \sqrt{52} \approx 7.21$

$\|\vec{b}\|_2 = \sqrt{1+25+4+36+1} = \sqrt{67} \approx 8.19$

$\cos\theta = \frac{27}{7.21 \times 8.19} \approx 0.457$

A cosine similarity of 0.46 is low. The two users have quite different tastes (A prefers action/sci-fi, B prefers romance/drama).

In practice: Collaborative Filtering recommendation systems are built on this principle.

$h_1 = \sigma(W_1 x + b_1) \quad (4 \times 3)(3 \times 1) + (4 \times 1) = (4 \times 1)$

$y = W_2 h_1 + b_2 \quad (2 \times 4)(4 \times 1) + (2 \times 1) = (2 \times 1)$

When processing a full batch ($n$ samples):

$H_1 = \sigma(X W_1^T + \mathbf{1}b_1^T) \quad (n \times 3)(3 \times 4) = (n \times 4)$

$Y = H_1 W_2^T + \mathbf{1}b_2^T \quad (n \times 4)(4 \times 2) = (n \times 2)$

As this shows, the entire neural network forward pass is a sequence of matrix multiplications. GPUs are optimized to parallelize exactly these large-scale matrix multiplications.

Backpropagation is the matrix version of the chain rule.

$\frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial h_1} \cdot \frac{\partial h_1}{\partial W_1}$

Step by step:

1. Compute gradient at the output layer: $\delta_2 = \frac{\partial L}{\partial y}$

2. Backpropagate to layer 1: $\delta_1 = (W_2^T \delta_2) \odot \sigma'(z_1)$

   • $W_2^T$: multiply by the transpose matrix to match dimensions
   • $\odot$: element-wise multiplication (Hadamard product)

3. Gradient for $W_1$: $\frac{\partial L}{\partial W_1} = \delta_1 x^T$

Key insight: The reason transpose matrices appear in backpropagation is that we are tracing the forward-pass linear transformations in reverse. The reverse of a linear transformation = multiplication by its transpose.

• $\det(A) = 4 - 4 = 0$ → Singular, no inverse
• $\det(B) = 6 - 1 = 5 \neq 0$ → Non-singular, inverse exists ✓
• $\det(C) = 0$ → Singular, no inverse

Answer: B

With multicollinearity, $X^T X$ becomes singular and its inverse does not exist.

L2 regularization adds $\lambda$ to the diagonal entries:

$(X^T X + \lambda I)^{-1}$

Adding the identity matrix $I$ shifts every eigenvalue by $\lambda$:

$\lambda_i(X^T X + \lambda I) = \lambda_i(X^T X) + \lambda \geq \lambda > 0$

This guarantees the determinant is always positive and the inverse always exists.

The covariance matrix $\Sigma$ is a symmetric matrix ($\Sigma = \Sigma^T$).

Properties of symmetric matrices:

• Eigenvectors corresponding to distinct eigenvalues are always orthogonal
• This is known as the Spectral Theorem

Because the principal components point in mutually orthogonal directions, each one captures independent, non-overlapping information. This is exactly why PCA automatically eliminates multicollinearity.