ML Calculus Chapter 1: Derivatives and Key Differentiation Rules

Training a machine learning model is ultimately about minimizing a loss function. The mathematical tool that makes this minimization possible is the derivative. In this post, we'll build intuition for what a derivative means, then work through the differentiation rules for the most important functions.

This post is based on Week 1 of DeepLearning.AI's Mathematics for Machine Learning and Data Science — Calculus.

What You'll Learn#

• The intuition behind the derivative — instantaneous rate of change and the slope of a tangent line
• The relationship between slope, maxima, and minima
• Differentiation rules for constant, linear, polynomial, exponential, logarithmic, and trigonometric functions
• Conditions for non-differentiability
• The four core differentiation properties: scalar multiplication, sum, product, and chain rules

---

Why Do We Need Derivatives in Machine Learning?#

Training a machine learning model is an optimization problem.

• Regression: find the line that best explains the data points
• Classification: find the decision boundary that best separates the data

In both cases, we start from an arbitrary line and gradually tweak it toward the best fit. This process is called optimization, and derivatives tell us which direction to move and by how much.

---

What Is a Derivative? — Instantaneous Rate of Change#

Think of a car's speedometer. You can easily calculate the average speed over a full trip.

$\text{Average speed} = \frac{\Delta \text{distance}}{\Delta \text{time}} = \frac{y(t_2) - y(t_1)}{t_2 - t_1}$

But how do you find the speed at a specific instant? You shrink the time interval down to zero.

$\text{Instantaneous speed} = \lim_{\Delta t \to 0} \frac{\Delta y}{\Delta t}$

That is exactly the derivative — the instantaneous rate of change of a function.

Example: Average speed between $t = 10$ and $t = 15$ seconds

$v = \frac{y(15) - y(10)}{15 - 10} = \frac{202 - 122}{5} = 16 \text{ m/s}$

Narrowing to $t = 12$ to $t = 13$ seconds gives an approximation of the instantaneous speed at $t = 12.5$ seconds.

Key analogy: distance = function $f(x)$, speed = derivative $f'(x)$

---

Tangent Lines and Derivatives#

The instantaneous rate of change is equal to the slope of the tangent line at a point on the curve.

A tangent line touches the curve at exactly one point without crossing through it.

$f'(x) = \lim_{\Delta x \to 0} \frac{f(x + \Delta x) - f(x)}{\Delta x}$

$\boxed{\text{Derivative} = \text{Slope of the tangent line}}$

---

Slope, Maxima, and Minima#

Where do a function's maximum and minimum values occur?

Core principle: A function's maximum or minimum occurs at a point where the derivative equals zero (slope = 0).

$f'(x) = 0 \Rightarrow \text{candidate for maximum or minimum}$

In practice: The point where the loss function's derivative equals zero corresponds to the model's optimal parameters. Gradient descent moves toward that point step by step.

---

Derivative Notation#

There are several common ways to write a derivative.

---

Derivatives of Common Functions#

Constant Function#

$f(x) = c \Rightarrow f'(x) = 0$

A constant never changes, so its rate of change is zero.

Linear Function#

$f(x) = ax + b \Rightarrow f'(x) = a$

The slope $a$ is constant everywhere, so the derivative equals $a$ at every point. The constant term $b$ has no effect on the rate of change.

Quadratic Function#

$f(x) = x^2 \Rightarrow f'(x) = 2x$

Intuitive derivation:

$\frac{(x + \Delta x)^2 - x^2}{\Delta x} = \frac{2x\Delta x + (\Delta x)^2}{\Delta x} = 2x + \Delta x \xrightarrow{\Delta x \to 0} 2x$

Higher-Degree Polynomials (Power Rule)#

$f(x) = x^n \Rightarrow f'(x) = nx^{n-1}$

Rule: Bring the exponent down as a coefficient, then reduce the exponent by one.

---

Derivatives of Inverse Functions#

An inverse function undoes what the original function did.

$g = f^{-1} \Rightarrow g'(x) = \frac{1}{f'(g(x))}$

The derivative of an inverse function is the reciprocal of the original function's derivative.

---

Derivatives of Trigonometric Functions#

$\frac{d}{dx}\sin(x) = \cos(x)$

$\frac{d}{dx}\cos(x) = -\sin(x)$

$\frac{d}{dx}\tan(x) = \sec^2(x)$

Intuition: As $\Delta x$ approaches zero, the hypotenuse of the triangle converges to $\Delta x$, and working through the algebra yields the trig differentiation formulas.

---

Euler's Number $e$ and the Derivative of Exponential Functions#

What Is $e$?#

Using compound interest as an analogy:

$e = \lim_{n \to \infty} \left(1 + \frac{1}{n}\right)^n \approx 2.71828\ldots$

As $n$ grows larger, the expression converges to $e$.

Derivative of $e^x$#

$\frac{d}{dx}e^x = e^x$

$e^x$ is the only function that is its own derivative — differentiation leaves it unchanged.

In practice: The derivative of the sigmoid function $\sigma(x) = \frac{1}{1+e^{-x}}$ is central to backpropagation in deep learning.

---

Derivative of the Logarithmic Function#

$\frac{d}{dx}\ln(x) = \frac{1}{x}$

Why: $\ln(x)$ is the inverse of $e^x$, so by the inverse function rule, its derivative is $\frac{1}{x}$.

---

When a Derivative Does Not Exist#

Not every function is differentiable at every point.

Functions like these are called non-differentiable functions.

In practice: The ReLU activation $\max(0, x)$ is non-differentiable at $x=0$, but in practice the network uses a subgradient approximation — treating the derivative as either 0 or 1 — to keep training working.

---

The Four Core Differentiation Properties#

1. Scalar Multiplication Rule#

$\frac{d}{dx}[c \cdot f(x)] = c \cdot f'(x)$

Multiplying a function by a constant multiplies its derivative by the same constant.

$\frac{d}{dx}[5x^3] = 5 \cdot 3x^2 = 15x^2$

2. Sum Rule#

$\frac{d}{dx}[f(x) + g(x)] = f'(x) + g'(x)$

The derivative of a sum equals the sum of the derivatives.

$f(x) = 2x,\ g(x) = x^2 \Rightarrow (f+g)'(x) = 2 + 2x$

Intuition: A child running on a moving train — their speed relative to the ground equals the train's speed plus the child's speed.

3. Product Rule#

$\frac{d}{dx}[f(x) \cdot g(x)] = f'(x)g(x) + f(x)g'(x)$

Example: $f(x) = xe^x$

$f'(x) = (x)' \cdot e^x + x \cdot (e^x)' = e^x + xe^x = e^x(1+x)$

4. Chain Rule#

The differentiation rule for composite functions.

$\frac{d}{dx}[f(g(x))] = f'(g(x)) \cdot g'(x)$

Intuition: If you know how altitude affects temperature and how altitude changes over time:

$\frac{d\text{temperature}}{d\text{time}} = \frac{d\text{temperature}}{d\text{altitude}} \times \frac{d\text{altitude}}{d\text{time}}$

Example: $f(x) = e^{2x}$

$g(x) = 2x,\quad f(g) = e^g$

$f'(x) = e^{2x} \cdot (2x)' = 2e^{2x}$

In practice: Deep learning's backpropagation algorithm is essentially the repeated application of the chain rule to compute the gradient at each layer.

---

Summary of Differentiation Formulas#

---

Quiz#

Q1. Differentiate the following function.

$f(x) = 3x^4 - 2x^2 + 5x - 7$

Show answer

Apply the sum rule, scalar multiplication rule, and power rule:

$f'(x) = 12x^3 - 4x + 5$

---

Q2. Differentiate $f(x) = x^2 e^x$.

Show answer

Product rule: $(fg)' = f'g + fg'$

$f'(x) = 2x \cdot e^x + x^2 \cdot e^x = e^x(2x + x^2) = xe^x(x+2)$

---

Q3. Differentiate $f(x) = \ln(x^2 + 1)$.

Show answer

Chain rule: $g(x) = x^2 + 1$, $f(g) = \ln(g)$

$f'(x) = \frac{1}{x^2+1} \cdot 2x = \frac{2x}{x^2+1}$

---

Q4. Which of the following functions is NOT differentiable at $x = 0$?

1. $f(x) = x^2$
2. $f(x) = |x|$
3. $f(x) = e^x$
4. $f(x) = \sin(x)$

Show answer

Option 2: $f(x) = |x|$

At $x = 0$, the left-hand derivative is $-1$ and the right-hand derivative is $+1$ — they differ, so the derivative does not exist. (This is a cusp.)

---

Q5. Differentiate $f(x) = \sin(3x^2)$.

Show answer

Apply the chain rule twice: $h(x) = 3x^2$, $g(h) = \sin(h)$

$f'(x) = \cos(3x^2) \cdot 6x = 6x\cos(3x^2)$

---

The next post will cover optimization, partial derivatives, gradients, and gradient descent.