An opinionated introduction to deep learning

Note: This post is a writeup of a set of slides from May 2025. The text is essentially unchanged from the slides, just reformatted as a post; the original slides are available as a PDF.

Outline

We will begin by formalizing deep learning as an optimization problem and briefly cover:

terminology
training
gradient descent
backpropagation

Then we will discuss how deep learning is not quite an optimization problem, at least in the way we formulated it. How do our goals vary from the simple formulation and what can we do about it?

Note: This is “Eric’s take” rather than anything comprehensive. Everything discussed has many alternatives, variants, and generalizations — it’s a big field.

Deep learning as optimization

Deep learning terminology

A deep learning model (a network) is a parametrized function $𝑓 (𝑥; 𝜃)$ , where:

𝜃∈ℝ𝑝 are the parameters of the network. In 2025, 𝑝 is typically in the millions or billions, although smaller networks can still be practical and useful.
- Llama 3 70B, an open-weights large language model (LLM) from Meta, has 70B parameters
𝑥∈ℝ𝑛 is the input to the network. While we often think of inputs as living in other domains, we will assume they have been injected into ℝ𝑛 prior to being passed to the network.
- For a small 28x28 pixel grayscale image, $𝑛 = 28 * 28 = 784$ .
- For Llama 3 70B, $𝑛 = 8192$ token ids (its “context length”); the first layer embeds each into $ℝ^{8192}$ (its “embedding dimension”), $8192^{2} \approx 67 𝑀$ numbers in all
𝑓 is the architecture, typically a composition of simple functions called layers.
- The architecture can be quite important, as a “good” architecture can make a problem tractable with much less data.
𝑦=𝑓(𝑥;𝜃)∈ℝ𝑚 is the output of the network.
- The output dimension $𝑚$ varies widely. In classic tasks like regression or classification, $𝑚$ is usually much less than $𝑛$ ; for example, $𝑚 = 1$ in binary classification.
- However for generative models and LLMs, $𝑚$ can be quite large; for Llama 3 70B, for example, $𝑚 \approx 1 𝐵$ .

Training

Training is optimizing a network $𝑓$ over its parameters $𝜃 \in ℝ^{𝑝}$ to minimize a loss function $ℓ : ℝ^{𝑚} \times ℝ^{𝑚} \to ℝ$ .

To train a network, one needs a “train set” consisting of pairs $(𝑥, 𝑦) \in ℝ^{𝑛} \times ℝ^{𝑚}$ . Here, $𝑥$ are the inputs to the network, and $𝑦$ are the corresponding desired outputs.

Typically, training proceeds by attempting to solve the following optimization problem:

\underset{𝜃 \in ℝ^{𝑝}}{minimize} \sum_{(𝑥, 𝑦) \in train set} ℓ (𝑓 (𝑥; 𝜃), 𝑦)

That is, we want to optimize over the parameters $𝜃 \in ℝ^{𝑝}$ to minimize the loss $ℓ$ of $𝑓$ over the train set.

This approach is sometimes called “empirical risk minimization”: we seek to minimize the “risk” (loss) empirically over the data we have (the train set).

Gradient descent

Often, the optimization problem

\underset{𝜃 \in ℝ^{𝑝}}{minimize} \sum_{(𝑥, 𝑦) \in train set} ℓ (𝑓 (𝑥; 𝜃), 𝑦)

is solved by variants of gradient descent. Gradient descent is a simple iterative algorithm. To minimize some function $𝐹 : ℝ^{𝑝} \to ℝ$ over some values $𝜃 \in ℝ^{𝑝}$ , we start with some initial point $𝜃_{0}$ , then iteratively compute

𝜃_{𝑖} ≔ 𝜃_{𝑖 - 1} - 𝛾 \nabla 𝐹 (𝜃_{𝑖 - 1})

where $𝛾 > 0$ is a hyperparameterA hyperparameter is a value that is chosen before training and fixed, rather than optimized over called the learning rate (or step size). Here $\nabla$ is the gradient of $𝐹$ with respect to $𝜃$ , also denoted $\nabla_{𝜃}$ for clarity sometimes.

In some scenarios, this procedure can be guaranteed to converge, meaning the sequence of $𝜃_{𝑖}$ tends toward some fixed limit. This is neither necessary nor sufficient for it to be useful.

Applying gradient descent to deep learning models

Recall that the optimization problem we want to solve is:

\underset{𝜃 \in ℝ^{𝑝}}{minimize} \sum_{(𝑥, 𝑦) \in train set} ℓ (𝑓 (𝑥; 𝜃), 𝑦)

To solve this with gradient descent (directly/naively), our function $𝐹$ must be

𝐹 (𝜃) = \sum_{(𝑥, 𝑦) \in train set} ℓ (𝑓 (𝑥; 𝜃), 𝑦)

That is, to perform one update of gradient descent we pass through the entire train set. In practice, we often perform stochastic gradient descent instead:

Initialize with some point $𝜃_{0}$
At step $𝑖$ , draw a random pair $(𝑥_{𝑖}, 𝑦_{𝑖})$ from the train set at random
Update the parameters as $𝜃_{𝑖} ≔ 𝜃_{𝑖 - 1} - 𝛾 \nabla_{𝜃} ℓ (𝑓 (𝑥_{𝑖}; 𝜃_{𝑖 - 1}), 𝑦_{𝑖})$

Advantages of stochastic gradient descent (SGD)

Stochastic gradient descent has several important advantages over ordinary gradient descent:

if the train set is very large (or infinite, in the case of randomly generated data), we might never finish completing a single update. Instead, with SGD, a single update can be finished relatively quickly.
the random noise incurred by performing individual random updates can actually be helpful. The model “jumps around” more in parameter space over the course of training, which can help escape local minimaThis is a thing people say, and it makes sense, but I haven’t read any papers about it nor ever tried full-dataset gradient descent..

SGD variants

In practice, there are often many more adjustments made to SGD. For example:

rather than performing updates with a single sample $(𝑥_{𝑖}, 𝑦_{𝑖})$ , a batch (or equivalently, minibatch) of such $𝑏$ samples is used, so the update step becomes $𝜃_{𝑖} ≔ 𝜃_{𝑖 - 1} - 𝛾 \nabla_{𝜃} \sum_{𝑗 = 1}^{𝑏} ℓ (𝑓 (𝑥_{𝑖_{𝑗}}; 𝜃_{𝑖 - 1}), 𝑦_{𝑖_{𝑗}}),$ where ${(𝑥_{𝑖_{𝑗}}, 𝑦_{𝑖_{𝑗}}) : 𝑗 = 1, \dots, 𝑏}$ is the $𝑖$ th batch of training data.
the learning rate $𝛾$ may be set to decay over time (“learning rate scheduler”) or depend on previous updates in some deterministic fashion (adaptive step size) or be divided by the batch size (convention)
a “momentum” term may be added to the update rule, consisting of an exponentially decaying average of previous gradients
many more…

Review: Jacobians

Let $𝑓 : ℝ^{𝑛} \to ℝ^{𝑚}$ be a vector-valued function. Then we can write it as $𝑚$ scalar-valued functions $𝑓_{1}, \dots, 𝑓_{𝑚}$ as:

𝑓 (𝜃) = (\begin{matrix} 𝑓_{1} (𝜃) \\ 𝑓_{2} (𝜃) \\ ⋮ \\ 𝑓_{𝑚} (𝜃) \end{matrix})

Then the Jacobian matrix of $𝑓$ at $𝜃$ is defined as:

𝐽_{𝑓} (𝜃) = (\begin{matrix} \frac{𝜕 𝑓_{1}}{𝜕 𝜃_{1}} & \frac{𝜕 𝑓_{1}}{𝜕 𝜃_{2}} & \dots & \frac{𝜕 𝑓_{1}}{𝜕 𝜃_{𝑛}} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ \frac{𝜕 𝑓_{𝑚}}{𝜕 𝜃_{1}} & \frac{𝜕 𝑓_{𝑚}}{𝜕 𝜃_{2}} & \dots & \frac{𝜕 𝑓_{𝑚}}{𝜕 𝜃_{𝑛}} \end{matrix})

which is the $𝑚 \times 𝑛$ matrix of partial derivatives.

Computing the gradient

Recall that $𝑓$ is frequently composed of simpler functions called layers. The gradient of $𝑓$ can thus be computed via the chain rule from the gradient of each individual layer.

The chain rule says if $𝑓 = ℎ \circ 𝑔$ , then we can take the derivative with respect to $𝜃$ as

𝐽_{𝑓} (𝜃) = 𝐽_{ℎ} (𝑔 (𝜃)) \cdot 𝐽_{𝑔} (𝜃)

This is a matrix-multiplication between $𝐽_{ℎ} (𝑔 (𝜃))$ and $𝐽_{𝑔} (𝜃)$ . If we had $𝑓 = ℎ_{1} \circ ℎ_{2} \circ \dots \circ ℎ_{𝑘}$ for some value $𝑘$ , then

𝐽_{𝑓} (𝜃) = 𝐽_{ℎ_{1}} (ℎ_{2} (\dots ℎ_{𝑘} (𝜃) \dots)) \cdot 𝐽_{ℎ_{2}} (ℎ_{3} (\dots ℎ_{𝑘} (𝜃) \dots)) \cdot \dots \cdot 𝐽_{ℎ_{𝑘}} (𝜃)

we have $𝑘$ matrix multiplications. The computational complexity of this operation depends on the sizes of all the individual matrices, and the order in which the product is conducted. Since matrix multiplication is associative, $𝐴 𝐵 𝐶 = 𝐴 (𝐵 𝐶) = (𝐴 𝐵) 𝐶$ , we can choose the order to minimize the computation time.

Backpropagation as optimized matrix multiplication associativity

Wikipedia gives the concrete case that if A is a 10 × 30 matrix, B is a 30 × 5 matrix, and C is a 5 × 60 matrix, then

computing $(𝐴 𝐵) 𝐶$ needs (10×30×5) + (10×5×60) = 1500 + 3000 = 4500 operations, while
computing $𝐴 (𝐵 𝐶)$ needs (30×5×60) + (10×30×60) = 9000 + 18000 = 27000 operations.

using that the straightforward multiplication of a matrix that is $𝑋 \times 𝑌$ by a matrix that is $𝑌 \times 𝑍$ requires $𝑋 𝑌 𝑍$ ordinary multiplications and $𝑋 (𝑌 - 1) 𝑍$ ordinary additions.

Backpropagation is the insight that when $𝑚 ≪ 𝑛$ , that is, there are many fewer outputs than inputs, the best order in which to conduct this large matrix product is often from the output layer back toward the input.

Deep learning as human-in-the-loop meta-optimization

Deep learning is not a tractable mechanical optimization problem

The goals of deep learning frequently differ from solving the optimization problem

\underset{𝜃 \in ℝ^{𝑝}}{minimize} \sum_{(𝑥, 𝑦) \in train set} ℓ (𝑓 (𝑥; 𝜃), 𝑦)

as follows:

the loss function is often chosen for computational convenience (nice smooth derivatives). Perhaps the actual evaluation of the output on a sample requires an extensive simulation or is otherwise intractable.
we don’t actually care about the train set! Those are the examples we already labeled, we know the idealized output $𝑦$ for each $𝑥$ . We typically care about performance on new, potentially never-seen-before data.

Typically, what we want to solve is more like:

\underset{𝜃 \in ℝ^{𝑝}}{minimize} \sum_{𝑥 \in real world} weight (𝑥) evaluate (𝑓 (𝑥; 𝜃))

where:

“real world” represents the set of all possible inputs $𝑥$
$weight (𝑥)$ captures that some inputs are more important than others and may have a higher weight. We may not know in advance which these are.
$evaluate (𝑓 (𝑥; 𝜃))$ represents some intractable evaluation like “is $𝑓 (𝑥; 𝜃)$ the same output that a panel of human experts would provide after deliberative review of $𝑥$ ” or “is $𝑓 (𝑥; 𝜃)$ a ‘good answer’ to a plain-text question $𝑥$ encoded as a vector?”

and typically each of those items is infeasible to obtain or compute in general, and we have little hope that we can run some mechanical code (e.g. gradient descent) to solve this.

Deep learning as meta-optimization

So, how do we tackle deep learning problems? We manually approximate each quantity in the previous intractable problem and iterate, iterate, iterate (“grad-student descent”).

Typically, this involves:

collecting a train and test dataset
iterating over update rules (“optimizers”), architectures, loss functions, initialization procedures, hyperparameters, and an increasing variety of other techniques (regularization, data augmentation, self-supervised pretraining, data resampling, ensembles of networks, etc.)
until one finds a network that performs “well” on the test dataset, typically evaluated in many different domain-specific ways (not just an average loss)

One test dataset is not enough

The process of developing a network sometimes “burns” the test dataset: while the network wasn’t trained on it, the optimization of generating the network has encoded dataset-specific properties into the network, such that it won’t perform quite as well on truly unseen data. Alternatively, the test dataset may not be truly representative of real-world data.

So, a truly held-out test dataset must be collected and evaluated. If performance is not up-to-snuff, or the network shows performance issues down the line, the whole process is repeated.

Alternative philosophy: bootstrapping proxies

Another complementary way to see deep learning is as an opportunistic exploitation of proxies to the true problem we want to solve:

Gradient-based optimization is an incredibly effective tool, so let’s change our problem to one we can use it on
Statistical sampling is powerful and the performance of statistical estimates typically depends on the number of samples you have, not the total population size.
- e.g. you can get opinion polls of 300M Americans with ~3% margin of error by surveying just 1,000 people… as long as they are uniformly randomly sampled (they aren’t)
- truly representative test sets don’t need to be huge (the problem is they aren’t truly representative)

Instead of solving intractable problems, find a sequence of similar-but-tractable oneswith “tractable” being more important than “similar”.

Why do all that?

Machine learning approaches (deep learning + classical ML) can take some problems that have been at < 50% solved for decades to 80% solved in days/weeks/months and to 95%+ solved in months/years, and frequently the best performance comes from deep learning models. This approach fundamentally improves the capabilities of computers.

That is not to say we can’t do better than deep learning, nor that deep learning techniques can tackle every problem. We almost certainly can and will do better in time. But deep learning is likely an important step in the journey.

XKCD 1425, Tasks: checking whether a photo is in a national park is easy, checking whether it is of a bird needs a research team and five years — XKCD#1425 (September 2014)