72 Gradient Descent Interview Questions and Answers (2026)

Gradient descent is an iterative way to minimize a function: from your current position, find the direction in which the loss decreases fastest and take a small step that way, repeating until you reach a low point.

• The hiker-in-fog analogy:

  • You can't see the valley floor, only the slope under your feet, so you step downhill and re-check the slope after each step.

  • The loss surface is the landscape; the parameters are your coordinates.

• Two ingredients:

  • Direction: the negative gradient, which points to the locally steepest decrease.

  • Step size: the learning rate. Too small means slow progress. Too large means you overshoot and may diverge.

• It is purely local:

  • It uses only first-order (slope) information, so it can settle in local minima or stall at saddle points and plateaus.

  • For convex losses, any minimum it reaches is the global one.

• Why it dominates ML: each step is cheap (one gradient), and it scales to millions of parameters where second-order methods like Newton's become impractical.

Each step moves the parameters against the gradient, scaled by the learning rate: θ ← θ − η ∇θ J(θ).

• Terms:

  • θ: the parameter vector (weights, biases).

  • η: the learning rate, a positive scalar step size.

  • ∇θ J(θ): the gradient of the loss with respect to the parameters, evaluated at the current θ.

• Why the minus sign: the gradient points uphill, so subtracting it decreases the loss for a small enough η.

• Per-parameter view: θ_j ← θ_j − η ∂J/∂θ_j, applied to all parameters simultaneously, using gradients computed at the old θ.

• Variants only change how the gradient is estimated:

  • Batch: the full dataset.

  • Stochastic: one example.

  • Mini-batch: a subset.

  • Momentum and Adam modify the step built from that gradient, but the core idea is unchanged.

Locally, the change in a function along a unit direction u is the dot product ∇f · u. A dot product is most negative when u points exactly opposite the gradient, so −∇f gives the steepest decrease.

• First-order Taylor expansion:

  • f(x + εu) ≈ f(x) + ε ∇f(x) · u, for small ε.

  • So the directional derivative is D_u f = ∇f · u.

• Minimize over unit vectors:

  1. ∇f · u = ‖∇f‖ ‖u‖ cos θ = ‖∇f‖ cos θ.

  2. cos θ = −1 when θ = 180°, which means u = −∇f / ‖∇f‖.

  3. The rate of decrease there is ‖∇f‖, the steepest possible. By the same argument, +∇f is the steepest ascent.

• Geometric view: the gradient is perpendicular to the level sets (contours), so moving along it crosses contours as fast as possible.

• Caveats:

  • The result is only local: it holds for an infinitesimal step, which is why large learning rates can overshoot.

  • "Steepest" assumes the Euclidean norm. Under other metrics (e.g. Newton's method or natural gradient), the steepest direction differs.

Gradient descent stops when further progress is negligible, a budget runs out, or (in ML) generalization stops improving. In practice several criteria are combined.

• Convergence-based:

  • Gradient norm: stop when ‖∇J‖ < ε, meaning you are near a stationary point.

  • Loss change: stop when |J_t − J_{t−1}| (absolute or relative) falls below a tolerance.

  • Parameter change: stop when ‖θ_t − θ_{t−1}‖ < ε.

• Budget-based:

  • A maximum number of iterations or epochs.

  • A wall-clock or compute limit.

• Generalization-based:

  • Early stopping: monitor validation loss and stop after it fails to improve for a set number of evaluations (patience), then restore the best checkpoint.

  • It also acts as a regularizer against overfitting.

• Failure guards: abort on NaN/inf or exploding loss, which signals divergence (often a learning rate set too high).

• Caveats:

  • With SGD, loss and gradients are noisy, so use smoothed or epoch-level metrics.

  • A small gradient can also mean a saddle point or plateau, not a minimum.

Backpropagation computes the gradients; gradient descent uses them to update the parameters. One is a differentiation algorithm, the other an optimization algorithm.

• Backpropagation:

  • Applies the chain rule backward through the computation graph to get ∂L/∂w for every weight. This is reverse-mode automatic differentiation.

  • Efficient: one backward pass costs about the same as a forward pass, regardless of parameter count.

  • It says nothing about how to change the weights.

• Gradient descent:

  • The update rule θ ← θ − η∇L.

  • It doesn't care how the gradient was obtained: analytically, by finite differences, or by backprop.

• They are decoupled:

  • Backprop gradients can feed other optimizers (Adam, RMSprop, L-BFGS).

  • Gradient descent applies to models with no network at all (e.g. linear or logistic regression).

• In PyTorch the split is explicit:

  • loss.backward() is backprop.

  • optimizer.step() is the gradient descent update.

Each component is a partial derivative ∂J/∂θ_i. It measures how fast the loss changes when you nudge only that one parameter and hold all the others fixed.

• Sign gives the direction:

  • Positive: increasing θ_i increases the loss, so gradient descent decreases it.

  • Negative: the opposite.

• Magnitude gives the sensitivity:

  • A large |∂J/∂θ_i| means the loss is very sensitive to that parameter locally, so it receives a bigger update.

  • Near zero means it has little local influence at present.

• Together, the components form the vector of steepest ascent, and each parameter's update is proportional to its own component.

• Practical implications:

  • Components can differ by orders of magnitude across parameters (poor feature scaling, deep layers). This causes zig-zagging and motivates feature normalization and adaptive optimizers like Adam.

  • Persistently tiny components in early layers signal vanishing gradients.

At a stationary point the gradient is the zero vector: every partial derivative equals 0. As a result, the update θ − η∇J leaves the parameters unchanged.

• Stationary does not mean minimum:

  • A stationary point can be a local minimum, a local maximum, or a saddle point.

  • Second-order information (the Hessian) distinguishes them:

• Classifying with the Hessian:

  • All eigenvalues positive (positive definite): local minimum.

  • All negative: local maximum.

  • Mixed signs: saddle point.

  • Some zero: the test is inconclusive.

• Relevance to deep learning:

  • In high dimensions, saddle points are far more common than poor local minima.

  • Plain GD slows down near them because the gradient approaches zero.

  • SGD noise and momentum help escape.

• In practice the gradient only gets close to zero, which is why ‖∇J‖ < ε is used as a stopping criterion.

• For convex functions, any stationary point is a global minimum.

A batch is the subset of examples used to compute one gradient estimate. An iteration is one parameter update made using one batch. An epoch is one full pass through the entire training set.

• Batch (size B):

  • B = N gives batch GD.

  • B = 1 gives SGD.

  • 1 < B < N gives mini-batch GD, the usual choice (e.g. 32 to 512).

• Iteration (step): One forward pass, backward pass, and optimizer update on a single batch.

• Epoch: Every example is seen once. Data is typically reshuffled between epochs.

• The relationship:

  • Iterations per epoch = ⌈N / B⌉.

  • Example: N = 10,000 and B = 100 gives 100 iterations per epoch, so 10 epochs equals 1,000 updates.

• Why it matters:

  • Schedules, logging, and early stopping may be defined per step or per epoch.

  • Changing B changes the number of updates per epoch, so learning rates often need retuning.

Shuffling keeps each mini-batch a roughly random sample of the data, so its gradient stays an unbiased, representative estimate of the full gradient. It also stops the optimizer from seeing the same fixed batch sequence every epoch.

• Preserves the unbiasedness assumption:

  • SGD theory assumes batches are i.i.d. draws. Data stored in order (sorted by class, time, or source) breaks this, and batch gradients become systematically skewed.

  • Example: if a dataset is sorted by label, a batch of all "cats" pulls the weights toward "cat", and the next batch pulls them back. The result is oscillation instead of progress.

• Avoids order artifacts across epochs:

  • With a fixed order, the same examples always land in the same batches at the same point in the epoch. The model can then drift toward whatever it saw last (a recency bias).

  • The same batch sequence also produces correlated, repeating update patterns, which can create cycles that slow convergence.

• Varies the gradient noise: New batch compositions each epoch give fresh noise. That helps exploration and acts as mild regularization, which improves generalization.

• Exceptions: Time series and curriculum learning deliberately keep an order. Even then, you usually shuffle whole windows or sequences rather than individual timesteps.

• In practice: DataLoader(shuffle=True) in PyTorch, or dataset.shuffle(buffer_size) in tf.data.

Mini-batch gradient descent updates parameters using the average gradient over a small random subset (batch) of the training data. It sits between full-batch GD and single-example SGD, and it is the default way neural networks are trained.

• Update rule:

  • θ ← θ − η · (1/B) Σᵢ∈batch ∇L(xᵢ, θ)

  • One epoch is about N/B updates.

• Algorithm:

  1. Shuffle the data each epoch.

  2. Split it into batches of size B (typically 32 to 512).

  3. For each batch, compute the loss, backpropagate, and take a step.

• Why it is the sweet spot:

  • Compared with full-batch, it gives many cheap updates per epoch and fits in memory.

  • Compared with pure SGD, it has lower gradient variance and uses vectorized, parallel GPU math efficiently.

  • It keeps some noise, which helps escape saddles and sharp minima.

• Batch size is a hyperparameter that interacts with the learning rate. Optimizers such as momentum and Adam are applied on top of mini-batch gradients.

Full-batch gradient descent computes the exact gradient over the whole dataset, giving stable but expensive steps; SGD uses one example (or a small batch), giving cheap but noisy steps. The trade-off is gradient accuracy and stability versus per-step cost and update frequency.

• Full-batch GD:

  • Exact gradient, smooth monotone descent on convex problems; deterministic and easy to analyze.

  • Cost per step is O(N): one update per full pass, memory-heavy, impractical for large datasets.

  • Can settle into sharp minima or saddle regions since there is no noise to escape them.

• SGD:

  • Cost per step is O(1) in N, so many updates per epoch and fast early progress.

  • Gradient is an unbiased but high-variance estimate: the loss curve is noisy and it oscillates near the minimum unless the learning rate decays.

  • Noise acts as implicit regularization and helps escape saddle points and poor local minima.

• In practice: mini-batch (e.g. 32 to 1024) is the compromise: Reduces variance (roughly by 1/batch size) while staying cheap, and exploits GPU vectorization.

Momentum treats the parameters like a heavy ball rolling downhill. It accumulates a velocity from past gradients, so consistent directions speed up and oscillating directions cancel out.

• The update:

  • v = β·v + g, then θ -= η·v, typically with β = 0.9.

  • The velocity is an exponentially weighted sum of past gradients.

• Damps zig-zagging in ravines: In steep directions the gradient flips sign every step, so those components cancel in v.

• Accelerates along consistent directions:

  • In shallow but consistent directions the gradients add up.

  • The effective step approaches η/(1-β), which is about 10× at β = 0.9.

• Smooths minibatch noise:

  • Averaging over recent steps reduces the variance of the update direction.

  • Inertia helps carry the parameters across plateaus and small bumps.

• Nesterov variant:

  • It evaluates the gradient at the look-ahead point θ - η·β·v.

  • This corrects overshoot earlier and is often slightly more stable.

Too large a learning rate makes each step overshoot the minimum. Training then oscillates, the loss rises instead of falling, and in the worst case it diverges to inf or NaN. Slightly too large rates show up as noisy, stalled loss.

• Why it diverges:

  • Consider a quadratic with curvature λ. Gradient descent multiplies the error by (1 - η·λ) each step.

  • If η > 2/λ, that factor's magnitude exceeds 1 and the error grows every step.

  • The sharpest direction (largest Hessian eigenvalue) sets the stability limit.

• Symptoms by severity:

  1. Mild: the loss plateaus at a high, noisy level and bounces around the valley instead of settling.

  2. Moderate: zig-zagging, loss spikes, and dead ReLUs from huge weight updates.

  3. Severe: exploding loss and gradients, ending in NaN.

• Fixes:

  • Lower the learning rate, or find a good one with a learning-rate range test.

  • Use warmup: start small while curvature and Adam's moment estimates settle.

  • Add gradient clipping and normalization layers.

• Nuance:

  • A large but still stable learning rate is often beneficial.

  • It adds implicit regularization and biases training toward flatter minima (the "edge of stability" regime).

  • This is why schedules start high and decay.

Step decay keeps the learning rate constant for a stretch of epochs and then cuts it by a fixed factor. Each phase makes fast progress at its current scale until the loss plateaus, and the drop then lets the optimizer descend into finer detail of the basin.

• Form: lr = lr0 * gamma ** (epoch // step_size): Classic choice: divide by 10 at fixed milestones (e.g. epochs 30, 60, 90 for ResNet on ImageNet), or gamma=0.5 more often.

• Intuition:

  • At a fixed LR, SGD reaches an equilibrium where gradient noise balances progress, so the loss plateaus.

  • Cutting the LR shrinks the noise and step size, allowing a lower equilibrium, which shows up as a sharp drop in loss right after each step.

  • It is coarse to fine: explore broadly first, then refine.

• Pros: simple, interpretable, and well proven on vision benchmarks.

• Cons:

  • Extra hyperparameters (milestones, factor) that depend on the training length.

  • Abrupt changes. Smooth schedules (cosine) often match or beat it with less tuning.

  • The adaptive variant ReduceLROnPlateau drops the LR when the validation metric stalls rather than at fixed epochs.

On a convex surface any local minimum is the global minimum, so following the gradient is guaranteed to reach the best solution. A non-convex surface can have many local minima, saddle points, and plateaus, so the result depends on initialization, step sizes, and noise.

• Convex:

  • Definition: the line segment between any two points lies on or above the function; equivalently, the Hessian is positive semi-definite everywhere.

  • Zero gradient implies global optimum, with provable convergence rates.

  • Initialization affects speed, not the final loss value.

  • Examples: linear regression (MSE), logistic regression, SVM.

• Non-convex:

  • Zero gradient can be a local min, local max, or saddle.

  • Only guarantee: convergence to a stationary point.

  • Different seeds can find different solutions; symmetries (permuting hidden units) create many equivalent minima.

  • Examples: neural networks, matrix factorization, k-means objective.

• Practical consequence:

  • Convex problems can use exact or second-order solvers and stop confidently.

  • Non-convex problems rely on good init, momentum, SGD noise, normalization, and LR schedules to reach good regions.

GPU memory has a fixed part that does not depend on batch size (weights, gradients, optimizer state) and a part that grows roughly linearly with batch size: the activations saved for backpropagation. For large models and inputs, activations usually dominate, so batch size is the main lever on memory.

• Independent of batch size:

  • Parameters, their gradients, and optimizer state.

  • Adam keeps 2 extra copies of the parameters, so the fixed part is about 4× parameter memory.

• Scales with batch size:

  • Intermediate activations from every layer are stored during the forward pass for use in the backward pass.

  • Activation memory is about B × (per-sample activations).

  • Per-sample activations grow with input size, such as image resolution or sequence length. Naive attention grows quadratically in sequence length.

  • Temporary workspaces (for example cuDNN buffers) also grow with B.

• Rough formula:

  • Memory ≈ fixed + B·a, so doubling B roughly doubles activation memory, not total memory.

  • Usage peaks at the start of the backward pass, when all activations are held at once.

• Ways to fit a larger effective batch:

  • Mixed precision (fp16 or bf16) roughly halves activation size.

  • Activation checkpointing recomputes activations in the backward pass instead of storing them.

  • Gradient accumulation splits a large batch into micro-batches.

Mini-batches turn many small sequential updates into a few large parallel matrix operations. GPUs and TPUs run those efficiently, and averaging over a batch also reduces gradient noise compared with single-example SGD.

• Hardware utilization:

  • Accelerators have thousands of cores. A batch of size 1 leaves most of them idle and is dominated by kernel-launch and memory-transfer overhead.

  • Batching turns per-example matrix-vector products into matrix-matrix products (GEMM). Throughput per example rises dramatically: a batch of 64 often costs barely more wall time than a batch of 1.

• Better gradient estimates: Variance shrinks roughly as 1/B. Updates are more stable, so you can use a larger learning rate.

• Still cheap per step compared with full-batch: You get many updates per epoch, enough noise to help generalization, and memory that stays bounded.

• Works well with normalization and distributed training: BatchNorm needs batch statistics. Data parallelism splits a batch across devices.

• Trade-off: very large batches show diminishing returns and can generalize worse. Typical sizes are 32 to 512 per device, often powers of 2 to align with memory.

Each mini-batch gradient is the true gradient plus random noise. That noise occasionally pushes parameters uphill or sideways, so the optimizer can climb out of narrow, shallow basins that exact gradient descent would settle into.

• Full-batch GD is deterministic: Once it is inside a basin, every step points downhill, so it stays there. That holds even for poor minima and near saddle points where the gradient is close to zero.

• SGD noise acts as random perturbations:

  • Noise scale grows roughly with learning rate / batch size. A sharp, narrow minimum is unstable under this jitter, because a small kick sends the loss sharply up and the iterate bounces out.

  • A wide, flat basin tolerates the noise, so SGD tends to settle there.

  • Loosely, this resembles simulated annealing or Langevin dynamics: noise is a temperature that lets the optimizer explore.

• Escaping saddle points: In high dimensions, saddles are far more common than bad local minima. Noise breaks the symmetry so the iterate slides off along a negative-curvature direction.

• Why it matters: Flat minima are empirically linked to better generalization. This is part of why small batches sometimes generalize better than very large ones.

• Caveat: noise also prevents exact convergence. That is why the learning rate is decayed late in training, which lowers the temperature and lets the optimizer settle.

For i.i.d. samples, the variance of the mini-batch gradient falls inversely with batch size: Var ≈ σ²/B. The standard deviation, which sets the noise level, therefore falls only as 1/√B.

• Why 1/B:

  • The batch gradient is the average of B independent per-example gradients, each with variance σ².

  • Averaging B independent terms divides the variance by B.

  • When sampling without replacement from a finite set of N examples, a correction factor (N−B)/(N−1) applies, and variance reaches 0 at B = N (full batch).

• Diminishing returns:

  • Doubling B doubles compute but cuts noise (std) by only about 29%.

  • Past a "critical batch size", extra samples barely speed up convergence per unit of compute.

• Interaction with learning rate: Effective noise scales like η/B. Lower variance lets you raise the learning rate. That is the basis of the linear scaling rule: scale η with B, using warmup.

• Trade-off:

  • Small B gives noisy but cheap steps and an implicit regularization effect.

  • Large B gives precise but expensive steps and less exploration.

Full-batch gradient descent gives exact gradients but is slow and memory-hungry. Because it is noise-free, it also lacks the exploration and regularization that stochastic noise provides.

• Compute and memory cost:

  • Each update requires a pass over every example. For large datasets this means very few updates per unit of time.

  • Activations for the whole dataset rarely fit in memory, so you need gradient accumulation, which is just as slow.

• Redundant work: Real data is redundant, so a small batch already gives a good gradient direction. Spending N examples per step wastes compute that could have bought many more updates.

• No gradient noise:

  • Deterministic descent can stall near saddle points and settles into whatever basin it starts in, often a sharp minimum.

  • It tends to generalize worse than small-batch SGD (the generalization gap).

• Learning-rate tuning and conditioning: The step size is bounded by the sharpest curvature (η < 2/λ_max). Progress along flat directions is then slow on ill-conditioned problems.

• Other practical issues: It cannot handle streaming or online data, and the full set must be revisited for every update.

• When it is fine: small datasets and convex problems, such as logistic regression with L-BFGS, where exact gradients enable second-order or line-search methods.

If the batch is sampled uniformly at random, each example's gradient appears with equal probability. The expected value of the batch-average gradient is then exactly the full-dataset average gradient.

• The math:

  • The true gradient is g = (1/N) Σⱼ ∇Lⱼ.

  • For a uniformly drawn index i, E[∇Lᵢ] = Σⱼ (1/N) ∇Lⱼ = g.

  • The batch gradient ĝ = (1/B) Σᵢ ∇Lᵢ, so by linearity of expectation E[ĝ] = g.

  • This holds for sampling with or without replacement.

• Unbiased does not mean exact: Any single batch gradient is noisy, with variance about σ²/B. It is correct only on average, and that is enough for SGD convergence guarantees with a decaying learning rate.

• When it breaks:

  • Non-random batching, such as unshuffled sorted data, introduces bias.

  • Importance or weighted sampling introduces bias unless you reweight the gradients by 1/(N·pᵢ).

  • Using a sum instead of a mean only rescales the gradient and does not create bias. BatchNorm makes the per-example losses depend on the batch, so the estimate is only approximately unbiased.

• Note: the gradient is unbiased for the training loss, not for the true population risk, except on a single pass over fresh data.

Offline mini-batch training repeatedly samples a fixed dataset to minimize empirical risk. Online SGD updates on each new example as it arrives, never revisits it, and in effect minimizes expected risk on a possibly shifting data stream.

• Data assumptions:

  • Offline training uses a fixed, finite dataset that is shuffled and revisited over many epochs. Batches are i.i.d. samples of the training set.

  • Online training uses a stream in which each example is seen once, often in arrival order. The distribution may drift over time (non-stationary).

• What is being optimized:

  • Offline training minimizes training loss, so overfitting is possible and you rely on a held-out validation set.

  • Online training follows fresh samples, so each gradient is an unbiased estimate of the population gradient. Evaluation is "predict, then update" (progressive validation or regret).

• Learning-rate behavior:

  • Offline training decays the learning rate to converge to a fixed point.

  • Online training often keeps a non-zero or adaptive rate so it can keep tracking drift. The trade-off is stability versus adaptability, plus a risk of catastrophic forgetting.

• Practical differences:

  • Online training uses constant memory, needs no stored dataset, and gives instant model updates. Typical uses are ad click prediction and recommendations.

  • Online training is noisier and sensitive to bad or adversarial data. In production, "online" usually means small streaming mini-batches rather than strictly one example at a time.

Standard momentum computes the gradient at the current parameters and then adds velocity; Nesterov first takes the momentum step and computes the gradient at that look-ahead point, so it can correct the velocity before overshooting.

• Standard (heavy-ball) momentum:

  • v = μ·v − η·∇f(θ); θ = θ + v

  • Gradient reflects where you are, not where velocity is taking you, so it reacts late and overshoots.

• Nesterov (NAG):

  • v = μ·v − η·∇f(θ + μ·v); θ = θ + v

  • If the look-ahead point is past the minimum, its gradient points back and damps the step early: less oscillation.

• Why it matters:

  • For smooth convex functions NAG achieves the optimal O(1/t²) rate vs O(1/t) for plain GD.

  • Same cost per step (one gradient); in practice implemented via a reparameterized form, e.g. torch.optim.SGD(..., nesterov=True).

In a ravine the gradient is dominated by the steep direction, so plain GD bounces wall to wall while crawling along the shallow floor. Momentum averages gradients over time: oscillating components cancel and the consistent along-ravine component accumulates.

• The problem: ill-conditioning: High Hessian condition number (largest/smallest curvature) forces a small learning rate to avoid divergence in the steep direction, making progress in the flat direction tiny.

• How momentum fixes it:

  • Velocity v = μ·v − η·g is an exponential average of past gradients.

  • Across the ravine, gradient signs alternate and cancel, damping oscillation.

  • Along the ravine, gradients agree and add up, so effective step grows toward η/(1−μ) (10x for μ=0.9).

• Theory: on quadratics, tuned heavy-ball momentum improves convergence dependence from κ to √κ.

• Caveat: too high μ causes overshoot and its own oscillation; Nesterov reduces this.

AdaGrad divides the learning rate by the square root of the sum of all past squared gradients; that sum only grows, so the effective learning rate decays monotonically toward zero and training stalls. RMSProp replaces the sum with an exponential moving average so the denominator tracks recent gradient magnitudes.

• AdaGrad:

  • G = G + g²; θ = θ − η·g / (√G + ε)

  • Good for sparse convex problems (rare features get larger steps), but in long non-convex training (deep nets) steps shrink too early.

• RMSProp:

  • E = ρ·E + (1−ρ)·g²; θ = θ − η·g / (√E + ε), typically ρ≈0.9.

  • Old gradients are forgotten (window of roughly 1/(1−ρ) steps), so the step size stays usable and adapts as the loss landscape changes.

• Keeps AdaGrad's benefit: per-parameter scaling that normalizes steep vs flat directions.

Adam keeps two exponential moving averages per parameter: the first moment (mean gradient) acts as momentum for direction, and the second moment (mean squared gradient) acts like RMSProp to scale the step size. The update is momentum divided by the RMS of gradients, with bias correction.

• Momentum part: m = β₁·m + (1−β₁)·g (β₁≈0.9) smooths noisy gradients and accelerates consistent directions.

• RMSProp part: v = β₂·v + (1−β₂)·g² (β₂≈0.999) gives per-parameter adaptive learning rates: large-gradient parameters get smaller steps.

• Combined update:

  1. Bias-correct: m̂ = m/(1−β₁ᵗ), v̂ = v/(1−β₂ᵗ)

  2. Step: θ = θ − η·m̂/(√v̂ + ε)

• Effect: step magnitude is roughly bounded by η and invariant to gradient scale, making Adam robust with little tuning.

AdaGrad divides each parameter's step by the square root of that parameter's accumulated sum of squared gradients. Parameters that have seen large or frequent gradients get smaller steps, and rarely updated parameters keep relatively large steps.

• The update rule:

  1. Accumulate per parameter: G_t = G_{t-1} + g_t² (elementwise).

  2. Step: θ_t = θ_{t-1} - η · g_t / (√G_t + ε).

  3. The effective learning rate η / √G_t is different for every coordinate.

• Why it helps:

  • It balances steep and flat directions automatically, which acts as a cheap diagonal preconditioner.

  • Rare features (sparse NLP or click-through-rate inputs) keep a large effective step, so they still learn.

  • It needs less learning-rate tuning than plain SGD.

• Main weakness: the learning rate only shrinks:

  • G_t never decreases, so the effective learning rate decays roughly like 1/√t.

  • In long, non-convex deep-learning runs, training can stall before it converges.

  • RMSProp and Adam fix this by replacing the running sum with an exponential moving average of g².

Choose SGD with momentum when final generalization matters most and a proven tuning recipe exists (classic vision CNNs), or when memory is tight. Adam converges faster with less tuning, but on some tasks it reaches solutions that generalize slightly worse.

• Generalization:

  • On image classification (ResNets on ImageNet, CIFAR), a well-tuned SGD with momentum often matches or beats Adam on test accuracy.

  • Wilson et al. (2017) showed that adaptive methods can find solutions that generalize worse.

  • AdamW narrows this gap considerably.

• Established recipes:

  • Many vision baselines use a known setup: learning rate 0.1, momentum 0.9, step or cosine decay, and weight decay.

  • Reproducing those baselines is easiest with the optimizer they were tuned for.

• Memory and simplicity:

  • Momentum stores 1 extra buffer per parameter, versus 2 for Adam.

  • It has fewer hyperparameters that interact with each other.

  • Its behavior is easier to reason about, for example for large-batch scaling with LARS.

• When to prefer Adam or AdamW instead:

  • Transformers and LLMs, where SGD trains poorly because gradient scales vary widely across layers.

  • Sparse gradients, such as embeddings and recommender models.

  • Fast prototyping, or when you have little budget for learning-rate tuning.

• Rule of thumb: CNN with a known recipe, use SGD with momentum; transformers, sparse features or a quick start, use AdamW.

Epsilon is a small constant added to the denominator √v + ε. It prevents division by zero and caps the step size when the second-moment estimate is tiny. Its value also shifts the optimizer between adaptive and SGD-like behavior.

• Numerical safety:

  • Parameters with near-zero gradient history would otherwise get huge or infinite steps.

  • With epsilon, the per-coordinate step is bounded by roughly η·|m|/ε.

• Controls how adaptive the optimizer is:

  • Small ε (default 1e-8): fully per-parameter scaling.

  • Large ε, where ε ≫ √v: the denominator is roughly constant, so Adam behaves like SGD with momentum at learning rate η/ε.

  • Some setups tune it upward for stability: 1e-6 in RoBERTa-style training, and up to 0.1 or 1.0 suggested for some vision models.

• Mixed-precision pitfall:

  • 1e-8 underflows to zero in fp16.

  • Keep optimizer state in fp32, or use a larger ε, to avoid NaNs and exploding steps.

• Implementation detail:

  • Where ε is added (inside or outside the sqrt, before or after bias correction) differs across frameworks.

  • So the same ε value is not always exactly equivalent between libraries.

Adam keeps two extra state tensors per parameter: the first moment m and the second moment v. In fp32 that is +8 bytes per parameter, versus 0 extra for vanilla SGD and +4 bytes for SGD with momentum.

• fp32 accounting (bytes per parameter):

  • Vanilla SGD: weights 4 + gradients 4 = 8.

  • SGD with momentum: 8 + velocity 4 = 12.

  • Adam: 8 + m 4 + v 4 = 16, so 2× SGD's total.

• Mixed-precision training (ZeRO paper accounting):

  • fp16 weights 2 + fp16 gradients 2 + fp32 master weights 4 + m 4 + v 4 = 16 bytes.

  • Optimizer state makes up 12 of those 16 bytes.

• Scale example: For a 7B-parameter model, m and v alone take about 56 GB in fp32, before counting activations.

• Mitigations:

  • 8-bit Adam (bitsandbytes): quantizes the optimizer state.

  • Adafactor: stores a factored row and column second moment.

  • Lion: keeps only one state tensor.

  • ZeRO or FSDP: shards optimizer state across GPUs.

Adaptive optimizers scale each parameter's step by its own gradient history. Parameters tied to rare features, which seldom receive gradient, keep a large effective learning rate, while frequently updated parameters are damped. A single global learning rate can't strike that balance.

• The problem with a single learning rate:

  • With SGD, a learning rate that is safe for frequent features (common words, popular items) is far too small for rare ones.

  • Rare features get only a handful of updates, so they barely move.

• How adaptive scaling fixes it:

  • For rare features, the denominator √v stays small, so each update they do get is a large step.

  • For frequent features, √v is large, so the steps stay small and stable.

  • The result is roughly equalized progress across features, an idea from information geometry.

• Where this matters:

  • Word embeddings, click-through-rate and recommender models, and NLP with long-tail vocabularies.

  • AdaGrad was motivated by exactly this setting, and FTRL-style optimizers dominate large sparse logistic models.

• Practical notes:

  • Updating dense moment state for every embedding row is wasteful.

  • Sparse or lazy variants (torch.optim.SparseAdam) update only the rows present in the batch.

The LR range test trains briefly while increasing the learning rate exponentially from tiny to huge, logs the loss at each step, and picks a rate from the region where the loss falls fastest, well before it starts to blow up.

• Procedure:

  1. Start from a very small LR (e.g. 1e-7) and multiply it every mini-batch up to a large value (e.g. 10), over a few hundred iterations.

  2. Record the (usually exponentially smoothed) training loss against LR on a log axis.

  3. Stop once the loss diverges (e.g. exceeds about 4x its minimum).

• Reading the curve:

  • Flat region at low LR: steps are too small to make progress.

  • Steep descending region: the LR is effective.

  • Loss minimum, then a sharp rise: the LR is now unstable.

• Choosing the value:

  • Pick the point of steepest negative slope, or roughly 1/10 of the LR at the loss minimum (the minimum itself is already on the edge of instability).

  • For one-cycle, the LR near the minimum (slightly below it) is often used as the max LR.

• Why it works: one cheap sweep probes the stability boundary of the loss landscape, replacing a full grid search.

• Caveats:

  • It is a heuristic: the curve is noisy and depends on batch size, optimizer, and initialization.

  • It only reflects early-training dynamics, so you still need a schedule afterwards. Reset the model weights after the test.

Warmup ramps the learning rate up from near zero over the first steps so the model isn't hit with large updates while its weights, gradients, and optimizer statistics are still unreliable. This prevents early divergence and lets you use a higher peak LR.

• Early training is unstable:

  • Random initialization can produce large, poorly conditioned gradients. A full-size step can throw weights into a bad region or cause loss spikes or NaNs.

  • Curvature tends to be sharp early on and flattens as training proceeds, so a small LR is needed at first.

• Adaptive optimizers need warmup: Adam's second-moment estimate v is computed from very few samples early on, so it has high variance. Tiny v values give huge effective steps. RAdam was designed to address exactly this.

• Large-batch and Transformer training:

  • With linear LR scaling for large batches, the target LR is too large to apply from step 0 (Goyal et al., ImageNet in 1 hour).

  • Post-LayerNorm Transformers are notoriously unstable without warmup.

• Typical form: linear increase over a few hundred to a few thousand steps (often 1 to 10% of training), then a decay schedule (cosine, linear, inverse square root).

The shape of the training loss curve reveals the learning rate. Too high gives spikes, oscillation, or divergence. Too low gives slow, nearly linear decrease. A good LR gives a fast initial drop followed by a smooth plateau.

• LR too high:

  • Loss explodes to inf/NaN or increases: steps overshoot the minimum.

  • Large, persistent oscillations or sudden spikes, with no downward trend.

  • Quick early drop, then a plateau at a high value: the optimizer bounces around a basin instead of descending into it.

• LR too low:

  • Loss decreases slowly and almost linearly, with no sign of converging within budget.

  • The curve is very smooth but reaches a poor value; it may get stuck in a sharp or poor region.

• Well tuned: steep initial decrease bending into a gradual plateau (roughly exponential decay), with modest mini-batch noise.

• Distinguishing from other issues:

  • Noise alone may come from a small batch size. Look at the trend after smoothing.

  • Low train loss with rising validation loss is overfitting, not an LR problem.

  • A drop right after a scheduled LR decrease means the previous LR was too high for that phase.

• Confirm by rerunning short trials at about 3x higher and 3x lower LR, or by running an LR range test.

The ideal step size changes during training. Early on, large steps make fast progress and explore. Later, small steps are needed to settle into a minimum instead of bouncing around it. A constant LR forces a compromise that is either slow or unable to converge precisely.

• The noise floor of SGD:

  • With mini-batch gradients, a constant LR makes iterates hover in a region around the minimum, and the region's size grows with the LR. Decaying the LR shrinks that region.

  • Convergence theory for SGD requires the conditions Σ η_t = ∞ and Σ η_t² < ∞, i.e. a decaying LR.

• Speed early, precision late:

  • A high LR early covers distance quickly and its noise helps escape sharp minima and saddle points (which often generalizes better).

  • A low LR late refines the solution. The characteristic loss drop after an LR decay shows this.

• Stability at the start: warmup avoids divergence while statistics and curvature are unfavorable.

• Common schedules: step decay, exponential, cosine, linear, warmup plus decay, one-cycle, ReduceLROnPlateau.

• Adaptive optimizers (Adam) still benefit: they rescale per parameter but still use a global LR whose decay improves the final loss.

Step decay lowers the LR monotonically in discrete jumps. Cosine annealing with warm restarts (SGDR) decays the LR smoothly along a cosine curve to a minimum, then periodically resets it back to the maximum and starts a new cycle.

• Shape of the schedule:

  • Step: piecewise constant, with sudden drops at milestones.

  • Cosine: η_t = η_min + ½(η_max − η_min)(1 + cos(π·T_cur/T_i)). The LR stays high for a while, decreases fastest mid-cycle, and flattens near the end.

• Restarts (non-monotonic):

  • Jumping back to a high LR kicks the model out of the current basin, which encourages exploration and helps escape sharp minima.

  • Cycle length can grow each time (T_mult, e.g. 2), giving longer refinement later.

  • Each cycle's end is a well-converged model, which enables cheap snapshot ensembles.

• Tuning: cosine needs only the max/min LR and the cycle length, instead of hand-picked milestones and a decay factor.

• Practice: plain cosine decay without restarts (usually after warmup) is now the most common default, especially for Transformers.

One-cycle (Leslie Smith) runs a single cycle over the whole training run. The LR ramps up from low to a high maximum, then anneals down to far below the starting value, while momentum moves in the opposite direction. This gives fast training and good regularization in fewer epochs ("super-convergence").

• Phases:

  1. Warm-up (around 25 to 45% of steps): LR increases from max_lr / div_factor to max_lr.

  2. Annealing: LR decreases back down (cosine or linear).

  3. Final phase: LR drops to a tiny value (final_div_factor) for fine convergence.

• Inverse momentum cycling: Momentum drops (e.g. 0.95 to 0.85) while the LR rises, then rises back as the LR falls. This keeps the effective step size stable when the LR is high.

• Why it works:

  • The high-LR middle acts as a regularizer and moves quickly through flat regions toward wide minima.

  • The ramp-up gives warmup-like stability. The long anneal provides the precision.

• Picking max_lr: use the LR range test (near the loss minimum, slightly below divergence).

• Note: the schedule is tied to the total number of steps, so you must know the training length in advance and call scheduler.step() per batch.

Clipping by value caps each gradient component independently, which can change the gradient's direction. Clipping by norm rescales the whole gradient vector when its norm exceeds a threshold, which preserves direction and only shrinks magnitude.

• Clip by value:

  • Each component becomes clamp(g_i, -c, c).

  • Large components are cut while small ones are untouched, so the vector is distorted. It no longer points along the true descent direction.

  • It is simple, but the threshold depends on each parameter's scale.

• Clip by norm:

  • If ||g|| > c, set g ← g · c / ||g||. Otherwise leave g unchanged.

  • Direction is preserved, and only the step size is bounded.

  • It is usually applied to the global norm across all parameters, so the relative magnitudes between layers are kept.

• In practice:

  • Norm clipping is the standard choice for RNNs and transformers (e.g. max_norm=1.0).

  • Value clipping is rarer: it is used when individual extreme entries are the concern.

  • With mixed precision, unscale the gradients before clipping, or the threshold is meaningless.

Backprop multiplies many local derivatives (Jacobians) together via the chain rule. If those factors are consistently smaller than 1 in magnitude, the product shrinks exponentially with depth or sequence length. Early layers then receive near-zero gradients and barely learn.

• Saturating activations:

  • The sigmoid derivative is at most 0.25, so stacking 10 layers gives at most 0.25^10 ≈ 1e-6.

  • Sigmoid and tanh have near-zero derivatives in their saturated regions.

• Weight scale and initialization: Weights that are too small shrink the signal at every layer, in both the forward and backward pass.

• Recurrent networks:

  • The same recurrent matrix is multiplied once per time step.

  • If its largest singular value is below 1, gradients from distant steps vanish, so the model can't learn long-range dependencies.

• Fixes:

  • Non-saturating activations: ReLU, GELU.

  • Variance-preserving initialization: Xavier/Glorot, He.

  • Residual connections: an identity path lets gradient flow directly.

  • Normalization layers: BatchNorm, LayerNorm.

  • Gated RNNs: LSTM/GRU, whose additive cell state avoids repeated multiplication.

Exploding gradients are the mirror image of vanishing ones. The chain rule multiplies many Jacobians, and when their magnitudes are consistently greater than 1, the gradient grows exponentially with depth or time steps. The result is huge updates that wreck the weights.

• Large weights or poor initialization: Each layer amplifies the backward signal.

• Recurrent networks: If the recurrent matrix's largest singular value is above 1, gradients blow up over long sequences.

• Learning rate feedback loop:

  • Too high a learning rate makes a large step that lands in a steeper region.

  • That produces a larger gradient and an even larger step, which compounds.

• Loss landscape cliffs: Sharp regions of very high curvature, common in RNNs, produce sudden enormous gradients.

• Symptoms:

  • Loss spikes, then inf/NaN values.

  • Wildly growing weight norms.

• Fixes:

  • Gradient clipping by norm.

  • Proper initialization.

  • Lower learning rate plus warmup.

  • Normalization layers and residual connections.

  • LSTM/GRU for sequences.

NaN loss almost always comes from one of three sources:

• numerical instability, where values overflow to inf (via exploding gradients or too high a learning rate) and then inf - inf or 0 · inf gives NaN;

• an invalid math operation;

• bad data.

The common causes in each category:

• Optimization blow-up:

  • Learning rate too high, or no warmup.

  • Exploding gradients with no clipping.

• Unsafe math:

  • log(0) in hand-written cross-entropy.

  • Division by zero, e.g. normalizing by a zero variance or zero count.

  • sqrt of a negative number, or the gradient of sqrt at 0.

  • exp overflow in a naive softmax.

  • Fix: use numerically stable fused ops such as log_softmax and F.cross_entropy on logits, plus an epsilon in denominators.

• Precision issues:

  • FP16 overflows above roughly 65504.

  • An optimizer epsilon such as Adam's 1e-8 underflows to zero in FP16.

  • Missing or broken loss scaling.

• Data problems:

  • NaN/inf values in inputs or labels.

  • Unnormalized features with huge magnitudes.

  • Label indices out of range.

• How to localize it:

  • Assert torch.isfinite on each batch.

  • Log the gradient norm every step.

  • Use torch.autograd.set_detect_anomaly(True) to find the op that produced the first NaN.

Gradient checking verifies that your analytic (backprop) gradients are correct by comparing them to a numerical approximation from finite differences. It catches bugs in hand-written backward passes or custom ops that would otherwise just make training silently worse.

• The approximation:

  • Centered difference: (f(θ+ε) − f(θ−ε)) / 2ε.

  • Its error is O(ε²), better than the O(ε) of a one-sided difference.

• Comparing the two:

  • Use relative error: |g_a − g_n| / max(|g_a|, |g_n|).

  • Around 1e-7 or lower (in float64) is good; around 1e-2 almost certainly means a bug.

• Practical rules:

  • Use float64 and an ε around 1e-5 to 1e-6. Too small an ε causes round-off error; too large causes truncation error.

  • Disable randomness: no dropout, fixed data.

  • Watch for non-differentiable kinks such as ReLU at 0 and max.

  • Check on a tiny model or a few sampled parameters.

• Why it is debugging-only:

  • It costs two forward passes per parameter, which is far too slow for training.

  • In PyTorch, torch.autograd.gradcheck automates it for custom autograd functions.

Divergence within ten steps is almost never a subtle modeling issue. Start by instrumenting the first steps: check the initial loss against its expected value, and log the learning rate actually applied and the gradient norm per step. Then cut the learning rate by about 10x to confirm or rule out step size as the culprit.

• Check step 0:

  • For K-class classification, the initial loss should be about ln(K).

  • If it is already huge or non-finite, the problem is in the data, initialization, or loss, not the optimizer.

• Verify the effective learning rate:

  • Look for warmup not applied, a scheduler bug, or the wrong units.

  • Look for a learning rate scaled up for a large batch, or a loss summed rather than averaged (gradients scale with batch size).

• Watch gradient norm and update/weight ratio:

  • A ratio far above about 1e-3 per step means updates are too large.

  • If a 10x lower learning rate or clipping fixes it, add warmup and clipping.

• Sanity-check the inputs:

  • Look for unnormalized features, NaN/inf values, and corrupt labels.

  • Make sure probabilities aren't passed where logits are expected.

• Isolate further if needed:

  • Try overfitting a single batch in FP32 with plain SGD.

  • If that is stable, reintroduce mixed precision, the optimizer, and the schedule one at a time.

Convexity is what turns gradient descent from a heuristic into a method with guarantees: on a convex function every local minimum is global, so with a suitable step size GD provably converges to the optimal value at a known rate. Without convexity, the only general guarantee is reaching a stationary point.

• Convex (plus L-smooth gradient):

  • Any point with zero gradient is a global minimum, so "gradient is zero" means "done".

  • With step η ≤ 1/L, f(x_k) − f* = O(1/k): sublinear but global.

• Strongly convex: Unique minimizer and linear (geometric) convergence: error shrinks by a constant factor like (1 − μ/L) each step.

• Non-convex:

  • Guarantee is only that the gradient norm goes to zero: min ||∇f||² = O(1/k).

  • The stationary point may be a local minimum, saddle, or plateau, and the result depends on initialization.

• Practical implication:

  • Linear/logistic regression and SVMs: convex, so tuning is about speed, not about finding the answer.

  • Deep nets: non-convex, yet they train well because of overparameterization and benign landscapes, which is an empirical and partly theoretical observation rather than a convexity guarantee.

A small gradient alone can't tell you, because the gradient is near zero at a minimum, a saddle, and a plateau. You have to look at the curvature (Hessian) and at how the loss behaves over time and under perturbation.

• Curvature is the definitive test:

  • Minimum: all Hessian eigenvalues are positive (or non-negative), so the loss rises in every direction.

  • Plateau: many eigenvalues are near zero, so the surface is flat over a wide region and the gradient stays tiny.

  • Saddle: some eigenvalues are negative, so there is a descent direction the optimizer hasn't found yet.

  • For large nets, estimate the top or bottom eigenvalues with Hessian-vector products plus power iteration or Lanczos. You never form the full Hessian.

• Loss-curve signals:

  • A plateau is often followed by a sudden drop. A minimum shows a stable loss that also responds normally to LR decay.

  • Compare against what's achievable. An overparameterized net stuck at a training loss far above near-zero is more likely on a plateau than at a good minimum.

• Active probes:

  • Perturb the weights or briefly raise the learning rate. At a minimum, the loss returns to the same value. On a plateau or saddle, it often escapes to a lower loss.

  • Plot the loss along a random or top-eigenvector direction (1D interpolation) to see whether it is bowl-shaped or flat.

• Practical remedies for plateaus: momentum or Adam (which keep moving through flat regions), LR warm restarts, and better initialization or normalization.

Newton's method needs the full n×n Hessian and must solve a linear system with it at every step. With n in the millions to billions, that is impossible in memory and compute, and the method is also ill-suited to nonconvex, noisy objectives.

• Cost scaling:

  1. Memory: O(n²) entries. For n = 10⁹, that is 10¹⁸ numbers (exabytes).

  2. Computing it: roughly n Hessian-vector products, each costing about one backward pass.

  3. Solving H·d = −∇f: O(n³) for a direct factorization. Gradient descent, by contrast, costs O(n) per step.

• Problems beyond cost:

  • In nonconvex losses the Hessian is indefinite, so the Newton step can point uphill or be attracted to saddle points.

  • Mini-batch Hessians are very noisy, which undermines the quadratic model.

  • Near-zero eigenvalues make H⁻¹ blow up, so damping or trust regions are required.

• What people use instead:

  • Hessian-free / truncated Newton: conjugate gradient using only Hessian-vector products (Pearlmutter trick, O(n) each).

  • Structured approximations: K-FAC and Shampoo (Kronecker-factored), and L-BFGS (low-rank).

  • Diagonal per-parameter scaling, as in Adam, as a cheap curvature proxy.

Gradient descent moves all parameters at once along the negative gradient. Coordinate descent moves along one coordinate axis (or block) at a time, often minimizing exactly along it, with every other variable held fixed.

• Direction choice:

  • GD: direction −∇f(x), the steepest descent in Euclidean norm.

  • CD: direction is a standard basis vector e_i. The coordinate can be chosen in several ways:

  • Cyclic: sweep coordinates in order.

  • Randomized: sample the coordinate uniformly or in proportion to its smoothness L_i.

  • Greedy (Gauss–Southwell): pick the coordinate with the largest |∂f/∂x_i|.

• Step size:

  • CD can do an exact 1D minimization, which is often closed form, or take a step of 1/L_i per coordinate.

  • Each coordinate thus gets its own step size, instead of one global LR limited by the sharpest direction.

• Where CD shines: Problems with cheap 1D subproblems and separable non-smooth regularizers. For example, the Lasso uses a soft-thresholding update per coordinate (as in glmnet and scikit-learn's Lasso), and liblinear uses CD on the SVM dual.

• Limitations:

  • It zig-zags when variables are strongly coupled, because it can't move diagonally.

  • It can stall on non-smooth, non-separable objectives.

  • It is sequential by nature, which makes it a poor fit for GPU-parallel deep learning.

First-order methods use only the gradient (a local linear model of the loss) and need a hand-chosen step size. Second-order methods also use curvature (the Hessian, a local quadratic model), which lets them choose both the direction and the scale of the step.

• Update rules:

  • First-order: x ← x − η∇f(x).

  • Newton: x ← x − H⁻¹∇f(x), which jumps to the minimum of the local quadratic.

• Behavior:

  • First-order is sensitive to conditioning. One LR must suit both sharp and flat directions, which causes zig-zagging, and convergence is linear at best.

  • Second-order rescales each direction by its curvature. It is affine-invariant and converges quadratically near a minimum.

• Cost:

  • First-order: O(n) memory and compute per step, which scales to billions of parameters.

  • Second-order: O(n²) memory and O(n³) to solve. It also needs care with indefinite Hessians in nonconvex problems.

• Middle ground:

  • Quasi-Newton methods (BFGS, L-BFGS), Gauss-Newton/natural gradient, K-FAC, and Shampoo approximate curvature cheaply.

  • Adam's per-parameter scaling is still first-order (it uses gradient moments), but it loosely mimics a diagonal preconditioner.

Newton's method uses the Hessian (curvature) as well as the gradient, so it fits an exact quadratic model of the function and jumps straight to that model's minimum. On a quadratic, the model is the function itself, so one Newton step lands exactly on the minimum. Gradient descent only uses slope and must take many small steps.

• The one-step result:

  • For f(x) = ½xᵀAx − bᵀx with A positive definite, the gradient is Ax − b and the Hessian is A.

  • The Newton step x − A⁻¹(Ax − b) = A⁻¹b is the exact minimizer, from any starting point.

• Why gradient descent needs many steps:

  • The negative gradient points toward steepest descent, not toward the minimum. In elongated valleys it zig-zags across the narrow direction.

  • The step size must stay below 2/λ_max for stability. That makes progress along flat directions (small λ_min) very slow.

  • The error shrinks by about (κ−1)/(κ+1) per step, where κ = λ_max/λ_min is the condition number. Ill-conditioned problems therefore need many iterations.

• Newton rescales every direction by its curvature:

  • Multiplying by H⁻¹ takes big steps in flat directions and small steps in steep ones. This removes the dependence on κ.

  • It is affine invariant: rotating or stretching the coordinates does not change its behavior.

  • On non-quadratic functions it converges quadratically near the minimum.

• The cost:

  • Storing the Hessian takes O(n²) memory, and solving with it takes O(n³) time per step.

  • It needs a positive-definite Hessian. At saddle points or in non-convex regions it can move toward a maximum or a saddle.

  • For large models, the cheaper options are quasi-Newton methods (L-BFGS) or diagonal preconditioners such as Adam.

A line search picks the step size at each iteration by searching along the descent direction. This removes the need to hand-tune a fixed learning rate and guarantees the loss actually decreases. Exact line search finds the best step along the line; backtracking settles for a step that is good enough, at much lower cost.

• Exact line search:

  • Chooses t = argmin_t f(x − t∇f(x)).

  • On a quadratic it has a closed form: t = gᵀg / gᵀAg.

  • Each gradient ends up orthogonal to the previous one, which produces the classic zig-zag path. Even optimal step sizes do not fix poor conditioning.

  • It is usually too expensive on general functions, since every step requires a 1-D optimization.

• Backtracking line search (Armijo condition):

  1. Start with a trial step such as t = 1.

  2. While f(x − t g) > f(x) − α t ‖g‖², shrink the step: t ← βt. Typical values are α ≈ 0.3 and β ≈ 0.5.

  3. Take the step with the accepted t.

• Why it matters:

  • It guarantees sufficient decrease and convergence without knowing the Lipschitz constant L.

  • It adapts the step to local curvature: larger steps in flat regions, smaller ones in steep regions.

• Why deep learning rarely uses it:

  • Mini-batch losses are noisy, so the sufficient-decrease test becomes unreliable.

  • Each trial step costs an extra forward pass.

  • Learning-rate schedules and adaptive optimizers fill this role in practice.

A larger batch cuts wall-clock time while it raises hardware utilization and still reduces the number of steps needed to reach a target loss. Past the hardware saturation point and the "critical batch size", the gains flatten out, so speedup has diminishing returns.

• Time per step:

  • Small batches underuse the GPU, so step time grows sublinearly with B and throughput (samples/sec) rises.

  • Once compute is saturated, step time grows roughly linearly with B and throughput plateaus.

• Steps needed to converge:

  • Below the critical batch size (McCandlish et al., 2018), doubling B roughly halves the steps needed. This gives near-perfect scaling.

  • Above it, the gradient is already accurate. Extra samples barely reduce the step count, so total compute is wasted.

• Distributed training: A larger global batch amortizes gradient all-reduce communication over more computation, which improves scaling efficiency across devices.

• Caveats:

  • Gradient accumulation simulates a big batch but gives no speedup. Micro-batches still run sequentially.

  • Large batches may need more epochs or careful tuning to match accuracy, which can erase the time saved.

Gradient accumulation runs several small micro-batches forward and backward and adds up their gradients. It calls the optimizer only once every k micro-batches. Activation memory is set by the micro-batch size, but the update matches a batch k times larger.

• How it works:

  1. For each micro-batch, compute the loss and divide it by k, so the sum equals the mean over the full batch.

  2. Call loss.backward(). PyTorch adds the result into .grad instead of overwriting it.

  3. After k micro-batches, call optimizer.step() and then optimizer.zero_grad().

• Why memory stays low:

  • Activations are freed after each micro-batch's backward pass.

  • Only one extra gradient-sized buffer persists between micro-batches.

• Caveats:

  • No speedup: micro-batches run sequentially, and wall-clock time is about the same as small-batch training per sample.

  • BatchNorm computes statistics per micro-batch, so results are not exactly identical to a true large batch. LayerNorm or GroupNorm avoid this.

  • With DDP, wrap the non-final micro-batches in model.no_sync() to skip redundant all-reduces.

  • Learning-rate schedules should count optimizer steps, not micro-batches.

Online SGD updates the model on each new example (or small batch) as it arrives, so it naturally tracks a drifting distribution, provided the learning rate stays non-zero so recent data keeps influencing the weights.

• Why it adapts:

  • Each update moves weights toward minimizing loss on current data; older data's influence decays geometrically over subsequent updates.

  • Effectively an exponentially weighted fit to recent history.

• Learning rate is the key knob:

  • Decaying schedules (e.g. 1/t) needed for convergence on stationary data would freeze the model under drift.

  • Use a constant or lower-bounded rate: larger tracks drift faster but is noisier (stability vs plasticity trade-off).

  • Adaptive optimizers with exponential averages (RMSProp, Adam) also forget old statistics; AdaGrad's ever-growing accumulator does not.

• Complementary techniques:

  • Drift detection (monitor loss/metrics) to bump the learning rate or retrain.

  • Sliding windows or sample reweighting favoring recent data.

  • Regularization or replay buffers to limit catastrophic forgetting of still-relevant patterns.

Adam's moment estimates are exponential moving averages initialized at zero, so in early steps they are biased toward zero. Bias correction divides by (1 − βᵗ) to make them unbiased estimates of the true gradient moments.

• Where the bias comes from:

  • m_t = β₁·m_{t−1} + (1−β₁)·g_t with m_0 = 0 gives E[m_t] ≈ (1 − β₁ᵗ)·E[g].

  • At t=1 with β₁=0.9, m is only 10% of the gradient; with β₂=0.999, v is only 0.1% of g².

• Why it matters most for v:

  • β₂ is close to 1, so v stays tiny for hundreds of steps; dividing by √v would produce huge, unstable early updates.

  • Since m and v are biased differently, their ratio is distorted too.

• The fix:

  • m̂ = m / (1 − β₁ᵗ), v̂ = v / (1 − β₂ᵗ)

  • Correction factor goes to 1 as t grows, so it only affects early training (partly why warmup is sometimes still used).