RMSProp Optimizer Step with Momentum
Adam Optimizer With Bias Correction
AdaGrad Optimizer With Per-Coordinate Step Sizes
AdamW Versus Adam With L2 Penalty
Linear Warm-up with Cosine Annealing
Stateful Step and Exponential LR Scheduler
Learning-Rate Range Test With Smoothed Loss
One-Cycle Learning Rate and Momentum Schedule
Linear LR Scaling With Warm-up
ReduceLROnPlateau Learning Rate Scheduler
Cosine Annealing With Warm Restarts
Line Search Second Order Methods
Cyclic Coordinate Descent for Convex Quadratics
Backtracking Line Search with Armijo Condition
Damped Multivariate Newton Step
L-BFGS with Two-Loop Recursion
Training Stability Robustness
Gradient Clipping by Value and Global Norm
Early Stopping with Best Checkpoint Restore
Gradient Accumulation Over K Micro-Batches
Polyak and Stochastic Weight Averaging
AdamW With Decay and No-Decay Parameter Groups
Training Loss Monitor With LR Recommendation
Dynamic Loss Scaler for Mixed Precision
Guarded Momentum SGD With LR Backoff
SGD with Classical Momentum
Nesterov Accelerated Gradient Step
Gradient Descent Fundamentals
Mini-Batch SGD with Gradient Variance Tracking
Streaming L2-Regularised SGD with Welford Standardisation
Conditioning Convergence Analysis
Feature Scaling and Gradient Descent Convergence
Step-Size Stability on a Quadratic Objective
Escaping a Saddle with Noisy Gradient Descent