Training a model often produces a rather strange scene.

We spend days debating the model architecture, analyze the dataset down to duplication rates and quality, and tune GPU settings to the decimal point. Then the moment comes to choose an optimizer, and the discussion usually ends like this:

optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)

We may not be entirely sure why it is AdamW, but it worked yesterday, it appears in the Hugging Face example, and most importantly, the GPUs are already sitting idle while we contemplate reading another optimizer paper. This is a perfectly reasonable decision. I have started plenty of training runs this way myself.

Following the history of optimizers, however, reveals more than an exhibition of nearly identical algorithms with different names. Each method was an answer to a fairly specific problem encountered by an earlier one.

• The gradient is too noisy → remember past directions with Momentum
• Gradient scales differ across parameters → adjust the stride per coordinate with AdaGrad and RMSProp
• We want both direction and scale → use Adam
• State for a gigantic matrix is too expensive → compress it with Adafactor
• Layers belong to different weight classes → use LARS and LAMB
• We want to model row-column relationships, not isolated coordinates → use Shampoo and SOAP
• We trust direction more than gradient magnitude → use Lion
• We want curvature information without paying full price → use Sophia
• We want to remove an explicit learning-rate schedule → use Schedule-Free
• We want to straighten distorted directions in a matrix update → use Muon

This article will not list every optimizer proposed through August 2026 as though it were a dictionary. There are already too many named variants to count, and including every method used in a single paper would turn the article into a phone book. Instead, we will follow the representative lineage needed to understand modern deep-learning optimizers, while keeping the 2025–2026 proposals separate from broadly validated defaults.

Preliminary: What Does an Optimizer Decide?

Let the model parameters be $\theta$, and let $\mathcal{L}_{B_t}$ be the loss computed from mini-batch $B_t$ at step $t$. The gradient is

$$g_t = \nabla_\theta \mathcal{L}_{B_t}(\theta_t).$$

The gradient points in the direction in which the loss increases most rapidly at the current position. To reduce the loss, we therefore move in the opposite direction.

$$\theta_{t+1} = \theta_t-\eta g_t.$$

Here, $\eta$ is the learning rate. This equation makes an optimizer look like nothing more than a number placed in front of a gradient. Real problems immediately raise several other questions:

1. Is this gradient wobbling merely because of the batch we happened to sample?
2. If previous gradients kept pointing the same way, could we move a little more aggressively?
3. If one parameter has a gradient of $10^{-1}$ and another has a gradient of $10^{-6}$, should they really use the same stride?
4. In a narrow, tilted valley, is correcting each coordinate independently enough?
5. If computing a better direction makes every step three times slower, are we actually training faster?

An optimizer decides more than “where to go.” It determines how much to trust the current gradient, how much history to remember, whether to reason in units of coordinates, layers, or matrices, and how much memory and computation to pay for those decisions.

Unless stated otherwise, $\odot$, squaring, division, and square roots in this article are element-wise operations. Implementations differ in whether momentum includes $(1-\beta)$, whether the learning rate is folded into velocity, and other notational details, so all equations here use one consistent convention. Constant factors can often be absorbed into the learning rate, while the essential behavior remains the same.

1. GD and SGD: How Many People Should Attend the Meeting?

Batch Gradient Descent

Suppose the full dataset contains $N$ samples. Batch Gradient Descent averages the gradients of every sample at each step.

$$g_t = \frac{1}{N} \sum_{i=1}^{N} \nabla_\theta \ell_i(\theta_t),$$ $$\theta_{t+1} = \theta_t-\eta g_t.$$

It is like surveying every employee before changing the company lunch menu. The opinion is stable, but lunch may be over by the time the all-hands meeting ends. In deep learning, where datasets are enormous, reading the entire dataset for every update is simply too expensive.

Stochastic Gradient Descent

Strictly speaking, SGD samples a single example $i_t$ at every step.

$$g_t = \nabla_\theta \ell_{i_t}(\theta_t).$$

This gradient is a noisy estimate of the full gradient. It may point in the correct direction on average, but individual steps can wobble considerably. It is like stopping one citizen to ask for directions, only to discover that they are also a lost tourist.

Mini-batch SGD

What practitioners commonly call SGD is usually mini-batch SGD.

$$g_t = \frac{1}{|B_t|} \sum_{i\in B_t} \nabla_\theta \ell_i(\theta_t).$$

Combining several opinions reduces noise compared with asking one person, and GPUs can process the resulting matrix operations efficiently in parallel. Increasing the batch size stabilizes the gradient estimate but consumes more memory; beyond some point, it also reduces the number of updates made while processing the same amount of data.

Importantly, SGD's wobble is not always bad. Noise can help escape sharp valleys or saddle points and may sometimes improve generalization. A more accurate compass does not guarantee a better journey—especially if reading it takes all day.

2. Momentum and NAG: Put Wheels on the Shopping Cart

Imagine SGD descending a narrow valley. It should consistently travel downhill, but the sideways slope alternates from left to right. Following only the current gradient makes the parameters bounce between the walls in a zigzag.

Heavy-ball Momentum

Momentum uses an exponential moving average of past gradients like a velocity.

$$m_t = \beta m_{t-1} + (1-\beta)g_t,$$ $$\theta_{t+1} = \theta_t-\eta m_t.$$

Gradients that repeatedly point in the same direction accumulate in $m_t$ and build speed. Components whose signs alternate left and right cancel one another. It is much like a shopping cart that keeps rolling after you briefly let go.

The problem is that a shopping cart does not recognize its destination on its own. It may overshoot the minimum because yesterday's enthusiasm is still stored in the wheels. In supermarkets as in optimization, inertia is particularly dangerous near the checkout.

Nesterov Accelerated Gradient

NAG evaluates the gradient not at the current position, but at the lookahead position where momentum is about to carry us.

$$\widetilde\theta_t = \theta_t-\eta\beta m_{t-1},$$ $$m_t = \beta m_{t-1} + (1-\beta) \nabla_\theta \mathcal{L}(\widetilde\theta_t),$$ $$\theta_{t+1} = \theta_t-\eta m_t.$$

If Momentum says, “start running and inspect the situation later,” NAG sends an intern to the cart's expected destination to check the slope first. If the road is already turning uphill there, it can slow down in advance.

Momentum remembers previous directions; NAG uses that memory to ask for a new direction at the future position. This lookahead idea was later combined with Adam in methods such as NAdam.

3. AdaGrad, RMSProp, and AdaDelta: Give Every Parameter Its Own Shoe Size

Momentum smooths the direction, but it still applies the same learning rate to every coordinate. Real parameters do not share a common unit. Some coordinates receive large gradients on every step, while coordinates connected to sparse features may receive a signal only occasionally.

AdaGrad

AdaGrad accumulates squared gradients separately for every coordinate.

$$s_t = s_{t-1}+g_t\odot g_t,$$ $$\theta_{t+1} = \theta_t - \eta \frac{g_t}{\sqrt{s_t}+\epsilon}.$$

A coordinate that frequently receives large gradients grows a large $s_t$, reducing its next stride. A rarely updated coordinate retains a relatively large stride. This is why AdaGrad can be useful for problems with sparse features.

But $s_t$ never decreases. AdaGrad is the manager who remembers every mistake you made as an intern for the rest of your career. As training continues, the denominator keeps growing until every step can become excessively small.

RMSProp

RMSProp replaces the cumulative sum with an exponential moving average of squared gradients.

$$s_t = \rho s_{t-1} + (1-\rho)g_t\odot g_t,$$ $$\theta_{t+1} = \theta_t - \eta \frac{g_t}{\sqrt{s_t}+\epsilon}.$$

The influence of old gradients gradually decays by powers of $\rho$. If AdaGrad's bucket keeps every drop of water forever, RMSProp's bucket has a small hole in the bottom.

Historically, RMSProp became widely known through Geoffrey Hinton's 2012 Coursera lecture materials, rather than through a formal original paper. If your paper search cannot find “the RMSProp paper,” the internet is not necessarily broken.

AdaDelta

AdaDelta tracks the RMS of gradients and the RMS of past parameter updates.

$$E[g^2]_t = \rho E[g^2]_{t-1} + (1-\rho)g_t^2,$$ $$\Delta\theta_t = - \frac{ \operatorname{RMS}[\Delta\theta]_{t-1} }{ \operatorname{RMS}[g]_t } g_t,$$ $$E[\Delta\theta^2]_t = \rho E[\Delta\theta^2]_{t-1} + (1-\rho)\Delta\theta_t^2.$$

Here, $\operatorname{RMS}[x]_t=\sqrt{E[x^2]_t+\epsilon}$. Put simply, AdaDelta converts the units of the gradient into the units of past updates. It attempts to avoid AdaGrad's endlessly shrinking steps while reducing dependence on a global learning rate.

All three methods adjust scales per coordinate. If the coordinate axes themselves are tilted relative to the natural directions of the loss, however, adjusting one coordinate at a time may not be enough. This issue will return when we discuss Shampoo and Muon.

4. Adam and AdamW: Install Both a Compass and a Road-Vibration Gauge

Adam combines the ideas of Momentum and RMSProp. The original Adam paper maintains two central states.

$$m_t = \beta_1m_{t-1} + (1-\beta_1)g_t,$$ $$v_t = \beta_2v_{t-1} + (1-\beta_2)g_t^2.$$

$m_t$ estimates the first moment of the gradient, and $v_t$ estimates the gradient's second raw moment by averaging squared gradients. Although $v_t$ is often called a “variance,” it is not a statistical variance in the strict sense because the mean has not been subtracted.

Because $m_0=v_0=0$, early estimates are biased toward zero. Adam corrects this bias.

$$\widehat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t = \frac{v_t}{1-\beta_2^t}.$$

The final update is

$$\theta_{t+1} = \theta_t - \eta \frac{\widehat m_t} {\sqrt{\widehat v_t}+\epsilon}.$$

$m_t$ remembers “which direction have we recently kept traveling?” while $v_t$ measures “how large have the squared gradients on this coordinate recently been?” It is like equipping a car with both a compass and a gauge for the magnitude of shocks coming from the road.

Adam is convenient because it automatically adjusts the stride for each parameter and usually starts stably even with noisy gradients. The cost is storing both $m_t$ and $v_t$, each as large as the parameters themselves. Two additional FP32 values per parameter make this convenience expensive for a gigantic model.

AdamW: Give Weight Decay Its Own Department

If we implement L2 regularization in Adam by adding $\lambda\theta_t$ to the gradient, this term also passes through Adam's coordinate-wise preconditioning.

$$g_t \leftarrow g_t+\lambda\theta_t.$$

For plain SGD, L2 regularization and weight decay are equivalent under an appropriate relationship between constants. That equivalence breaks for an optimizer such as Adam that divides each coordinate by a different value.

AdamW decouples weight decay from the adaptive gradient update.

$$\theta_{t+1} = (1-\eta\lambda)\theta_t - \eta \frac{\widehat m_t} {\sqrt{\widehat v_t}+\epsilon}.$$

While Adam handles the driving, a separate custodian gradually shrinks the parameters. The W specifically refers to moving decay outside gradient preconditioning, rather than simply enabling a weight-decay option.

What Did Adam's Relatives Change?

A larger family tree does not mean that every relative is unconditionally better than AdamW. AMSGrad fixed an important convergence issue, but that does not make it faster on every practical task; RAdam and AdaBelief also depend on the dataset, model, and schedule. Optimizer-paper titles usually contain the benefit. The receipt contains the conditions.

5. Handling Layer Weight Classes and Optimizer Memory

LARS and LAMB: A Separate Volume Knob for Every Layer

When the batch size becomes very large, we would like to increase the global learning rate and exploit the available throughput. Applying the same absolute step to every layer, however, can produce an enormous relative change in a layer with a small parameter norm and an almost invisible change in a large one.

LARS compares the weight and gradient norms separately for each layer $l$.

$$r_l = \frac{ \lVert w_l\rVert }{ \lVert g_l\rVert + \lambda\lVert w_l\rVert + \epsilon },$$ $$w_l \leftarrow w_l-\eta r_l(g_l+\lambda w_l).$$

In other words, it controls the relative update with respect to the current size of a layer, rather than only the absolute update.

LAMB first constructs an Adam-style direction $u_l$, then applies a layer-wise trust ratio to it.

$$u_l = \frac{\widehat m_l} {\sqrt{\widehat v_l}+\epsilon} + \lambda w_l,$$ $$r_l = \frac{\phi(\lVert w_l\rVert)} {\lVert u_l\rVert+\epsilon}, \qquad w_l \leftarrow w_l-\eta r_lu_l.$$

Giving exactly one kilogram to everyone is exercise for a child and dust to a building. LARS and LAMB install a scale on each layer and adjust the relative update size. Their main purpose is to stabilize large-batch training; they are not magic buttons that automatically win on small batches as well.

Adafactor: Compress a Giant Ledger into Row and Column Receipts

Storing Adam's second moment for a matrix parameter $W\in\mathbb{R}^{m\times n}$ requires another $mn$ values. Adafactor stores only row statistics $r_t$ and column statistics $c_t$ of the squared-gradient matrix, then approximates the full matrix.

$$V_t \approx \frac{r_tc_t^\top} {\operatorname{mean}(r_t)}.$$

Optimizer state drops from $O(mn)$ to $O(m+n)$. It is like discarding the original household ledger and keeping only one receipt of row totals and another of column totals.

Factorization applies naturally only to matrix parameters. Vector parameters require separate state, and actual Adafactor behavior also depends on options such as relative step size, update clipping, and whether momentum is used.

The 2025 method Adam-mini takes a different route: it partitions parameters into blocks informed by Hessian structure and shares a scalar second moment within each block. Meanwhile, 8-bit Adam stores state at low precision, and GaLore or APOLLO projects gradients into a lower-dimensional space. All of these reduce memory, but changing the update rule itself should be distinguished from reducing only the representation or precision of optimizer state.

6. What If We Look at Matrices and Curvature Instead of Coordinates?

Newton, L-BFGS, K-FAC, and Sophia

A gradient tells us only the slope at the current position. It does not say whether the floor is gentle or bends sharply just ahead. Using the Hessian, which describes curvature, yields the following Newton step for a local quadratic model.

$$\theta_{t+1} = \theta_t - \eta H_t^{-1}g_t, \qquad H_t = \nabla_\theta^2\mathcal{L}(\theta_t).$$

In a tilted elliptical valley, $H_t^{-1}$ corrects for curvature in different directions and can produce a more direct step toward the minimum. The problem is that if there are billions of parameters, the Hessian is billions by billions. The terrain-aware glasses are excellent; unfortunately, they cost an entire GPU cluster.

Instead of storing a full Hessian, L-BFGS approximates the action of its inverse from a short history of parameter differences $s_t=\theta_{t+1}-\theta_t$ and gradient differences $y_t=g_{t+1}-g_t$. It can be powerful for small full-batch problems, but multiple function evaluations, history, and line-search conditions become burdensome for noisy mini-batches and enormous neural networks.

K-FAC approximates each layer's Fisher block with two smaller Kronecker factors.

$$F_l \approx A_l\otimes G_l, \qquad F_l^{-1} \approx A_l^{-1}\otimes G_l^{-1}.$$

This imitates natural-gradient geometry much more cheaply, but estimating the factors, taking matrix inverses, damping, and distributed communication still add cost.

Sophia periodically updates a lightweight diagonal Hessian estimate $h_t$ instead of computing the full Hessian.

$$m_t = \beta_1m_{t-1} + (1-\beta_1)g_t,$$ $$\theta_{t+1} = (1-\eta\lambda)\theta_t - \eta\, \operatorname{clip} \left( \frac{m_t}{\max(\gamma h_t,\epsilon)}, 1 \right).$$

Coordinates with high curvature receive smaller steps, while element-wise clipping prevents updates from exploding because of non-convex regions or rapidly changing Hessians. Sophia is not a “full second-order optimizer”; it is closer to a stochastic second-order method that occasionally computes a diagonal curvature estimate.

Shampoo and SOAP: If the Room Is Tilted, Rotate the Coordinate System

Adam adjusts the scale of each coordinate but does not directly model correlations between coordinates. For a matrix gradient $G_t\in\mathbb{R}^{m\times n}$, Shampoo accumulates statistics separately along the row and column directions.

$$L_t = \epsilon I + \sum_{\tau=1}^{t} G_\tau G_\tau^\top,$$ $$R_t = \epsilon I + \sum_{\tau=1}^{t} G_\tau^\top G_\tau,$$ $$\Delta W_t = -\eta L_t^{-1/4} G_t R_t^{-1/4}.$$

Preconditioning the matrix from both sides straightens distorted scales across rows and columns together. The picture is closer to combing tangled hair in both horizontal and vertical directions, but this is not Newton's method computing an exact Hessian. It is better understood as structural preconditioning related to gradient covariance or the Fisher.

The downside is matrix state and inverse-matrix-power computation. Practical implementations split large matrices into blocks and update preconditioners periodically rather than on every step.

SOAP performs an Adam-style update inside Shampoo's eigenbasis.

$$\widetilde G_t = U_L^\top G_tU_R.$$

SOAP tracks Adam's second moment and constructs the update in the rotated $\widetilde G_t$ space, then rotates it back. If the room is tilted, instead of letting a cleaning robot repeatedly crash into the wall, rotate the room upright, let Adam clean it, and rotate everything back.

Adafactor compresses state into row and column statistics; Shampoo uses row and column preconditioners to straighten the update; SOAP runs Adam in the eigenbasis of those preconditioners. Looking at matrices does not make the three methods equivalent.

7. signSGD and Lion: Forget How Strong, Remember Which Way

One of the boldest simplifications is to discard gradient magnitude and keep only its sign.

$$\theta_{t+1} = \theta_t - \eta\operatorname{sign}(g_t).$$

signSGD asks only whether each coordinate is positive or negative. In two dimensions, if neither component is zero, the update points in one of four diagonal directions. Two gradients whose magnitudes differ by a factor of ten produce the same coordinate-wise step when their signs match.

Lion is a sign-momentum optimizer discovered through program search.

$$c_t = \beta_1m_{t-1} + (1-\beta_1)g_t,$$ $$\theta_{t+1} = \theta_t - \eta \left( \operatorname{sign}(c_t) + \lambda\theta_t \right),$$ $$m_t = \beta_2m_{t-1} + (1-\beta_2)g_t.$$

Lion mixes the current gradient with past momentum to choose a direction, then takes its sign. Because there is no Adam-style $v_t$, it needs only one momentum buffer the size of the parameters.

Put simply, Lion forgets “how angry were we?” and remembers only “which direction were we angry in?” Since every coordinate moves by a fixed magnitude, however, the update norm can become large. Lion often needs a much smaller learning rate and different weight decay than AdamW. A small state and a short equation do not automatically make the hyperparameters simple.

8. Tools That Wrap an Optimizer or Change Its Step Policy

The methods in this section do not all live at the same conceptual level. Lookahead and SAM are close to wrappers around a base optimizer. D-Adaptation and Prodigy are optimizers that choose the step scale adaptively. Schedule-Free is an optimizer transformation that changes both the gradient-measurement point and the parameter sequences. They nevertheless share a concern that goes beyond “what should multiply the gradient?”: how to operate exploration points, step sizes, and iterates.

Lookahead

Lookahead keeps separate fast weights $\theta$ and slow weights $\phi$. After the base optimizer updates the fast weights $k$ times, the slow weights move partway toward them.

$$\theta_{t,i+1} = \operatorname{BaseOpt}(\theta_{t,i}), \qquad i=0,\ldots,k-1,$$ $$\phi_{t+1} = \phi_t + \alpha(\theta_{t,k}-\phi_t), \qquad \theta_{t+1,0}=\phi_{t+1}.$$

The fast weights scout around while the slow weights inspect the report and follow cautiously. Separating the scouting party from the main group can reduce oscillation in the base optimizer.

SAM

SAM prefers a broad minimum, where the loss remains low after a small displacement, over a point that has low loss only at its exact current position.

A simple form using an L2 perturbation is

$$\epsilon_t^* = \rho \frac{g_t}{\lVert g_t\rVert_2+\epsilon},$$ $$g_t^{\mathrm{SAM}} = \nabla_\theta \mathcal{L}(\theta_t+\epsilon_t^*),$$ $$\theta_{t+1} = \operatorname{BaseOpt} (\theta_t,g_t^{\mathrm{SAM}}).$$

The orange perturbation is not an actual move. It is a temporary visit to check how noisy the apartment next door is. SAM computes another gradient there and uses it for the real update. This generally requires one extra gradient evaluation and increases the cost per step.

D-Adaptation and Prodigy

Choosing an optimizer's learning rate requires some sense of the problem scale, such as the distance between the initial point and a solution. D-Adaptation grows an estimate of this unknown distance $D$ during training and uses it to adjust the step size. Prodigy belongs to a family that tries to adapt this estimate more rapidly.

It is like measuring the remaining distance with an automatic tape measure while choosing your stride. This does not mean that the learning rate disappears entirely: base scales, schedulers, weight decay, and other settings still matter. Popularity in fine-tuning a particular class of generative models does not automatically make a method the default for every pretraining problem.

Schedule-Free

Cosine and linear decay typically assume that the total number of training steps $T$ is known in advance. If we do not know when training will stop, where should the schedule end?

Schedule-Free separates an exploration point $z_t$, an averaged point $x_t$, and a gradient-measurement point $y_t$. The core form of Schedule-Free SGD is

$$y_t = (1-\beta)z_t + \beta x_t,$$ $$z_{t+1} = z_t - \gamma\nabla f(y_t,\zeta_t),$$ $$x_{t+1} = (1-c_{t+1})x_t + c_{t+1}z_{t+1}.$$

$c_{t+1}$ determines how strongly the new exploration point enters the average; equal averaging uses $c_{t+1}=1/(t+1)$.

$z_t$ is the explorer running ahead, $x_t$ is the cartographer averaging the route so far, and $y_t$ is where the two meet to measure the next gradient. Iterate averaging and momentum take over the role of an explicit decay schedule.

The name is easy to misread: Schedule-Free is not learning-rate-free. A base learning rate and warmup may still be needed. Implementations must also distinguish which parameter sequence is used in training and evaluation modes, and BatchNorm running statistics require care.

Update Modules from 2025–2026

MARS constructs a corrected gradient $c_t$ that reduces variance using the gradient difference between the current and previous positions on the same sample.

$$c_t = g(\theta_t,\xi_t) + \gamma_t \frac{\beta_1}{1-\beta_1} \left[ g(\theta_t,\xi_t) - g(\theta_{t-1},\xi_t) \right].$$

This $c_t$ is fed into a base optimizer such as AdamW, Lion, or Shampoo. It resembles noise-canceling headphones, but the exact version requires gradients at two positions. A cheaper approximation should not be confused with the exact theoretical method.

Cautious Optimizers pass only those coordinates where the preconditioned direction $u_t$ produced by the base optimizer, before subtraction from the parameters, agrees in sign with the current gradient.

$$\phi_t = \mathbf{1}[u_t\odot g_t>0],$$ $$\widetilde u_t = \frac{\phi_t\odot u_t} {\operatorname{mean}(\phi_t)+\epsilon}.$$

It is a bouncer admitting only guests on whom momentum and the current gradient agree. The mechanism can be attached to AdamW, Lion, Muon, and others, but the trade-off is that it may discard a direction that is temporarily opposed to the current gradient yet useful over the longer term.

9. Muon and the Matrix-Native Optimizers of 2026

Adam ultimately treats a matrix parameter as a collection of coordinates. A linear layer's weight is a matrix where input and output directions meet, rather than an arbitrary long vector. Would it be better to handle the update according to matrix geometry as well?

Muon: Iron the Momentum Matrix Flat

Muon stands for MomentUm Orthogonalized by Newton-Schulz. As the name suggests, it constructs momentum for a two-dimensional weight matrix, then approximately orthogonalizes that update with a Newton–Schulz iteration.

Let the gradient matrix be $G_t$ and the momentum be $M_t$.

$$M_t = \beta M_{t-1} + (1-\beta)G_t.$$

Now form a Nesterov-style update matrix $U_t$ and consider its SVD.

$$U_t = \beta M_t + (1-\beta)G_t,$$ $$U_t = P\Sigma Q^\top.$$

The conceptual target is the matrix sign, or polar factor,

$$\operatorname{msign}(U_t) = PQ^\top.$$

Singular values that originally had different magnitudes are flattened so that each direction receives one vote. Rather than computing an SVD on every step, practical Muon applies several Newton–Schulz polynomial steps composed only of matrix multiplications. One important qualification is that the official default five-step quintic is not a classical iteration that converges exactly to $PQ^\top$. It aims toward the polar direction while leaving output singular values roughly in the $0.5\sim1.5$ range, trading exactness for computation.

$$W_{t+1} = (1-\eta\lambda)W_t - \eta s_t \operatorname{NS}(U_t).$$

$s_t$ is an update scale adjusted for matrix shape. For a rectangular matrix, the result is a semi-orthogonal polar factor, not a square matrix whose rows and columns can all be orthogonal simultaneously.

Two points are particularly important.

First, Muon orthogonalizes the momentum update, not the weight itself. Second, it is not normally applied to every parameter. Hidden-layer two-dimensional weight matrices commonly use Muon, while embeddings, output heads, biases, norm gains, and other vector-shaped or structurally different parameters continue to use AdamW.

A large-scale Muon study from 2025 reported that weight decay and per-parameter update scaling are important for scaling the method. Newton–Schulz matrix multiplications, distributed communication, parameter-group separation, and retuning still have costs. Replacing the single word AdamW with Muon does not make every model faster by drop-in magic.

Scion: Give Every Layer a Differently Shaped Playing Field

Scion defines norm balls appropriate to layer structure and solves a Linear Minimization Oracle for the momentum direction.

$$d_t = \alpha_tg_t + (1-\alpha_t)d_{t-1},$$ $$\operatorname{LMO}(d_t) = \arg\min_{\lVert v\rVert\leq 1} \langle d_t,v\rangle,$$

Constrained Scion keeps parameters inside a norm ball of radius $\rho$ by interpolating between the current parameters and the boundary point selected by the oracle.

$$\theta_{t+1} = (1-\gamma_t)\theta_t + \gamma_t\rho \operatorname{LMO}(d_t).$$

Unconstrained Scion, or uSCG, does not explicitly preserve the norm constraint and instead uses

$$\theta_{t+1} = \theta_t + \gamma_t\rho \operatorname{LMO}(d_t).$$

A spectral-norm ball can be chosen for matrix layers, with different norms for vectors and embeddings. Under spectral-norm geometry, the resulting direction connects to Muon's polar update. Scion broadens Muon's idea into the claim that every layer deserves a playing field shaped for its own structure.

The choice of norm and radius for each layer becomes a new design problem. Current evidence is also closer to comparatively limited language-model experiments than to broad validation across the largest commercial scales.

AdaMuon and NorMuon

AdaMuon combines the orthogonalized direction with element-wise second-moment adaptation. It tries to retain the advantage of flattening directional scales through orthogonalization while adding Adam-style coordinate adaptation.

NorMuon applies neuron-wise, or row-wise, second-moment normalization after the Muon update. It targets the problem that even after Muon improves conditioning, an unusually large row norm can allow a particular neuron to dominate the update.

Both names suggest “adaptive Muon,” but their units differ. AdaMuon is closer to cell-wise adaptation, whereas NorMuon balances rows.

The 2026 Observation Deck: Interesting, but Not Yet the Default

Proposals extending post-Muon matrix geometry have appeared rapidly in 2026.

• FISMO tries to reintroduce anisotropic Fisher curvature into Muon's uniform singular-value geometry.
• Newton-Muon first right-preconditions the gradient using the covariance of inputs $Z$, then applies the matrix sign.

$$W \leftarrow W - \eta\, \operatorname{msign} \left( G(ZZ^\top)^{-1} \right).$$

• Hyperball fixes the Frobenius norm of a matrix weight to its initial radius $R$ and projects it back onto the sphere after each update.

$$\widehat u_t = \frac{u_t}{\lVert u_t\rVert_F}, \qquad \widetilde W_t = W_t-\eta_tR\widehat u_t,$$ $$W_{t+1} = R \frac{\widetilde W_t} {\lVert\widetilde W_t\rVert_F}.$$

This allows the learning rate to be interpreted like angular motion relative to the weight norm, but it relies on the matrix being scale-invariant enough that radial learning can be removed. A follow-up analysis soon argued that much of the apparent benefit may arise from an implicit learning-rate schedule.

In addition, an August 2026 convergence analysis presented counterexamples where the original Muon can fail to converge for almost every mini-batch size in particular stochastic settings. This does not prove that Muon is useless, nor that AdamW is the eternal answer. It merely suggests that scheduling Adam's funeral may be a little premature.

Optimizers at a Glance

The state sizes below are rough optimizer-state counts excluding weights and gradients. Actual memory depends on mixed precision, master weights, sharding, quantization, momentum options, and implementation details.

Here, $P$ is the number of elements in the relevant parameter group. AdamW's $2P$, for example, means that FP32 $m$ and $v$ alone require 8 bytes per parameter. ¹Schedule-Free can replace conventional momentum state with the exploration sequence $z$, so SF-SGD with Momentum uses about $1P$ and SF-AdamW about $2P$, broadly matching their base optimizers. Total training memory additionally includes weights, gradients, activations, master weights, and communication buffers. “Optimizer state reduced by 50%” and “total VRAM reduced by 50%” are entirely different statements.

So Which Optimizer Should We Use in Practice?

Optimizer choice needs a diagnosis of the bottleneck more than it needs a winner's podium.

We Need a Stable Baseline

AdamW remains a strong starting point for Transformers and most fine-tuning tasks. Its implementations are broadly validated, and considerable experience has accumulated around schedulers, weight decay, and mixed precision. A new optimizer should first beat a well-tuned AdamW baseline before claiming a benefit.

We Are Training a Vision Model with a Traditional Recipe

Momentum SGD remains a strong baseline for CNNs and some vision tasks. Training loss may fall somewhat more slowly while final generalization is better. This difference should not be attributed to the optimizer alone; the learning-rate schedule, augmentation, and weight decay must be considered together.

We Are Scaling the Batch Size Dramatically

If relative updates across layers begin to break down, consider LARS or LAMB. Verify that the large batch actually improves throughput after communication and data efficiency are included.

Optimizer State Is Pushing the Model Out of Memory

Compare Adafactor, Adam-mini, 8-bit state, and state sharding. Separate the effect of changing the optimizer update from the effect of merely compressing its state.

We Want to Study a Matrix-Aware Optimizer

Muon, Shampoo, and SOAP are interesting choices. Measure loss at the same token count alongside real wall-clock time, including Newton–Schulz iterations or eigendecompositions, distributed communication, and the AdamW state for parameter groups to which the matrix optimizer cannot be applied.

We Do Not Know the Total Number of Steps in Advance

Schedule-Free may be a good candidate. It does not mean “no schedule and no configuration whatsoever”; it replaces decay tied to a known endpoint with an iterate-averaging structure.

Personally, I find the following order safest:

1. Start from a well-known baseline and recipe.
2. Determine whether the current bottleneck is step count, wall-clock time, VRAM, or tuning time.
3. Change only one optimizer component that directly targets that bottleneck.
4. Retune the learning rate and weight decay for the new optimizer.
5. State whether the comparison holds tokens, FLOPs, or wall-clock time constant.

Dropping in a new optimizer, reusing AdamW's learning rate unchanged, and declaring the new method bad when performance falls is like changing running shoes, leaving the laces untied, and blaming the shoes.

Common Misconceptions

Is the Second Moment a Variance?

Adam's $v_t$ is an uncentered second moment, an exponential average of $g_t^2$. It is not the statistical variance obtained by subtracting a mean, as in $(g_t-m_t)^2$.

Are Weight Decay and L2 Regularization Always the Same?

For plain SGD, they can be equivalent under an appropriate scaling relationship. With an adaptive preconditioner, adding $\lambda\theta$ to the gradient differs from directly shrinking the parameters as AdamW does.

Are Second-Order Optimizers Always Faster?

They may require fewer iterations while making every step much more expensive in computation and communication. The word “faster” is incomplete until wall-clock time and memory are included.

Does Muon Completely Replace AdamW?

Usually not. Muon is applied to hidden two-dimensional matrices, while vectors, biases, embeddings, and output heads commonly remain on AdamW. Total optimizer state and communication should be measured for this mixed configuration.

If a Paper Reports a 2× Speedup, Will My Training Be 2× Faster?

First ask whether “2×” refers to step count, FLOPs, loss at equal tokens, or actual wall-clock time. The large 2025 comparison Fantastic Pretraining Optimizers and Where to Find Them shows that benefits from matrix-aware optimizers depend on model scale, token count, and the decay phase, and that the gap may narrow as scale increases.

An optimizer result in a paper is not the report card of one algorithm. It is closer to a group exam taken jointly by the learning rate, schedule, weight decay, model scale, kernels, and distributed implementation.

Key Takeaways

• SGD follows the current mini-batch gradient directly.
• Momentum remembers past directions; NAG checks the gradient in advance at an expected future position.
• AdaGrad, RMSProp, and AdaDelta adjust strides using per-coordinate gradient scales.
• Adam combines first and second raw moments; AdamW decouples weight decay from the adaptive update.
• LARS and LAMB control relative updates per layer, while Adafactor compresses a matrix second moment into row and column statistics.
• Shampoo preconditions a matrix from both sides; SOAP performs an Adam-style update in the resulting eigenbasis.
• Lion mixes momentum and the current gradient, then uses only the coordinate-wise sign.
• Sophia uses a diagonal Hessian estimate and clipping to obtain inexpensive second-order information.
• Lookahead and SAM wrap a base optimizer to manage exploration points and sharpness. Prodigy adapts the step scale, while Schedule-Free operates the gradient-measurement point and parameter average.
• Muon approximately orthogonalizes momentum updates for two-dimensional weights using Newton–Schulz iterations.
• Scion and several 2025–2026 proposals incorporate layer and matrix geometry more directly, but their validation scope still matters.

Conclusion

The history of optimizers is not a story in which increasingly complicated equations simply became increasingly good equations. It is closer to a history of changing opinions about what the actual problem is: gradient noise, coordinate scale, layer norm, curvature, memory, or matrix geometry.

SGD trusted the current slope. Momentum remembered the past. AdaGrad and Adam accepted that different coordinates live under different circumstances. Shampoo and SOAP questioned whether the coordinate axes themselves were aligned correctly. Muon and Scion began to embrace more directly the fact that parameters have matrix and layer structure.

None of this means that the newest method always displaces an older one. More information can produce a better update, but computing, storing, and communicating that information also has a price. The best optimizer is ultimately not the one with the cleverest equation, but the one that reaches the desired performance at the lowest cost on the model, data, and hardware we actually have.

Perhaps choosing an optimizer is itself an optimization problem. The only difference is that this time, Autograd will not calculate the gradient for us.