Optimizer Laboratory

Race down the loss curves

Every machine learning model is optimized by rolling down a loss surface. This sandbox lets you select a landscape $J(w_1, w_2)$, choose optimizers, set the learning rate, and **click anywhere on the map** to release particles. Watch them race to the minimum.

Loss Surface:

J(w₁, w₂) = 0.5 · (w₁² + 2·w₂²)

SGD Momentum RMSprop Adam Click anywhere on the contour grid to launch particles for all four optimizers simultaneously. Trace lines show their history.

Learning Rate α 0.020

Momentum Beta (β) 0.90

SGD w

–

Momentum w

–

RMSprop w

–

Adam w

–

Step Count

Optimizer Mathematics

SGD `Stochastic Gradient Descent`	Updates weights directly in the direction of the negative gradient. Simple, but easily stalls in valleys or oscillates wildly if parameters differ in scale. `w_t+1 = w_t − α · g_t`
Momentum `Classical Momentum`	Adds a fraction of the previous step's velocity vector to the update. Simulates a heavy ball rolling with inertia, helping it slide past noisy gradients and escape flat saddle points. `v_t = β·v_t-1 + α·g_t` `w_t+1 = w_t − v_t`
RMSprop `Root Mean Squared Prop`	Maintains an exponentially decaying average of squared gradients. It divides the learning rate by this running standard deviation, shrinking steps in steep directions and growing them in flat areas. `s_t = β·s_t-1 + (1−β)·g_t²` `w_t+1 = w_t − [α / √(s_t + ε)] · g_t`
Adam `Adaptive Moment Estimation`	Combines the ideas of Momentum (first moment) and RMSprop (second moment) together, including bias corrections for initialization. The industry standard optimizer for deep neural networks. `m_t = β₁·m_t-1 + (1−β₁)·g_t ; v_t = β₂·v_t-1 + (1−β₂)·g_t²` `m̂_t = m_t / (1−β₁ᵗ) ; v̂_t = v_t / (1−β₂ᵗ)` `w_t+1 = w_t − [α / √(v̂_t + ε)] · m̂_t`