The model reads a word and guesses the next one. The bar is the probability of the true next word. The point’s height is the loss: how wrong those guesses are, as cross-entropy. Each step follows the gradient, the direction that lowers the loss, and moves a distance set by the learning rate. That update is gradient descent. Train all uses the mean gradient of every sentence. Train this sentence uses only that sentence’s gradient, so the other bars stay short. The tabs are backpropagation: the same gradient, walked from the logits back to the embedding. The Transformer page runs a published model forward and does not train.
Gradient descent
Up is the loss. Each step moves along the gradient, scaled by the learning rate. The line bends when later gradients point a different way. Drag to look around. Scroll zooms.
Backpropagation is computed in this page. Finite differences check the gradient.