Sequence models

Order is information

A dense network over a fixed window treats its inputs as a set with positions wired in. It connects every input to every unit. It can learn a rule for the word at position three, but not the same rule wherever the pattern appears. It also cannot accept a sequence longer than its input width. Both limits are structural, not a matter of training longer.

A bag of words counts words and ignores their order. “Dog bites man” and “man bites dog” have the same counts. A sentiment model that sees only counts cannot tell them apart. Order is often the signal. A negation before an adjective changes the meaning, and a verb at the end of a clause changes the parse.

The idea that makes sequence models work is weight sharing across positions. One set of parameters is applied at every step, with a state that carries what came before. The parameter count stops depending on the length. A pattern learned at one position then transfers to every other.

The recurrence

A simple recurrent layer computes h at step t from the input at t and the hidden state from t minus one. The hidden state is the summary the layer carries between steps. In symbols, h equals tanh of Wx times x plus Wh times h previous plus a bias. The same two matrices are used at every step, and the state is a fixed-size summary of the prefix.

An output can be read at every step for tagging, or only at the end for classification. The state is a bottleneck. Everything the model knows about the past has to fit in that one vector. Information from far back is overwritten by everything since. That compression is the price of a fixed-size interface.

The initial state is usually zeros or a learned vector. Dropout on the state, not only on the weights, is a common regulariser. It must use the same mask across the sequence if the recurrence is to stay coherent. A different mask at every step turns the state into noise rather than a summary.

The same recurrence in a library

import torch
from torch import nn

layer = nn.LSTM(input_size=32, hidden_size=64, batch_first=True)
sequence = torch.randn(8, 20, 32)          # batch, steps, features
output, (hidden, cell) = layer(sequence)
print(output.shape)                         # (8, 20, 64)
print(hidden.shape, cell.shape)             # (1, 8, 64) each

Backprop through time

Training unrolls the graph across time and applies the chain rule to the whole unrolled network. The gradient at an early step includes a product of Jacobians from every later step. A Jacobian is the matrix of local derivatives for one step. A norm is a measure of a vector’s size. If the norms are below one the product vanishes, and if they are above one it explodes. That is the same problem as a very deep feed-forward network, with the depth set by the sequence.

Truncated backpropagation limits the window. The state is carried forward, but it is detached from the graph every k steps. Gradients then travel only k steps back. That bounds memory and the product, at the cost of long-range credit. The model can use the past but cannot learn from an error that depends on it.

Gradient clipping by global norm is the standard guard. If the norm of the whole gradient exceeds a threshold, every component is scaled down by the same factor. The direction is preserved, and the step cannot destroy the weights. Clipping does not fix a bad architecture. It does keep one bad batch from ending a run.

Gates and the end of recurrence

An LSTM adds a cell state and three gates. LSTM stands for long short-term memory, a gated recurrent layer. A gate is a learned valve that controls how much passes through. The input gate decides what to write, the forget gate decides what to keep, and the output gate decides what to expose. The cell state is a mostly additive path, so gradients can flow further than through a stack of tanh layers. The forget gate learns how long a memory should last.

A GRU, a gated recurrent unit, uses two gates and a single state. It is cheaper and often close in quality. Neither removes the sequential dependency. Step t cannot be computed before t minus one, so training cannot parallelise across time, however wide the hardware is.

Encoder-decoder models put one recurrent network in front of another. The encoder compresses the input into a state, and the decoder generates from it. Attention lets the decoder look at every encoder state instead of one bottleneck vector. Transformers keep that attention and drop the recurrence. That is why they train on long sequences in parallel. Recurrent layers remain useful for streaming, for tiny state, and for problems where the sequence arrives one element at a time.