Follow a token through a tiny decoder. Change the position encoding, attention head, causal mask and temperature to see what each part actually does.
Follow a token through a tiny decoder. Change the position encoding, attention head, causal mask and temperature to see what each part actually does.
Each head projects the input into its own Q, K and V. Select a query row: dot products become attention weights, which mix value vectors. The causal mask blocks future tokens.
A = softmax(QKᵀ / √2 + mask) · V
| Q ↓ / K → | AI0 | can1 | help2 |
|---|---|---|---|
| 100.0% | — | — | |
| 64.6% | 35.4% | — | |
| 26.8% | 42.4% | 30.8% |
Try query 0, then turn the mask off: future-token weights become nonzero. Switch heads to compare two different projections, using fixed teaching weights here.
Computed with fixed, untrained weights. One simplified post-norm decoder block; no cross-attention or training loop. The original Transformer also has an encoder. This explorer focuses on autoregressive decoder computation. Attention Is All You Need ↗