Hardware track

Start here: the machines that run the models

Four sites, one question: what does it take to do a transformer’s arithmetic fast? Kernels follows the maths through one GPU; Systolic builds the other kind of matrix engine, register by register; Accelerators runs the same decoder on every family of AI chip; Racks joins the chips into a data centre. Read them in order: each assumes the one before.

Reading order for the Hardware track

Each site's chapters build on each other; start at chapter 1 of Kernels. Every number is computed by the site's tested model.

1Kernels (this site): how a GPU executes the maths, from the memory hierarchy and the roofline to a Blackwell tensor-core pipeline; assumes the Foundations track.
2Systolic: the other way to build a matrix engine, cycle by cycle and checked against RTL, up to the TPU; assumes Kernels' roofline.
3Accelerators: eight families of AI chip running the same decoder, from GPUs to wafers, LPUs and in-memory compute; assumes Kernels and Systolic.
4Racks: from one package to a data centre: NVLink, networks, power, cooling, disaggregated serving and cost; assumes the three before it.

What you need first

What a transformer computes. Matrix multiplies of activations by weights, attention, and decoding one token at a time. The Foundations track (Decoder and Training) covers it.
Bytes, bandwidth and latency. That a number takes 1, 2 or 4 bytes, that bandwidth is bytes per second and latency the wait before the first one arrives.
A little computer architecture. Caches, DRAM and the idea that memory close to the arithmetic is small and fast. The NVIDIA GPU slide series starts from there.
Reading CUDA or RTL is optional. The Code layer shows the model’s TypeScript and, where it helps, CUDA or SystemVerilog (the CUDA slide series teaches it); hide it with the layer switch.

How a chapter works

A chapter opens with an animation of its mechanism, computed by the site’s tested model, then explains it in three layers you can switch on and off: Concept (words and pictures), Maths (the formulas) and Code (the code that runs). It ends with Check yourself, three questions to commit to before you reveal the answers (the numbers come from the same model), and What to remember, the chapter in a few lines.

Glossary

Each term links to the section that defines it, on the site that defines it. Some terms mean different things on different sites (goodput, ρ, the pipeline bubble); the shared definitions give each one definition and one formula, and say where a site differs.

Memory hierarchy: registers, shared memory, L2 and HBM: the closer to the ALUs, the faster and the smaller
Roofline, arithmetic intensity, ridge point: attainable speed is min(peak, bandwidth × flops per byte); the ridge is peak over bandwidth
Warp, SIMT, divergence: 32 threads share one instruction stream; a branch they disagree on runs both ways
Coalescing, sector: a warp's addresses become 32-byte sectors; consecutive threads should read consecutive addresses
Bank conflict: different words in one of shared memory's 32 banks are served one per pass
Occupancy: active warps over the SM's maximum, limited by warp slots, registers, shared memory or blocks
Tiling, register blocking: reusing each value from shared memory and registers so a GEMM stops waiting on memory
Warp shuffle: lanes of a warp read each other's registers directly, with no shared memory or barrier
Online softmax, FlashAttention: a running maximum and sum let attention stream K and V without storing the score matrix
Double buffering, split-K: loading the next tile while computing this one; splitting k to fill every SM
TMA, warp specialisation, Tensor Memory: Hopper and Blackwell give loads, MMAs and the epilogue their own engines and warps
Systolic array: fetch each number once and pass it neighbour to neighbour every cycle
Weight-stationary, skew: weights stay in the PEs; skewed activations meet their partial sums exactly
Output- and input-stationary: the other two dataflows, named after what stays in each PE
Wavefront, fill and drain, utilisation: the fixed overhead of each pass, and the share of MAC slots doing useful work
Shadow weights: a second weight register so the next tile loads while this one streams
bfloat16, FP32 accumulation: multiply narrow, accumulate wide: the product is exact, only the add rounds
MXU, ICI torus, all-reduce: a TPU's systolic matrix units, and its pods' torus of chip-to-chip links
Row-stationary, in-memory compute: Eyeriss's dataflow for convolutions, and computing where the weight is stored
im2col, lowering: turning a convolution into a GEMM so the array can run it
Accelerator families: where the weights live, who schedules the data, how big the engines are
Decode bound, batching: batch-1 decode is a full read of the weights; batching shares it
Wafer-scale yield: Poisson yield, and why a whole wafer must survive its defects
Static schedule: every instruction and transfer on a cycle fixed by the compiler
Kernel fusion, dataflow: a layer laid out as a circuit, its activations never leaving the chip
Crossbar, analogue compute: weights as conductances; the column currents are the dot products
Per-layer floor: the fixed time per layer that sets an SRAM machine's speed
NVLink domain, scale-up: the GPUs that share NVLink; crossing its edge costs an order of magnitude
Package, HBM stack: compute dies and HBM on an interposer; capacity sets the GPU count
Ring all-reduce, α–β model: bandwidth-optimal collective; small messages are latency-bound
Fat tree, oversubscription: leaf, spine and core switches; fewer uplinks trade bandwidth for hardware
Busbar, power smoothing: DC power down the rack; I²R loss falls with the voltage squared
Liquid cooling, CDU: cold plates and a coolant loop: Q = ṁ c_p ΔT
Disaggregated serving: separate prefill and decode pools, and the KV hand-off between them
Tensor, pipeline and data parallelism: how a training job is laid out across NVLink and the network
Cost per token: capital and energy over tokens served; capital dominates