I'm working through How to Scale Your Model and keeping notes as I go. Chapter 1 is about rooflines: a way to tell whether a computation is limited by math, memory bandwidth, or the network. These are the pointers I'd want to remember; the derivations and exercises are in the notes.
- Three limits. Compute (FLOPs/s), bandwidth (bytes/s) and memory capacity. Runtime is bounded below by max(Tmath, Tcomms) when the two overlap, and above by their sum.
- Arithmetic intensity is FLOPs per byte moved, a property of the algorithm. You are compute-bound only when it exceeds the hardware's peak FLOPs/s divided by bandwidth.
- The roofline is achievable FLOPs/s = min(peak, AI × bandwidth). Taking the reciprocal flips the max into a min, which I got wrong on my first pass.
- Matmul has AI ≈ B (batch size) when B ≪ D, F. That makes the threshold about 240 on a TPU v5e and about 298 on an H100 (3.35 TB/s HBM, dense bf16).
- Weight precision matters. With int8 weights and bf16 activations the threshold halves to about 120, because each weight costs one byte instead of two.
- Reuse is where intensity comes from. A dot product has AI ≈ ½, and a matmul with a different weight matrix per batch element is stuck at AI = 2, about 0.4% of int8 peak. Decode attention over per-sequence KV caches is the same story.
- Network rooflines count only the bytes crossing the link. I had mixed HBM traffic into the network term; each chip has its own HBM roofline separately.
The full notes include the worked exercises, a roofline plot across several D and F, and my corrections: typeset notes (PDF) and handwritten notes (PDF). More chapters in the repo as I get to them.