LLM Visualization: Walk Through a Working GPT Model, One Matrix Multiply at a Time
🛠️ LLM Visualization
| What it is | A 3D interactive walkthrough of a GPT-style LLM running real inference, visualized down to individual matrix operations |
| Platform | Web (requires WebGL2, best in Chrome or Firefox) |
| Price | Free, open source |
| Link | bbycroft.net/llm |
If you have ever looked at the Transformer diagram from “Attention Is All You Need” and wished you could watch the numbers actually move, this is for you.
Brendan Bycroft built a 3D visualization of a working GPT model, loaded with real weights from Karpathy’s minGPT. You press Space and it steps through inference token by token, animating every embedding lookup, every attention head, every layer norm, and every matrix multiply along the way. No simplification, no hand-waving. The actual model, running in your browser.
It is a real model, not a diagram. The tiny GPT inside this visualization actually runs inference. Its weights sort a three-element list of letters. You can watch the input token get embedded, flow through the transformer blocks, and emerge as a prediction. The numbers are real. Every add, multiply, and softmax is computed live.
The 3D walkthrough actually teaches you. The site is structured as a guided tour. Each component gets its own chapter with a clear 3D visualization you can orbit and zoom. Layer norm is shown as a real vector being normalized. Self-attention renders the Q, K, V matrices and highlights which tokens attend to which. The MLP shows the expansion and projection. It maps the standard transformer diagram onto real computation, showing you what the arrows actually mean.
You control the pace. Press Space to advance one step. Pause on any operation. Rotate the view to examine which dimensions are being multiplied. The visualization updates every value as the computation progresses. There is no video to scrub through, no animation you cannot stop. You go as fast or as slow as you need.
Multiple model sizes to explore. The default is the tiny minGPT model (easy to understand, fast to render). But you can switch to GPT-2 small, GPT-2 XL, or GPT-3 scale to see how the architecture grows. The same visualization applies at every size, which is a useful lesson in itself: the architecture scales, the principles do not change.
Simon Willison called it “beautifully crafted.” He is not wrong. Clean visual design, consistent color coding, the 3D view mirrors the standard transformer diagram exactly so you always know where you are. You will wonder why nobody thought of this before.
What to know
The page uses WebGL2 and can be heavy on older hardware. Chrome and Firefox work well. The tiny model renders instantly; GPT-2 and larger need to download weights. The source is on GitHub at bbycroft/llm-viz if you want to run it locally.
Worth your time if
You want to understand how a transformer actually processes a token, or you teach others about LLMs and need something better than a static diagram.
Related TMFNK Content
- hfviewer: Visualize Any Hugging Face Model A complementary tool: paste a model URL and get its architecture graph. LLM Visualization shows you the computation; hfviewer shows you the structure.
- How to Train Your GPT Build a GPT from scratch, tokenizer to inference. The visualization shows you what the code you write actually does.
- Learning How Mixture of Experts Works The next step after understanding the dense transformer: how sparse MoE layers route tokens through expert sub-networks.
Crepi il lupo! 🐺