Verify a design against PyTorch¶
The analysis is arithmetic on a graph. This checks that arithmetic against a model PyTorch actually built.
Setup¶
Run it¶
bun run scripts/codegen-demo.ts llama-3-8b
python -m tensorcad_runtime verify out/llama-3-8b/model.py
The model is built on the meta device, so a 405B design costs no memory to
count. You get the real parameter total, a per-module breakdown, and — where the
model is small enough — a CPU forward pass, a FLOP count and a torch.export
check.
Reading the output¶
"matches": true is the assertion that matters: PyTorch's parameter count equals
the design's.
params_by_module is where a disagreement becomes actionable. A mismatch is
almost always one module, and the breakdown names it.
tied_parameters lists weights that are shared. A tied embedding and output head
appear here, and the head contributes zero to the total.
Options¶
For a convolutional design pass --seq 1: its token is one image.
The harness reads the design's own input block for the shape and dtype of the
tensor to feed, so a vision model gets B C H W floats rather than token ids.
When the forward pass is skipped¶
Above a few billion parameters the message is "forward": "skipped: too large".
That is fp32 weights not fitting in RAM, not a failure — the parameter count is
still verified, because that happens on the meta device.
FLOPs will not always match exactly¶
For a convolutional model they do, to the FLOP.
For a transformer, torch.utils.flop_counter reports more than the analysis:
GPT-2 small measures 251.78 MFLOP/token against a reported 249.42. The whole
difference is the causal mask. A profiler counts the attention operator as if
nothing were masked, because the operator's shape does not depend on the mask.
flops.fwdTotalUnmasked reproduces the profiler exactly; flops.fwdTotal is
what a fused causal kernel actually does. Both are correct answers to different
questions, and a test pins both.