Blog · November 2, 2026 · Measurement

Measure, then tune: how we decide a change is real

NotFoxLabs5 min read

Performance work has a quiet failure mode: you change something, run the benchmark once, see a bigger number, and keep the change. Sometimes the change helped. Often the machine was just cooler, or a background task had finished. This post describes the rules every result on this site goes through before we call it real.

Noise is bigger than you think

On our RX 580 test machine, prompt throughput for a partly offloaded 4B model varied by 25% between identical runs. Generation for 10B and 12B models moved by about 15% between processes when free RAM was low. Most tuning tips promise gains smaller than that. A single run cannot tell them apart from luck.

Five rules

1. Start from stock. The baseline is the unmodified runtime. If stock fails, the report says what replaced it.
2. Never drop a slow sample. A sample is excluded only if its run failed outright. Discarding slow results is how benchmarks lie.
3. Interleave. Confirmation runs alternate A-B-B-A, then B-A-A-B, each in a fresh process. Heat and background load land on both arms equally.
4. Use an interval, not a winner. We compute Welch's t-test and a 95% confidence interval on the relative change. "Improved" means the whole interval sits above +1%.
5. Say "inconclusive". If the interval straddles zero, or the run-to-run variation exceeds 10%, the result is reported as inconclusive and nothing is saved.

Interleaved confirmation, Qwen3-4B Q4_K_M, RX 580
  baseline   pp512 187.00   tg128 22.81
  candidate  pp512 198.36   tg128 38.14
  prompt      +9.7 %  [+9.5, +9.9]   p < 1e-4   improved
  generation +68.3 %  [+67.9, +68.8] p < 1e-4   improved

What the rules caught

These rules are not bureaucracy. In one week they caught three mistakes we would otherwise have published.

• A fake 150% speed-up. Speculative decoding looked 2.5 times faster, with every draft accepted. The test was sending the same request every round, and the speculator was replaying its last answer. With varied requests it was 1.5% slower.
• A wrong ranking. Three-sample screening ranked one 12B placement first. Interleaved confirmation found it 9.7% slower than the runner-up, and the better plan was kept.
• A promising result we could not prove. A smaller batch raised prompt speed in every single sample of a hybrid run, but the baseline varied by 25%. We reported it as inconclusive and saved nothing.

The full protocol, with every number, is in the Flux-GPU paper.