Paper · October 25, 2026 · Inference systems

Flux-GPU: measured autotuning for local inference on consumer GPUs

NotFoxLabsNotFox DevelopmentUpdated November 7, 2026

Abstract

Most advice for running language models locally is folklore: turn on flash attention, shrink the batch, quantize the cache. Whether any of it helps depends on the GPU, the driver, the runtime build and the model, and the effects are often smaller than run-to-run noise. Flux-GPU replaces the folklore with a measurement loop. It fingerprints the machine, screens candidate execution configurations for llama.cpp, confirms the best one against stock defaults with interleaved A-B-B-A runs and a Welch's t-test, and stores the winner only if its 95% confidence interval clears a 1% effect. On an AMD Radeon RX 580 (Polaris, 8 GB) the search found a single Vulkan backend option that raised generation throughput on Qwen3-4B from 22.8 to 38.1 tokens per second (+68.3%, 95% CI +67.9 to +68.8%), reproduced in an independent second run. A placement planner extends the same discipline to models that do not fit in VRAM, running a 12B model on the same 8 GB card at 7.5 to 7.8 tokens per second with exact weights. We report the rejected candidates, the inconclusive results and a benchmark artefact we caught, alongside the gains.

1. Introduction

Local inference matters for privacy, cost and availability, and most of the GPUs that could run it are not the newest ones. An eight-year-old RX 580 has 8 GB of fast memory, but no FP16 arithmetic, no accelerated 8-bit dot product and no current ROCm support. The runtimes that serve such a card expose dozens of switches, and their defaults are tuned for very different hardware.

Tuning by hand has two failure modes. The first is believing a change helped when the difference is noise: on our test machine, prompt throughput for a partially offloaded model varied by 25% between identical runs. The second is generalising a result that only holds for one configuration: a backend option that made one model 68% faster made a larger one 8% slower once some of its layers ran on the CPU.

Flux-GPU treats every configuration change as a hypothesis. It measures the machine and the workload, tests candidate settings against a baseline, and keeps a change only when the evidence is strong enough. The result is stored against a fingerprint of the hardware and software, and is measured again when any part of that fingerprint changes. This paper describes the method and reports what it found on one consumer GPU.

Our contributions are:

• A screening and confirmation protocol for inference settings that cancels thermal and time drift and reports "inconclusive" instead of guessing.
• Hardware fingerprinting that keys every stored profile to the GPU, driver, runtime build and host, so a tuned result is never applied where it was not measured.
• A placement planner for models larger than VRAM that predicts candidates from a measured bandwidth profile, then benchmarks them and recalibrates itself from the results.
• Measured results on an RX 580 from 1.2B to 12B parameters, including the negative ones.

2. How Flux decides

Flux orchestrates mature engines rather than replacing them. Inference runs on llama.cpp's kernels; Flux decides how it is launched, where each tensor lives and which backend options are set. Components are replaced with Flux's own code only where Flux measures them faster, and so far none of our kernel prototypes has beaten llama.cpp for real inference.

2.1 Capabilities from the device, not the product name

flux scan queries Vulkan features and properties, memory heaps, queue families, PCI identifiers and vendor telemetry. Our test card reports PCI id 1002:6FDF, a cut-down Polaris 20 sold mainly in China rather than a standard RX 580. Classifying by product name would have been wrong; classifying by device-id range and queried features was not. Those features shape the search space: because integerDotProduct4x8BitPackedSignedAccelerated is false on this card, int8 dot-product kernel paths are never offered as candidates.

2.2 Fingerprints

Every result is keyed to a hash of the GPU PCI id, architecture, VRAM, driver version, Vulkan driver and API version, CPU, RAM size class, operating system and the performance-relevant runtimes, including the exact llama.cpp build. Any change produces a new fingerprint, and profiles from the old one are shown as stale rather than silently reused.

2.3 The search

The autotuner runs a coordinate search over flash attention, micro-batch size, KV-cache precision and a whitelist of backend options. Each candidate is screened with at least three repetitions. The best screening candidate then goes to confirmation against the stock baseline, described in Section 3. Only a confirmed winner becomes a profile, and flux run --explain cites the screening run and the confirmation that introduced each setting.

Step	What Flux does	Output
Scan	Query devices, features, heaps, queues, telemetry	Capability report
Fingerprint	Hash hardware, driver and runtime versions	Profile key
Screen	Benchmark each candidate, n ≥ 3	Ranked candidates
Confirm	Interleaved A-B-B-A against stock, Welch's t-test	Improved, regressed, unchanged or inconclusive
Persist	Store the winner and every sample in SQLite	Profile used by flux run

3. Measurement protocol

The protocol is deliberately conservative. A false "faster" stored in a profile costs every later run on that machine; a missed improvement costs only a future search.

1. Baseline first. The baseline is the stock runtime with no Flux settings. If stock defaults fail, for example by running out of memory, the planner's configuration becomes the baseline and the report says so.
2. Valid samples only, none dropped for being slow. A sample is invalid only if it is non-finite or non-positive, or its run failed: a non-zero exit, out-of-memory, a lost device or unparsable output.
3. Interleaving. Confirmation alternates the arms A-B-B-A and B-A-A-B, each in a fresh process, so thermal drift and background load fall evenly on both.
4. Statistics. Welch's unequal-variance t-test on the difference of means, a 95% confidence interval on the relative change, and Hedges' g. A change is improved only if the interval's lower bound exceeds 1%; it is inconclusive if the interval straddles zero or the coefficient of variation exceeds 10%.
5. Secondary metrics and memory. A candidate that improves its primary metric but regresses the secondary one by more than 2%, or whose device buffers exceed free VRAM minus a safety reserve, is rejected even if it is faster.

Telemetry is sampled on its own thread every 250 ms, and its cost, one to two milliseconds per sample, is measured and reported with the results.

4. Results on an RX 580

Machine. AMD Radeon RX 580 2048SP, 8 GB (Polaris, Adrenalin 22.20.27.09, Vulkan 1.3.217), Intel i7-8700, 16 GB RAM, Windows 11. Runtime: llama.cpp build 11149, Vulkan backend. Model: Qwen3-4B-Instruct-2507 at Q4_K_M, all layers on the GPU. Metrics are llama-bench prompt processing over 512 tokens (pp512) and generation of 128 tokens (tg128).

Generation throughput (tg128) tokens/s · higher is better

Graphics queue on (Flux)38.20

Stock defaults22.81

Flash attention on22.78

Micro-batch 102422.70

Graph optimizer off21.75

Flash attention off20.24

q8_0 KV cache19.83

Op fusion off16.96

Prompt processing (pp512) tokens/s · higher is better

Graphics queue on (Flux)197.9

Flash attention off192.9

Stock defaults187.0

Flash attention on185.7

Micro-batch 1024180.8

Graph optimizer off179.6

q8_0 KV cache179.5

Micro-batch 128106.0

Micro-batch 25692.7

4.1 Confirmation

The winning candidate was confirmed against stock defaults in eight interleaved llama-bench processes, eight samples per arm. A second autotune, run independently after a fingerprint change, reproduced it.

Through a live llama-server, with a chat template, sampling and a 1,496-token prompt varied on every run so the server cache could not skip prefill, time to first token fell from 9,547 ms to 8,852 ms (−7.3%) and decode rose from 19.57 to 30.68 tokens per second (+56.8%). At 87 W average board power the tuned profile generates 0.448 tokens per joule.

4.2 Why one queue matters

By default llama.cpp's Vulkan backend submits compute work to a compute-only queue family. The option Flux found lets it use the universal graphics-and-compute family instead. The effect is large on generation, which issues many small dispatches per token, and small on prompt processing, which issues few large ones. That pattern points to per-submission or scheduling overhead on the compute-only queue under this Windows driver rather than to shader throughput. This is a hypothesis: we have not yet captured GPU timestamps on both queue families to prove it, and Flux does not hard-code the option. It is a profile because it was measured faster on this fingerprint, and it will be measured again when the driver or llama.cpp changes.

4.3 Kernel bounds

Flux's kernel lab measured the ceiling any memory-bound decode kernel faces on this card: a shader streaming copy reached 169.4 GB/s and the best f32 matrix-vector kernel 197.3 GB/s effective, about 88% of the card's theoretical 224 GB/s. The naive one-thread-per-row variant reached 2.3 GB/s. The same operation spans an 85-fold range depending on how work is laid out, which is why we measure layouts per GPU rather than assume them.

5. Models larger than VRAM

The placement planner chooses among five tiers per model and machine: GPU-resident; GPU and RAM split by layer; GPU and RAM split by tensor class, with attention and the KV cache on the GPU and feed-forward weights or experts in RAM; GPU, RAM and SSD; and CPU only. It predicts each candidate from a measured bandwidth profile of the machine, then benchmarks the ones that fit and keeps the fastest that meets the quality policy. The default policy keeps exact weights and an f16 KV cache; reduced-precision caches are candidates only when the user allows them.

Flux (chosen plan)Stock llama.cpp8B candidates10B candidates12B candidates

Three findings came out of these runs. First, whole-layer offload beat tensor-level offload on this machine. Moving feed-forward weights to RAM keeps attention on the GPU, but costs two GPU-to-CPU hand-offs per offloaded layer per token, and on Vulkan under Windows each hand-off costs several milliseconds. A least-squares fit over the 10B and 12B runs put it at 4.87 ms, with CPU decode at 19.0 GB/s, and the planner now uses those fitted values instead of assumed ones.

Second, the planner's first predictions underestimated partial offload, as the right panel of Figure 4 shows; it now recalibrates from every benchmark it runs. Third, interleaved confirmation corrected a wrong screening order. Under memory pressure, three-sample screening ranked 34-layer offload above 36-layer offload for the 12B model; the A-B-B-A confirmation found the opposite (7.68 against 6.93 tokens per second, −9.7%) and the better plan was kept.

The graphics-queue option that gave +40 to +68% on fully offloaded models cost 7.8% of generation on the 12B once layers ran on the CPU, so it remains a per-model measurement, not a GPU-wide default.

6. What did not work

We report the negative results because they are where most local-inference advice goes wrong.

• Common tuning tips. Flash attention, micro-batch sizes and a q8_0 KV cache were each neutral or harmful for generation on this card (Figure 2).
• Draft-free speculative decoding. On requests with varied subjects, n-gram speculation measured −14.8% on code (inconclusive) and −1.5% on prose. It stays off. A first version of this test sent the same request every round and reported +150% with 100% draft acceptance; the n-gram pool was replaying the previous identical answer. We caught the artefact and the test now varies every request.
• Hybrid placement under a forced budget. With only 1.37 GB of VRAM allowed, a micro-batch of 128 raised prompt speed in every sample, but the baseline's 25% coefficient of variation exceeded the gate, so Flux reported the result as inconclusive and saved nothing.
• Streaming weights from disk. Double buffering hid 88% of the consumer's time, but a SATA drive at 0.52 GB/s bounds per-token streaming of even half of an 8B model's layers to about 0.27 tokens per second. Flux keeps weights resident instead.
• Custom kernels. None of our Vulkan kernel prototypes has yet beaten llama.cpp's kernels for real inference, so none replaces them.

7. Limitations

All results come from one machine. AMD Polaris on Windows is benchmark-validated; NVIDIA, ROCm and Apple Silicon paths are implemented and tested against simulators but have not run on real hardware, and Linux has been compiled but not run. The queue-family explanation in Section 4.2 is a hypothesis pending GPU timestamp captures. Efficiency figures use GPU board power only, not CPU or wall power. Mixture-of-experts placement has been unit-tested and simulated, but no MoE model has been benchmarked. Hybrid measurements were taken with 3 to 5 GB of free system RAM and vary by about 15% between processes; differences smaller than that are not decisive.

8. Conclusion

On a GPU most guides would call obsolete, the largest available speed-up was not a new kernel or a lower precision. It was a queue-family setting that a measured search found and an interleaved test confirmed. The same discipline rejected eight plausible tips, caught a misleading benchmark and corrected a wrong ranking. We think local inference gets faster the way this result did: by measuring each machine instead of assuming it, keeping only what the evidence supports, and saying "inconclusive" when it does not.

Next, we are capturing GPU timestamps to test the queue-family hypothesis, running the NVIDIA and Apple paths on real hardware, and benchmarking mixture-of-experts placement.

References

1. G. Gerganov et al. llama.cpp: LLM inference in C/C++, build 11149, Vulkan backend.
2. B. L. Welch. The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1–2), 1947.
3. L. V. Hedges. Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6(2), 1981.
4. The Khronos Group. Vulkan 1.3 Specification: queue families and device features.
5. Qwen Team. Qwen3 technical report, 2025.