Blog · October 27, 2026 · Inference systems
One queue, 68% faster: what an autotuner found on an RX 580
If you search for how to speed up a local language model on an older AMD card, you get the same handful of tips: turn on flash attention, change the batch size, quantize the KV cache. We tried all of them, properly, on an RX 580. None of them helped generation. The change that did was one we would never have guessed.
The setup
The card is an RX 580 2048SP with 8 GB of memory, a Polaris GPU from 2018 with no FP16 arithmetic and no ROCm support. It runs llama.cpp through Vulkan. Out of the box, Qwen3-4B at Q4_K_M generated 22.8 tokens per second.
We pointed Flux-GPU's autotuner at it. Flux screens each candidate setting three times, confirms the best one against stock defaults with interleaved runs in fresh processes, and keeps it only if the 95% confidence interval of the gain clears 1%. The full method is in the paper.
What the search tried
Flash attention did nothing. Smaller batches made prompt processing up to twice as slow. A quantized cache cost 13% of generation. Then one candidate came back at 38.2 tokens per second: GGML_VK_ALLOW_GRAPHICS_QUEUE=1.
A number that big deserves suspicion, so Flux ran eight interleaved processes against stock and computed the interval: +68.3% generation (95% CI +67.9 to +68.8%) and +9.7% prompt processing. A second, independent autotune, run after the machine's fingerprint changed, found the same thing. Through a real chat server with a long prompt, decode went from 19.6 to 30.7 tokens per second.
Why a queue?
A Vulkan GPU exposes several queue families. llama.cpp normally submits compute work to a compute-only family. This option lets it use the universal graphics-and-compute family instead.
The shape of the gain is a clue. Generation, which launches many small GPU jobs for every token, got 68% faster. Prompt processing, which launches a few large ones, got 10% faster. That suggests the cost was in submitting and scheduling work on the compute-only queue with this Windows driver, not in the shaders themselves. Others have reported Polaris on Windows behaving differently on non-graphics queues, which fits. We have not proved it yet; the next step is capturing GPU timestamps on both queues.
The lesson
The same option is not a universal fix. On a 12B model that does not fit in memory, with some layers on the CPU, it made generation 7.8% slower. That is exactly why Flux stores results per machine and per model, and measures again when the driver or runtime changes.
The fastest setting on your machine is an empirical question. Ask it with a baseline, interleaved runs and a confidence interval, and be ready for the answer to surprise you.