Qwen3.8 27B GSQ-RCO on RX 6800M: IQ2_XS vs IQ3_S – VelsTech Lab

🧪 VelsTech Lab Can a 27B dense model be usable on a 12 GB mobile GPU? I ran two quants of Qwen3.8 27B GSQ-RCO – IQ2_XS (with an MTP draft head) and IQ3_S – side by side on the RX 6800M. The quant you pick decides both your context budget and your speed, and one llama.cpp flag mistake quietly cost the IQ3_S run 92% of its context.

🔧 Check your own fit first

LLM VRAM Calculator – does 27B IQ2/IQ3 + your context fit your VRAM? · GPU AI Performance Calculator – what tok/s to expect before you run.

Open calculators →

The test rig

Machine: ASUS ROG Strix G513QY · Ryzen 9 5900HX (8C/16T) · AMD RX 6800M 12GB (RDNA2, mobile) · 32 GB RAM · 2 TB NVMe
OS: Ubuntu 26.04.1 LTS (Wayland) · ROCm · llama.cpp (current build), n_threads=8
Model A: Qwen3.8-27B-GSQ-RCO-IQ2_XS-mtp.gguf – 27B dense, i-quants at ~2 bpw, ships a block-64 MTP draft head
Model B: Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf – same model, ~3 bpw, no draft head
Both runs: -ngl 999 --cache-type-k q8_0 --cache-type-v q8_0 -fa on --fit on --reasoning-preserve --jinja

System details panel showing ASUS ROG Strix G513QY, Ryzen 9 5900HX, 32 GB RAM and AMD Radeon RX 6800M on Ubuntu 26.04 — The test machine – a 2021 gaming laptop with a 12 GB RDNA2 mobile GPU.

How we ran it

Run 1 – IQ2_XS (with MTP tensors)

export LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH
llama-server \
 -m /home/user-name/models/Qwen3.8-27B-GSQ-RCO-IQ2_XS-mtp.gguf \
 -ngl 999 \
 --cache-type-k q8_0 \
 --cache-type-v q8_0 \
 -fa on \
 --fit on \
 --reasoning-preserve \
 --jinja

Run 2 – IQ3_S

export LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH
llama-server \
 -m /home/user-name/models/Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf \
 -ngl 999 \
 --cache-type-k q8_0 \
 --cache-type-v q8_0 \
 -fa on \
 --fit on \
 --reasoning-preserve \
 --jinja

What we measured

Metric               IQ2_XS-mtp          IQ3_S           Delta
──────────────────────────────────────────────────────────────
Decode (eval)        20.77 tok/s         18.97 tok/s     +9.5%
Decode (tg)          20.70 tok/s         18.93 tok/s     +9.4%
Prompt eval          113.51 tok/s        178.32 tok/s    -36%  (IQ3_S faster)
Context fitted       52,736 tok          4,096 tok       12.9×
VRAM used            11.57 / 12.87 GB    12.52 / 12.87 GB
                     (90%)               (97%)
Graphs reused        124                 252
Chat UI avg speed    20.9 t/s            19.0 t/s

Takeaway: IQ2_XS wins decode while carrying 13× more context – and a bigger KV cache makes decode slower, not faster, so the win is real. IQ3_S reads more bytes per token (~3 bpw vs ~2 bpw) and the GPU is bandwidth-bound on this card; that gap shows up as ~1.8 tok/s lost. The surprise is prompt eval: IQ3_S processes input 57% faster (178 vs 114 tok/s) – i-quants at 2 bpw pay a heavier dequant/codebook cost per token during the compute-bound prompt pass, while decode is bandwidth-bound and flips the ranking.

The proof – chat runs

llama.cpp chat UI showing the IQ2_XS model introducing itself with 498 of 52.74K context used and average speed 20.9 tokens per second — IQ2_XS: 498 / 52.74K context used, 20.9 t/s average in the chat UI.

llama.cpp chat UI showing the IQ3_S model introducing itself with 711 of 4.10K context used at 17 percent and average speed 19.0 tokens per second — IQ3_S: a longer answer, but only 4.10K context existed to fill – 19.0 t/s.

VRAM – the quiet story

GPU monitor showing AMD Radeon RX 6800M video memory usage at 11.57 of 12.87 GB, 90 percent — IQ2_XS at 90% VRAM – weights plus a 52K q8_0 KV cache, with ~1.3 GB spare.

GPU monitor showing AMD Radeon RX 6800M video memory usage at 12.52 of 12.87 GB, 97 percent — IQ3_S at 97% VRAM with only the default 4K context – the weights alone nearly fill the card.

What the logs said – and what broke

Run 2: W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort – --fit wants to choose -ngl and context together; forcing -ngl 999 by hand makes it bail, so IQ3_S fell back to the default 4096-token context. (The follow-up below shows that removing -ngl is actually worse on this model – keep it forced.)
Run 1: fifteen W model has unused tensor blk.64.* lines – the whole extra transformer block (attention + FFN + nextn.* MTP head, ~348 MB) is ignored because no --spec-type draft-mtp flag was passed. The MTP head is dead weight in this run; enabling it is the obvious next test (it gave +14.6% on Tiel-Coder 35B-A3B).
Both runs: W srv llama_server: CORS is set to allow all origins ('*') and no API key is set – harmless while the server binds 127.0.0.1 only; never expose this port to a network without an API key.
Both runs: n_ctx_slot = 52736 / 4096, kv_unified = 'true' – unified KV across 4 slots; the 52K figure is what --fit proved could fit alongside the IQ2_XS weights.

Follow-up: letting `--fit` choose, and turning MTP on

I re-ran both configs to answer the two open questions from the first pass. Both answers were unexpected.

IQ3_S without `-ngl` – auto-fit

prompt eval:   3329.43 ms / 373 tokens (   8.93 ms/tok,  112.03 tok/s)
eval (decode): 45080.30 ms / 330 tokens ( 137.02 ms/tok,    7.30 tok/s)
context:       n_ctx_slot = 4096  (fit did NOT grow it)
W resolve_fused_ops: layer 0 is assigned to device CPU but fused
  Gated Delta Net (chunked) is assigned to device ROCm0
W fused Gated Delta Net (chunked) not supported, set to disabled

Dropping -ngl 999 made things dramatically worse: 7.30 tok/s, a third of the manual setting. --fit decided to put layer 0 on the CPU and never grew the context past the default 4K. The warning also reveals what this model actually is: Qwen3.8 is a Gated Delta Net (GDN) hybrid – most layers use linear attention, and ROCm has no fused chunked-GDN kernel, so the hybrid CPU/GPU split collapses throughput. On this card, -ngl 999 (all layers on GPU, accept the small context) is the correct IQ3_S config – my original "mistake" was the right answer.

IQ2_XS with `--spec-type draft-mtp`

prompt eval:   3568.06 ms / 373 tokens (   9.57 ms/tok,  104.54 tok/s)
eval (decode): 8148.60 ms / 189 tokens (  43.34 ms/tok,   23.07 tok/s)
generate:      103 tokens @ 26.73 tok/s (tg), 26.99 tok/s (tg_3s)
draft:         acceptance = 0.48052 (111 accepted / 231 generated), mean len = 2.44
context:       n_ctx_slot = 4096  (--fit aborted: -ngl 999 + draft context)
W llama_sampler_backend_support: device 'ROCm0' does not have
  support for op TOP_K needed for sampler 'top-k'  (×4)

The MTP head is alive: 23.07 tok/s (peaking at 26.7) versus 20.77 without it. Two caveats keep this from being a clean +11%: the MTP run got only 4K context (the draft context eats the headroom --fit would otherwise spend on KV, and with -ngl 999 set, fit bails), and a smaller KV cache flatters decode – at equal context the net MTP gain is closer to +5–8%. The four TOP_K warnings matter too: ROCm can't run the draft sampler's top-k on GPU, which likely caps acceptance at 48% (vs 54.7% on Tiel-Coder). Still the fastest IQ2_XS number on this card so far.

Bottom line

On a 12 GB RX 6800M, IQ2_XS is the practical choice for this 27B: ~21 tok/s decode at 52K context, and ~23 tok/s with --spec-type draft-mtp if you can live with 4K context (or wait for fit/MTP to cooperate). IQ3_S is the better-looking quant this card can't afford: 97% VRAM before any real context, and the "proper" auto-fit path is a trap – it CPU-offloads a GDN layer and decode falls off a cliff at 7.3 tok/s. If you run IQ3_S here, force -ngl 999 and accept 4K. The open questions now belong to the software stack, not the hardware: a fused GDN kernel for ROCm and TOP_K support for the MTP sampler would unlock both the 3 bpw quant and higher draft acceptance.

🧪 More Lab

Back to VelsTech Lab →