🧪 VelsTech Lab – Coming soon This is the first measured post in the Lab format: exact hardware, software, command, and raw llama-server timings – including what broke.

🔧 Check your own fit first

LLM VRAM Calculator – does 27B 3.7bpw + 16K fit your VRAM? · GPU AI Performance Calculator – what tok/s to expect before you run.

Open calculators →

What we tested

How we ran it

export LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH
llama-server \
 -m /home/user-name/models/Qwen3.8-27B-Ridge-3.7bpw.gguf \
 -ngl 999 \
 --cache-type-k q8_0 --cache-type-v q8_0 \
 -c 16384 -fa on --fit on --reasoning-preserve --jinja

Two backends, identical flags. Server reports listening on http://127.0.0.1:8080 then processes a 373-token prompt and generates ~228-243 tokens (600 token budget, truncated 0).

What we measured

Run            Prompt eval                Decode (tg)                Total
─────────────────────────────────────────────────────────────────────────────
ROCm  (build)  1713 ms / 373 tok = 4.60 ms/tok = 217.6 tok/s   100:18.42  155:18.12  210:18.08 → 18.12 tok/s   14.24s / 601 tok  graphs reused 226
Vulkan        2493 ms / 373 tok = 6.68 ms/tok = 149.6 tok/s   100:22.09  166:21.83  232:21.82 → 21.84 tok/s   13.57s / 616 tok  graphs reused 241

Takeaway: Vulkan is ~20% faster at decode (21.84 vs 18.12 tok/s) – the part that matters for chatting. ROCm is ~45% faster at prompt eval (217 vs 149 tok/s) – the time to first token on a long prompt. End-to-end for 615 tokens, Vulkan wins by 0.67s (13.57 vs 14.23) because decode dominates.

What the logs actually said – and what broke

  1. W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort – You forced -ngl 999 so --fit on is ignored. On a 12 GB card this 27B q8-KV run must partially offload; forcing 999 hides how many layers actually fit. Fix next run: drop -ngl 999 and let --fit on choose, or set -ngl 60 and log n_gpu_layers.
  2. 14× W model has unused tensor blk.64.* (size ~20KB–73MB) – Model header advertises 65 blocks (0-64) but block 64 is Qwen3 nextn speculative head (eh_proj/enorm/hnorm/shared_head_norm). llama.cpp ignores it when not drafting with --jinja --reasoning-preserve under this build. Not an error, but you load ~300 MB of dead weight.
  3. kv_unified=true, n_slots=4, n_ctx_slot=16384 – 4 slots share one KV pool. If you hammer parallel requests at 16K each, you’ll OOM faster than the single-slot test shows.

How this maps to the calculators

Weights: 27B × 0.4625 bytes (3.7bpw/8) ≈ 12.5 GB. KV q8 at 16K: our VRAM Calculator estimates ~1.0 GB at 8K with 2×layers×kv_heads×head_dim×2 bytes×context – at 16K q8 it's ~2× that, so ~2.2 GB. Plus 6% overhead + 0.25 GB → total ~15.6 GB, well above 12 GB. The fact it ran at all proves partial offload (via system RAM) – which explains the ~18-21 tok/s vs the GPU AI Performance Calculator upper bound of ~30 tok/s if fully resident. That delta is the offload tax.

How we calculate – the formulas behind the numbers

Same logic as the calculators, linked so you can verify.

VRAM: LLM VRAM Calculator

weights = params × bytes_per_weight / GiB
kv_per_token = 2 × layers × kv_heads × head_dim × 2 bytes
kv_cache = kv_per_token × context × batch / GiB

Speed: GPU AI Performance Calculator

decode_tok/s ≈ (memory_BW × 0.8) / bytes_per_token

Your measured 18-21 tok/s is below the 30 tok/s bound because bytes_per_token includes PCIe + RAM offload, not just VRAM BW.

Bottom line

On a 12 GB RX 6800M at 16K, Qwen 27B Ridge 3.7bpw does run, but not fully resident. If you chat (decode-bound), Vulkan’s +20% is real. If you feed long documents, ROCm’s faster prefill matters more. For a Lab verdict: usable but offloaded – expect 18-21 tok/s at 16K q8 KV, slower at 32K. Want it comfortably resident? Drop to c 8192, use q8_0→f16 only if you have headroom, or step up to 16/24 GB.

🧪 More Lab

Next: same model with --fit on without -ngl 999 (show fitted layers), and q8_0 vs f16 KV at 16K. Then 70B range is cloud only.

Back to VelsTech Lab →