# Ornith 35B MoE at 262K on RX 6800M: ROCm vs Vulkan – VelsTech Lab > Tested Ornith 1.5 35B-A3B MoE at 262K on RX 6800M – ROCm 25.6 tok/s vs Vulkan 19.6 tok/s decode, 83 tok/s prompt, with 28 CPU experts and q8 KV. *Source: https://velstech.net/ornith-35b-moe-262k-rocm-vs-vulkan · Updated: 2026-08-27 · Category: AI · Tags: VelsTech Lab, MoE, Ornith 35B, RX 6800M, ROCm, Vulkan, Benchmark* *Markdown version of [Ornith 35B MoE at 262K on RX 6800M: ROCm vs Vulkan – VelsTech Lab](https://velstech.net/ornith-35b-moe-262k-rocm-vs-vulkan). [Read the full guide with interactive tools](https://velstech.net/ornith-35b-moe-262k-rocm-vs-vulkan).* *Also as Markdown: [Hindi](https://velstech.net/ornith-35b-moe-262k-rocm-vs-vulkan.hi.md) · [Tamil](https://velstech.net/ornith-35b-moe-262k-rocm-vs-vulkan.ta.md).* --- 🧪 VelsTech Lab Second measured Lab: same 12GB RX 6800M, same q8 KV, but a 35B MoE with 28 CPU experts and a 262K context window. Tiny prompt, huge KV reservation. 🔧 Does this fit your GPU? [LLM VRAM Calculator](https://velstech.net/llm-vram-calculator) – weights + KV at 262K for your quantization · [GPU AI Performance Calculator](https://velstech.net/gpu-ai-calculator) – tok/s upper bound before offload. [Open calculators →](https://velstech.net/llm-vram-calculator) ## What we tested - GPU: RX 6800M 12GB (RDNA2, ~384 GB/s) – same card as Qwen 27B 16K test, so MoE vs dense is apples-to-apples on hardware - Model: Ornith-1.5-35B-A3B-AD-Q5_K-Q4_K.gguf – 35B total, 3B active (A3B MoE), Q5_K + Q4_K mixed quant - MoE CPU offload: --n-cpu-moe 28 – 28 experts forced to CPU (hence tensor overrides to CPU with mmap warning) - Context: -c 262144 -fa on --cache-type-k q8_0 --cache-type-v q8_0, n_slots=1, kv_unified=false – one slot, 262K KV pool reserved - Other: -ngl 999 --jinja --reasoning-preserve --parallel 1 --temp 0.6 --top-p 0.95 --top-k 20, n_threads=8 - Builds: Run 1 – llama.cpp/build/bin/llama-server (ROCm); Run 2 – llama.cpp_vulkan/build-vulkan/bin/llama-server (Vulkan) ## How we ran it ``` llama-server \ -m Ornith-1.5-35B-A3B-AD-Q5_K-Q4_K.gguf \ -ngl 999 --n-cpu-moe 28 \ --cache-type-k q8_0 --cache-type-v q8_0 \ -c 262144 -fa on --jinja --reasoning-preserve \ --parallel 1 --temp 0.6 --top-p 0.95 --top-k 20 ``` Same 335-token prompt both runs (`progress 322→331`), generate ~274 vs 307 tokens. Server: `listening on 127.0.0.1:8080`, `llama threadpool n_threads=8`. ## What we measured ``` Run Prompt eval Decode (tg) Total Graphs ───────────────────────────────────────────────────────────────────────────────────────────── ROCm 3995 ms /335 tok = 11.93 ms/tok = 83.84 tok/s 100:25.16 179:25.52 257:25.60 → 25.62 tok/s 14.65s /609 tok 272 Vulkan 4029 ms /335 tok = 12.03 ms/tok = 83.13 tok/s 100:19.65 160:19.66 219:19.65 → 19.66 tok/s 19.59s /642 tok 305 ``` **Takeaway:** At 262K + MoE, **ROCm is ~30% faster at decode** (25.62 vs 19.66). Prompt eval is tied ~83 tok/s – 335 tokens is trivial vs 262K capacity, so prefill doesn’t stress context. Total wall time reflects decode: ROCm 14.65s vs Vulkan 19.59s for ~600 tok. ## What the logs actually said – and what broke - W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort (ROCm run) – same as Qwen: you forced -ngl 999 so --fit is ignored. Fix: drop -ngl 999 and log fitted layers, or explicitly set -ngl 60 and --n-cpu-moe together. - W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance – with --n-cpu-moe 28 you’re mmap’ing tensors that immediately get copied to CPU RAM. Add --load-mode none for MoE CPU offload. - kv_unified=false, n_slots=1, n_ctx_slot=262144 – one slot, 262K KV pool reserved even though prompt is 335 tokens. That reservation is why you need q8 KV – f16 at 262K would be ~2× larger and likely OOM. Decode speed here is with the pool reserved, not filled. ## How this maps to the calculators Weights ~35B mixed Q5/Q4 ≈ 13–14 GB (Q5 0.66 + Q4 0.61 averaged). KV q8 at 262K for a MoE is not the simple `2×layers×kv_heads×head_dim×2×context` – MoE shares KV per token, so our [VRAM Calculator](https://velstech.net/llm-vram-calculator) overestimates if you enter 35B; enter `3B active` + `28 CPU experts` separately or use custom model with reduced KV. Even at 3B active, 262K q8 is ~262K × ~0.5KB → ~130 GB if fully resident – you’re **not resident**, you’re MoE-sparse + q8 + CPU offload, hence 19–25 tok/s is plausible, not 60 tok/s bound from [GPU AI Performance Calculator](https://velstech.net/gpu-ai-calculator) which assumes full VRAM. How we calculate – the formulas behind the numbers #### VRAM: LLM VRAM Calculator ``` weights = params × bytes_per_weight / GiB kv_per_token = 2 × layers × kv_heads × head_dim × 2 bytes (f16) or ×1 (q8) kv_cache = kv_per_token × context × batch / GiB ``` For MoE at 262K, use active params for KV per token, not total 35B, or toggle custom model. #### Speed: GPU AI Performance Calculator ``` decode_tok/s ≈ (memory_BW × 0.8) / bytes_per_token ``` MoE decodes fewer bytes per token (active 3B, not 35B), so bound is higher than a 35B dense would be. Your 25 tok/s is ~40% of the bound – the gap is CPU MoE + PCIe. ## Bottom line Ornith 35B MoE at 262K **does run on 12GB** with q8 KV + 28 CPU experts, but you pay in decode: ROCm 25.6 tok/s is usable for short answers, Vulkan 19.6 is slower here (inverse of Qwen 16K). Both need `--load-mode none` and no `-ngl 999` for optimal. Next Lab will be **MoE vs dense on the same RX 6800M**: this Ornith 35B-A3B (3B active) vs [Qwen 27B dense 16K](https://velstech.net/qwen-27b-ridge-rocm-vs-vulkan) – active params vs context trade, same card, both engines. 🧪 VelsTech Lab – MoE vs dense next We’ll compare Ornith 35B MoE (3B active, 262K) vs Qwen 27B dense (27B, 16K) on the same RX 6800M – ROCm vs Vulkan, q8 vs f16 KV, with fitted layers logged. [Read Qwen 27B 16K test →](https://velstech.net/qwen-27b-ridge-rocm-vs-vulkan) --- *VelsTech – technology explained for everyone. Original: https://velstech.net/ornith-35b-moe-262k-rocm-vs-vulkan*