# MoE vs Dense on RX 6800M: 3B Active vs 27B at 16K/262K – VelsTech Lab

> Same RX 6800M 12GB – Ornith 35B MoE (3B active, 262K) vs Qwen 27B dense (27B, 16K) at q8 KV, ROCm vs Vulkan head-to-head.

*Source: https://velstech.net/moe-vs-dense-rx6800m-16k-vs-262k · Updated: 2026-08-27 · Category: AI · Tags: VelsTech Lab, MoE vs Dense, RX 6800M, Benchmark*

*Markdown version of [MoE vs Dense on RX 6800M: 3B Active vs 27B at 16K/262K – VelsTech Lab](https://velstech.net/moe-vs-dense-rx6800m-16k-vs-262k). [Read the full guide with interactive tools](https://velstech.net/moe-vs-dense-rx6800m-16k-vs-262k).*
*Also as Markdown: [Hindi](https://velstech.net/moe-vs-dense-rx6800m-16k-vs-262k.hi.md) · [Tamil](https://velstech.net/moe-vs-dense-rx6800m-16k-vs-262k.ta.md).*

---

🧪 VelsTech Lab Same card, same engines, two opposite ways to scale: Qwen 27B dense at 16K vs Ornith 35B-A3B MoE (3B active) at 262K, both at q8 KV on a 12GB RX 6800M. Which wins depends on what you ask vs how much you generate.

🔧 Re-run the math for your model

MoE: enter *active* params (3B, not 35B) + custom KV. Dense: enter full params. Keep `q8_0` vs `f16` honest.

[LLM VRAM Calculator →](https://velstech.net/llm-vram-calculator) [GPU Performance →](https://velstech.net/gpu-ai-calculator)

## The two runs side-by-side

Same GPU **RX 6800M 12GB** (~384 GB/s), same `q8_0` KV, same `n_threads=8`, same 335-373 prompt tokens. Only model + context + MoE offload differ.

```
Model               Params   Context  KV type  Offload          Build   Prompt eval          Decode tg          Total
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Qwen 27B Ridge       27B dense  16K   q8_0     -ngl 999 (+fit abort)  ROCm    1713 ms /373 = 217.6 tok/s   18.12 tok/s   14.23s /601 tok
3.7bpw Ridge                                                Vulkan  2493 ms /373 = 149.6 tok/s   21.84 tok/s   13.57s /616 tok

Ornith 35B-A3B MoE   35B/3B*    262K  q8_0     --n-cpu-moe 28        ROCm    3995 ms /335 = 83.8 tok/s    25.62 tok/s   14.65s /609 tok
Q5_K/Q4_K mix               kv_unified=false              Vulkan  4029 ms /335 = 83.1 tok/s    19.66 tok/s   19.59s /642 tok
* 35B total, 3B active – MoE decodes 3B, not 35B.
```

First two rows: [Qwen 27B 16K full report](https://velstech.net/qwen-27b-ridge-rocm-vs-vulkan); last two: [Ornith 35B 262K full report](https://velstech.net/ornith-35b-moe-262k-rocm-vs-vulkan). All at `-c` reserved KV, `fa on`, 373/335 prompt tokens, ~600 gen tokens.

## What actually happens

- Decode is MoE’s win at scale, but not universally. At 16K dense, Vulkan wins +20% (21.84 vs 18.12). At 262K MoE, ROCm wins +30% (25.62 vs 19.66). Same card, opposite engine winner – the bottleneck flips from pure VRAM BW (dense 27B) to CPU-MoE + PCIe (28 experts on CPU) at 262K.

- Prompt eval collapses with context reservation, not prompt size. Qwen 16K prompt 217 tok/s vs Ornith 262K prompt 83 tok/s – same 335 tokens, but Ornith reserves a 262K KV pool (kv_unified=false, n_slots=1) that costs prefill bookkeeping. You pay for -c even when prompt is short.

- MoE decodes far fewer bytes per token. Dense decodes 27B weights per token; MoE decodes ~3B active. That’s why Ornith at 262K can still do 25.6 tok/s on 12GB while Qwen at 16K does 18-21 – active, not total, matters. Our GPU calculator bound decode ≈ BW×0.8 / bytes_per_token predicts ~30 tok/s if resident; the gap to 25.6 is CPU MoE + offload.

- Both forced -ngl 999 → failed to fit params... abort. --fit on was ignored. Next Lab will drop -ngl and log fitted layers, plus add --load-mode none for MoE (fixes tensor overrides to CPU with mmap).

## VRAM math – why 12GB still runs

Qwen 27B 3.7bpw: `27B ×0.4625 ≈12.5 GB` weights + q8 KV 16K ≈2.2 GB + 6% overhead → ~15.6 GB (from [VRAM Calculator](https://velstech.net/llm-vram-calculator)) – well over 12GB, so partial offload explains 18-21 tok/s vs 30 tok/s bound.

Ornith 35B MoE 3B active: 35B mix Q5/Q4 ≈13-14 GB if counted as 35B, but KV per token scales with *active* 3B, so 262K q8 is ~262K×0.5KB ≈131 GB if naively counted as dense. That’s why we force 28 experts to CPU – you’re not resident, you’re sparse. Enter `3B active` + custom `layers 48 / kv 4 / hd 128` in the calculator, not 35B, or toggle `Custom model…`.

How we calculate – the formulas behind the numbers

#### VRAM

```
weights = params × bytes_per_weight / GiB
kv_per_token = 2 × layers × kv_heads × head_dim × bytes
kv_cache = kv_per_token × context × batch / GiB
total = weights + kv_cache + (weights+kv)×0.06+0.25
```

For MoE, use *active* params for KV, not total. See [LLM VRAM Calculator](https://velstech.net/llm-vram-calculator) → Custom model.

#### Speed bound

```
decode_tok/s ≈ (memory_BW ×0.8) / bytes_per_token (weights + KV per token)
```

MoE bytes_per_token ≈ active 3B, dense ≈27B – hence Ornith can be faster than Qwen despite larger total.

## So which should you pick on a 12GB card?

- Short prompts + long answers (chat, 16K window): Qwen 27B dense on Vulkan (21.8 tok/s) is the better chat feel.

- Huge context + MoE (262K window, 28 CPU experts): Ornith 35B MoE on ROCm (25.6 tok/s) wins decode despite the window, but needs --load-mode none and patience on first prompt (83 tok/s).

- Next optimization: Both need no -ngl 999, log fitted layers, and test q8 vs f16 KV at each context. That’s Lab #4.

🧪 Dive into the raw runs

Full logs, commands and warnings are in the two source Labs – start with Qwen 16K, then Ornith 262K, then compare here.

[Qwen 27B 16K →](https://velstech.net/qwen-27b-ridge-rocm-vs-vulkan) [Ornith 35B 262K →](https://velstech.net/ornith-35b-moe-262k-rocm-vs-vulkan)

---

*VelsTech – technology explained for everyone. Original: https://velstech.net/moe-vs-dense-rx6800m-16k-vs-262k*