Ternary-Bonsai-2-27B on RX 6800M: PTQ1_0 vs PQ2_0 – VelsTech Lab

🧪 VelsTech Lab Can a 27B ternary model run on a 12 GB mobile GPU? I tried both packings of Ternary-Bonsai-2-27B – PTQ1_0 (1.75 bits/weight, 5.95 GB) and PQ2_0 (2.13 bits/weight, 7.21 GB) – on the RX 6800M. Stock llama.cpp refuses both files outright, the Prism prebuilt loads them but dies on ROCm, and only a source build for gfx1031 works. Once built, PQ2_0 is nearly 2× faster on this card.

The test rig

• Machine: ASUS ROG Strix G513QY · Ryzen 9 5900HX (8C/16T) · AMD RX 6800M 12GB (RDNA2 gfx1031, mobile) · 30 GB RAM · NVMe
• OS / stack: Ubuntu · ROCm driver 7.1.3 / HIP 7.15 · LD_LIBRARY_PATH=/opt/rocm/lib
• Stock build (fails): ggml-org/llama.cpp 5f436dddb, GGML_HIP=ON, Release
• Working build: PrismML-Eng/llama.cpp branch prism, b10709-9a9394a89, built with -DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1031
• Model A: Ternary-Bonsai-2-27B-PTQ1_0.gguf – 5946648928 bytes, qwen35 arch, ternary g128 {−1,0,+1} + FP16 scales, Hadamard block 1024
• Model B: Ternary-Bonsai-2-27B-PQ2_0.gguf – 7206168928 bytes, same architecture, trit-in-2-bit packing
• Both runs: -ngl 99 -fa on -c 32768 --temp 1.0 --top-p 0.95 --top-k 20 -n 256

How we ran it

Run 0 – stock llama.cpp (fails)

export LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH
llama-cli \
 -m /home/velsrocky/models/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
 -ngl 99 -fa on -c 32768 \
 --temp 1.0 --top-p 0.95 --top-k 20 \
 -p "Explain quantum computing in simple terms." -n 256

gguf_init_from_reader: tensor 'output.weight' has invalid ggml type 143.
llama_model_load: error loading model: failed to load model from
  /home/velsrocky/models/Ternary-Bonsai-2-27B-PTQ1_0.gguf
llama_server exited with code 1

Run 1 – Prism prebuilt ROCm 7.2 (loads header, dies on GPU)

~/llama-prism/llama-cli -m .../Ternary-Bonsai-2-27B-PTQ1_0.gguf \
 -ngl 99 -fa on -c 32768 ...

ROCm error: device kernel image is invalid
ggml_cuda_kernel_launch at ggml/src/ggml-cuda/common.cuh:1715
hipGetLastError()

Even -ngl 0 fails on this binary – HIP backend init probes device 0 and aborts.

Run 2 – Prism fork built for gfx1031 (works)

git clone -b prism https://github.com/PrismML-Eng/llama.cpp.git ~/llama-prism-src
cmake -B build -DCMAKE_BUILD_TYPE=Release \
 -DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1031
cmake --build build -j$(nproc)

export LD_LIBRARY_PATH=$HOME/llama-prism-src/build/bin:/opt/rocm/lib:$LD_LIBRARY_PATH
~/llama-prism-src/build/bin/llama-cli \
 -m /home/velsrocky/models/Ternary-Bonsai-2-27B-PQ2_0.gguf \
 -ngl 99 -fa on -c 32768 \
 --temp 1.0 --top-p 0.95 --top-k 20 \
 -p "Explain quantum computing in simple terms." -n 256

build : b10709-9a9394a89
ftype : PQ2_0 - 2.13 bpw (group 128)
[ Prompt: 64.2 t/s | Generation: 32.8 t/s ]

build : b10709-9a9394a89
ftype : PTQ1_0 - 1.75 bpw ternary (group 128)
[ Prompt: 22.5 t/s | Generation: 18.8 t/s ]

What we measured

Metric               PQ2_0               PTQ1_0            Delta
──────────────────────────────────────────────────────────────
File size            7.21 GB             5.95 GB           -17%
True bits/weight     2.13                1.75              -18%
Prompt eval          64.2 tok/s          22.5 tok/s        +185% (PQ2_0 faster)
Decode (gen)         32.8 tok/s          18.8 tok/s        +74%
Context              32768               32768             same
Backend              HIP gfx1031         HIP gfx1031       same

Takeaway: on this RDNA2 card PQ2_0 wins everywhere – prompt processing is compute-bound and dense-trit unpacking costs arithmetic, so the 2-bit-slot packing is ~2.9× faster at prompt eval and ~1.7× faster at decode. PTQ1_0 only wins on footprint: 1.3 GB less weight traffic, the pick when 32K+ KV plus projector must squeeze into 12 GB. This matches Prism's own guidance – PQ2_0 is preferred on HIP/CUDA/Metal/CPU.

What the logs said – and what broke

1. Type 143 / 142: GGUF dump shows general.architecture = qwen35, general.file_type = 143, prism.hadamard.* rotation, output.weight type=143 (PTQ1_0) / 142 (PQ2_0). Upstream ggml only defines types 0-42 – these are fork packings with a Hadamard activation transform. Stock llama.cpp has no such runtime, so it refuses them. The files are complete, not truncated.
2. Kernel image invalid: the prism-b10709 ROCm 7.2 prebuilt targets datacenter gfx (CDNA), not gfx1031. HIP init fails before any token is generated. Fix is a source build with -DCMAKE_HIP_ARCHITECTURES=gfx1031 – same flag pattern as the working upstream HIP build. See the ROCm & Vulkan guide and llama.cpp guide.
3. Q2_0 trap: per Prism, never substitute a legacy *-Q2_0.gguf here – stock llama.cpp loads it without warning and outputs garbage because it skips the Hadamard transform. Use PQ2_0 / PTQ1_0 only with fork binaries.

Bottom line

On a 12 GB RX 6800M, use the Prism fork built for gfx1031 and run PQ2_0: ~33 tok/s decode at 32K context. Keep PTQ1_0 for the tightest VRAM budgets. Do not debug type 143/142 as a download error – it is a backend mismatch by design. The authoritative setup is Bonsai-demo (setup.sh + start_llama_server.sh), which pins the right binary per backend.