Ternary-Bonsai-2-27B on RX 6800M: PTQ1_0 vs PQ2_0 – VelsTech Lab

🧪 VelsTech Lab 12 GB mobile GPU-வில் 27B ternary model இயங்குமா? RX 6800M-ல் Ternary-Bonsai-2-27B – PTQ1_0 (1.75 bits/weight, 5.95 GB) and PQ2_0 (2.13 bits/weight, 7.21 GB) – இரு packing-களையும் சோதித்தேன். Stock llama.cpp இரு கோப்புகளையும் நிராகரிக்கிறது, Prism prebuilt header-ஐ ஏற்றி ROCm-ல் நிற்கிறது, gfx1031 க்கான source build மட்டுமே வேலை செய்கிறது. Build செய்த பிறகு, PQ2_0 இந்த card-ல் கிட்டத்தட்ட 2× வேகமானது.

சோதனை அமைப்பு

• Machine: ASUS ROG Strix G513QY · Ryzen 9 5900HX (8C/16T) · AMD RX 6800M 12GB (RDNA2 gfx1031, mobile) · 30 GB RAM · NVMe
• OS / stack: Ubuntu · ROCm driver 7.1.3 / HIP 7.15 · LD_LIBRARY_PATH=/opt/rocm/lib
• Stock build (fails): ggml-org/llama.cpp 5f436dddb, GGML_HIP=ON, Release
• Working build: PrismML-Eng/llama.cpp branch prism, b10709-9a9394a89, built with -DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1031
• Model A: Ternary-Bonsai-2-27B-PTQ1_0.gguf – 5946648928 bytes, qwen35 arch, ternary g128 {−1,0,+1} + FP16 scales, Hadamard block 1024
• Model B: Ternary-Bonsai-2-27B-PQ2_0.gguf – 7206168928 bytes, same architecture, trit-in-2-bit packing
• Both runs: -ngl 99 -fa on -c 32768 --temp 1.0 --top-p 0.95 --top-k 20 -n 256

எப்படி இயக்கினோம்

Run 0 – stock llama.cpp (fails)

export LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH
llama-cli \
 -m /home/velsrocky/models/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
 -ngl 99 -fa on -c 32768 \
 --temp 1.0 --top-p 0.95 --top-k 20 \
 -p "Explain quantum computing in simple terms." -n 256

gguf_init_from_reader: tensor 'output.weight' has invalid ggml type 143.
llama_model_load: error loading model: failed to load model from
  /home/velsrocky/models/Ternary-Bonsai-2-27B-PTQ1_0.gguf
llama_server exited with code 1

Run 1 – Prism prebuilt ROCm 7.2 (loads header, dies on GPU)

~/llama-prism/llama-cli -m .../Ternary-Bonsai-2-27B-PTQ1_0.gguf \
 -ngl 99 -fa on -c 32768 ...

ROCm error: device kernel image is invalid
ggml_cuda_kernel_launch at ggml/src/ggml-cuda/common.cuh:1715
hipGetLastError()

Even -ngl 0 fails on this binary – HIP backend init probes device 0 and aborts.

Run 2 – Prism fork built for gfx1031 (works)

git clone -b prism https://github.com/PrismML-Eng/llama.cpp.git ~/llama-prism-src
cmake -B build -DCMAKE_BUILD_TYPE=Release \
 -DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1031
cmake --build build -j$(nproc)

export LD_LIBRARY_PATH=$HOME/llama-prism-src/build/bin:/opt/rocm/lib:$LD_LIBRARY_PATH
~/llama-prism-src/build/bin/llama-cli \
 -m /home/velsrocky/models/Ternary-Bonsai-2-27B-PQ2_0.gguf \
 -ngl 99 -fa on -c 32768 \
 --temp 1.0 --top-p 0.95 --top-k 20 \
 -p "Explain quantum computing in simple terms." -n 256

build : b10709-9a9394a89
ftype : PQ2_0 - 2.13 bpw (group 128)
[ Prompt: 64.2 t/s | Generation: 32.8 t/s ]

build : b10709-9a9394a89
ftype : PTQ1_0 - 1.75 bpw ternary (group 128)
[ Prompt: 22.5 t/s | Generation: 18.8 t/s ]

அளவீடுகள்

Metric               PQ2_0               PTQ1_0            Delta
──────────────────────────────────────────────────────────────
File size            7.21 GB             5.95 GB           -17%
True bits/weight     2.13                1.75              -18%
Prompt eval          64.2 tok/s          22.5 tok/s        +185% (PQ2_0 faster)
Decode (gen)         32.8 tok/s          18.8 tok/s        +74%
Context              32768               32768             same
Backend              HIP gfx1031         HIP gfx1031       same

Takeaway: on this RDNA2 card PQ2_0 wins everywhere – prompt processing is compute-bound and dense-trit unpacking costs arithmetic, so the 2-bit-slot packing is ~2.9× faster at prompt eval and ~1.7× faster at decode. PTQ1_0 only wins on footprint: 1.3 GB less weight traffic, the pick when 32K+ KV plus projector must squeeze into 12 GB. This matches Prism's own guidance – PQ2_0 is preferred on HIP/CUDA/Metal/CPU.

Logs சொன்னது – உடைந்தது என்ன

1. Type 143 / 142: GGUF dump shows general.architecture = qwen35, general.file_type = 143, prism.hadamard.* rotation, output.weight type=143 (PTQ1_0) / 142 (PQ2_0). Upstream ggml only defines types 0-42 – these are fork packings with a Hadamard activation transform. Stock llama.cpp has no such runtime, so it refuses them. The files are complete, not truncated.
2. Kernel image invalid: the prism-b10709 ROCm 7.2 prebuilt targets datacenter gfx (CDNA), not gfx1031. HIP init fails before any token is generated. Fix is a source build with -DCMAKE_HIP_ARCHITECTURES=gfx1031 – same flag pattern as the working upstream HIP build. See the ROCm & Vulkan guide and llama.cpp guide.
3. Q2_0 trap: per Prism, never substitute a legacy *-Q2_0.gguf here – stock llama.cpp loads it without warning and outputs garbage because it skips the Hadamard transform. Use PQ2_0 / PTQ1_0 only with fork binaries.

முடிவு

On a 12 GB RX 6800M, use the Prism fork built for gfx1031 and run PQ2_0: ~33 tok/s decode at 32K context. Keep PTQ1_0 for the tightest VRAM budgets. Do not debug type 143/142 as a download error – it is a backend mismatch by design. The authoritative setup is Bonsai-demo (setup.sh + start_llama_server.sh), which pins the right binary per backend.