Ternary-Bonsai-2-27B on RX 6800M: PTQ1_0 vs PQ2_0 – VelsTech Lab

🧪 VelsTech Lab क्या 12 GB मोबाइल GPU पर 27B ternary मॉडल चल सकता है? मैंने RX 6800M पर Ternary-Bonsai-2-27B – PTQ1_0 (1.75 bits/weight, 5.95 GB) and PQ2_0 (2.13 bits/weight, 7.21 GB) – दोनों packings आज़माईं। Stock llama.cpp दोनों files को सीधे reject कर देता है, Prism prebuilt header load करता है पर ROCm पर crash होता है, और सिर्फ gfx1031 के लिए source build ही चलता है। Build के बाद, PQ2_0 इस card पर लगभग 2× तेज़ है।

टेस्ट rig

• Machine: ASUS ROG Strix G513QY · Ryzen 9 5900HX (8C/16T) · AMD RX 6800M 12GB (RDNA2 gfx1031, mobile) · 30 GB RAM · NVMe
• OS / stack: Ubuntu · ROCm driver 7.1.3 / HIP 7.15 · LD_LIBRARY_PATH=/opt/rocm/lib
• Stock build (fails): ggml-org/llama.cpp 5f436dddb, GGML_HIP=ON, Release
• Working build: PrismML-Eng/llama.cpp branch prism, b10709-9a9394a89, built with -DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1031
• Model A: Ternary-Bonsai-2-27B-PTQ1_0.gguf – 5946648928 bytes, qwen35 arch, ternary g128 {−1,0,+1} + FP16 scales, Hadamard block 1024
• Model B: Ternary-Bonsai-2-27B-PQ2_0.gguf – 7206168928 bytes, same architecture, trit-in-2-bit packing
• Both runs: -ngl 99 -fa on -c 32768 --temp 1.0 --top-p 0.95 --top-k 20 -n 256

हमने कैसे चलाया

Run 0 – stock llama.cpp (fails)

export LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH
llama-cli \
 -m /home/velsrocky/models/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
 -ngl 99 -fa on -c 32768 \
 --temp 1.0 --top-p 0.95 --top-k 20 \
 -p "Explain quantum computing in simple terms." -n 256

gguf_init_from_reader: tensor 'output.weight' has invalid ggml type 143.
llama_model_load: error loading model: failed to load model from
  /home/velsrocky/models/Ternary-Bonsai-2-27B-PTQ1_0.gguf
llama_server exited with code 1

Run 1 – Prism prebuilt ROCm 7.2 (loads header, dies on GPU)

~/llama-prism/llama-cli -m .../Ternary-Bonsai-2-27B-PTQ1_0.gguf \
 -ngl 99 -fa on -c 32768 ...

ROCm error: device kernel image is invalid
ggml_cuda_kernel_launch at ggml/src/ggml-cuda/common.cuh:1715
hipGetLastError()

Even -ngl 0 fails on this binary – HIP backend init probes device 0 and aborts.

Run 2 – Prism fork built for gfx1031 (works)

git clone -b prism https://github.com/PrismML-Eng/llama.cpp.git ~/llama-prism-src
cmake -B build -DCMAKE_BUILD_TYPE=Release \
 -DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1031
cmake --build build -j$(nproc)

export LD_LIBRARY_PATH=$HOME/llama-prism-src/build/bin:/opt/rocm/lib:$LD_LIBRARY_PATH
~/llama-prism-src/build/bin/llama-cli \
 -m /home/velsrocky/models/Ternary-Bonsai-2-27B-PQ2_0.gguf \
 -ngl 99 -fa on -c 32768 \
 --temp 1.0 --top-p 0.95 --top-k 20 \
 -p "Explain quantum computing in simple terms." -n 256

build : b10709-9a9394a89
ftype : PQ2_0 - 2.13 bpw (group 128)
[ Prompt: 64.2 t/s | Generation: 32.8 t/s ]

build : b10709-9a9394a89
ftype : PTQ1_0 - 1.75 bpw ternary (group 128)
[ Prompt: 22.5 t/s | Generation: 18.8 t/s ]

क्या मापा

Metric               PQ2_0               PTQ1_0            Delta
──────────────────────────────────────────────────────────────
File size            7.21 GB             5.95 GB           -17%
True bits/weight     2.13                1.75              -18%
Prompt eval          64.2 tok/s          22.5 tok/s        +185% (PQ2_0 faster)
Decode (gen)         32.8 tok/s          18.8 tok/s        +74%
Context              32768               32768             same
Backend              HIP gfx1031         HIP gfx1031       same

निष्कर्ष: on this RDNA2 card PQ2_0 wins everywhere – prompt processing is compute-bound and dense-trit unpacking costs arithmetic, so the 2-bit-slot packing is ~2.9× faster at prompt eval and ~1.7× faster at decode. PTQ1_0 only wins on footprint: 1.3 GB less weight traffic, the pick when 32K+ KV plus projector must squeeze into 12 GB. This matches Prism's own guidance – PQ2_0 is preferred on HIP/CUDA/Metal/CPU.

Logs ने क्या कहा – और क्या टूटा

1. Type 143 / 142: GGUF dump shows general.architecture = qwen35, general.file_type = 143, prism.hadamard.* rotation, output.weight type=143 (PTQ1_0) / 142 (PQ2_0). Upstream ggml only defines types 0-42 – these are fork packings with a Hadamard activation transform. Stock llama.cpp has no such runtime, so it refuses them. The files are complete, not truncated.
2. Kernel image invalid: the prism-b10709 ROCm 7.2 prebuilt targets datacenter gfx (CDNA), not gfx1031. HIP init fails before any token is generated. Fix is a source build with -DCMAKE_HIP_ARCHITECTURES=gfx1031 – same flag pattern as the working upstream HIP build. See the ROCm & Vulkan guide and llama.cpp guide.
3. Q2_0 trap: per Prism, never substitute a legacy *-Q2_0.gguf here – stock llama.cpp loads it without warning and outputs garbage because it skips the Hadamard transform. Use PQ2_0 / PTQ1_0 only with fork binaries.

निष्कर्ष

On a 12 GB RX 6800M, Prism fork को gfx1031 and run PQ2_0: ~33 tok/s decode at 32K context. Keep PTQ1_0 for the tightest VRAM budgets. Do not debug type 143/142 as a download error – it is a backend mismatch by design. The authoritative setup is Bonsai-demo (setup.sh + start_llama_server.sh), which pins the right binary per backend.