If you run local AI on an AMD Radeon card, you'll hit two competing backends: ROCm and Vulkan. Which one you use changes your tokens-per-second, your setup effort, and how many GPUs are supported. This guide explains the difference, how to set each up, and – based on our RX 6800M benchmarks – which to pick.
What's the difference?
Both are ways for software like llama.cpp and Ollama to talk to your GPU. They solve the same problem (run the model on the graphics card instead of the CPU) but go about it differently:
- ROCm is AMD's CUDA equivalent – a purpose-built compute stack with HIP (AMD's CUDA-like language). It's what AMD pushes for AI and HPC.
- Vulkan is a cross-vendor graphics and compute API. It's lower-level and more general, but llama.cpp's Vulkan backend works broadly because it doesn't depend on AMD-specific driver quirks.
The practical differences: ROCm is usually faster on supported cards, but officially supports a narrower list. Vulkan works on more GPUs (including older and unsupported Radeons) and is easier to get running, but can be a bit slower in some workloads.
Which one is faster?
The honest answer from our testing on an RX 6800M (12 GB): it depends on the workload.
- For Qwen 27B at 16K context (dense): Vulkan was ~20% faster at decode (21.8 vs 18.1 tok/s), while ROCm won prompt eval by ~45%.
- For Ornith 35B MoE at 262K: ROCm was ~30% faster at decode (25.6 vs 19.7 tok/s).
Why the flip? The bottleneck changes. On a dense model that's partially offloaded, the slower part is feeding bytes to the GPU – where Vulkan's simpler path wins. On a huge-context MoE with many CPU experts, the constraint moves elsewhere and ROCm's compute advantage shows. The full numbers are in the benchmark database and the MoE vs Dense Lab test.
Setting up ROCm
ROCm is the more involved install. On Ubuntu, you use AMD's installer, choosing the
right GPU version flag for your card. Our step-by-step
ROCm on Ubuntu guide covers the full flow –
from the apt repo to rocminfo verifying your GPU is seen. The short
version:
sudo amdgpu-install --usecase=rocm,graphics --gfxversion=gfx1031 # RX 6800M rocminfo # verify the GPU shows up
Then build llama.cpp with ROCm support:
cmake .. -DLLAMA_HIPBLAS=ON. Note that RDNA2 (gfx1030) and RDNA3
(gfx1100) need different flags, and you may need --load-mode none in
llama.cpp to avoid splitting weights across GPU and CPU.
Setting up Vulkan
Vulkan is simpler. Install the Vulkan drivers, then build llama.cpp with Vulkan:
# Ubuntu: install the Vulkan driver sudo apt install mesa-vulkan-drivers # llama.cpp with Vulkan cmake .. -DLLAMA_VULKAN=ON make -j$(nproc)
Ollama also ships Vulkan builds for AMD. The big win: if your card isn't on ROCm's official support list, Vulkan is almost certainly your working option.
How to choose
| Your situation | Best backend |
|---|---|
| Officially supported Radeon, want max compute speed | ROCm |
| Unsupported / older / laptop Radeon | Vulkan |
| Chat (decode-bound) on a dense model | Vulkan (often) |
| Huge context / MoE workloads | ROCm (often) |
| Simplest setup, widest compatibility | Vulkan |
The safest advice: try both and benchmark. Building both backends and running the same prompt through each takes minutes, and the winner genuinely depends on your exact card and model. Our Lab does exactly this – see the Qwen 27B ROCm vs Vulkan test and Ornith 35B ROCm vs Vulkan test.
Bottom line
ROCm and Vulkan both run local AI on AMD, and neither is universally better. ROCm targets supported cards for max compute; Vulkan covers everything else and often wins on chat-speed. Start with the simplest backend that works on your card (usually Vulkan), and don't be afraid to benchmark the other – the numbers, not the marketing, should decide.
Want to know what your Radeon can do? Check the benchmark database or the GPU AI Performance Calculator.