# ROCm & Vulkan on AMD: running local AI on Radeon

> ROCm vs Vulkan on AMD GPUs for local AI – what each backend is, which to choose, how to install, and real benchmark results on the RX 6800M.

*Source: https://velstech.net/rocm-vulkan-amd-guide · Updated: 2026-08-30 · Category: AI · Tags: ROCm, Vulkan, AMD, Local AI*

*Markdown version of [ROCm & Vulkan on AMD: running local AI on Radeon](https://velstech.net/rocm-vulkan-amd-guide). [Read the full guide with interactive tools](https://velstech.net/rocm-vulkan-amd-guide).*
*Also as Markdown: [Hindi](https://velstech.net/rocm-vulkan-amd-guide.hi.md) · [Tamil](https://velstech.net/rocm-vulkan-amd-guide.ta.md).*

---

If you run local AI on an AMD Radeon card, you'll hit two competing backends:
**ROCm** and **Vulkan**. Which one you use changes your
tokens-per-second, your setup effort, and how many GPUs are supported. This guide
explains the difference, how to set each up, and – based on our
[RX 6800M benchmarks](https://velstech.net/benchmarks/index) – which to pick.

## What's the difference?

Both are ways for software like llama.cpp and Ollama to talk to your GPU. They solve
the same problem (run the model on the graphics card instead of the CPU) but go about
it differently:

- ROCm is AMD's CUDA equivalent – a purpose-built compute stack with HIP (AMD's CUDA-like language). It's what AMD pushes for AI and HPC.

- Vulkan is a cross-vendor graphics and compute API. It's lower-level and more general, but llama.cpp's Vulkan backend works broadly because it doesn't depend on AMD-specific driver quirks.

The practical differences: ROCm is usually *faster* on supported cards, but
officially supports a narrower list. Vulkan works on more GPUs (including older and
unsupported Radeons) and is easier to get running, but can be a bit slower in some
workloads.

## Which one is faster?

The honest answer from our testing on an **RX 6800M (12 GB)**:
**it depends on the workload.**

- For Qwen 27B at 16K context (dense): Vulkan was ~20% faster at decode (21.8 vs 18.1 tok/s), while ROCm won prompt eval by ~45%.

- For Ornith 35B MoE at 262K: ROCm was ~30% faster at decode (25.6 vs 19.7 tok/s).

Why the flip? The bottleneck changes. On a dense model that's partially offloaded,
the slower part is feeding bytes to the GPU – where Vulkan's simpler path wins. On a
huge-context MoE with many CPU experts, the constraint moves elsewhere and ROCm's
compute advantage shows. The full numbers are in the
[benchmark database](https://velstech.net/benchmarks/index) and the
[MoE vs Dense Lab test](https://velstech.net/moe-vs-dense-rx6800m-16k-vs-262k).

## Setting up ROCm

ROCm is the more involved install. On Ubuntu, you use AMD's installer, choosing the
right GPU version flag for your card. Our step-by-step
[ROCm on Ubuntu guide](https://velstech.net/install-rocm-ubuntu) covers the full flow –
from the apt repo to `rocminfo` verifying your GPU is seen. The short
version:

```
sudo amdgpu-install --usecase=rocm,graphics --gfxversion=gfx1031   # RX 6800M
rocminfo   # verify the GPU shows up
```

Then build llama.cpp with ROCm support:
`cmake .. -DLLAMA_HIPBLAS=ON`. Note that RDNA2 (gfx1030) and RDNA3
(gfx1100) need different flags, and you may need `--load-mode none` in
llama.cpp to avoid splitting weights across GPU and CPU.
If you haven't run it directly before, start with the
[llama.cpp guide](https://velstech.net/llama-cpp-guide) – it covers install, GPU offload, and reading benchmark output.

## Setting up Vulkan

Vulkan is simpler. Install the Vulkan drivers, then build llama.cpp with Vulkan:

```
# Ubuntu: install the Vulkan driver
sudo apt install mesa-vulkan-drivers
# llama.cpp with Vulkan
cmake .. -DLLAMA_VULKAN=ON
make -j$(nproc)
```

Ollama also ships Vulkan builds for AMD. The big win: if your card isn't on ROCm's
official support list, **Vulkan is almost certainly your working option**.

## How to choose

| Your situation | Best backend |
| --- | --- |
| Officially supported Radeon, want max compute speed | ROCm |
| Unsupported / older / laptop Radeon | Vulkan |
| Chat (decode-bound) on a dense model | Vulkan (often) |
| Huge context / MoE workloads | ROCm (often) |
| Simplest setup, widest compatibility | Vulkan |

The safest advice: **try both and benchmark**. Building both backends and
running the same prompt through each takes minutes, and the winner genuinely depends
on your exact card and model. Our Lab does exactly this – see the
[Qwen 27B ROCm vs Vulkan test](https://velstech.net/qwen-27b-ridge-rocm-vs-vulkan) and
[Ornith 35B ROCm vs Vulkan test](https://velstech.net/ornith-35b-moe-262k-rocm-vs-vulkan).

## Bottom line

ROCm and Vulkan both run local AI on AMD, and neither is universally better. ROCm
targets supported cards for max compute; Vulkan covers everything else and often wins
on chat-speed. Start with the simplest backend that works on your card (usually
Vulkan), and don't be afraid to benchmark the other – the numbers, not the marketing,
should decide.

Want to know what your Radeon can do? Check the
[benchmark database](https://velstech.net/benchmarks/index) or the
[GPU AI Performance Calculator](https://velstech.net/gpu-ai-calculator).

## FAQ

**Which is faster, ROCm or Vulkan on AMD?**

It depends on the workload. In our RX 6800M tests, Vulkan won decode on a dense 27B model (+20%), while ROCm won on a huge-context MoE (+30%). Try both and benchmark on your own card.

**Does my AMD GPU support ROCm?**

ROCm officially supports a specific list of cards. If yours isn't on it, Vulkan is almost always the working option – it's the simpler, broader-compatibility backend.

---

*VelsTech – technology explained for everyone. Original: https://velstech.net/rocm-vulkan-amd-guide*