The LLM Ready USB Stick: Arch ISO + Vulkan + llama.cpp
How a Ventoy USB stick turned into a portable LLM workstation: the stock Arch ISO, booted from any machine, with a persistent partition and nothing but the bare minimum installed: llama.cpp compiled with Vulkan, GPU drivers, and a local model.
- Linux
- Arch Linux
- llama.cpp
- Vulkan
- AMD GPU
- Local LLMs
- Agentic Coding
The Original Purpose
I had a USB stick set up with Ventoy, the universal boot manager. Its job was simple: install Linux distros on bare machines without burning a new stick every time. A utility tool, nothing more.
From Installer to Workstation
Then I did two small changes:
- I took the official Arch ISO (the very one used to install Arch, the live bootable image).
- I added a persistent partition, so that everything done on top of the ISO survives reboots.
That turned an installer into a portable workstation. Plug the stick in any machine with UEFI, boot, and you get an Arch environment that follows you instead of the hardware.
The key point: there is basically nothing on it. No installed system, no desktop, no bloat. Just the live ISO and, on the persistent side, the bare minimum I actually need:
- The AMD GPU drivers and Vulkan stack
- llama.cpp compiled from source with the Vulkan backend
- A couple of model files
- A small set of CLI tools
That’s it. A live system plus a handful of packages. Anything that is not needed for launching, benchmarking, and running local models simply does not exist on the stick.
Why Vulkan and Not CUDA
Because the hardware is AMD. The server this stick lives on has a Radeon 7800 XT, and the whole point of compiling llama.cpp with Vulkan is to run inference on AMD GPUs without any CUDA lock-in. Compiling from source on the persistent volume was the only non-trivial part of the setup, but once it builds, the stick is LLM ready.
What the Stick Can Do
From a cold boot on any compatible machine:
- Launch LLMs locally, from the command line or through an API server
- Benchmark models, quantization levels, and context lengths on identical hardware
- Run inference for real tasks, offline, with no API calls and no data leaving the machine
Where It Runs: the 7800 XT Server
The stick is hosted on one of my servers with the Radeon 7800 XT. Its 16 GB of VRAM is the hard ceiling, and the current record holder is Qwen3.8-27B-i1, which fits the VRAM budget and runs at a usable speed.
The Vulkan backend drives the GPU through an open standard, which is the whole appeal: no vendor tax, no proprietary stack, just a GPU doing matrix multiplications.
What I Actually Use It For
This started as a question: “can I boot an Arch ISO with persistence and run local LLMs on whatever machine I plug it into?”
Now it does real work:
1. Local LLM inference Running Qwen3.8-27B-i1 for tasks that benefit from a local model: summarization, quick reasoning, drafting. Fully offline.
2. Light agentic coding I use the local model to power harnesses like opencode, codex, or my own flowcode for simpler tasks. Not the full frontier-model agentic workflow, but enough to handle refactors, boilerplate, and small features without touching the cloud.
3. Benchmarking Testing models and quantizations on the same hardware, from the same environment. That reproducibility is exactly what the persistent ISO setup gives me.
4. Portability The stick boots on the server for heavy work, or on a laptop for lighter sessions. Same environment, different hardware ceiling.
The Connection to My Writing
This project is the physical side of what I wrote in my agentic coding piece: the models are good enough, the harnesses are good enough, and the infrastructure to run them locally is now a USB stick away.
I am not running 70B models here. I am running real models, for real tasks, offline, on hardware I own, from a stick that boots anywhere. That is more than enough for experimentation, and increasingly enough for day to day work.
What I Learned
1. A live ISO plus persistence is a legitimate workstation You do not need an installed system to have a stable, portable environment. The official installer image is a perfectly good base, as long as your footprint on it stays minimal.
2. Keep the persistent side tiny The stick works precisely because it only carries what is needed: drivers, Vulkan, llama.cpp, models. Anything else is dead weight that slows boot and complicates support.
3. Vulkan is the underdog that works on AMD CUDA is the default assumption in the LLM world. If you are on AMD, Vulkan is the reality. Compiling llama.cpp against it is a rite of passage, but it pays off.
4. Local inference has no magic No rate limits, no per-token cost, but also no abstraction: you manage the context window, the VRAM budget, and the model files yourself. More work, total understanding.
5. The stick is the lab Every experiment starts there. If it works on the stick, it works on any machine that can boot it.