Hermes llama.cpp support is built in since Hermes Agent v0.22.0: Hermes installs and maintains llama.cpp for you, picks the right engine for your hardware, rates models against your GPU and RAM, and runs them entirely on your machine with no account and no API key. You can also point Hermes at a llama.cpp server you run yourself.
📺 Watch: Hermes Desktop Can Now Set Up Local AI in ONE Click
🔥 Get the Agent OS as a free bonus: AI Profit Boardroom members get the full Agent OS zip, prompt libraries, daily tutorials and weekly live coaching calls. → Get inside
Before v0.22, running a local model meant compiling llama.cpp, guessing a quantization, fiddling with GPU layers and hoping the context fit. Now it's a few clicks. Here's the one-click setup, how Hermes decides what fits, the provider config for your own server, and the limits you should know about.
What one-click local models actually do
Nous Research shipped this in v0.22.0, published on GitHub on 14 October 2026. Per the release notes, Hermes:
- Installs and maintains llama.cpp with the right engine for your hardware: CPU, CUDA, Metal, Vulkan or ROCm.
- Rates every catalog model against your GPU and RAM and picks the highest-quality build that fits.
- Downloads Hugging Face GGUFs with pause and resume.
- Starts and stops the server with Hermes, and engine updates are one click.
- Shows a self-started llama.cpp server in /model, so a server you launch by hand appears in the model picker too.
The catalog named in the notes includes Qwen3.8 27B, Qwen3.8 Flash Next with its MTP speculative head, Qwen3.6 35B-A3B and DeepSeek V4 Flash, plus any GGUF from Hugging Face. At v0.22.0 every backend ran llama.cpp build b11370.
The docs put the privacy side simply: nothing leaves your computer. No account, no API key and no network access after a model is downloaded.
I covered this flow in my one-click local AI video, from the four-click setup to the green, amber and red fit labels. If you want the full Agent OS around it, the AI Profit Boardroom ($69/mo) gives you the Agent OS, a 30-day AI agent roadmap, daily tutorials as new tools drop, 4 coaching calls a week and the prompt library, with 3,400+ members inside.
📺 Watch: Run Hermes Desktop With Ollama for FREE
Hermes llama.cpp setup in four steps
In the desktop app, Local Models is on by default on macOS, Windows and Linux, x64 and arm64. The docs give this flow:
- Open Settings, then Providers, then Local Models. Or choose Run models locally during onboarding.
- Click Install runtime. Hermes downloads the official llama.cpp build for your hardware, a few hundred MB, verifies it and keeps it updated.
- Pick a model and click Download.
- Click Use. New chats now run on the local model.
Switching back to a cloud model is one click in the model picker. Stuck on which route suits your work? A free AI strategy session sorts that out fast.
Prefer a terminal? Run hermes model and choose Local models. The docs say it lists the catalog priced for your machine with a star on the recommendation, installs the pinned engine, downloads with live progress, starts the server and makes the model your default. Ctrl+C pauses a download. Pick the model again to resume.
📺 Watch: Hermes Desktop + Ollama is Insane (FREE!)
How Hermes decides what fits your machine
This is the part that saves you from out-of-memory crashes. Every catalog model is priced against your machine before you download anything, and each row shows a memory-fit label:
- Green, Fits your GPU: runs entirely in GPU memory.
- Amber, Uses system RAM: works, but slower.
- Red, Too big for this machine.
Hermes recommends the highest-quality model that runs entirely in GPU or unified memory at a predicted 20 tokens per second or more. If nothing hits that floor, it recommends nothing, and you can still pick any model that fits. It never offers builds below 4-bit, because the docs say the quality loss is too severe.
Memory is handled for you too. Every recommended model gets at least a 64K context window, which grows toward the model's native maximum as your conversation needs room. When a model spills past GPU memory, Hermes puts the overflow in system RAM in the order that hurts least: expert weights first, never the attention cache. Idle models unload after 15 minutes and reload on your next message.
Want live numbers? Right-click the status bar and turn on System resources to see GPU utilization, GPU memory and RAM.
Engines by platform
The requirements section of the docs lists which engine you get where:
| Platform | Engines | Notes from the docs |
|---|---|---|
| Windows | CUDA, Vulkan, CPU | CUDA on supported NVIDIA targets, Vulkan on x64 |
| Linux | CUDA, Vulkan, CPU | CUDA on x64 and arm64 NVIDIA, needs a CUDA 13 driver |
| macOS | Metal, CPU | Metal or CPU |
| Supported x64 targets | HIP/ROCm | An explicit choice, not picked automatically |
On auto, Hermes prefers CUDA for NVIDIA and Vulkan for AMD and Intel GPUs, then falls back to CPU when no GPU is found. A discrete card behind Vulkan or HIP is sized from its own memory. As a rough guide, the docs say a GPU with 8 GB or more runs the small catalog models comfortably, and 16 GB or more runs the 27 to 35B models at high quality.
Running this on a Windows laptop? v0.22.2 sized model launch and the prompt cache to free memory on Windows unified-memory machines. More in my Hermes Agent Windows guide.
Hugging Face GGUFs and models already on disk
The catalog is a starting point. The Find more models section on the same page searches all of Hugging Face, with download counts and a per-file fit check sized to your machine. A model you download there behaves like a catalog model, because Hermes reads the model file itself to pick its context window and memory placement. The only difference: community models don't carry the "validated" badge.
Already have a .gguf file? Add model file links it into your library without copying it, and it's usable straight away.
Downloads are built for flaky connections. Pause stops at a chunk boundary, paused jobs stay visible when you come back, and hosts that support ranges resume only the missing bytes.
Hermes llama.cpp provider config for your own server
The managed runtime is a default, not a requirement. The docs confirm two manual routes.
Auto-detect. If a llama-server is already running on your machine, Hermes detects it and uses it instead of starting its own. The default probe is port 8080 only. Running on another port? Add it to local_runtime.detect_ports in config.yaml.
Explicit provider. Define the endpoint under providers in config.yaml, with a llamacpp entry holding base_url: http://127.0.0.1:8081/v1 and model: my-model. The docs say an explicit entry wins over detection. Your main model then uses model.provider: llamacpp with model.default set to the model.
You can also add a custom endpoint pointing at any OpenAI-compatible server. Enter the server root, like http://127.0.0.1:8080, or the full /v1 URL. The endpoint test tries both and saves whichever actually served /models.
Other config keys in the local_runtime section:
- enabled: true starts the managed server with Hermes. The desktop Use button sets this for you.
- backend: auto, cuda, metal, vulkan, hip or cpu.
If no server is reachable, the error now says the local model server isn't running, instead of a confusing missing-API-key message. Engine versions come only from Hermes's own lockfile, so update the engine with the desktop update button.
Using Ollama instead of llama.cpp? Follow my guide on how to set up Hermes with Ollama. And if you want one profile on a local model and another on a cloud model, that's what Hermes multiple profiles are for.
Not sure which model your business work needs? Check Goldie Bench for how models handle real agent tasks, read the Agent OS guide for the bigger setup, or book a free AI strategy session.
FAQ
Do I need to install llama.cpp myself?
No. Click Install runtime and Hermes downloads, verifies and updates the official llama.cpp build for your hardware. You only run your own server if you want full manual control.
Can Hermes use a GGUF I already downloaded?
Yes. Add model file links an existing .gguf into your library without copying it.
Why doesn't Hermes recommend any model on my machine?
Hermes only recommends a model that runs fully in GPU or unified memory at a predicted 20 tokens per second or more. If none hits that, you can still choose any model that fits.
Does a local model need internet?
Only to download the engine and model. After that, the docs say nothing leaves your computer.
Ready to run AI agents on your own hardware? Join the AI Profit Boardroom for the Agent OS and daily tutorials, or book a free AI strategy session and we'll plan your local setup.
