Run Hermes Fully Locally with QVAC

The system we are building

Hermes Agent is the agent layer around a language model. It manages conversations, state, memory, skills, tool descriptions, and the loop that connects a model request to an actual action. QVAC is responsible for running the model locally and exposing it through an OpenAI-compatible HTTP interface.

User
  │
  ▼
Hermes Agent
  ├── local sessions
  ├── persistent memory
  ├── skills and policies
  └── restricted tools
          │
          │ HTTP over 127.0.0.1
          ▼
    QVAC OpenAI server
          │
          ▼
      local model
          │
          ▼
      CPU / GPU / RAM

The roles must remain separate. The large language model produces a response or requests a function. Hermes controls the agent loop, memory, permissions, and function execution. QVAC loads the model weights, applies the model's chat template, and performs local inference.

In this guide, “fully local” has a precise and limited meaning:

• Hermes and QVAC run on one computer, or inside an isolated local environment;
• model requests are sent only to 127.0.0.1;
• sessions, memory, skills, configuration, and logs remain on local storage;
• the tools used by the acceptance test require no external services;
• after dependencies and weights have been downloaded, the scenario passes without internet access.

Concrete case: a private decision journal

A greeting is too weak to prove that an agent works. Our acceptance case therefore uses a small project directory and tests four independent layers: generation, tool calling, durable memory, and the absence of a required cloud route.

Hermes must:

1. read a local meeting note;
2. extract the project, decision, and next action;
3. write a Markdown summary inside an allowed output directory;
4. remember a harmless formatting preference;
5. recover that preference in a new session;
6. repeat a new file task after external networking is disabled.

Create a dedicated laboratory directory. Do not begin with your home directory, a repository containing credentials, or a cloud-synchronized folder.

mkdir -p "$PWD/hermes-local-lab/inbox"
mkdir -p "$PWD/hermes-local-lab/outbox"

printf '%s\n' \
  'Project: local assistant.' \
  'Decision: store final notes as Markdown.' \
  'Next action: verify operation without a network.' \
  > "$PWD/hermes-local-lab/inbox/meeting.txt"

export LAB_DIR="$PWD/hermes-local-lab"

find "$LAB_DIR" -maxdepth 2 -type f -print
printf 'Laboratory directory: %s\n' "$LAB_DIR"

Keep this terminal open or record the absolute value of LAB_DIR. Agent prompts should use an absolute path rather than relying on an uncertain working directory.

Prerequisites and version discipline

You need a Unix-like environment, Node.js and npm for QVAC, a Python environment supported by the Hermes release you install, Git, curl, and enough disk space for the selected model. On Windows, use WSL2 unless the current release explicitly documents another supported route.

Before changing the machine, record the software and available resources:

uname -a
node --version
npm --version
python3 --version
git --version
curl --version | head -n 1

df -h .
free -h 2>/dev/null || true

CLI flags, model identifiers, and configuration formats can change between releases. Treat every command below as a configuration for the installed version, not as permission to ignore its built-in help. If a flag is rejected, stop and inspect --help instead of guessing a replacement.

After installing QVAC, use its diagnostics to identify the available compute backend:

qvac doctor

A supported GPU backend can reduce latency, while CPU execution may still be adequate for a small model. The actual result depends on the machine, model build, context length, and runtime version; this article does not claim universal speed or memory measurements.

Choose a model that fits before optimizing quality

Begin with a model variant using 4-bit quantization. Quantization reduces the storage and memory cost of weights, but it does not eliminate runtime buffers or the memory consumed by the context cache.

The QVAC model registry available in your installed release is the source of truth. List or inspect that registry before copying a model identifier into the configuration. If the following identifiers are not present in your release, select the corresponding supported instruct model and use its exact identifier.

A mixture-of-experts model activates only part of its expert network for each token, but its weights and working data still have to fit the runtime's memory strategy. Leave capacity for the operating system, Hermes, the context, and the KV cache.

The smallest model is useful for validating the route, but it may emit invalid function arguments or skip steps. A larger model can improve agent behavior, yet no model size guarantees correct tool use. First complete the whole test with a model that fits comfortably; compare alternatives only after the infrastructure is stable.

Step 1: install the QVAC CLI

Internet access is required at this stage to retrieve the package and, later, the model weights.

npm install -g @qvac/cli

command -v qvac
qvac --help
qvac doctor

If global npm installation asks for administrator privileges, do not solve it by piping an unreviewed process into sudo. Configure a user-owned npm prefix or use a managed Node.js installation. Confirm that the discovered qvac binary belongs to the installation you intended.

Create a separate QVAC project directory:

mkdir -p "$PWD/qvac-hermes"
cd "$PWD/qvac-hermes"
pwd

Step 2: declare the model and enable functions

The HTTP server loads models declared under serve.models. The object key becomes the model name visible to clients. Using a stable alias keeps Hermes independent of the underlying registry identifier.

Create qvac.config.json in the current directory:

{
  "serve": {
    "models": {
      "hermes-local": {
        "model": "QWEN3_6_35B_A3B_MULTIMODAL_Q4_K_M",
        "default": true,
        "preload": true,
        "config": {
          "ctx_size": 65536,
          "tools": true
        }
      }
    }
  }
}

For a smaller model, change only the model value to an identifier verified in your installed QVAC registry, for example:

"QWEN3_5_4B_MULTIMODAL_Q4_K_M"

or:

"QWEN3_5_9B_MULTIMODAL_Q4_K_M"

ctx_size is the maximum context the runtime attempts to hold. Hermes may send a large system instruction plus multiple tool schemas before the user's task begins. A small context can overflow early, while a large token limit increases KV-cache memory.

If 65,536 tokens do not fit, lower the value, expose fewer tools, and configure Hermes with the same effective limit. Do not advertise a larger context to Hermes than QVAC can actually serve.

The tools option enables function handling for the selected model. Without it, ordinary chat can succeed while the agent returns prose instead of a structured call.

Step 3: start the OpenAI-compatible server

Bind QVAC to the loopback interface so that other devices cannot reach the port through the LAN.

cd /absolute/path/to/qvac-hermes

qvac serve openai \
  --config "$PWD/qvac.config.json" \
  --host 127.0.0.1 \
  --port 11434 \
  --verbose

Keep this process in a dedicated terminal. The first start may download weights before loading them into memory. Wait until server initialization has completed.

From another terminal, inspect the model endpoint:

curl --fail --silent --show-error \
  http://127.0.0.1:11434/v1/models

The returned JSON should contain the client-facing identifier hermes-local. An empty list usually means the model is not ready. A 404 indicates a route, configuration, or CLI-version mismatch.

Now send a minimal completion:

curl --fail --silent --show-error \
  http://127.0.0.1:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "hermes-local",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly one word: ready"
      }
    ],
    "temperature": 0,
    "max_tokens": 16
  }'

Validate the response structure rather than punctuation or capitalization. It must contain a choices array with an assistant message. If HTTP succeeds but content is empty, inspect the verbose QVAC log and verify that the chosen registry entry is a chat-capable instruct model.

Step 4: test function output before installing Hermes

This test separates QVAC and model behavior from the agent runtime. The described function performs no action; the request only asks the model to return a structured intention.

curl --fail --silent --show-error \
  http://127.0.0.1:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "hermes-local",
    "messages": [
      {
        "role": "user",
        "content": "Find the size of /lab/inbox/meeting.txt. Do not guess; use the function."
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_file_size",
          "description": "Return the size of an allowed local file",
          "parameters": {
            "type": "object",
            "properties": {
              "path": {
                "type": "string",
                "description": "Absolute path to the file"
              }
            },
            "required": ["path"],
            "additionalProperties": false
          }
        }
      }
    ],
    "tool_choice": "auto",
    "temperature": 0,
    "max_tokens": 256
  }'

A successful result is not an alleged byte count. It is an item in message.tool_calls whose function name is get_file_size and whose arguments value contains valid JSON with the requested path.

Fail the test if:

• the model says that it will call a function but returns no tool_calls;
• the function name is changed;
• arguments cannot be parsed as JSON;
• the model invents a file size;
• the requested path is replaced with another path.

Do not install or debug Hermes until this boundary works. Confirm that "tools": true is active, restart QVAC after configuration changes, and test a more capable model if necessary. Successful chat completion is not evidence of successful function calling.

Step 5: install Hermes Agent

Review remote installation scripts before executing them. One cautious sequence is:

curl --fail --silent --show-error \
  https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh \
  -o /tmp/hermes-install.sh

less /tmp/hermes-install.sh
bash /tmp/hermes-install.sh

If your environment prohibits remote installation scripts, follow the manual installation procedure for the release you selected. Avoid mixing system Python packages with unrelated virtual environments.

Open a new terminal if the installer changed PATH, then verify the command:

command -v hermes
hermes --help

Hermes commonly stores user state under ~/.hermes/. That directory can contain sessions, memory, skills, logs, and provider configuration. Do not publish it or commit it to Git.

find "$HOME/.hermes" -maxdepth 2 -type d -print 2>/dev/null

Step 6: connect Hermes to QVAC

Use Hermes' interactive model configuration so that the installed release writes its own supported configuration structure:

hermes model

Select a custom OpenAI-compatible endpoint and enter:

API base URL: http://127.0.0.1:11434/v1
API key: local-qvac
Model name: hermes-local
Context length: 65536

If QVAC was started without an API-key requirement, local-qvac is merely a non-empty local placeholder for clients that require a value. It is not a cloud credential and must not be copied from a real account.

The corresponding configuration is conceptually equivalent to:

model:
  default: hermes-local
  provider: custom
  base_url: http://127.0.0.1:11434/v1

The exact YAML structure can change. Do not replace the entire Hermes configuration with this short example: the existing file may also contain memory, channel, and skill settings.

Confirm the saved route without printing credential fields:

grep -nE 'provider:|base_url:|default:' \
  "$HOME/.hermes/config.yaml"

The base URL should begin with http://127.0.0.1:11434/v1, and no cloud provider domain should appear in the active model configuration.

Step 7: perform the first launch without dangerous tools

hermes

Begin with a generation-only request:

State the configured model name and base URL.
Do not use tools and do not access the network.

The model's answer is not authoritative. It can repeat prompt text or hallucinate configuration. The sources of truth are the Hermes configuration, QVAC request logs, and network observation.

Next, establish an explicit working boundary:

Working directory: /absolute/path/hermes-local-lab
You may read only from inbox.
You may write only to outbox.
Do not modify source files.
Do not make network requests.
First present a plan, then wait for my next instruction.

If the agent immediately changes files, stop it. A prompt influences model behavior but does not create an isolation boundary. File permissions and tool-level validation must enforce the same policy.

Step 8: restrict file tools

The precise skill and permission controls depend on the installed Hermes version, but the security rule does not: expose the smallest directory and the smallest set of operations required by the task.

The laboratory needs only these capabilities:

• read a file from the approved input directory;
• list files inside the laboratory;
• create a new file in outbox;
• read and write Hermes' local memory through its dedicated interface;
• request approval for anything outside those boundaries.

Do not enable a browser, email, messaging, unrestricted shell access, or external MCP servers for this test. If Hermes runs in a container, expose individual directories rather than the entire home directory:

/host/hermes-local-lab/inbox  -> /lab/inbox:ro
/host/hermes-local-lab/outbox -> /lab/outbox:rw
/host/hermes-state            -> /home/hermes/.hermes:rw

For a native process, use a dedicated operating-system account or another isolation mechanism. Natural-language instructions remain useful, but the operating system must reject access that the agent should not have.

An allowlist must compare normalized absolute paths. A raw string-prefix check can be bypassed with .., alternate separators, or symbolic links.

requested path
      │
      ▼
resolve with realpath
      │
      ├── inside inbox  → read
      ├── inside outbox → read and create
      └── elsewhere     → deny

Step 9: verify the complete agent loop

Give Hermes this task after replacing <LAB_DIR> with the absolute laboratory path:

Read:
<LAB_DIR>/inbox/meeting.txt

Create:
<LAB_DIR>/outbox/meeting-summary.md

The new file must contain exactly these three sections:
# Project
# Decision
# Next action

Do not modify the source file. After writing the result,
read the new file again and report its path and section names.

Do not accept the final chat message as evidence. Inspect the filesystem independently:

test -f "$LAB_DIR/outbox/meeting-summary.md"
test -f "$LAB_DIR/inbox/meeting.txt"

sed -n '1,80p' "$LAB_DIR/outbox/meeting-summary.md"

find "$LAB_DIR" -maxdepth 2 -type f \
  -printf '%p\n' 2>/dev/null || \
find "$LAB_DIR" -maxdepth 2 -type f -print

The loop passes only if:

• the agent reads the source instead of inventing its contents;
• the write call targets a path inside outbox;
• exactly the expected output file is created;
• the input file remains unchanged;
• the agent rereads the output before completing;
• QVAC logs show local model requests only.

Record the source hash before and after a repeated run:

sha256sum "$LAB_DIR/inbox/meeting.txt"

A matching hash proves that this particular source file did not change. It does not prove that no file outside the laboratory was touched, which is why operating-system isolation remains important.

Step 10: test persistent memory

Agent memory must be tested across independent sessions. A correct answer inside the original conversation may come from the active context rather than durable storage.

In the first session, store a harmless synthetic preference:

Remember this durable preference for the test project:
final notes must contain no more than three sections.
Do not store file paths, personal data, or secrets.
Confirm exactly what you sent to persistent memory.

Exit Hermes through its normal exit command. Start a new session and ask:

What formatting limit did I set for final notes?
If persistent memory does not contain the answer, say so plainly.

The test passes when the new session recovers the three-section rule from persistent storage. Do not include the answer in the retrieval question.

You can inspect modification times without printing all memory contents:

find "$HOME/.hermes" -type f \
  -printf '%TY-%Tm-%Td %TH:%TM %p\n' 2>/dev/null \
  | sort \
  | tail -n 30

On macOS, use its supported find options or file metadata tools. Do not infer the memory format from a directory name; internal storage can change between Hermes versions.

Step 11: prove offline operation

Run the offline test only after the complete model has been downloaded. Otherwise, a startup failure merely proves that required files were never cached.

Before disconnecting, confirm that:

1. QVAC starts without another download;
2. /v1/models returns hermes-local;
3. Hermes sends completions through QVAC;
4. the restricted tools can read and write laboratory files;
5. memory survives a Hermes restart.

Stop Hermes and QVAC. Disable Wi-Fi and wired networking, or use a temporary network namespace or firewall policy that denies external connections while preserving loopback. Do not block 127.0.0.1, because Hermes needs it to reach QVAC.

Cold-start QVAC again:

cd /absolute/path/to/qvac-hermes

qvac serve openai \
  --config "$PWD/qvac.config.json" \
  --host 127.0.0.1 \
  --port 11434 \
  --verbose

In a second terminal:

curl --fail http://127.0.0.1:11434/v1/models
hermes

Give Hermes a new task that was not completed in the earlier session:

Create offline-check.md in the allowed outbox.
Write this exact line: OFFLINE_ROUTE_OK
Add a short summary of inbox/meeting.txt.
Do not use a browser, search, URL, or external source.
After writing the file, read it again.

Verify the artifact independently:

test -f "$LAB_DIR/outbox/offline-check.md"

grep -F 'OFFLINE_ROUTE_OK' \
  "$LAB_DIR/outbox/offline-check.md"

sed -n '1,80p' \
  "$LAB_DIR/outbox/offline-check.md"

Confirm that QVAC listens only on loopback:

ss -ltnp 2>/dev/null | grep ':11434' || \
lsof -nP -iTCP:11434 -sTCP:LISTEN

The listening address should be 127.0.0.1:11434, not 0.0.0.0:11434. Disabling Wi-Fi alone may leave VPN, virtual-machine, container, or wired interfaces active. For a strict test, observe outbound connections or isolate the processes so that only loopback traffic is permitted.

Automated QVAC smoke test

A small script can verify the HTTP boundary before every Hermes session. It checks model discovery and an ordinary completion; the earlier raw function test is still required.

#!/usr/bin/env bash
set -euo pipefail

base_url="${QVAC_BASE_URL:-http://127.0.0.1:11434/v1}"
model="${QVAC_MODEL:-hermes-local}"

models_json="$(curl --fail --silent --show-error \
  "$base_url/models")"

printf '%s' "$models_json" |
  grep -F "\"id\":\"$model\"" >/dev/null ||
printf '%s' "$models_json" |
  grep -F "\"id\": \"$model\"" >/dev/null

response_json="$(curl --fail --silent --show-error \
  "$base_url/chat/completions" \
  -H 'Content-Type: application/json' \
  -d "{
    \"model\": \"$model\",
    \"messages\": [
      {\"role\": \"user\", \"content\": \"Reply: LOCAL_OK\"}
    ],
    \"temperature\": 0,
    \"max_tokens\": 32
  }")"

printf '%s' "$response_json" |
  grep -F 'choices' >/dev/null

printf '%s\n' 'QVAC_HTTP_OK'

For a maintained test, parse JSON with jq and validate exact fields. The two grep patterns are only a minimal portable check and do not validate the complete response.

Troubleshooting by layer

Common failure cases

The base URL is wrong

The expected value in this setup includes /v1:

http://127.0.0.1:11434/v1

Some clients append /v1 themselves. A duplicated path will appear in QVAC's HTTP log. Change one parameter at a time so that an address problem is not confused with a model or provider problem.

The model alias does not match

Hermes sends the key declared under serve.models: hermes-local. The longer QVAC registry identifier tells the server which weights to resolve; it is not necessarily the name that the client should send.

The context is too large

A configured value of 65,536 does not guarantee that the machine can allocate it efficiently. Reduce the context, expose only necessary tools, restart QVAC, and repeat the raw HTTP tests. Keep the effective limit consistent between server and client.

The model prints JSON as ordinary text

Text that resembles JSON is not a valid function call. Inspect the raw response. If message.tool_calls is absent, Hermes should not execute the text. Repairing this output with regular expressions can transform malformed model prose into unintended actions.

The tool loops indefinitely

Set a maximum number of agent steps and preserve the full message sequence. Verify that Hermes returns each tool result to the model with the expected role and call identifier. Give the task an explicit stopping condition: after one successful write and one verification read, finish.

The first generation appears frozen

A long system prompt and multiple schemas require prompt processing before output begins. Watch QVAC logs and resource use. If generation never starts, reduce the model, context, or active tool set. Do not assign a universal timeout based on one machine.

Memory stores information but retrieves the wrong item

Store short, atomic, unambiguous preferences rather than full conversational transcripts. Test retrieval in a fresh session without embedding the answer in the question. Never place passwords, API tokens, or confidential document content in persistent memory.

The offline test passes while another route remains open

A disabled browser or Wi-Fi icon does not prove network isolation. Check listening sockets and outbound connections. VPNs, container bridges, virtual interfaces, and wired adapters can remain available. Use a firewall or network namespace when the proof matters.

A safer daily operating profile

A local agent inherits the filesystem permissions of the account that runs it. The production boundary must therefore limit the consequences of an incorrect tool call, not merely its internet access.

• Bind QVAC to 127.0.0.1 unless remote access is explicitly required.
• Do not expose the complete home directory to Hermes.
• Mount or permission input data as read-only.
• Use a separate directory for generated artifacts.
• Prefer narrow file functions over a general-purpose shell.
• Require an approval gate for deletion, sending, publication, or access outside the working directory.
• Resolve symbolic links before applying path policy.
• Keep real credentials out of laboratory configuration and memory.
• Record the model identifier, package versions, and configuration hash.
• Back up ~/.hermes before upgrading or changing memory storage.

sha256sum qvac.config.json
npm list -g @qvac/cli --depth=0
hermes --help | head -n 5

Hermes releases may expose version information through different commands. Record only information returned by the installed build; do not invent a version string when no version flag is available.

Compare local models without inventing a benchmark

A valid benchmark gives every model the same prompts, function schemas, permissions, context budget, and acceptance checks. Do not compare a 4B model with one small function against a 35B model loaded with every Hermes skill.

Use five local tasks:

1. read one file and return an exact fact;
2. create a file that follows a three-section schema;
3. select the correct function from two similar functions;
4. repair an argument after the path allowlist rejects it;
5. stop after receiving the successful result instead of calling the function again.

For each run, record:

• the model registry identifier and context size;
• the complete sanitized input;
• the ordered sequence of tool calls;
• whether names and JSON arguments are valid;
• the number of unnecessary calls;
• whether every path remained inside the boundary;
• whether the filesystem result passed deterministic checks;
• cold-start and warmed-run times measured on your own machine.

A fast, persuasive answer that does not create the required file is a failed run. A model that reaches the result but repeats a dangerous call also fails. Determine the number of repetitions and pass criteria before reviewing results, and preserve raw logs. This guide intentionally reports no success percentages or latency figures because none can be honestly transferred to the reader without the original hardware, tasks, versions, and logs.

Limitations of the local architecture

• Model quality. A compact local model may select tools, plan long chains, and recover from errors less reliably than a larger model.
• Hardware memory. Weight files are only part of consumption. The runtime needs working buffers, KV cache, Hermes memory, and operating-system capacity.
• Latency. Large system prompts and many function schemas make the first agent step expensive.
• Context. A model's advertised maximum does not prove that the selected runtime and device can use it efficiently.
• External tools. A local model does not make email, web search, SaaS APIs, or remote MCP servers local.
• Updates. CLI flags, registries, model templates, and configuration formats change. Pinning a working setup creates an upgrade responsibility.
• Security. A malicious local document can contain a prompt injection. Removing cloud access does not prevent the model from requesting a dangerous local action.
• Observability. Without logs, model, template, server, agent, and tool failures are difficult to distinguish.
• Backups. Local memory can disappear with disk failure unless it is backed up separately.

Offline autonomy is not the same as universal usefulness. Without a network, the agent cannot retrieve current news, use remote business systems, or synchronize data. This architecture is best suited to local documents, private knowledge collections, drafting, classification, and workflows where data locality matters more than live external information.

Final acceptance checklist

• qvac doctor recognizes an available backend.
• The selected model exists in the installed QVAC registry.
• The model is declared in qvac.config.json.
• "tools": true is enabled for that model.
• QVAC listens only on 127.0.0.1:11434.
• /v1/models returns hermes-local.
• A direct curl request returns a structured chat completion.
• The direct function test returns structured tool_calls.
• Hermes uses the custom provider and local /v1 URL.
• Hermes can access only the laboratory directories.
• The input directory is read-only at the system or container level.
• Filesystem results are checked independently of the agent's claims.
• Persistent memory is verified in a new session.
• Both processes cold-start after external networking is disabled.
• The offline task creates and rereads a new file.
• No real credentials are present in configuration, prompts, logs, or memory.

If every item passes, you have more than a local chat interface. You have a minimal reproducible personal-agent environment: QVAC serves the model, Hermes manages memory and function execution, operating-system controls restrict access, and the offline test demonstrates that the working path does not require a cloud API.

Where to go next

Add one specialized skill and one narrowly defined tool, then repeat the complete test suite. Avoid enabling a browser, unrestricted shell, email, and messaging at the same time: when a failure occurs, you need to know which layer violated its contract.

Continue with the practical material in Agent Lab Journal guides, and use the glossary for definitions of agents, language models, context, tool calling, memory, MCP, and related engineering terms.