Building Hermes, Part 10: Adding Jarvis to Hermes

The earlier parts of this series covered the path from installing Hermes to adding planner, evaluation, Second Brain, VPS migration, incident response, container durability, and a visual agent office. Jarvis is the next layer in that progression: giving Hermes a realtime voice and HUD surface without turning it into a separate assistant.

Jarvis was the next boundary. I did not want another chatbot with a microphone attached. I wanted voice to become a real runtime surface for Hermes, with the same safety model as the terminal: durable events, explicit approvals, replay, cancellation, and privacy rules that survive reconnects.

That decision changed the architecture. Jarvis became a production voice and HUD runtime on top of Hermes, not a separate AI brain.

Why voice could not be a thin wrapper

A simple voice assistant usually looks like this:

Microphone
→ Speech-to-text
→ LLM request
→ Text-to-speech

That works for demos. It does not work for an agent that can read files, run commands, ask for approvals, publish drafts, inspect services, and coordinate multi-step work.

The moment a voice interface sits in front of real tools, it needs answers to harder questions:

• What state is the task in right now?
• Is Hermes thinking, using a tool, waiting for approval, speaking, or done?
• Can the user interrupt safely?
• If the browser reconnects, what should the HUD replay?
• If a tool payload contains a path, command, token, URL, or SQL query, can any of it leak into spoken progress?
• If an approval request appears while TTS is running, which one wins?

Those are runtime questions, not model questions. The solution was to make Jarvis deterministic and keep Hermes canonical.

The boundary between Hermes and Jarvis

The working rule is simple:

Hermes owns the brain.
Jarvis owns the realtime voice experience.

Hermes remains responsible for the planner, memory, MCP tools, approvals, run lifecycle, durable Event Bus, and project context. That matches the larger Hermes architecture where the VPS is the dedicated AI OS host and external systems connect through controlled interfaces rather than ad hoc local state [Hermes Architecture Audit](sb://01_Projects/Hermes/Architecture/Hermes Architecture Audit 2026-07-10.md#chunk-0).

Jarvis is responsible for the browser HUD, WebSocket connection, voice session runtime, local voice commands, STT routing, browser TTS, barge-in behavior, approval UI routing, replay, voice session metadata, and proactive alerts.

The result is not two assistants. It is one assistant with two layers:

Browser HUD
  ↓ WebSocket
Jarvis voice runtime
  ↓ Hermes Event Bus and run API
Hermes runtime
  ↓ Planner, memory, MCP, tools

That separation matters because it keeps the voice layer from becoming a second source of truth. Jarvis can make the interaction feel immediate, but it should not decide what Hermes remembers, which tools exist, or whether a command is approved.

The production flow

The deployed shape is split across three roles:

Browser
  - HUD
  - push-to-talk or listening UI
  - browser TTS
  - WebSocket client

Jarvis on VPS
  - WebSocket server
  - voice session runtime
  - speech-likeness gate
  - local command router
  - event adapter
  - approval, cancel, replay, proactive alerts

Hermes on VPS
  - planner
  - memory
  - Event Bus
  - run lifecycle
  - MCP and tools

Speech recognition is intentionally not coupled to the VPS container. The browser and Jarvis route audio through a private worker path to a Mac-based Whisper worker. That keeps the heavy STT workload out of the small VPS runtime while still letting the HUD behave like a single voice surface.

The voice request path looks like this:

User speaks
→ browser HUD captures audio
→ Jarvis routes STT
→ transcript passes speech-likeness and local command checks
→ local command runs immediately, or Hermes run starts
→ Hermes emits canonical events
→ Jarvis converts events into HUD actions and speech decisions
→ browser TTS speaks the final answer

The key point is that Jarvis does not need to understand the whole task. It needs to track the interaction state faithfully.

The deterministic voice state machine

The core runtime pattern is a deterministic Voice Task Orchestrator. It consumes canonical Hermes Event Bus envelopes plus voice lifecycle markers, then emits a sanitized state snapshot and replay-safe action descriptors.

A normal path looks like this:

IDLE
→ LISTENING
→ TRANSCRIBING
→ SUBMITTING
→ THINKING
→ USING_TOOL
→ THINKING
→ SPEAKING
→ COMPLETED

An approval path branches like this:

THINKING or USING_TOOL
→ WAITING_FOR_APPROVAL
→ approval decision is posted back to Hermes
→ USING_TOOL
→ SPEAKING
→ COMPLETED

This is deliberately boring. The orchestrator does not call tools. It does not approve commands. It does not render the HUD. It does not speak audio. It turns events into state and action descriptors such as:

• submit the run
• show the approval UI
• pause TTS
• post an approval decision
• start final TTS
• stop TTS

That makes the boundary testable. If the same event stream is replayed after reconnect, Jarvis can reconstruct the same user-facing state without inventing a new story.

This mirrors the broader Hermes design principle: keep durable project knowledge and runtime projections separate. The Second Brain design keeps Markdown as source of truth and SQLite as a rebuildable search projection [Hermes Memory Layer Specification](sb://01_Projects/Hermes/Architecture/Hermes Memory Layer Specification.md#chunk-0). Jarvis uses the same idea at runtime. Hermes events are the source. HUD state is the projection.

Spoken progress without leaking private context

Voice changes the privacy model. A terminal can show a path or command in a scrollback that only the operator is watching. A speaker can say it out loud in the room.

So Jarvis uses a separate spoken progress policy. It maps tool categories to fixed utterances rather than reading raw payloads aloud.

Examples:

web_search        → "I’m searching the web."
file_retrieval    → "I’m checking your files."
code_execution    → "I’m running the requested task."
approval.required → "This action needs your approval."

For Korean sessions, the same policy can use Korean templates, for example:

웹에서 검색하고 있습니다.
파일을 확인하고 있습니다.
요청한 작업을 실행하고 있습니다.
이 작업은 승인이 필요합니다.

The important part is what the policy does not say. It should not speak URLs, paths, SQL queries, command arguments, authorization headers, tokens, or environment values. It should not expose raw run IDs or tool call IDs either. If a visible identifier is useful, it should be a stable public hash, not the original internal value.

This is one of the places where voice forces better engineering. The system cannot rely on the model to be discreet. It needs a policy that makes private data structurally unavailable to the spoken layer.

Local commands, barge-in, and personal use

Not every utterance should start a Hermes run. Some voice actions are local and immediate:

• cancel this
• stop speaking
• repeat that
• open or focus the HUD state
• handle short navigation or control commands

That is why Jarvis has a local command router before the Hermes run path. Fast local commands keep the system responsive and reduce unnecessary agent runs.

Continuous listening and barge-in belong in the same category. They are not about making the model smarter. They are about making the runtime feel interruptible without corrupting the task state. If the user interrupts while Jarvis is speaking, the voice layer can stop TTS and route the new intent. If Hermes is waiting for approval or running a tool, cancellation needs to go through the correct Event Bus path instead of being treated as a browser-only action.

For a personal system, this is enough to move from push-to-talk demo to usable voice runtime. It does not mean the system is ready for every shared office or multi-user environment. It means the single-user control loop is now real.

Voice Session Memory is not Hermes memory

Jarvis has voice session memory, but it is intentionally narrow. It stores metadata about the voice interaction, not the conversation itself.

Allowed metadata includes:

• digest
• run id
• terminal state
• locale
• tool category
• approval outcome
• timestamps

Disallowed data includes:

• transcript
• audio
• TTS output
• reasoning
• tool arguments
• shell commands
• credentials
• raw environment values

This split is important. Hermes memory is where durable project knowledge belongs. Jarvis memory is only for voice-session continuity and UX recovery. If Jarvis starts storing transcripts and tool arguments, it becomes a shadow memory system with a worse privacy model.

Proactive alerts complete the loop

The final production step was proactive alerts. A voice HUD should not only respond when the user is watching. It should also surface important runtime events from Hermes.

The alert flow is:

Hermes Event Bus
→ ProactiveAlertController
→ browser alert queue
→ HUD displays or speaks
→ browser ACK
→ alert is removed

The useful events are the ones that affect the user's attention:

• run completed
• run failed
• run cancelled
• approval required

This is where replay and dedupe matter. If the browser reconnects, the user should not receive a storm of duplicate alerts. If an approval is still pending, it should survive reconnect. If a run already completed and was acknowledged, it should not reappear as a new event.

Again, this is runtime engineering, not prompt engineering.

What shipped

By the end of this phase, Jarvis was no longer a gateway experiment. It was a production voice runtime for Hermes with these pieces in place:

• realtime Event Bus transport
• Voice Task Orchestrator
• spoken tool progress
• approval routing
• fast local voice commands
• continuous listening and barge-in for personal use
• Voice Session Memory metadata
• proactive alerts
• reconnect and replay behavior
• independent Jarvis deployment without restarting Hermes

The remaining roadmap is less about voice infrastructure and more about what Hermes can do through that voice surface: safer self-update, stronger planning, coding-agent workflows, browser-agent workflows, workflow automation, and vision.

Lessons from the build

The first lesson is that voice should be a runtime surface, not a second agent. If voice becomes another brain, memory and authority split in ways that are hard to debug.

The second lesson is that deterministic state beats conversational guessing. The HUD should not infer whether the agent is waiting for approval. It should receive a state derived from canonical events.

The third lesson is that spoken progress needs a privacy contract. A safe terminal log is not automatically safe speech.

The fourth lesson is that replay is part of UX. Browser reconnects, network hiccups, and container restarts are normal. A voice runtime that cannot reconstruct state will feel haunted.

The fifth lesson is that local commands matter. Not every utterance deserves an LLM call. A good voice runtime knows which actions are local, which are agentic, and which require approval.

Jarvis made Hermes feel less like a CLI tool and more like an operating layer. The architecture worked because Jarvis did not replace Hermes. It gave Hermes a realtime body.

References

1. Hermes Agent Documentation
2. Hermes Architecture Audit
3. Hermes Second Brain Bridge Design
4. Hermes Memory Layer Specification