From cb09d4230ceaba50925726c4a1cfe334f563abae Mon Sep 17 00:00:00 2001 From: Nicholas Ward Date: Tue, 4 Aug 2026 19:04:40 -0500 Subject: [PATCH] =?UTF-8?q?=E2=98=8E=EF=B8=8F=20Local=20Voice=20Agent?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../index.md | 186 ++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 container/site/content/posts/2026-08-03-self-hosted-voice-agent/index.md diff --git a/container/site/content/posts/2026-08-03-self-hosted-voice-agent/index.md b/container/site/content/posts/2026-08-03-self-hosted-voice-agent/index.md new file mode 100644 index 0000000..7239c44 --- /dev/null +++ b/container/site/content/posts/2026-08-03-self-hosted-voice-agent/index.md @@ -0,0 +1,186 @@ ++++ +categories = ["software"] +tags = ["voice-agent","freepbx","livekit","sip","ollama","openwebui","speech-to-text","text-to-speech"] +date = 2026-08-03T10:00:00-05:00 +description = "Building a self-hosted telephone voice agent with FreePBX, LiveKit, streaming speech recognition, local language models, and retrieval-backed specialist voices." +draft = true +slug = "2026-08-03-local-voice-agent" +title = "☎️ Local Voice Agent" +author = "nicholas" ++++ + +In a [previous post]({{< relref "posts/2026-03-01-voip" >}}), I built a small telephone system with FreePBX and a low-cost SIP carrier, BulkVS. This gave me a real phone number, extensions, softphones, a ring group, and voicemail. This would serve as the perfect entry point for a self-hosted voice agent. + +I have configured the system such that when my phone number is dialed, a voice describes a telephone menu where the caller can dial a selection: +- `1` to speak with an AI bot about me, +- `2` to leave voicemail, or +- `3` to try to reach me (ring my ring group, which are each of my IP phone extensions) + +This is still a hobby project. It is not a customer-service platform, or a production-ready architecture. It is, however, a complete working system and a useful experiment in real-time audio, local AI, retrieval, and telephone routing. + +## What I Wanted + +I wanted the system to meet a few requirements: + +- Keep the telephone system in existing FreePBX. +- Run speech recognition, the language model, and speech synthesis on my own modest hardware. +- Answer questions only from information that I intentionally published. +- Route different subjects to agents with different voices and knowledge bases. +- Preserve normal telephone features such as voicemail and a ring group. + +I did not want the language model to control the initial phone menu. The first decision is deterministic: the caller presses a number, and the router follows a fixed route. The AI becomes involved only after the caller selects the conversational option. + +## Architecture + +The more or less complete path: + +```text +Public telephone network + ↓ +SIP carrier (BulkVS) + ↓ +FreePBX / Asterisk + ↓ +LiveKit SIP + ↓ +LiveKit room + ↓ +Deterministic telephone router + ↓ +Python knowledge agent + ├── Silero voice activity detection + ├── sherpa-onnx + NVIDIA Nemotron streaming speech recognition + ├── Open WebUI retrieval and model management + ├── Ollama + Granite language model + └── Kokoro speech synthesis + ↓ +The same path back to the caller +``` + +FreePBX remains the public telephone system. It owns the inbound route, voicemail, the ring group, and the connection to the SIP carrier. The AI services are on one of my local application hosts with access to my GPU. FreePBX sends only the voice-agent route to a private LiveKit SIP trunk, so I do not expose LiveKit's SIP or media ports directly to the internet. + +LiveKit converts the SIP call into a room with participants and audio tracks. A small Python router, which I call the Usher, joins first. It is a deterministic menu application, not a language-model agent: it has no speech recognition, language model, tools, or autonomous routing decisions. It plays fixed menu options and receives telephone keypad events. + +If the caller presses `1`, the telephone router dispatches the knowledge agent into the same room. The system plays a short telephone routing tone to indicate the handoff. The Usher remains in the room without speaking so it can retain the active-call lease, enforce the call-duration limit, and clean up when the caller disconnects. If the caller selects voicemail or my ring group, LiveKit sends the call back to the appropriate FreePBX destination with a SIP transfer. + +## The House Steward and Specialists + +The first conversational profile is the House Steward. Its greeting is deliberately simple, it will ask something simple like "Hello. Question about Nicholas?" This general profile can explain that the caller is speaking with an AI, describe what the service does, and answer broad questions. + +The specialists cover subjects such as bicycling, books, career history, cooking, gardening, movies, music, photography, quotations, self-hosted services, travel, and early life info. This list will probably change. Each profile has its own: + +- instructions +- Open WebUI model wrapper +- knowledge base +- retrieval rules +- Kokoro voice + +A separate local classifier reads a completed caller turn and a small amount of recent conversation. It returns one profile and a confidence score. A high-confidence change causes a handoff inside the existing LiveKit session. The current voice announces the transfer, a short telephone routing tone plays, and the new voice gives a short introduction. The room and conversation remain intact. + +## Published Knowledge + +The personal information does not live in the voice-agent image or its prompt. I keep published Markdown and other data files in a separate knowledge repository. Each conversational profile has exactly one dedicated Open WebUI knowledge base. A synchronization workflow copies each published file into the appropriate knowledge bases. The general profile's knowledge base contains all published personal material, while each specialist's knowledge base contains a smaller subject-specific selection. A document can appear in several knowledge bases when it is relevant to several profiles. + +For an ordinary completed turn that does not trigger a handoff, the Python agent makes three requests: + +```text +Caller transcript + │ + ├── 1. Ask the local Ollama classifier to select a profile + │ ↓ + │ profile + confidence + │ + ├── 2. Search the active profile's Open WebUI knowledge base + │ ↓ + │ relevant excerpts + │ + └── 3. Send the question, excerpts, and conversation to the model + ↓ + answer text +``` + +If classification selects another profile with sufficient confidence, the fixed handoff sequence consumes that turn instead of performing retrieval and generating an answer. Otherwise, the agent limits and deduplicates the retrieved excerpts before adding them to the temporary context. The profile prompt tells the model not to invent personal details that are absent from those excerpts. Open WebUI owns the knowledge bases, embeddings, access grants, and named model configuration. Ollama runs the local Granite model. + +## Open WebUI Configuration + +Each conversational profile has a corresponding Open WebUI custom model wrapper attached only to that profile's knowledge base. + +Every wrapper uses `granite4.1:3b` as its Ollama base model, with an 8,192-token context and a 96-token response limit. The wrapper gives the profile a stable model ID and supplies its role-specific system prompt, attached knowledge base, and model parameters. Retrieved evidence is added at request time. + +Access is managed through two Open WebUI groups. The knowledge-synchronization account can write to the knowledge bases, while the voice-agent account receives read access to the managed knowledge bases and model wrappers. For each voice profile, the built-in knowledge tool is enabled and scoped to that profile's single attached knowledge base. Web search, memory, chat history, code execution, terminal access, image generation, notes, tasks, calendars, automations, custom tools, skills, filters, and actions are disabled so the model stays very dumb and safe. + +The active profile selects its matching knowledge base for retrieval. Because the knowledge repository and synchronization workflow form a separate publication boundary, I can update published information without rebuilding the voice-agent container; drafts and private notes are not synchronized. + +The general profile acts as a sort of FAQ for questions about the call itself. It can explain the AI identity, transfers, limits, logging, voicemail, knowledge boundaries, etc. + +## Streaming Speech Recognition + +Speech recognition runs through `sherpa-onnx` with NVIDIA's Nemotron 3.5 streaming model on my NVIDIA GPU. + +A streaming speech recognizer's "final" result does not necessarily mean that the caller's entire conversational turn is finished. It means that a portion of the transcript is stable. A caller can speak a long sentence with a pause and produce several final transcript fragments. + +LiveKit collects those stable fragments into one turn. Two timing thresholds coordinate this (values given here change as I tune to my liking): + +- sherpa-onnx finalizes a stable segment after some time (something like 0.8 seconds) of trailing silence +- Silero marks the end of the caller's turn after, say, 1.5 seconds of silence + +The gap gives the final transcript fragment time to reach LiveKit before the turn is committed. LiveKit can therefore combine several final fragments produced during one speaking state and send the complete request to the classifier. + +## Interruptions and Handoffs + +Telephone conversation is mostly open-ended. A caller can interrupt the agent while it is answering, change subjects during model generation, or ask the system to stop. + +The agent tracks a generation number for each caller turn. When the caller starts speaking, it invalidates older work and interrupts active generated speech. Retrieval or model output from an older generation is discarded before it can reach the caller. The words `stop` and `cancel` are also handled as direct cancellation commands. + +Handoffs use a stricter sequence. The current profile finishes its fixed transfer sentence, the system changes the active instructions, model, knowledge base, and voice, and then the receiving profile speaks. These fixed handoff messages are not generated by the language model. This prevents two profiles from speaking over each other or a delayed response from the old profile appearing after the transfer. + +## Public-Call Safeguards + +Connecting any automated service to a public telephone number creates opportunities for accidents and abuse. I added limits before routing the public DID to the agent: + +- one active public AI call at a time; +- a maximum of 15 minutes per call; +- a maximum of 60 minutes per caller number per UTC day; +- caller IDs that parse as valid United States numbers only; +- a 60-second inactivity timeout; and +- fixed voicemail and ring-group routes outside the language model. + +Redis holds the active-call lease and daily counters. Caller numbers are normalized for validation and hashed for the usage-counter keys. This is caller-ID validation, not a determination of the caller's physical location: caller ID is not authentication and can be spoofed. These controls limit casual abuse, while carrier-side restrictions remain useful defense in depth. + +The service does not save an audio recording of the AI conversation. It does log final caller and agent transcripts, the complete retrieved excerpts, the assembled model-message context, timing data, warnings, and errors. Some diagnostic values are logged in both plain text and base64; base64 is only another representation, not encryption or privacy protection. Logs rotate daily and normally retain seven daily rotations. They are excluded from backup archives. Voicemail is different: when the caller chooses voicemail, FreePBX intentionally records the message and sends the normal voicemail notification. + +## Performance + +The system normally feels conversational, but it is not instant. A typical completed turn contains several distinct delays: + +1. 1.5 seconds of silence to determine that the caller has finished. +2. A few hundred milliseconds for topic classification. +3. Usually tens of milliseconds for local knowledge retrieval. +4. Model time to the first generated token. +5. Kokoro time to the first audio frame. + +In recent calls, the complete delay before speech commonly landed around three to five and a half seconds. The largest variable was often text-to-speech startup, which sometimes took more than two seconds. The local Granite model usually began producing text quickly, but a fast model does not make the whole voice pipeline fast. + +I added per-stage metrics because a single "the agent is slow" measurement is not actionable. End-of-turn detection, transcription, classification, retrieval, model generation, and speech synthesis need separate timing records. + +## Current Limitations + +The system is operational, but it is not finished. + +Speech recognition sometimes loses the end of a word. Topic classification can still select the wrong specialist. The small local model can ignore a grounding instruction, invent an unsupported recipe step, or produce an answer that is too long. Things like this. A hard token limit can stop a long answer in the middle of a sentence, which sounds particularly bad on a telephone call. + +There are also two configuration layers: the Python runtime supplies current turn instructions, while Open WebUI supplies named-model configuration and permissions. Those layers must remain aligned. + +## Was This the Simple Way? + +No. A hosted voice-agent API could replace much of this stack. My goal, however, was to connect the phone system I already built to speech and language models running on my own hardware. For that goal, the separation now makes sense: + +- FreePBX owns telephone behavior. +- LiveKit owns real-time media and agent sessions. +- sherpa-onnx owns streaming transcription. +- Open WebUI owns published knowledge and model access. +- Ollama owns local language-model inference. +- Kokoro owns speech synthesis. +- The Python agent owns routing, grounding, handoffs, and cancellation. + +✅ works.