Give your agent a voice.

VoiceManager is a local speech server for voice agents. One WebSocket carries speech-to-text and text-to-speech at the same time, so your agent listens while it talks. Stream LLM tokens in, and the speech starts before the reply is complete. When the person talks over it, the agent stops at once.

git clone https://github.com/TacoTakumi/VoiceManager

Get started View on GitHub

Free and open source (MIT). Python 3.12 on Linux. Runs on your own GPU. Whisper and OpenVoice also run on a CPU.

One turn, one socket Example

→ your agent to VoiceManager← VoiceManager to your agent

→binary 16 kHz PCM from the microphone
←transcription "will it rain" is_final=false
←transcription "Will it rain tomorrow?" is_final=true
→tts_stream "No, tomorrow"
→tts_stream " looks dry."
←audio_start sample_rate=24000
←binary PCM, playing while text still arrives
→tts_stream " Rain comes on"
→binary the person starts to talk
→tts_cancel barge-in
←audio_end reason="cancelled", no more audio

The microphone keeps streaming the whole time. STT and TTS share the socket, so the agent hears the interruption while it speaks.

Built for conversation

Batch STT and TTS make a voice agent wait at every step. VoiceManager streams both directions, so a reply can start while the person is still finishing a sentence.

A conversation turn

1
Hear it as it is said
Send audio as you capture it. Interim transcripts arrive while the person speaks, and a final transcript arrives when they stop. The default engine finds the end of speech with Silero VAD, also when the microphone never pauses.
2
Speak before the reply is complete
Forward every LLM token with tts_stream. The server finds the sentence boundaries and starts the audio. flush speaks the buffered text now.
3
Stop when interrupted
tts_cancel ends a reply at once: one audio_end with reason: "cancelled", and no more audio for that request.
4
Keep the turns apart
turn_start opens a new turn, and each transcript carries its turn_id. Late transcripts from a turn you ended are easy to drop.

Fast first audio

With the default audio.cpp engine, the first TTS audio of a warm session arrives in about 110-170 ms with Breeze-TTS 2, and 50-85 ms with Qwen3-TTS.

Any language, one contract

The WebSocket protocol is the contract, and it is documented frame by frame. The Python client ships with the server. HermesConsole talks to it from Kotlin.

Users, tokens and voices

Each client gets its own bearer token, and each token can have its own voice. Manage them with the voicemanager command or a small web UI that uses no JavaScript.

On your own hardware

Audio and text stay on your machine. Each session runs its audio.cpp engine in its own sidecar process, so a crash ends one session, not the server.

Pick your engines

Choose one STT engine and one TTS engine in .env. Clients see the same protocol whichever engines run behind it.

Speech to text VM_STT_ENGINE

audio.cpp default
Nemotron-3.5 ASR streaming or Voxtral Realtime, as GGUF models in a native sidecar. The lowest latency.
faster-whisper
With Silero VAD. Runs on a CPU.
NeMo Nemotron
Cache-aware streaming through NVIDIA NeMo.
Qwen3-ASR
Interim transcripts through vLLM.

Text to speech VM_TTS_ENGINE

audio.cpp default
Breeze-TTS 2, Qwen3-TTS, dots.tts or VoxCPM2, with preset voices from voices/designed/.
OpenVoice v2
With MeloTTS. The only engine that clones a voice from an uploaded sample.
VibeVoice
VibeVoice-Realtime-0.5B with preset voices.

The Python client

voicemanager.client wraps the protocol. One async with block holds one socket, and every STT and TTS call shares it.

agent.py

from voicemanager.client import VoiceManagerClient

async with VoiceManagerClient(host="voice.local", token="vmp_...") as vm:
    # Listen: interim and final transcripts as they arrive
    async for t in vm.stream_stt_live(mic_chunks()):
        if t["is_final"]:
            question = t["text"]
            break

    # Speak: send LLM tokens in, play PCM out
    async for pcm in vm.stream_tts_live(llm_tokens(question)):
        speaker.write(pcm)

Stream or batch

stream_stt_live and stream_tts_live yield results as they arrive. transcribe_file and synthesize_tts return the whole result at once.

Turn taking built in

cancel_tts, flush_tts and turn_start map one to one to the protocol frames for barge-in, flush and turns.

Pick a voice per request

Pass voice= to any TTS call. Without it, the server uses the voice of the token, then the server default.

Get started

Install the server, build the audio.cpp engine for the default models, make a token, and start it. The README has the full steps, including the other engines.

Terminal

git clone https://github.com/TacoTakumi/VoiceManager
cd VoiceManager
python3.12 -m venv venv && source venv/bin/activate
pip install -r requirements.txt && pip install -e .
cp .env.example .env
# build audio.cpp and the sidecar (README, "audio.cpp engine")
python setup_models.py
voicemanager user add me
voicemanager user token add me my-agent
voicemanager serve

Or let your agent do the setup. Point it at the repo and ask:

Set up VoiceManager from https://github.com/TacoTakumi/VoiceManager on this machine.

What you need

Linux and Python 3.12. An NVIDIA GPU with CUDA for real-time speed. For the default engine, CMake, Ninja, a C++ compiler and the CUDA toolkit.

Connect

Open ws://<host>:8255/ws/stream with the header Authorization: Bearer <token>. The first frame tells you the engines, the voices and the default voice.

Check the model licenses

The code is MIT. Each model has its own license. Breeze-TTS 2, the default voice model, is for non-commercial use.

Pairs with HermesConsole

HermesConsole is an LCARS-style voice console for an Android wall tablet. Say "hey computer", ask a question, and it streams your speech through VoiceManager, sends the turn to a Hermes agent and speaks the reply back sentence by sentence.

HermesConsole on GitHub

What HermesConsole uses

Streaming STT
After the wake word, the tablet streams 16 kHz audio and gets the transcripts back on the same socket.
Streaming TTS
The agent's reply goes to tts_stream as it arrives, so the console speaks while the agent still writes.
The protocol, from Kotlin
It uses no Python client. The documented WebSocket frames are all it needs.