An open-source Python project for building low-latency voice agents with a modular speech pipeline and an OpenAI Realtime-compatible WebSocket API.
This repository provides a voice-agent stack called Speech To Speech. It is built around a modular pipeline for voice activity detection, speech-to-text, language modeling, and text-to-speech, and it exposes that pipeline through an OpenAI Realtime-compatible WebSocket API. The README also says the system is designed to let you swap components, and that it can run with local models or with OpenAI-compatible hosted backends.
It addresses the need for a low-latency way to build local voice agents without depending on a single proprietary stack. The project is positioned for users who want to combine speech recognition, reasoning, and speech synthesis while choosing their own models and backends, including fully local and open setups.
Conceptually, the system processes audio in four stages: it first detects when a person is speaking, then transcribes the speech, then sends the text to a language model, and finally converts the model’s reply back into audio. The README says each stage runs separately and communicates through queues, and that the component choices are configurable through CLI flags. It also notes that the language-model slot speaks OpenAI-compatible protocols, which allows either hosted providers or local servers such as vLLM or llama.cpp to be used.
It is gaining attention because it combines several currently popular themes in one package: local AI, speech agents, open-source models, and compatibility with widely used OpenAI-style APIs. The README also highlights practical entry points such as quickstart commands, offline-capable local operation, multiple backend options, and production use as the conversation backend for thousands of Reachy Mini robots, which adds credibility and visibility.
The README points to several comparable component choices rather than direct one-to-one competitors: Silero VAD for voice activity detection, Parakeet TDT, Whisper, Faster Whisper, Lightning Whisper MLX, and Paraformer for speech recognition, and Qwen3-TTS, Kokoro, Pocket TTS, and ChatTTS for speech synthesis. At the system level, the main alternative approach described is to use another OpenAI-compatible LLM backend, such as a hosted provider or a local server like vLLM or llama.cpp, instead of this repository’s packaged pipeline.
AI-explained · grounded in each repo's README