Real-time voice intelligence · WebSocket API
Your voice agent's
ears & instincts.
Stream raw audio, get structured events: who is speaking, when they finished their thought, whether they may interrupt — and whether your agent should answer. In under a second.
Free tier · 5,000 min/mo · no credit card
Works with your existing voice stack
250ms
barge-in detection
<1s
verified response after last word
0
recordings stored — ever
1
WebSocket to integrate
Developer platform
The complete sensor layer for voice agents.
VSIP sits between the microphone and your agent stack — the piece most voice frameworks leave out: who spoke, when they finished, and whether your agent should respond.
Explore the API- 01
WebSocket streaming API
Frame-level audio in, typed JSON events out. One connection per session.
- 02
Realtime inference pipeline
VAD, noise, identity, semantic endpointing, and verification in one pass.
- 03
SDKs for Python and TypeScript
VoiceSession, reconnect, speak-gate, and verification-gated callbacks baked in.
- 04
Sidecar mode for S2S agents
Run beside OpenAI Realtime without touching the agent audio loop.
- 05
Observability per stream
Session metadata, event timelines, and billing minutes — no raw audio stored.
How it works
User speaks to your agent
Browser mic, mobile app, or SIP trunk — any surface that can stream 16kHz PCM.
Twilio μ-law, WebRTC, or raw PCM — same contract.
Audio streams to VSIP
One WebSocket connection. Frame-by-frame ingestion with no uploads, no batch jobs.
TLS 1.3 · tick 80ms · per-tenant isolation.
Inference runs in real time
VAD, noise class, identity, semantic endpointing, and verification — layered in milliseconds.
Five models · one pass · sub-second verdicts.
Events drive your stack
turn_end, barge_in, speaker_verification — your STT, LLM, and TTS react to structured truth.
VSIP never touches your STT, LLM, or TTS.
Scroll to advance · 1/4
Input capture
16kHz mono PCM · 20ms frames
Architecture
One stream in.
Structured truth out.
Five inference stages inside one WebSocket. Audio never leaves your tenant boundary as recordings — only embeddings and event metadata persist.
The product
Everything between the microphone and your agent.
Voice agents fail in the gaps — cut-off sentences, phantom turns from coughs, answering the wrong person. VSIP closes the gaps.
01
Semantic turn detection
Knows "I'd like to order… umm" isn't finished. A turn model judges whether the thought is complete — turns close ~300ms after the last word, and mid-thought pauses survive up to 2s.
See how it works02
Identity-gated barge-in
Only the enrolled speaker can interrupt the agent. Five gate layers — VAD, noise class, spectral stability, identity, duration — reject coughs, TVs, and bystanders before your TTS ever stops.
View gate layers03
Speaker verification
Every turn ends with an authoritative should_respond verdict, checked against the enrolled voiceprint with the similarity score attached.
View event schema04
Noise intelligence
Coughs, breaths, laughter and ambient noise are classified in real time — never opened as turns, never forwarded to your LLM as phantom utterances.
See noise classes05
Invisible enrollment
The first sentence a user speaks both enrolls their voiceprint and gets answered. No setup phrase, no onboarding step.
How enrollment works06
Overlap separation
When two people talk at once, source separation isolates the enrolled speaker's channel — quality-guarded, with per-speaker audio delivered.
Try the demoBuild faster · SDK
Or skip the protocol.
Write one callback.
The vsip SDK bakes in every integration pattern — onset-lookback audio slicing, verification-gated responses, queued turns, transparent reconnect. You write on_turn; it handles the rest.
pip install vsip-sdkPython client + VoiceSessionnpm install vsip-sdkJavaScript / TypeScriptPython · JS · Pipecat · LiveKit · Twilio — all available
| 1 | from vsip import VSIPClient, VoiceSession, Turn |
| 2 | |
| 3 | client = VSIPClient(api_key="vsip_...", |
| 4 | url="wss://api.vsip.online/v1/stream") |
| 5 | |
| 6 | async def on_turn(turn: Turn): |
| 7 | # verified speaker only — never the wrong voice |
| 8 | if not turn.should_respond: |
| 9 | return |
| 10 | text = await my_stt(turn.audio) |
| 11 | reply = await my_llm(text) |
| 12 | await session.speak_gate() # don't talk over a turn |
| 13 | await my_tts(reply) # stop this on barge-in |
| 14 | |
| 15 | session = VoiceSession( |
| 16 | client, |
| 17 | on_turn=on_turn, |
| 18 | on_barge_in=lambda e: my_tts.stop(), |
| 19 | ) |
| 20 | |
| 21 | async with client: |
| 22 | await session.start() |
| 23 | async for frame in mic(): # 16kHz PCM |
| 24 | await session.send_audio(frame) |
Quickstart paths
Voice agent quickstart
Connect a WebSocket, stream PCM, handle turn_end in under 10 minutes.
Try it outLiveKit + Pipecat
Drop VSIP beside your existing agent framework with sidecar or native hooks.
View guidesTwilio telephony
Connect SIP or Media Streams with μ-law audio_format out of the box.
Integrate telephonyDeploy to production
API keys, rate limits, concurrent streams, and billing on the dashboard.
Start buildingLive demo
Don't read about it — talk to it.
This agent's ears are the live VSIP API. Interrupt her mid-sentence, cough at her, or let someone else try to ask a question — and watch the real events stream.
Why teams pick VSIP
The only barge-in that knows who is interrupting.
Every voice stack can detect sound during TTS. VSIP verifies the sound is your user — not a cough, not a colleague, not a TV. Interruption stops the agent instantly (fail-safe); the response is released only after the voice is verified (fail-secure).
FAQ
Frequently asked questions.
What does VSIP do — is it a speech-to-text service?
No. VSIP is the sensor layer for voice agents: it detects turns, interruptions, speaker identity, and noise in real time, and tells your agent exactly when to listen, stop talking, and respond. Transcription, LLM reasoning, and TTS stay entirely in your stack — VSIP never touches them.
Does it work with my existing voice agent?
Yes. The full integration contract is four steps: send 16kHz PCM over the WebSocket, tell VSIP when your TTS starts and stops (one JSON message each), and stop your TTS when a barge_in event arrives. That works identically for custom Python backends, LiveKit, Pipecat, Daily, and raw WebRTC. Twilio calls connect natively with audio_format=mulaw8k.
How does identity-gated barge-in work?
VSIP builds a voiceprint of your primary speaker (auto-enrolled from their first sentence, or explicitly via the enroll_speaker command) and locks the session to it. When someone speaks during agent TTS, five gate layers decide whether it's a genuine interruption by the right person. A cough or a different voice fails the gates and your agent keeps talking.
Do users have to enroll their voice before talking?
No. The first sentence a user speaks both enrolls their voiceprint and gets answered — enrollment is invisible. For known users you can also pre-enroll from a stored sample with one API call.
What happens when the user coughs or someone else talks?
Coughs, breaths, and laughter are classified in real time and reported as noise_event — they don't open turns or trigger barge-in. Background speakers are rejected by the identity gate: they can be heard (and reported as speaker_change), but the agent's response is gated on the verified primary speaker.
Can I use it with speech-to-speech agents like OpenAI Realtime?
Yes — connect with mode=sidecar. VSIP runs listen-only beside the S2S agent's own turn-taking and gives you the layer those stacks lack: speaker identity, verification verdicts, and noise events, without interfering with the agent's audio loop.
How fast is it?
Turn end fires ~300ms after the last word (semantic endpointing), the speaker verification verdict lands 0–450ms after that, and barge-in is detected in 250–450ms when identity is already confirmed. Net effect: your agent can respond to a verified speaker in under a second after they stop talking.
Is my users' audio stored?
No. Audio is processed in memory frame-by-frame and discarded — VSIP keeps no recordings. What persists: 192-dimensional voice embeddings (per your enrolled speakers, isolated to your account) and billing metadata (session IDs, minutes).
Is there a free tier?
Yes — 1 concurrent stream and 5,000 minutes per month, no credit card required. Paid tiers add concurrent streams and volume: Starter $49/mo, Growth $149/mo, Scale $499/mo.
Ship it
Give your agent ears.
Free tier, five-minute quickstart, no credit card. Bring your own STT, LLM and TTS — VSIP handles everything the microphone hears.