Documentation.

import asyncio, json, pyaudio, websockets

API_KEY = "vsip_YOUR_KEY_HERE"

async def main():
    uri = "wss://api.vsip.online/v1/stream"
    headers = {"Authorization": f"Bearer {API_KEY}"}

    async with websockets.connect(uri, additional_headers=headers) as ws:
        print(json.loads(await ws.recv()))  # session_start

        p = pyaudio.PyAudio()
        mic = p.open(format=pyaudio.paInt16, channels=1, rate=16000,
                     input=True, frames_per_buffer=1600)

        async def send_audio():
            while True:
                await ws.send(mic.read(1600, exception_on_overflow=False))
                await asyncio.sleep(0)

        asyncio.create_task(send_audio())

        async for raw in ws:
            event = json.loads(raw)
            if event["type"] in ("turn_start", "turn_end", "barge_in",
                                 "speaker_verification"):
                print(event)

asyncio.run(main())

Authorization: Bearer vsip_YOUR_API_KEY

wss://api.vsip.online/v1/stream?tick_ms=80&session_id=call-abc-123

You are an expert voice systems engineer. Build a complete voice agent using the VSIP Voice Events API (Full Mode) + OpenAI APIs. Use Python + asyncio. Do not use any VSIP SDK — connect directly over WebSocket.

━━━ CREDENTIALS (replace these) ━━━
VSIP_API_KEY  = "vsip_YOUR_KEY_HERE"
OPENAI_API_KEY = "sk-YOUR_OPENAI_KEY_HERE"

━━━ ARCHITECTURE ━━━
Microphone (16kHz int16 PCM)
  │
  ├─► VSIP WebSocket (identity sensor + turn detector)
  │     wss://api.vsip.online/v1/stream?mode=full&tick_ms=80
  │     Authorization: Bearer {VSIP_API_KEY}
  │
  └─► On VSIP speaker_verification (should_respond=true):
        buffered audio → OpenAI Whisper STT → text
        text → GPT-4o-mini → reply text
        reply text → OpenAI TTS → speaker output

━━━ EXACT IMPLEMENTATION RULES ━━━

1. AUDIO STREAMING
   - Capture mic at 16000 Hz, 1 channel, int16 PCM using PyAudio
   - Chunk size: 1600 samples (100ms)
   - Send each chunk as a binary WebSocket frame to VSIP
   - Maintain a rolling buffer of the last 50 chunks (~5 seconds)

2. SESSION START
   - Connect with header: Authorization: Bearer {VSIP_API_KEY}
   - First message received from VSIP will be: {"type": "session_start", ...}
   - Log it and begin streaming audio

3. ON turn_start EVENT
   - Record the buffer position at this moment
   - Start accumulating all new mic frames into a dedicated turn_audio list
   - Also prepend 1.0 second of audio from the rolling buffer (25 chunks)
     so the first word of the user is never clipped
   - If the agent was mid-speech when this fires (barge-in case),
     use 1.5 seconds of lookback (37 chunks) instead

4. ON turn_end EVENT
   - Stop accumulating into turn_audio
   - Do NOT call STT yet — wait for speaker_verification

5. ON speaker_verification EVENT
   - If should_respond == False: drop the turn, log "impostor/noise blocked", do nothing
   - If should_respond == True AND the processing lock is NOT held:
       → acquire asyncio.Lock() so only one response runs at a time
       → proceed to step 6

6. WHISPER STT
   - Convert turn_audio (list of bytes) to an in-memory WAV file:
       16000 Hz, 1 channel, 16-bit PCM
   - Call: openai.audio.transcriptions.create(model="whisper-1", file=("audio.wav", wav_bytes, "audio/wav"))
   - If transcript is empty or whitespace: release lock, return

7. GPT-4o-mini CHAT
   - Maintain a conversation history list (system + user/assistant pairs)
   - System prompt: "You are a helpful voice assistant. Keep answers to 1-2 short sentences."
   - Append {"role": "user", "content": transcript} to history
   - Call: openai.chat.completions.create(model="gpt-4o-mini", messages=history, max_tokens=80)
   - Append {"role": "assistant", "content": reply} to history

8. AGENT STATE — BEFORE TTS
   - Send on the VSIP WebSocket: {"command": "set_agent_state", "value": "speaking"}

9. OPENAI TTS PLAYBACK
   - Call: openai.audio.speech.create(model="tts-1", voice="alloy", input=reply, response_format="pcm")
   - The response is raw 24000 Hz 16-bit mono PCM
   - Play it through PyAudio output stream at 24000 Hz
   - Play in 4800-byte chunks (100ms) so barge-in can interrupt mid-sentence
   - Check an asyncio.Event() cancel_flag before each chunk; if set, stop playback

10. AGENT STATE — AFTER TTS
    - Send on the VSIP WebSocket: {"command": "set_agent_state", "value": "idle"}
    - Release the asyncio.Lock()

11. ON barge_in EVENT
    - Set the cancel_flag asyncio.Event() to interrupt TTS immediately
    - Send: {"command": "set_agent_state", "value": "idle"}
    - Clear any pending turn queue

12. ERROR HANDLING
    - Wrap all WebSocket sends in try/except — ignore send errors if connection closed
    - On WebSocket disconnect: log and exit cleanly
    - On Whisper error: log and release lock (do not crash)
    - On GPT error: log and release lock

━━━ DEPENDENCIES ━━━
pip install websockets openai pyaudio numpy python-dotenv

━━━ OUTPUT ━━━
One Python file: agent_full_mode.py
Run with: python agent_full_mode.py
No web UI needed. Console logs show turn events, transcripts, and replies.

You are an expert voice systems engineer. Build a complete voice agent using VSIP (Sidecar Mode) alongside the OpenAI Realtime API. Use Python + asyncio. Do not use any VSIP SDK — connect directly over WebSocket.

━━━ CREDENTIALS (replace these) ━━━
VSIP_API_KEY   = "vsip_YOUR_KEY_HERE"
OPENAI_API_KEY = "sk-YOUR_OPENAI_KEY_HERE"

━━━ ARCHITECTURE ━━━
Microphone (16kHz int16 PCM)
  │
  ├─► VSIP WebSocket (identity guard only — does NOT transcribe)
  │     wss://api.vsip.online/v1/stream?mode=sidecar&tick_ms=80
  │     Authorization: Bearer {VSIP_API_KEY}
  │     → binary PCM frames
  │
  └─► OpenAI Realtime WebSocket (STT + LLM + TTS — all in one)
        wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview
        Authorization: Bearer {OPENAI_API_KEY}
        OpenAI-Beta: realtime=v1
        → input_audio_buffer.append (base64 PCM frames)

VSIP controls WHEN OpenAI responds.
OpenAI Realtime controls HOW it responds (STT + LLM + TTS).

━━━ EXACT IMPLEMENTATION RULES ━━━

1. DUAL CONNECTIONS
   - Open both WebSockets concurrently at startup
   - VSIP: Authorization: Bearer {VSIP_API_KEY}
   - OpenAI Realtime: Authorization: Bearer {OPENAI_API_KEY} + OpenAI-Beta: realtime=v1

2. SESSION CONFIGURATION (send to OpenAI Realtime after connect)
   Send this JSON immediately after connecting to OpenAI Realtime:
   {
     "type": "session.update",
     "session": {
       "modalities": ["text", "audio"],
       "instructions": "You are a helpful voice assistant. Keep answers to 1-2 short sentences.",
       "voice": "alloy",
       "input_audio_format": "pcm16",
       "output_audio_format": "pcm16",
       "turn_detection": null
     }
   }
   Setting turn_detection to null is CRITICAL — VSIP drives turn detection, not OpenAI.

3. DUAL AUDIO STREAMING (run concurrently)
   - Capture mic at 16000 Hz, 1 channel, int16 PCM, 1600 samples per chunk
   - For each chunk:
       → Send raw binary bytes to VSIP WebSocket
       → Send to OpenAI Realtime as:
         {"type": "input_audio_buffer.append", "audio": base64.b64encode(chunk).decode()}
   - This runs in one async loop — same chunk goes to both sockets

4. ON VSIP turn_end EVENT
   - VSIP has detected the user finished speaking
   - Commit the OpenAI audio buffer and request a response:
       Send: {"type": "input_audio_buffer.commit"}
       Send: {"type": "response.create"}
   - Log: "VSIP turn_end → committed to OpenAI Realtime"

5. ON VSIP speaker_verification EVENT
   - If should_respond == False (impostor / noise):
       Send to OpenAI: {"type": "response.cancel"}
       Send to OpenAI: {"type": "input_audio_buffer.clear"}
       Log: "VSIP: impostor blocked — response cancelled"
   - If should_respond == True: do nothing (response.create already sent in step 4)

6. ON VSIP barge_in EVENT
   - User is interrupting the agent mid-response
   - Send to OpenAI: {"type": "response.cancel"}
   - Stop PyAudio output stream playback immediately
   - Send to VSIP: {"command": "set_agent_state", "value": "idle"}
   - Clear any audio playback queue

7. OPENAI REALTIME EVENTS TO HANDLE
   - response.audio.delta:
       Decode base64 delta → raw 24000 Hz int16 PCM bytes
       Write to PyAudio output stream (24000 Hz, int16, mono)
       Before each chunk write, check a cancel_flag — if set, stop writing
   - response.audio.done or response.done:
       Send to VSIP: {"command": "set_agent_state", "value": "idle"}
       Log: "Response complete"
   - error:
       Log the error, do not crash

8. AGENT STATE SYNCHRONIZATION
   - When first response.audio.delta arrives (not yet playing):
       Send to VSIP: {"command": "set_agent_state", "value": "speaking"}
   - When response.done arrives:
       Send to VSIP: {"command": "set_agent_state", "value": "idle"}
   - When barge_in triggers:
       Send to VSIP: {"command": "set_agent_state", "value": "idle"}

9. OUTPUT AUDIO PLAYBACK
   - Open a PyAudio output stream at 24000 Hz, int16, mono
   - Write PCM chunks from response.audio.delta directly to the stream
   - Keep a cancel_flag asyncio.Event(); set it on barge_in, clear it when new response starts

10. ERROR HANDLING
    - Wrap all WebSocket sends in try/except — ignore errors if socket closed
    - If either WebSocket disconnects: log and shut down both cleanly
    - Never let one socket's error crash the other

━━━ DEPENDENCIES ━━━
pip install websockets pyaudio python-dotenv

━━━ OUTPUT ━━━
One Python file: agent_sidecar_mode.py
Run with: python agent_sidecar_mode.py
No web UI needed. Console logs show VSIP events and OpenAI Realtime state.

━━━ KEY DIFFERENCE FROM FULL MODE ━━━
In sidecar mode you do NOT call Whisper, GPT, or TTS separately.
OpenAI Realtime handles all of that internally.
VSIP's only job is: (a) tell you when a turn ends, (b) verify the speaker,
(c) fire barge_in when the user interrupts. Everything else is OpenAI Realtime.

pip install vsip-sdk              # typed client + VoiceSession
pip install "vsip-sdk[pipecat]"  # + drop-in Pipecat processor
# distribution is vsip-sdk; the import package is still 'vsip'

from vsip import VSIPClient, VoiceSession, Turn, int16_to_wav

client = VSIPClient(api_key="vsip_...", url="wss://api.vsip.online/v1/stream")

async def on_turn(turn: Turn):
    if not turn.should_respond:      # verified: not your enrolled user
        return
    text  = await my_stt(int16_to_wav(turn.audio))
    reply = await my_llm(text, interrupted=turn.interrupted_agent)
    await session.speak_gate()       # never talk over an open turn
    await client.agent_speaking()
    await my_tts_play(reply)         # stop this in on_barge_in
    await client.agent_idle()

session = VoiceSession(client, on_turn=on_turn,
                       on_barge_in=lambda e: my_tts_stop())

async with client:
    await session.start()
    async for frame in mic_frames():  # 16kHz mono int16 PCM
        await session.send_audio(frame)

from vsip import VSIPClient
from vsip.adapters.pipecat import VSIPProcessor

vsip = VSIPProcessor(VSIPClient(api_key="vsip_...", url="wss://.../v1/stream"))

pipeline = Pipeline([
    transport.input(),
    stt,
    vsip,              # ← identity-gated barge-in + verification gating
    context_aggregator,
    llm,
    tts,
    transport.output(),
])

from livekit.agents import AgentSession
from vsip import VSIPClient
from vsip.adapters.livekit import VSIPVAD

vsip_vad = VSIPVAD(VSIPClient(api_key="vsip_...", url="wss://.../v1/stream"))

session = AgentSession(vad=vsip_vad, stt=..., llm=..., tts=...)

from vsip import VSIPClient
from vsip.adapters.twilio import TwilioBridge

client = VSIPClient(api_key="vsip_...", url="wss://.../v1/stream",
                    audio_format="mulaw8k")
bridge = TwilioBridge(client, on_turn_end=..., on_verification=...)
await bridge.run(ws.iter_text(), send=ws.send_text)  # FastAPI/Starlette WS

import { VSIPClient, VoiceSession, type Turn } from "vsip-sdk";

const client = new VSIPClient({ apiKey: "vsip_...", url: "wss://.../v1/stream" });
const session = new VoiceSession(client, {
  onTurn: (t: Turn) => { if (t.shouldRespond) respond(t.audio); },
  onBargeIn: () => myTts.stop(),
});
await client.connect();
session.start();

from vsip import ProfilesClient

profiles = ProfilesClient(api_key="vsip_...", base_url="https://api.vsip.online")

# enroll (>=3s clean speech; wav/mp3/int16/float32/mulaw8k)
p = await profiles.enroll("alice", wav_bytes, audio_format="wav",
                          consent_obtained=True, retention_days=365)

# 1:1 verify
r = await profiles.verify(p["profile_id"], sample_bytes, audio_format="wav")
# -> {"match": true, "score": 0.82, "threshold": 0.75, ...}

# 1:N identify
who = await profiles.identify(sample_bytes, audio_format="wav")

from vsip import AnalyzeClient

analyze = AnalyzeClient(api_key="vsip_...", base_url="https://api.vsip.online")

job = await analyze.submit(open("call.wav", "rb").read(), "call.wav")
result = await analyze.wait(job["job_id"])   # polls until done

print(result["summary"]["talk_time_ratio"])  # {"speaker_0": 0.62, ...}
print(result["turns"], result["silences"], result["noise_events"])