# VSIP — Integration Guide for AI Coding Assistants

> **How to use this file:** paste this whole document into your AI coding
> assistant (Claude, Cursor, Copilot, ChatGPT…) together with a sentence about
> your stack, e.g. *"Integrate VSIP into my Python voice agent that uses
> Deepgram STT and ElevenLabs TTS."* Everything an assistant needs to write a
> correct integration is below. It is self-contained on purpose. The rules
> marked **MUST** each exist because skipping them broke a real deployment.

---

## 1. What VSIP is

VSIP is a real-time **voice-events API** for AI voice agents. You stream
microphone audio to one WebSocket. VSIP tells you:

- **when the user's turn starts and ends** (semantic end-of-turn detection),
- **who is speaking** (speaker identity, locked to an enrolled primary user),
- **whether to answer** (`speaker_verification.should_respond`), and
- **when the user interrupts your agent** (`barge_in`, identity-gated, so a
  cough or bystander cannot cut the agent off).

VSIP does **not** do speech-to-text, LLM reasoning or text-to-speech. You bring
those. VSIP decides *when* and *whether* they run.

Three more products share the same API key:

| Product | Interface | Purpose |
|---|---|---|
| Voice Events (core) | `wss://api.vsip.online/v1/stream` | Live turn-taking + identity |
| Speaker Profiles | `POST /v1/profiles/{enroll,verify,identify}` | Persistent voiceprints, 1:1 and 1:N |
| Batch Analytics | `POST /v1/analyze` | Speaker timeline for a recorded file |

## 2. Credentials and install

- Get an API key at `https://vsip.online` → Dashboard → API Keys. Keys start with `vsip_`.
- Store it **server-side** as `VSIP_API_KEY`. **MUST NOT** put it in a
  `NEXT_PUBLIC_*` / `VITE_*` / frontend variable or any code that runs in a browser.
- **Browser apps:** your backend calls `POST /v1/stream/token` with the API key
  and returns the short-lived `vst_…` token to the browser, which connects with
  it (see §8 "Browser"). Tokens expire (default 5 min, max 1 h), work only on
  the stream WebSocket, and die when the API key is revoked.
- Python: `pip install vsip-sdk` (import name `vsip`). Extras: `vsip-sdk[pipecat]`, `vsip-sdk[livekit]`.
- JS/TS (browser + Node 18+): `npm install vsip-sdk`. Node 18–20 need the `ws` package passed as `webSocketImpl`.

## 3. Connection contract

```
wss://api.vsip.online/v1/stream?mode=full&audio_format=int16
Authorization: Bearer <VSIP_API_KEY>        (server-side clients)
?token=<vst_ token from POST /v1/stream/token>   (browsers — never the vsip_ key)
```

```http
POST https://api.vsip.online/v1/stream/token          (from your backend)
Authorization: Bearer <VSIP_API_KEY>
Content-Type: application/json

{"ttl_seconds": 300, "mode": "full"}        // both optional; ttl 30–3600, mode pins the token

→ 200 {"token": "vst_…", "expires_at": 1793000000, "ws_url": "wss://api.vsip.online/v1/stream"}
```

| Query param | Values | Default | Notes |
|---|---|---|---|
| `mode` | `full`, `sidecar` | `full` | `sidecar` = listen-only beside a speech-to-speech model (OpenAI Realtime etc.); rejects `set_agent_state` |
| `audio_format` | `int16`, `float32`, `mulaw8k` | `int16` | `mulaw8k` for telephony (Twilio) |
| `aec` | `client`, `server` | `client` | `server`: you also send your TTS audio as an echo reference (§7) |
| `resumable` | `true`, `false` | SDK: `true` | Parks the session 60 s on disconnect; reconnect with `?resume=<stream_id>` |
| `tick_ms` | `20`–`500` (clamped) | `80` | Analysis cadence; leave at 80 unless told otherwise |

**Audio in:** binary WebSocket frames, **16 kHz mono**, little-endian, 20–100 ms
per frame (int16 = 640–3200 bytes). Send continuously, including silence. The
SDKs handle all of this.

## 4. Event reference (server → client JSON)

| Event | Key fields | What your app does |
|---|---|---|
| `session_start` | `stream_id`, `session_id` | Store `stream_id` for resume |
| `turn_start` | `speaker_id`, `interrupted_agent` | Start buffering the user's audio |
| `turn_end` | `audio_start`, `duration_ms`, `endpoint`, `speaker_id` | Turn complete → transcribe (but see gating) |
| `speaker_verification` | `should_respond`, `speaker_id`, `similarity`, `audio_start` | **THE response gate** — match to the turn by `audio_start` |
| `barge_in` | `speaker_id`, `confidence` | **Stop TTS immediately**, send `set_agent_state idle` |
| `lock_status` | `status` = `locked:<id>` / `unlocked` / `no_speaker` | Track whether gating is active |
| `enrollment_result` | `success`, `label`, `quality`, `locked`, … | Show enrollment quality; re-enroll on `poor` |
| `speaker_change`, `vad_start`, `vad_end`, `noise_event`, `telemetry` | — | Optional UI / analytics |
| `speaker_audio_ready` | `speakers[]` (base64 int16 per voice) | Optional: separated voices after overlap |
| `error` | `message` | Log it |

Unknown event types and fields **MUST** be ignored, not treated as errors. The
protocol only ever grows additively.

## 5. Control messages (client → server JSON text frames)

```json
{"command": "set_agent_state", "value": "speaking"}   // your TTS started   (full mode)
{"command": "set_agent_state", "value": "idle"}       // your TTS ended/stopped
{"command": "enroll_speaker", "label": "primary", "audio_base64": "<float32 PCM 16k mono>", "lock": true}
{"command": "set_barge_in_mode", "mode": "locked_only"}  // locked_only | locked_strict | any_enrolled | any_speaker | disabled
{"command": "toggle_lock"}
{"command": "reset"}
```

## 6. The integration loop and its rules

```
mic ──► VSIP (and optionally a streaming STT at the same time)
            │
   turn_end ─┼─► slice turn audio (with ≥1.0 s onset lookback) ──► STT ──► LLM
            │                                                         │
 speaker_verification(should_respond) ─────────── gate ───────────────┤
            │                                                         ▼
            │                        set_agent_state speaking ──► TTS playback
   barge_in ─┴─► stop TTS NOW + clear queued audio + set_agent_state idle
```

**MUST rules:**

1. **Gate every response on `speaker_verification.should_respond`** once a
   speaker is locked. Correlate verification with the turn by `audio_start`.
   Do not use `turn_end.speaker_id` as the gate; it can be a whole turn stale.
   Allow up to 12 s for the verdict. On timeout, do not respond.
2. **Send `set_agent_state` around every TTS playback** (full mode):
   `speaking` when audio starts, `idle` when it ends *or is stopped*. Without
   it, VSIP cannot detect barge-in.
3. **On `barge_in`:** stop playback within ~250 ms, drop queued TTS, cancel the
   in-flight LLM/TTS request, send `set_agent_state idle`. Then treat the
   following `turn_start`/`turn_end` (`interrupted_agent: true`) as the user's
   new utterance.
4. **Slice turn audio with lookback.** `turn_start` arrives a few hundred ms
   after speech onset. Keep a rolling buffer and start the slice 1.0 s before
   `turn_start` (1.5 s for barge-ins), or the first word is lost.
5. **Never start TTS while a user turn is open** (between `turn_start` and
   `turn_end`). The SDK's `speak_gate()` / `speakGate()` does this.
6. **Enroll with 15–30 s of natural speech** (e.g. a 2–3 sentence
   self-introduction), using `lock: true`. If `quality` is `poor`, ask the user
   to re-enroll somewhere quieter.

**Latency optimisation (recommended):** run STT and the LLM *as soon as
`turn_end` arrives*, in parallel with verification, and hold only *playback*
until `should_respond` is true. Discard the prepared reply if it is false. For
privacy, only speculate when the live tracker attributes the turn to the
locked speaker. Otherwise wait for the verdict before sending audio to STT.

## 7. Echo and simultaneous speech

If the agent plays through speakers, the mic hears the agent and the user
together. That degrades identity and barge-in.

- Best: `aec=server`. Prefix each binary frame with one tag byte: `0x01` + mic
  PCM, `0x02` + the exact TTS PCM you are playing (same format and rate). Python:
  `client.send_mic_frame()` / `client.send_tts_reference()`. With server AEC,
  barge-in identity is strict automatically.
- Otherwise: browser `echoCancellation: true` (client AEC), or headphones.
- `set_barge_in_mode locked_strict` blocks every voice except the locked one
  from interrupting, even when it reads "unknown".

## 8. Copy-paste implementations

### Python (any STT/LLM/TTS)

```python
import asyncio, os
from vsip import VSIPClient, VoiceSession, Turn, int16_to_wav

client = VSIPClient(api_key=os.environ["VSIP_API_KEY"], mode="full", audio_format="int16")

async def on_turn(turn: Turn):
    if not turn.should_respond:          # rule 1 (SDK already held it for the verdict)
        return
    text = await my_stt(int16_to_wav(turn.audio))     # turn.audio includes lookback (rule 4)
    reply = await my_llm(text, interrupted=turn.interrupted_agent)
    await session.speak_gate()                         # rule 5
    await client.agent_speaking()                      # rule 2
    try:
        await my_tts_play(reply)                       # must be cancellable by on_barge_in
    finally:
        await client.agent_idle()

def on_barge_in(event):
    my_tts_stop()                                      # rule 3 (SDK sends agent_idle)

session = VoiceSession(client, on_turn=on_turn, on_barge_in=on_barge_in)

async def main():
    async with client:
        await session.start()
        await client.enroll_speaker("primary", enrollment_pcm_float32_bytes, lock=True)  # rule 6
        async for frame in my_mic_frames():           # 16 kHz mono int16 bytes, 20 ms each
            await session.send_audio(frame)

asyncio.run(main())
```

`VoiceSession` implements rules 1, 3 (agent_idle), 4 and 5 for you, and queues
turns so a barge-in never drops the interrupting utterance.

A complete, runnable version of this loop with OpenAI STT, LLM and TTS wired
in (streamed replies, sentence-level synthesis, interruptible playback, file
mode for testing without a mic) is at
<https://vsip.online/examples/voice-agent/agent.py>, with its
[README](https://vsip.online/examples/voice-agent/README.md) and
[requirements](https://vsip.online/examples/voice-agent/requirements.txt).

### TypeScript (Node server, or browser with a token)

```ts
import { VSIPClient, VoiceSession, type Turn } from "vsip-sdk";

// Node / server: authenticate with the API key.
const client = new VSIPClient({ apiKey: process.env.VSIP_API_KEY!, url: "wss://api.vsip.online/v1/stream" });
// Browser: NEVER the API key — fetch a short-lived token from your backend instead:
// const client = new VSIPClient({
//   getToken: async () => (await (await fetch("/api/vsip-token", { method: "POST" })).json()).token,
// });
const session = new VoiceSession(client, {
  onTurn: async (turn: Turn) => {
    if (!turn.shouldRespond) return;                 // rule 1
    const text = await myStt(turn.audio);            // Int16Array, includes lookback
    const reply = await myLlm(text);
    await session.speakGate();                       // rule 5
    client.agentSpeaking();                          // rule 2
    try { await myTtsPlay(reply); } finally { client.agentIdle(); }
  },
  onBargeIn: () => myTtsStop(),                      // rule 3
});

await client.connect();
session.start();
const result = await client.enrollSpeaker("primary", enrollmentFloat32, { lock: true }); // rule 6
if (result.quality === "poor") askUserToReEnroll();
// for every 20 ms mic frame (16 kHz mono Int16Array): session.sendAudio(frame)
```

Capture mic audio with an `AudioWorklet` (not the deprecated
`ScriptProcessorNode`), resample to 16 kHz and convert to Int16.

The token route on your backend (Next.js / Express shown; the Python SDK has
the same helper as `vsip.create_stream_token`):

```ts
// app/api/vsip-token/route.ts — protect it with YOUR app's own auth
import { createStreamToken } from "vsip-sdk";

export async function POST() {
  const { token, expires_at } = await createStreamToken({ apiKey: process.env.VSIP_API_KEY!, ttlSeconds: 300 });
  return Response.json({ token, expires_at });
}
```

`getToken` is called before every connect and every automatic reconnect, so
an expired token never breaks session resume.

### Framework adapters (Python)

- **Pipecat:** `VSIPProcessor(VSIPClient(...))` placed between STT and the LLM
  context aggregator. Set `allow_interruptions=False` on the transport so VSIP
  is the only interruption authority.
- **LiveKit Agents:** `AgentSession(vad=VSIPVAD(VSIPClient(...), on_verification=...), ...)`.
- **Twilio Media Streams:** `TwilioBridge(VSIPClient(..., audio_format="mulaw8k"), on_turn_end=..., on_verification=...)`,
  then `await bridge.run(ws.iter_text(), send=ws.send_text)`. It sends Twilio `clear` on barge-in.

### Sidecar beside a speech-to-speech model

Use `mode=sidecar` when OpenAI Realtime (or similar) runs the conversation.
Stream the same mic audio to VSIP, and gate or cancel the S2S model's responses
using `speaker_verification.should_respond`. Do not send `set_agent_state` in
sidecar mode; the server rejects it.

## 9. Speaker Profiles (REST)

The machine-readable spec for every REST endpoint below (and
`POST /v1/stream/token`) is at <https://vsip.online/openapi.json>. Generate a
client from it if your language has no SDK.

All requests: `Authorization: Bearer <VSIP_API_KEY>`, JSON body. `audio_format`:
`int16`, `float32` (raw 16 kHz mono PCM), `wav`, `mp3`, `mulaw8k`. Audio is base64
(max ~4 MB of base64).

| Call | Body | Returns |
|---|---|---|
| `POST /v1/profiles/enroll` | `label`, `audio_base64`, `audio_format`, optional `consent_obtained`, `retention_days` (1–3650), `metadata` | `profile_id`, `label`, `sample_count`, … (≥3 s clean speech; same label adds a sample) |
| `POST /v1/profiles/verify` | `profile_id`, `audio_base64`, `audio_format`, optional `threshold` | `match`, `score`, `threshold` (≥1 s speech) |
| `POST /v1/profiles/identify` | `audio_base64`, `audio_format`, optional `threshold` | `matched`, `profile_id`, `label`, `score` |
| `GET /v1/profiles?limit=&offset=` | — | paginated list (no embeddings) |
| `GET /v1/profiles/usage` | — | monthly verify+identify usage vs allowance |
| `DELETE /v1/profiles/{id}` | — | `{"status": "deleted"}` |

SDKs: Python `ProfilesClient(api_key)`, JS `ProfilesClient`. To verify a caller
from their first word in a live stream, connect with `?profile_id=<id>`. This
is only available where the operator has enabled the profile bridge. Voiceprints
are biometric data: obtain consent and pass `consent_obtained: true`. Similarity
is not liveness. A good voice clone can pass, so don't use voice alone for
payments or account recovery.

## 10. Batch Analytics (REST)

`POST /v1/analyze` (multipart: `file` = wav/mp3/webm/ogg ≤100 MB, 0.5 s–1 h;
optional `tick_ms`) → `202 {job_id}`. Poll `GET /v1/analyze/{job_id}` until
`status` is `done` (`result` holds turns, speakers, talk time, interruptions,
silences, noise events and a `summary`) or `error`. Also `GET /v1/analyze` and
`GET /v1/analyze/usage`. SDKs: Python `AnalyzeClient(api_key).submit(...)`
then `.wait(job_id)`; JS `AnalyzeClient`.

## 11. Errors, limits and close codes

| Close code | Meaning | Client action |
|---:|---|---|
| 4001 | Bad/missing API key | Fatal — fix credentials |
| 4002 | Unsupported configuration | Fatal — fix query params |
| 4004 | Resume target expired | Start a fresh session |
| 4005 | `profile_id` not found | Fatal for this session |
| 4029 | Concurrent-stream limit for your plan | Back off; close idle streams |
| 4030 | Monthly minute quota exhausted (free tier) | Fatal until upgrade/next month |
| 4031 | Audio sent faster than 3× real time | Send at real-time pace |
| 1011 | Server error | Retry with backoff |

The SDKs treat every 4xxx code above as permanent: they stop
reconnecting and surface the code (`VSIPError.close_code` in Python, an `error`
event with `close_code` and `fatal: true` in JS). Raw WebSocket clients should
do the same.

HTTP: `401` auth, `413` payload too large, `422` invalid body/audio, `429`
rate limit or monthly quota (body says which). `GET /ready` returns `503`
while models are still loading after a deploy — retry.

## 12. Testing checklist for your integration

- [ ] Enrolled user asks a question → `should_respond: true` → agent answers.
- [ ] A second person asks a question → `should_respond: false` → agent stays silent.
- [ ] User interrupts the agent mid-sentence → playback stops < 0.5 s → the interruption is answered.
- [ ] A cough or background TV during agent speech does **not** stop playback.
- [ ] `set_agent_state` is sent on every TTS start/stop, including stops caused by barge-in.
- [ ] Network blip under 60 s → the session resumes (SDK) without re-enrollment.
- [ ] No `vsip_` API key in any frontend bundle; browsers connect with a `vst_` token from your backend.

Live reference implementation: `https://vsip.online/voice-lab`. Its
unminified client is readable at `https://vsip.online/voice-lab/app.js`. It
does live STT, speculative LLM with gated playback, server AEC, and atomic
enroll+lock.