Speech in, face out.

On this page

• How it works
• Install · Open a session
• iOS quick start · Web quick start
• Conversations: Yoob voice · your own OpenAI · Gemini Live · LiveKit
• Microphone controls
• Characters · States · Errors
• Security · Network · Content Security Policy · Offline and bundled characters
• Device support · Source and examples

How it works

• Your backend opens a session with your Yoob API key and hands the app or page a short-lived session token and download grant.
• The SDK downloads the character from cdn.yoob.com in signed, verified chunks and caches it. The character’s picture and idle motion appear while the models finish downloading.
• You pass speech as mono 16-bit PCM. The SDK plays it and renders the moving face on the device. Nothing it renders is streamed from a server.
• Speech comes from anywhere: your own LLM and TTS, a recording, or a live conversation with Yoob voice, your own OpenAI or Gemini account, or a LiveKit agent, with barge-in.
• Session time is metered by a heartbeat that carries only the session token. No frames are sent to Yoob, and audio reaches Yoob only if you use Yoob voice — see Network.

Install

iOS

Swift Package Manager. Requires iOS 17 or later and Xcode 16 or later. In Xcode choose File › Add Package Dependencies… and enter https://github.com/Yoob-com/yoob-ios-sdk, or add it to Package.swift:

// Package.swift
.package(url: "https://github.com/Yoob-com/yoob-ios-sdk", from: "0.3.0")

// in your target's dependencies
.product(name: "Yoob", package: "yoob-ios-sdk")

Character files are not in the package. They download on first use (see Characters). If you use the microphone, add NSMicrophoneUsageDescription to your Info.plist.

Web Preview

npm install @yoob/avatar

The package is on npm (about 180 KB, no dependencies; LiveKit support is an optional import, @yoob/avatar/livekit). Source: github.com/Yoob-com/yoob-web-sdk.

The web SDK needs WebGPU and runs in current desktop Chrome and Edge. Mobile browsers are not supported yet. Call YoobAvatar.isSupported() before loading anything and show a fallback if it says no.

Open a session on your backend

Create an API key in the Yoob console at la.yoob.com. Test keys start with yoob_test_ and open sandbox sessions; live keys start with yoob_live_ and draw on your prepaid credit. A key never goes in an app or a web page. Your server exchanges it for a short-lived session and returns that to the client. Check who the user is first: every session is metered to your workspace. See Security.

curl -X POST https://api2.yoob.com/api/v1/avatar/sessions \
  -H "Authorization: Bearer $YOOB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"characters": ["luna-anime"]}'

{ "session_token": "…", "download_token": "yg1.…", "heartbeat_seconds": 15, … }

The same call from Node, as a route your app or page calls:

// POST /yoob-session on your server (Node 18+)
export async function yoobSession(character) {
  const response = await fetch("https://api2.yoob.com/api/v1/avatar/sessions", {
    method: "POST",
    headers: {
      authorization: `Bearer ${process.env.YOOB_API_KEY}`,
      "content-type": "application/json",
    },
    body: JSON.stringify({ characters: [character] }),
  });
  if (!response.ok) throw new Error(`Yoob session failed: ${response.status}`);
  return response.json(); // pass this JSON to the SDK unchanged
}

The SDK calls your credentials function again when a session ends early — for example after the app was in the background — so the route must be safe to call more than once. Both SDK repositories include a complete token server: iOS, web.

iOS quick start

import SwiftUI
import Yoob

struct CharacterScreen: View {
    @State private var avatar = YoobAvatar(.cloud(character: "luna-realistic") {
        try await MyBackend.yoobSession()   // returns YoobCredentials, decoded from your route
    })

    var body: some View {
        YoobAvatarView(avatar)             // contentMode: .fill (default) or .fit
            .ignoresSafeArea()
            .task { try? await avatar.prepare() }
    }
}

prepare() downloads what is missing and warms up the renderer. It is safe to call again after a failure; a download resumes where it stopped. Watch avatar.phase for progress (see States). YoobCredentials decodes the session JSON directly.

Make it talk

// For each chunk as it streams in: PCM16, mono, little-endian.
try avatar.speak(pcm: chunk, sampleRate: 24_000)

// After the last chunk. The face settles back to idle when the audio ends.
avatar.endSpeech()

// Barge-in: stop now. Returns how many samples were heard.
let heard = avatar.interrupt()

// When the character leaves the screen: ends the metered session. Files stay cached.
await avatar.close()

The avatar holds the voice back until the first frame is ready, up to maxSyncDelay (1.8 s by default), so lips and voice start together. Audio always plays, even if the character failed to load. Keep one sample rate per utterance; 24 kHz is the default, and 16 kHz and 48 kHz streams are accepted too.

Your app already plays the audio

Render without playing, and tell the avatar how much the listener has heard:

try avatar.appendAudio(pcm: chunk, sampleRate: 24_000)
avatar.audioPlayed(samples: samplesHeardSoFar)   // call often, e.g. from your audio tap
avatar.endSpeech()

Web quick start

import { YoobAvatar } from "@yoob/avatar";

const support = await YoobAvatar.isSupported();
if (!support.supported) showFallback(support.reason);

const avatar = new YoobAvatar({
  container: document.querySelector("#character"),
  character: "luna-anime",
  getCredentials: () => fetch("/yoob-session", { method: "POST" }).then((r) => r.json()),
  onProgress: ({ fraction }) => (bar.value = fraction),
  onPhase: (phase) => console.log(phase),   // downloading → warming → ready ⇄ speaking
  onError: (error) => console.warn(error.code, error.message),
});
await avatar.prepare();

The character fills its container; pass fit: "contain" to letterbox instead. version pins a character version.

Make it talk

button.onclick = async () => {
  await avatar.unlockAudio();          // browsers allow sound only after a click or key press
  for await (const chunk of myTtsStream()) {
    avatar.speak(chunk);               // Int16Array or ArrayBuffer: 24 kHz mono PCM16
  }
  avatar.endSpeech();
};

const heardMs = avatar.interrupt();    // stop now; returns milliseconds heard
await avatar.destroy();               // ends the session and removes the character

Version 0.1 of the web SDK accepts 24 kHz audio only; request 24 kHz PCM from your voice provider. avatar.playedMs reports progress through the current utterance.

Conversations

The SDKs can run a live spoken conversation through the character, with barge-in and captions. Pick the voice: Yoob voice needs no provider key, or bring your own OpenAI account, Gemini Live or, on the web, a LiveKit agent. All of them are optional: the SDKs work with any speech source.

Yoob voice

No provider key needed. Yoob hosts the voice (OpenAI Realtime) and tunes it for the characters: server turn detection, far-field noise reduction, captions and a 1.08 speaking speed. Minutes are billed through your Yoob workspace at the Yoob voice rate.

Your backend asks Yoob for a voice session with your API key and returns the response unchanged:

curl -X POST https://api2.yoob.com/api/v1/voice/sessions \
  -H "Authorization: Bearer $YOOB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"voice": "marin", "instructions": "You are Luna, a warm, curious companion.", "max_seconds": 900}'

The 201 response has voice_session_id, voice_token, url (wss://voice.yoob.com/v1/realtime?model=…), model, max_seconds, credits_per_minute and expires_at. voice, instructions (up to 8,000 characters) and max_seconds (default 1800) are optional.

iOS

var options = YoobConversation.Options()
options.greet = true

let conversation = YoobConversation(avatar: avatar, options: options) {
    try await MyBackend.yoobVoiceSession()     // returns YoobVoiceSession
}
try await conversation.start()                 // asks for the microphone
conversation.stop()

Decode the response with JSONDecoder().decode(YoobVoiceSession.self, from: data). The iOS token server has a /yoob-voice route.

Web

const conversation = new YoobConversation(avatar, {
  getVoiceSession: () => fetch("/yoob-voice", { method: "POST" }).then((r) => r.json()),
  greet: true,
  onUserTranscript: (text) => (userCaption.textContent = text),
  onAssistantTranscript: (text) => (lunaCaption.textContent = text),
  onError: (error) => (status.textContent = error.message),
});

talkButton.onclick = () => conversation.start();   // asks for the microphone

• One session per conversation. A voice session opens exactly one connection, and its token must be used within 5 minutes. start() asks for a new one every time, so don’t cache it.
• Voice and instructions are set on your backend. The app or page can’t change them, and the prompt never reaches the device. voice is one of alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin or cedar. If the backend leaves them out, the conversation’s own voice and instructions options are used.
• Fixed settings. Yoob sets the model, turn detection, noise reduction, transcription and speed, so those options are ignored.
• Out of credit. The API answers 402 {"code": "quota_exceeded"}, which surfaces as .outOfCredit on iOS and out-of-credit on the web. A rejected API key is unauthorized.

If the voice service ends the call, iOS sets lastError to .voiceSession(code:message:), and the web calls onError with a voice-session error (the close code is in error.details.closeCode). Both carry a message you can show the user.

Your own OpenAI account

Pass a client secret instead of a voice session, and OpenAI bills your account directly.

• Client secret from your backend. Your server creates a short-lived secret with OpenAI’s POST https://api.openai.com/v1/realtime/client_secrets. Never put your OpenAI key in an app or page.
• Direct to OpenAI. Microphone audio goes from the device to OpenAI. It does not pass through Yoob.
• Barge-in. The microphone stays open while the character speaks, with echo cancellation removing the character’s own voice. When the user starts talking, the character stops and the model is told how much of its reply was heard.
• Captions. What the user says (transcribed) and what the character says (as it streams) are both available.

iOS

var options = YoobConversation.Options()
options.voice = "marin"
options.instructions = "You are Luna, a warm, curious companion."
options.greet = true

let conversation = YoobConversation(avatar: avatar, options: options) {
    try await MyBackend.openAIClientSecret()   // returns the secret string
}
try await conversation.start()     // prepares the character, asks for the microphone, connects

conversation.state                  // .connecting, .listening, .thinking, .speaking, .ended
conversation.userTranscript         // captions (observable)
conversation.assistantTranscript
conversation.send(text: "Tell me a story")
conversation.stop()                 // releases the microphone; the character stays on screen

Web

import { YoobConversation } from "@yoob/avatar";

const conversation = new YoobConversation(avatar, {
  getClientSecret: () =>
    fetch("/openai-secret", { method: "POST" }).then((r) => r.json()).then((s) => s.value),
  voice: "marin",
  instructions: "You are Luna, a warm, curious companion.",
  greet: true,
  onUserTranscript: (text, final) => (userCaption.textContent = text),
  onAssistantTranscript: (text, final) => (lunaCaption.textContent = text),
  onState: (state) => console.log(state),   // connecting → listening → thinking → speaking
  onError: (error) => (status.textContent = error.message),
});

talkButton.onclick = () => conversation.start();   // call from a click: asks for the microphone
endButton.onclick = () => conversation.stop();
conversation.sendText("Tell me a story");

Defaults

The defaults come from latency measurements on the Yoob demo. Override any of them in the options.

semantic_vad waits to judge whether a sentence is finished, so it tolerates mid-sentence pauses at the cost of that extra delay: .semantic() on iOS, { type: "semantic_vad" } on the web. In a noisy room, raise the server VAD threshold rather than muting the microphone.

Gemini Live

Bring your own Gemini voice with YoobGeminiConversation. It has the same states, captions and barge-in as YoobConversation.

iOS

var options = YoobGeminiConversation.Options()
options.voice = "Kore"
options.systemInstruction = "You are Luna, a warm, curious companion."
options.greet = true

let conversation = YoobGeminiConversation(avatar: avatar, options: options) {
    try await MyBackend.geminiToken()          // an ephemeral token's `name`, created on your server
}
try await conversation.start()

Web

import { YoobGeminiConversation } from "@yoob/avatar";

const conversation = new YoobGeminiConversation(avatar, {
  getToken: () => fetch("/gemini-token", { method: "POST" }).then((r) => r.json()).then((t) => t.name),
  voice: "Kore",
  systemInstruction: "You are Luna, a warm, curious companion.",
  greet: true,
});
talkButton.onclick = () => conversation.start();

The app or page never sees your Gemini API key. Your backend creates a single-use ephemeral token and returns its name:

// POST /gemini-token on your server (Node, @google/genai)
const token = await ai.authTokens.create({
  config: {
    uses: 1,
    expireTime: new Date(Date.now() + 30 * 60_000).toISOString(),    // messages stop after this
    newSessionExpireTime: new Date(Date.now() + 60_000).toISOString(), // must connect before this
    liveConnectConstraints: { model: "gemini-3.8-live" },              // optional: lock the model
  },
});
return { name: token.name };

Lock at least the model, so a leaked token can’t be used for anything else. Settings locked with liveConnectConstraints take precedence over the ones the client sends. The conversation converts the microphone’s 24 kHz audio to the 16 kHz Gemini expects and plays Gemini’s 24 kHz replies through the avatar. The default model is gemini-3.8-live, with high start and end sensitivity, a 450 ms silence window and captions for both sides. Audio-only Gemini sessions last up to 15 minutes; iOS sets goAwayTimeLeft and the web calls onGoAway(timeLeft) before Gemini closes the connection. Gemini doesn’t report when a user turn ends, so a spoken turn goes from listening straight to speaking.

LiveKit agents Web

@yoob/avatar/livekit shows a LiveKit voice agent as a Yoob character. The agent’s audio track is decoded in the browser and drives the face on the visitor’s GPU: there is no avatar worker in the room and no video track. Any normal voice agent works. It needs livekit-client 2.9 or newer.

import { Room } from "livekit-client";
import { YoobLiveKitSession } from "@yoob/avatar/livekit";

const room = new Room();
const session = new YoobLiveKitSession(avatar, {
  room,
  onUserTranscript: (text) => (userCaption.textContent = text),
  onAssistantTranscript: (text) => (lunaCaption.textContent = text),
});

talkButton.onclick = async () => {
  const sound = avatar.unlockAudio();     // while the click still allows sound
  const { url, token } = await fetch("/livekit-token", { method: "POST" }).then((r) => r.json());
  await room.connect(url, token);
  await sound;
  await session.start();                  // publishes the microphone
};

The session follows the first agent in the room (or agentIdentity) and captions come from the agent’s lk.transcription streams. The avatar plays the agent’s voice, so don’t also attach that track yourself (for example with RoomAudioRenderer), or it will be heard twice.

Microphone controls

Each avatar owns the user’s microphone as avatar.microphone. It captures through the same audio path that plays the character, so echo cancellation removes the character’s voice. YoobConversation uses it automatically; you can also use it with your own voice stack.

iOS

let mic = avatar.microphone
try await mic.start()                  // asks for permission if needed

mic.inputs                             // [Input(id:name:)] once started; refreshed when routes change
try mic.select(inputID: input.id)      // switches mid-conversation; nil = the system's choice
mic.setMuted(true)                     // nothing is captured or delivered while muted
mic.level                              // 0...1, about 20 times a second (observable)
if case .failed(let error) = mic.state { show(error.localizedDescription) }

mic.onAudio = { pcm in … }             // your own voice stack: 24 kHz mono PCM16
mic.stop()

mic.state is .off, .starting, .live, .muted or .failed(error). A refused permission fails with .permissionDenied and a message pointing to Settings.

Web

const mic = avatar.microphone;
select.replaceChildren(...(await mic.devices()).map((d) => new Option(d.label, d.deviceId)));
select.onchange = () => mic.select(select.value || null);   // switches mid-conversation
muteButton.onclick = () => mic.setMuted(!mic.muted);
mic.on("level", (level) => (meter.style.width = `${level * 100}%`));
mic.on("devices", refreshList);                              // plugged in or removed
mic.on("error", (error) => (status.textContent = error.message));

// Your own voice stack:
await mic.start();
const off = mic.on("audio", (pcm) => send(pcm));   // Int16Array, 24 kHz, 20 ms packets

on() returns a function that removes the listener. Device labels are empty until the user has granted microphone access once, so list devices again after start(). A stalled input is reopened automatically, and an unplugged device falls back to the system default. Microphones need HTTPS.

Characters

Character files are served by cdn.yoob.com as signed manifests and chunks. The SDK checks every file against its signature and checksum and discards anything that does not match. Without a pinned version the newest compatible one is used, and an update downloads only the files that changed. Pin one with YoobAvatar(source, version: "…") on iOS or the version option on the web. Yoob.clearCache() and YoobAvatar.clearCache() remove downloaded characters.

States

Both SDKs expose avatar.phase. On iOS it is observable; on the web, pass onPhase.

On iOS, avatar.stats counts frames shown and skipped, and avatar.lastRendererError says why rendering stopped, if it did. If the renderer stops, audio keeps playing and the idle face stays on screen.

Errors

Both SDKs throw YoobError: an enum case on iOS, and an Error with a code on the web. Messages are written to be shown to users.

Security

• Keys stay on your server. The API rejects keys sent from a browser.
• Your token server authenticates your users. Anyone who can call it can spend your credit.
• Request a specific character. Pass the one character the screen needs in characters.
• Grants are short-lived. A download grant covers that one character and lasts 15 minutes; heartbeats renew it while the session is open.
• Packs are signed. The SDK verifies every character file and discards anything that does not match.
• Spend caps. Set a monthly cap in the console. Sessions stop at the cap or when your balance reaches zero.
• Outages. If the SDK can’t reach Yoob, a running session keeps going for up to 10 minutes; new sessions need the API. An explicit denial — a revoked key, a suspended workspace, or no credit left — stops the session at once.

Network

This is everything the SDKs send to Yoob:

No rendered frames are sent to Yoob. Heartbeats are how session time is metered, so end the session when the character leaves the screen. With Yoob voice, microphone audio goes to voice.yoob.com, which relays it to OpenAI and meters the minutes. With your own OpenAI account or Gemini, audio goes directly from the device to that provider, and with a LiveKit agent it goes to your LiveKit server. Your own LLM and voice provider see whatever you send them; Yoob does not.

Content Security Policy (web)

If your site sets a Content Security Policy, allow:

connect-src https://cdn.yoob.com https://api2.yoob.com wss://voice.yoob.com;
script-src  'self' 'wasm-unsafe-eval';
worker-src  'self';
img-src     'self' blob:;
media-src   'self' blob:;

wss://voice.yoob.com is only needed for Yoob voice. With your own provider, allow wss://api.openai.com or wss://generativelanguage.googleapis.com instead, or your LiveKit server for LiveKit agents. Workers and audio worklets ship as files in the package and load from your own origin. The character’s poster and idle video are shown from downloaded data, which is why img-src and media-src need blob:.

Offline and bundled characters

iOS. A complete download is reused when the device is offline, as long as no version is pinned. To ship a character inside your app instead, create the avatar with YoobAvatar(.local(url)), pointing at a pack directory (character.json plus its files). Local packs make no network calls and are not metered.

Web. Downloaded chunks are kept in the browser’s cache, so a returning visitor only fetches what changed. The web SDK has no local-pack mode and needs a session to start.

Device support

Source and examples

• Yoob-com/yoob-ios-sdk — the iOS SDK. Examples/QuickStart is a one-screen app with both characters, a conversation with microphone controls, and a sample greeting.
• Yoob-com/yoob-web-sdk — the web SDK. apps/demo is a one-page demo: talk to Luna, pick a microphone, mute, see captions.

# iOS
cd Examples/QuickStart && xcodegen generate && open QuickStart.xcodeproj

# Web (at the repository root)
npm install && npm run build
YOOB_API_KEY=… node examples/token-server/server.mjs   # serves /yoob-session on :3100
npm run demo

The SDK source is Apache-2.0. Character model files are licensed separately and are not in either repository. Questions: sales@yoob.com. Security reports: security@yoob.com.

Covers Yoob for iOS 0.1.0 and @yoob/avatar 0.1.0. Last updated 17 September 2026.