Custom LLM & API

Custom LLM & API

Beyond the built‑in providers in the catalog, Telenow can drive an agent with your own model in two ways:

• Custom LLM (customllm) — point at any OpenAI‑compatible chat endpoint (self‑hosted Llama via Ollama/vLLM, a private gateway, etc.). Telenow speaks the standard OpenAI wire format; everything else (tools, RAG, transcripts) works normally.
• Custom API (customapi) — plug in your own agentic backend. Telenow sends the caller's turn to your HTTP endpoint and streams back whatever your service returns. Your service owns the "brain" (prompting, tools, RAG, memory); Telenow provides the real‑time voice (STT, TTS, telephony).

Both are selected by setting the agent's llmProvider and llmConfig when creating/updating an agent.

Custom LLM (OpenAI-compatible)

Use this when you have a model behind an OpenAI‑compatible /chat/completions API.

{
  "name": "Self-hosted agent",
  "llmProvider": "customllm",
  "llmModel": "llama-3.1-8b-instruct",
  "llmConfig": {
    "baseUrl": "https://llm.yourcompany.com/v1",
    "apiKey": "optional-key"
  },
  "sttProvider": "deepgram",
  "ttsProvider": "elevenlabs",
  "ttsVoice": "rachel",
  "sessionConfig": {}
}

llmModel is the model name your endpoint expects. Because it's the OpenAI wire format, tools and knowledge bases work the same as with hosted models.

Custom API (your agentic endpoint)

Use this when you already have an assistant/orchestration service and just want Telenow for voice. Telenow becomes a thin streaming relay to your endpoint.

{
  "name": "Bring-your-own-brain",
  "llmProvider": "customapi",
  "llmModel": "customapi",
  "llmConfig": { "apiEndpoint": "https://api.yourcompany.com/assistant" },
  "sttProvider": "deepgram",
  "ttsProvider": "elevenlabs",
  "ttsVoice": "rachel",
  "sessionConfig": {}
}

What Telenow sends

On each user turn, Telenow makes a streaming POST to your endpoint. The query string ?calling=true&stream=true is always appended to your apiEndpoint:

POST https://api.yourcompany.com/assistant?calling=true&stream=true
Authorization: Bearer <bearer>          # only if a bearer was provided at session start
Content-Type: application/json

{ "query": "the caller's latest utterance", "userId": "…", /* …your payload fields, merged at top level… */ }

Provide bearer and payload at session start via init-web-call. The bearer becomes the Authorization: Bearer … header on every turn; payload becomes the merged body fields above — handy for passing the signed‑in customer's identity/context to your backend.

What your endpoint returns

A Server‑Sent Events (SSE) stream. Each event is a data: line whose value is a JSON object with a type. Telenow reads two things: token deltas it speaks in real time, and control events it reacts to.

data: {"type":"assistant_token","delta":"Hi there, "}
data: {"type":"assistant_token","delta":"how can I help?"}
data: {"type":"call_end","msg":"Thanks for calling — goodbye!"}
data: [DONE]

Notes that match the parser exactly:

• Lines must start with data: (with the space). The JSON must be on a single data: line.
• A data: line (or even a multibyte UTF‑8 character) may be split across TCP chunks — Telenow buffers and reassembles, so you don't have to align your writes to event boundaries.
• A type of assistant_token with a missing/empty delta is dropped; an event with no type is ignored.

Tools, RAG and memory live on your server

With Custom API you already own the agentic workflow — tool calls, RAG, memory, and the system prompt all run inside your endpoint. Telenow just attaches STT + TTS and speaks your streamed tokens. So you don't define tools in Telenow for a customapi agent (the Tools step is skipped — Telenow never sends a tool list and never runs its tool loop). Implement tools inside your endpoint and emit assistant_token deltas back.

Want Telenow to dispatch tools for you (HTTP tools + native transfer/end‑call)? Switch the agent's Brain to OpenAI, Groq, Anthropic, or Custom LLM — those run Telenow's tool loop. customapi does not.

What still works. Because STT and TTS run normally, you still get a full transcript, call recording, post‑call analysis (it's computed from the transcript by Telenow's analysis model, independent of your brain), latency metrics, and webhooks. Billing: Telenow does not meter LLM tokens for customapi — that's your own model's cost on your side; you're billed for STT, TTS, and telephony as usual (see Billing & usage).

Two things still act on the live Telenow call, so your server drives them explicitly:

Hang up the call

Emit a call_end event in your SSE stream:

data: { "type": "call_end", "msg": "Thanks for calling — goodbye!" }

Telenow speaks the optional msg, then ends the call.

Transfer to a human

When your workflow decides to escalate, trigger a warm transfer from your server with the REST endpoint — pass the E.164 number to dial:

curl -X POST https://api.telenow.ai/api/sessions/{sessionId}/transfer \
  -H "x-api-key: vai_live_…" \
  -H "Content-Type: application/json" \
  -d '{ "to": "+15551234567" }'

Telenow bridges the live call to that number (respecting your Do‑Not‑Call list). Works on phone calls on every carrier (Plivo, Twilio, Vobiz, Exotel, Vonage, SIP trunks) and web (browser) calls — a web call dials the human on PSTN from the agent's bound number and bridges the browser audio to it (assign the agent a number on the Numbers page). Tata Tele Smartflo is the exception: a transfer on a live phone call is refused, because Tata offers no way to redirect or bridge a call already on a voice-streaming leg. A transfer from a web call whose agent is bound to a Smartflo number does work — there Telenow places a fresh outbound call to the human rather than redirecting an existing one.

Keep the sessionId handy: you get it back from init-web-call / initiate-call when the call starts. Pass it into your endpoint's payload so your server has it when it's time to transfer.

Set it up from the dashboard

You don't need the API to use either mode:

1. Open Agents in the sidebar → open (or create) an agent → the Brain step in the agent builder.
2. Pick Custom LLM or Custom API as the provider.
3. Custom LLM: enter your Base URL (the part before /chat/completions), an optional API key, and the Model name your endpoint expects. Custom API: enter your API endpoint URL.
4. Configure STT, TTS, and a voice as normal — those always run on Telenow.
5. Save, then use the Test panel (or the Publish tab) to try it.

For Custom API, pass per‑session context (bearer, payload, identifier) when you start the call via init-web-call / initiate-call.

Which one?

Troubleshooting