AI agents and test cases

An AiAgent is a named LLM configuration owned by a workspace. AiAgentVersion rows make every prompt change auditable, and AiAgent.latestVersionId points at the published one.

Agent types

type Role

caller

Speaks on outbound calls. Mirrored to Retell AI via retellAgentId; AiAgentVersion.configJson holds the full Retell agent config

analysis

Grades completed calls. Its mainPrompt supplies the grading criteria

sms

SMS conversations

other

Anything else

AiAgent.subscriptionId is nullable, allowing a shared agent across workspaces. AiAgent.retellAgentId is globally unique, so no two records can claim the same Retell agent.

Versioning

Field Meaning

version

1-based, unique per (aiAgentId, version)

mainPrompt

The script (caller) or the grading criteria (analysis)

configJson

Full Retell IAgentData for callers; model overrides and supplementary config for analysis agents

modelId

Vercel AI Gateway slug on the parent AiAgent, e.g. openai/gpt-5

llmModelType

Legacy gpt | claude fallback when modelId is unset

Model resolution is getModelSlug(modelId, llmModelType) in lib/ai/config.ts: an explicit modelId wins, otherwise gpt → openai/gpt-4o and claude → anthropic/claude-sonnet-4. Selectable models are enumerated in AI_GATEWAY_MODELS across OpenAI, Anthropic, xAI and Google.

Versions are never mutated in place. Call.agentVersionId records which version spoke, and CallAiAgentAnalysisResult.aiAgentVersionId records which version graded — this is what makes the per-agent performance report meaningful.

Caller agent selection at dial time

PhoneNumberCallingService.pickOverrideAgent(subscriptionId):

  1. Query active caller-type agents for the workspace (AiAgentRepository.findCallerAgentsBySubscription) and pick one at random. Random selection spreads load and voice variety across the pool rather than favouring one agent.

  2. If the workspace has none, fall back to the static RETELL_AGENT_LIST in scripts/retelai/agents/registry.ts — the single source of the fallback id/name list.

The chosen Retell agent id is sent as override_agent_id on the call. Call.aiAgentId and Call.aiAgentName are denormalised onto the call row so the Calls table can render "Brian v3" without a join. Call.agentId is the AiAgent UUID; do not confuse it with aiAgentId, which is the Retell identifier.

Retell synchronisation

RetellAgentService (lib/services/retell-agent-service.ts) wraps the Retell platform:

Method Purpose

createAgent, getAgent, updateAgent, deleteAgent

Agent CRUD

createAgentVersion(retellAgentId, baseVersion)

Branch a new Retell version from an existing one

publishAgent(retellAgentId, version)

Make a version live

createLLM, getLLM, updateLLM

The response engine behind an agent

syncAgent(…​)

Reconcile a local AiAgentVersion with Retell

Every create, update and delete through /api/ai-agents is mirrored to Retell, so the database and the platform do not drift.

Voice and behaviour configuration

lib/retellai/agent-config.ts holds the UI’s option lists and the non-negotiable defaults:

  • RETELL_VOICE_MODELS — the available voice models.

  • RETELL_LLM_MODELS — response-engine models offered by Retell.

  • RETELL_EXPRESSIVE_EMOTION_TAGS — emotion tags for platform voices only, offered as a multi-select.

  • HARDCODED_AGENT_CONFIG — applied to every caller agent:

Setting Value Why

stt_mode

fast

Latency matters more than transcript perfection on a cold call

end_call_after_silence_ms

25000

Hang up rather than burning minutes on dead air

max_call_duration_ms

80000

Hard ceiling on cost per call

ring_duration_ms

30000

Stop ringing before most voicemail systems answer

begin_message_delay_ms

1000

Wait for the callee to finish saying "hello"

allow_user_dtmf

true

IVR navigation, with a 5 s keypress timeout

is_public

false

Agents are never publicly listed

GET /api/retell/voices proxies Retell’s voice catalogue for the agent editor.

Agent definitions in the repository

scripts/retelai/agents/<name>/ holds each agent’s agent.md (the prompt, in version control) and index.ts (its config). scripts/retelai/sync.ts pushes these definitions to Retell. Agents currently defined: adrian, brian, cmon, kate, kathrine.

Post-call analysis schema

DEFAULT_POST_CALL_ANALYSIS in lib/retellai/post-call-analysis.ts defines the structured fields Retell extracts from each call — the shared vocabulary between the caller agent, the Retell platform and the analysis prompt. Per-agent overrides are merged by mergePostCallAnalysis, so an agent can add or reshape fields without forking the base schema.

Regression test cases

AgentTestCase captures a real conversation plus the correct answer, so a prompt change can be regression-tested before it goes live.

Field group Fields

Identity

name, notes, sourceCallId (null for hand-written cases)

Inputs

transcript, recordingUrl

Ground truth

disposition, isAmazonSeller, aiSuspicionDetected, callScreeningDetected, isDecisionMaker, userSentiment, nextAction, callSuccessful, inVoicemail

Running a suite

POST /api/test-cases/run invokes TestCaseRunnerService:

  1. Skip any case with no transcript — there is nothing to evaluate (skipped: true).

  2. Ask the model to predict the outcome from the transcript, given the caller agent version’s prompt and the formatted post-call analysis schema.

  3. Parse the response with PredictionSchema, tolerating a fenced ```json block.

  4. Compare field by field. A ground truth of null means "not asserted", producing pass: null — that field is excluded from the verdict rather than counted as a pass.

  5. passed is true only when every evaluated field passes.

The evaluated set is the intersection of DEFAULT_POST_CALL_ANALYSIS fields and test-case ground-truth columns: disposition, next_action, call_screening_detected, is_amazon_seller. String comparison is case-insensitive; booleans are compared strictly.

POST /api/test-cases/suggest-prompt proposes a prompt revision from failing cases.

Prompt suggestions from live issues

/agents/issues surfaces the issues arrays produced by call analysis, filterable by name, priority, agentName, agentVersion and date range. Selecting an issue and calling POST /api/ai-agents/issues/suggest-prompt runs PromptSuggestionService, which proposes a revision addressing it. The loop is: analyse calls → aggregate issues → suggest a prompt → regression-test it against saved cases → publish a new version.

Duplication

POST /api/ai-agents/{id}/duplicate copies an agent and its latest version, giving operators a safe way to experiment without editing a production agent.