Runbooks

Launch a call campaign

  1. Create the campaign (draft) and set maxConcurrentCalls, maxAttempts, waitHoursToRecall, windowStart/windowEnd, allowWeekends and preferredAiAgentId. Set maxAttempts explicitly — its fallback differs between code paths.

  2. Import targets. Timezones are derived from area codes; a HubSpot contact id may be supplied in the payload as Record ID, record_id, RecordId or hubspot_id.

  3. Check coverage — GET /api/campaigns/{id}/coverage. States with targets but no local source number will fall back to a non-local number, which lowers pickup rates.

  4. Confirm the workspace has active caller-type AI agents; otherwise the static RETELL_AGENT_LIST fallback is used.

  5. Set status = running.

  6. Create the pg_cron job for POST /api/campaigns/{id}/broadcast with the internal secret.

  7. Watch the first ticks: callsInitiated, callsSkipped and any throttle message in the broadcast response, plus callTarget runs in Inngest.

Verify the Redis counter is falling back to zero between ticks. If it only rises, callTarget is not reaching its finish() helper and the campaign will throttle itself into silence.

Pause or stop a campaign

Set status to paused or stopped. processBroadcast throws CAMPAIGN_NOT_RUNNING for anything other than running, so the next tick stops enqueueing.

Already-enqueued Inngest events still run — pausing does not recall in-flight calls. To stop those, disable the pg_cron job as well and let the queue drain.

Launch an email campaign

  1. Verify the domain in Resend, then create the Domain row with its resendDomainId from GET /api/email-domains/resend-domains.

  2. Add at least one active, verified EmailSendingDomain under it, with a dailyCap.

  3. Create the campaign with schedulerIntervalSeconds, emailsPerHour, maxEmailsPerDay and emailsPerVariationPerHour.

  4. Add at least one active variation. A campaign with no active variations halts with throttled: false — the condition never clears on its own.

  5. Import recipients. Contacts are upserted first, so every target gets a contactId and an emailId.

  6. Send a test through POST /api/email-campaigns/{id}/send and confirm rendering, the unsubscribe link and the Reply-To.

  7. Set status = running and create the pg_cron job.

  8. Watch the broadcast response: emailsEnqueued, throttled, throttleReason, variationGenerationRequested.

throttled: true clears on its own. throttled: false with zero enqueued means an operator has to act.

Warm a sending domain

  1. Create the Domain with warmupStatus = warming_up and a low dailyCap.

  2. Add one account with a low dailyCap.

  3. Raise both caps gradually over days, watching bouncedCount and unsubscribedCount per account.

  4. Set warmupStatus = active once volume is steady. The status is a label — it does not gate sending, so the caps are what actually protect the domain.

A single bad account damages the reputation of its whole parent domain, which is why the parent cap exists as well.

Suppress a recipient

Goal Action

Stop all marketing email to an address

Set newsletterOptOut on its TargetEmails row

Block an address entirely

Set isBlocked — treated as suppressed everywhere

Stop calling a number

Set isBlocked on its TargetPhoneNumbers row

Stop calling a contact

Set Contact.stopCalling = true

Registry-level blocks apply across every campaign, present and future. Marking a single campaign target canceled only affects that campaign.

A DNC outcome does the registry block automatically — callEndedAnalysis sets TargetPhoneNumbers.isBlocked = true.

Publish a new agent prompt

  1. Duplicate the agent (POST /api/ai-agents/{id}/duplicate) if you want to experiment without touching production.

  2. Review /agents/issues for recurring problems, and use POST /api/ai-agents/issues/suggest-prompt for a starting revision.

  3. Run the regression suite — POST /api/test-cases/run against the candidate version. passed is true only when every asserted field matches.

  4. Publish a new version (POST /api/ai-agents/{id}/versions). Versions are immutable and the change is mirrored to Retell.

  5. Watch /reports/daily-agent-performance — avg_score, correct_disposition_pct and total_critical_issues are per agent version, so the new prompt is directly comparable to the old one.

Re-analyse calls after a prompt change

  1. Open the re-analyse dialog on /calls or a campaign’s Calls tab.

  2. Set filters — campaign, status, date range, HubSpot sync status.

  3. Check the preview count before starting; it uses the same where builder as the run, so it cannot disagree.

  4. Start it. Results land as new CallAiAgentAnalysisResult rows keyed by (callId, aiAgentId, aiAgentVersionId), so the old verdicts are preserved for comparison.

Nooks calls have campaignId = null and are excluded from batch runs — re-run those individually.

Rotate a secret

  1. Update the value in Vercel for the target environment.

  2. Redeploy — module-load reads mean a running instance keeps the old value.

  3. For INTERNAL_API_SECRET, update every pg_cron job definition in Supabase in the same window. Broadcasts return 401 until both sides match.

  4. For provider webhook secrets, update the provider’s dashboard first, then the environment. A brief window of skipped or failed verification is preferable to a window of dropped events.

Add a carrier

  1. Add an entry to EXTERNAL_PROVIDER_CONFIGS in lib/constants.ts with label, source, trunkSid and terminationUrl.

  2. Add the corresponding environment variables.

  3. Deploy. PhoneNumberCallingService, MessagingAppPhoneService, the purchase route and the Phone Numbers page all read the registry dynamically — no other change is needed.

Release unused numbers

The Inngest function phoneNumberOptimizerCron runs weekly (Monday 11:15 UTC) and releases numbers unused for 3+ days with fewer than 10 lifetime calls. To run it now, open it in the Inngest dashboard and choose Invoke. Each release is its own step, so the run page shows which numbers were released and which failed.

Check /reports/phone-number-pickup-rate first — a number with poor pickup is often better rested than released, since a release means buying a new one later.

Replay a webhook

select "eventType", "rawPayload"
from "WebhookEvent"
where source = 'Retell' and "providerCallId" = '<call_id>'
order by "receivedAt" desc;

curl the payload back at the endpoint. The WebhookEvent row is idempotent on (source, eventId), but the handler runs again — which is the point. Downstream, Call.providerCallId uniqueness and the emailSend status guard absorb the duplicate side effects.