Troubleshooting

No calls are being made

Work down the chain; each step rules out the one above.

# Check If wrong

1

Campaign.status is running

Anything else and processBroadcast throws CAMPAIGN_NOT_RUNNING

2

The pg_cron job exists and is enabled in Supabase

No tick, no dials

3

The broadcast returns 200, not 401

INTERNAL_API_SECRET mismatch between Vercel and the job definition

4

The response’s message field

"Throttled: N/M in-flight runs" means the Redis counter is at the ceiling

5

The Redis counter falls back toward zero between ticks

If it only rises, callTarget is not reaching finish() and capacity is leaking

6

Targets are due — state = pending (or in_progress) with nextEarliestAttemptAt in the past and leaseExpiresAt expired

Everything may be inside its 24 h cooling-off window

7

callTarget runs appear in Inngest

If not, INNGEST_EVENT_KEY / INNGEST_SIGNING_KEY, or the function is missing from the serve() array

8

The check-call-window step result

outside_call_window means the current time is outside the window in the target’s timezone

9

The Inngest rate limit

2 runs per targetPhoneNumber per 24 h — excess events are dropped, not queued

A leaking Redis counter is the classic slow failure: dialing gradually reduces and then stops, with no error anywhere. Compare getCallTargetRuns(campaignId) against maxConcurrentCalls × 2.

Targets are being skipped

Read CampaignTarget.lastError and the workflow’s skipReason.

skipReason / lastError Cause

dnc

The number is on the DNC list

prior_disposition

A previous call to this number had Do_Not_Call, Not_a_Seller or Amazon_Seller

lifecycle_stage

The contact is a customer or marketingqualifiedlead

invalid_e164

The number does not match /^\+[1-9]\d{1,14}$/

outside_call_window

Outside the window in the target’s timezone

retell_error

Retell rejected the call; the target is failed with the message in lastError

Target phone number is blocked

The registry row has isBlocked = true

Target phone risk level (medium) must be below medium

Registry riskLevel is medium or worse

Target phone registry subscription mismatch

The registry row belongs to another workspace

Targets with attempts = 9999 were permanently disqualified by pre-validation and will not return to the pool even if their state is reset.

No emails are being sent

# Check If wrong

1

Campaign.status is running and type is email

The broadcast returns throttled: true with a status message

2

The pg_cron job exists

No tick, no sends

3

The broadcast returns 200, not 401

INTERNAL_API_SECRET

4

throttled and throttleReason in the response

true clears on its own; false needs an operator

5

At least one active variation exists

"No active email variations found" — throttled: false

6

At least one active sending account exists under an active domain

"No active email accounts on an active domain" — throttled: false

7

Accounts are under both their own and the parent domain’s dailyCap, counting only sends whose dailySentDate is today (UTC)

"All email accounts over daily cap" — clears at 00:00 UTC, even if the reset job does not run

8

countSentToday against maxEmailsPerDay

Daily cap reached

9

Pending targets exist whose target_emails row is not suppressed, is past its cadence gap, and is not leased (queuedAt null or older than 2 days)

"No pending recipients available" — see Pending targets hidden behind leases if pending targets exist but are all leased

10

emailSend runs appear in Inngest

Check the skip reason on each

emailSend skip reasons:

Reason Meaning

already_sent

The target has left pending — a duplicate event, or a retry after success

unsubscribed

newsletterOptOut, marketingOptOut or isBlocked on the address

newsletter_gap

Fewer than newsletterGapDays (default 7) since the last marketing send to that address. The lock is released so it returns later

variation_rate_limited

Every active variation is over its hourly rate. The lock is released

daily_limit_reached

Campaign daily cap. The lock is released

not_found

Campaign or variation missing

newsletter_gap is the one that most often looks like a bug: the campaign is healthy, the recipients are eligible, but the address received a different campaign’s email within the last week.

Pending targets hidden behind leases

Each tick stamps queuedAt on the batch it leases, and the lease query skips rows leased within EMAIL_TARGET_LOCK_WINDOW_SECONDS (2 days). A tick that halts after leasing gives its leases back. Before that fix, a run throttled by the daily cap left the batch leased, so a few hours of throttled ticks could hide thousands of recipients for 2 days.

Count leased pending targets:

SELECT count(*) FILTER (WHERE "queuedAt" >= NOW() - INTERVAL '2 days') AS leased,
       count(*) FILTER (WHERE "queuedAt" IS NULL OR "queuedAt" < NOW() - INTERVAL '2 days') AS available
FROM email_campaign_targets
WHERE "campaignId" = '<campaign-id>' AND status = 'pending';

If almost every pending target is leased and the timestamps are hours old, release them. The 1-hour guard leaves leases from a live tick alone, and emailSend skips any target that has already left pending, so a duplicate event cannot send twice:

UPDATE email_campaign_targets
SET "queuedAt" = NULL
WHERE "campaignId" = '<campaign-id>'
  AND status = 'pending'
  AND "queuedAt" < NOW() - INTERVAL '1 hour';

No AI analysis is being produced

Check Detail

An analysis agent is configured

The campaign’s preferredAiAgentId, else a workspace default. Neither → "No analysis AI agent configured"

The agent’s type is analysis

Otherwise "Analysis agent not found or invalid"

AI_GATEWAY_API_KEY is set

Otherwise every run fails

The disconnection reason

Reasons in BLACKLISTED_AI_ANALYSIS_REASONS are skipped by design

Whether a double tap intervened

An eligible double tap returns before the analysis step

The run-ai-analysis step result

Transient errors (EBUSY, ENOTFOUND, Cannot connect to API, Insufficient funds) are recorded as { saved: false, error }

Insufficient funds means the AI Gateway account is out of credit — the error is treated as transient and silently swallowed, so it produces missing analyses rather than alerts.

A sequence is stuck

Check Detail

ContactSequence.status

pending was never started; paused was stopped manually

executionId

Null means no run ever claimed it. Send sequence/execute via the start endpoint

The Inngest run

Look it up by executionId. A sleeping run is normal — a wait step or a gate retry delay

ContactSequenceStep logs

Each row’s status, outcome and response show exactly what happened

Gate steps

gateRetryCount against gateMaxRetries. Exhausted retries with no falseBranchStepIndex abandon the sequence

Skipped actions

no_contact_email, no_contact_phone, no_sms_number, no_sales_rep, no_hubspot_contact, already_called, invalid_crm_event_config

BigQuery

A contact with no hubspotContactId fails every gate condition, by design

already_called is intentional: a call_task step creates at most one task per enrollment, so a template that loops back to a call step will skip it the second time.

CRM data is not appearing in HubSpot

  1. Confirm the contact has a hubspotContactId. Every CRM publisher skips without one, deliberately.

  2. Confirm CRM_QUEUE_URL and AWS credentials are set — publishers return false rather than throwing, so a missing queue is silent.

  3. Check for the publish in the logs.

  4. Confirm call eligibility: duration ≥ 10 s, not voicemail, not screened, disposition in HUBSPOT_WHITE_LISTED_DISPOSITIONS, and isTestTarget not true.

  5. Check the downstream consumer — the application only enqueues. Call.hubspotSyncStatus is set by the consumer, not by this application.

Webhooks are not arriving

  1. Check the WebhookEvent table. A row means it arrived and the problem is in the handler.

  2. No row → verify the provider’s configured URL, and that the path is under /api/webhooks/ (excluded from middleware).

  3. 401 from HubSpot → HUBSPOT_WEBHOOK_CLIENT_SECRET mismatch; that route rejects when unset.

  4. A row exists but nothing changed → the event type may be unregistered. Only call_analyzed is registered for Retell; call_started is stored and ignored. Five9 payloads are stored and never dispatched.

  5. A Retell row with no effect → check for campaignId and campaignTargetId in call.metadata. Without both, the handler logs and returns.

A report is empty or stale

  1. Views prefixed vm_ are materialised — check the refresh schedule before suspecting the route.

  2. Date filters are Eastern Time. A dateFrom of today returns nothing before 00:00 ET, which is 05:00 UTC.

  3. Views are managed in Supabase, not by Prisma migrations. A new environment needs them created by hand.

  4. phone_number_pickup_rate joins "PhoneNumbers" — the quoted PascalCase name. An unquoted reference fails.

Duplicate sends or calls

The guards, in order of what to check:

Guard Where

WebhookEvent unique on (source, eventId)

Duplicate deliveries

Call.providerCallId unique

Duplicate call rows

callTarget rate limit — 2 per number per 24 h

Repeat dialing

callEndedAnalysis rate limit — 1 per callId per 24 h

Repeat post-processing

CampaignTarget.leaseExpiresAt

Concurrent broadcast ticks

EmailCampaignTarget.queuedAt plus FOR UPDATE SKIP LOCKED

Concurrent email ticks

emailSend step 1 status check

Duplicate send events

(subscriptionId, externalId) unique on Activity and Emails

Duplicate logs

POST /api/webhooks/five9 is the known exception — it generates a random eventId, so replays create new rows.

Build and type errors

Symptom Fix

@prisma/client did not initialize

pnpm prisma generate

Prisma model types missing after a schema edit

pnpm prisma generate — the client lives in prisma/generated

Stale build after dependency changes

rm -rf .next && pnpm install && pnpm build

Missing types in a production build

Use pnpm build:prod, which generates the client first