Operations and runbooks

Where to look first

Symptom First place to look

A pipeline stage produced nothing

The function’s CloudWatch log group; every handler logs a summary line on completion (… completed updated=N skipped=N total=N).

Messages disappearing

The relevant dead-letter queue. maxReceiveCount is 1 everywhere, so one handler failure sends the message straight to the DLQ.

A BigQuery-backed handler is a no-op

BIGQUERY_CREDENTIALS missing or empty in the secret — every such handler logs BigQuery credentials not configured — skipping … and returns.

Contacts not appearing on call lists

Daily call lists — the suppression rules. Then confirm Hubspot.SalesRepCallListRefresh ran.

Enrichment is silently skipped

The seller already has enrichedAt set, or the AI step failed. Enrichment is best-effort: the failure is logged at ERROR and the message is still acked.

CRM changes not landing in HubSpot

morpheus-sam-hubspot-sync logs, then CrmSyncDlq. Subscription mismatch and empty-properties events are dropped by design and logged at ERROR.

Log groups

Component Log group

enrichment-scraper

/aws/lambda/morpheus-sam-enrichment-scraper

enrichment-scorer

/aws/lambda/morpheus-sam-enrichment-scorer

morpheus-hubspot

/aws/lambda/morpheus-sam-hubspot-sync

email-agent

/aws/lambda/morpheus-sam-email-agent

phone-agent

/aws/lambda/morpheus-sam-phone-agent

AgentCore runtime

/aws/bedrock-agentcore/agents/agentcore

Every Lambda calls configure_logging(level, aws_request_id, event_type) at the start of an invocation, so each line carries the request id and the routing key. Log level is the LogLevel stack parameter (default INFO); DEBUG adds per-record detail.

aws logs tail /aws/lambda/morpheus-sam-hubspot-sync --follow --since 30m
aws logs tail /aws/lambda/morpheus-sam-hubspot-sync \
  --filter-pattern '"Hubspot.LostDealReEngagement"' --since 1d

Queues

Queue Physical name FIFO

Enrichment pipeline

morpheus-sam-enrichment-pipeline.fifo

yes

Enrichment pipeline DLQ

morpheus-sam-enrichment-pipeline-dlq.fifo

yes

CRM sync

morpheus-sam-crm-sync.fifo

yes

CRM sync DLQ

morpheus-sam-crm-sync-dlq.fifo

yes

Email agent

morpheus-sam-email-agent

no

Email agent DLQ

morpheus-sam-email-agent-dlq

no

Depth check:

aws sqs get-queue-attributes --queue-url <url> \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

Inspecting a DLQ message

aws sqs receive-message --queue-url <dlq-url> \
  --max-number-of-messages 1 --visibility-timeout 0 \
  --message-attribute-names All --query 'Messages[0].Body' --output text

Visibility timeout 0 leaves the message in place. Read the event_type in the body (or the event_type message attribute for the enrichment and email queues) and cross-reference the event catalog.

Replaying a DLQ

DLQ retention is 14 days. Use an SQS redrive:

aws sqs start-message-move-task \
  --source-arn <dlq-arn> \
  --destination-arn <source-queue-arn> \
  --max-number-of-messages-per-second 5

Fix the cause before replaying. With maxReceiveCount: 1 a replayed message that still fails lands back in the DLQ immediately, and FIFO ordering within a message group is not preserved by a redrive.

Kill switches

Amazon seller discovery

Set configs.key = 'enrichment.amazon.disabled' to 'true'. The handler checks this first and returns before touching SmartScout. Any other value re-enables it.

Any scheduled handler

Disable its EventBridge rule:

aws events disable-rule --name morpheus-sam-lost-deal-reengagement
A whole SQS consumer

Disable the event-source mapping:

aws lambda list-event-source-mappings --function-name <fn> --query 'EventSourceMappings[].UUID'
aws lambda update-event-source-mapping --uuid <uuid> --no-enabled

Messages accumulate on the queue for up to four days (MessageRetentionPeriod 345600 s) rather than being lost.

Slack notifications

Clear SLACK_BOT_TOKEN in the secret. phone-agent logs a warning and skips both reminder phases.

All BigQuery segmentation

Clear BIGQUERY_CREDENTIALS. Every BigQuery-backed handler becomes a no-op.

Watermarks and cursors

Incremental syncs keep their position in the configs table rather than in memory. To re-run a window, move the watermark backwards.

configs.key Owner Effect of resetting

bigquery_engagement_sync_watermark

Hubspot.BigQueryEngagementSync

Re-reads HubSpot engagements from that timestamp. Writes are idempotent upserts on (subscriptionId, externalId), so replay is safe.

bq_contact_sync_watermark

Scraper.BigQueryContactSync

Re-reads BigQuery contacts. Contact upserts are find-or-create; lifecycle-change Activity notes may be written again.

five9_call_sync_watermark

Hubspot.Five9CallSync

Re-processes Files.com CSVs modified after that time. Call upserts key on providerCallId, but HubSpot call engagements would be created again — reset with care.

enrichment.amazon.next_start_row

SellerDiscovery.Amazon

Sets the SmartScout row offset for the next run. 0 restarts discovery from the top.

dialer_follow_up_rep_rr:<subscriptionId>

Round-robin assignment

Resets which rep is next. Harmless.

 
-- inspect
SELECT key, value, "updatedAt" FROM configs WHERE key LIKE '%watermark%';

-- rewind one sync
UPDATE configs SET value = '2026-08-01T00:00:00+00:00'
WHERE key = 'bigquery_engagement_sync_watermark';

The contact sync has a guard against stalling: if a full batch returns without advancing the maximum updatedat, it bumps the watermark by one second and logs a warning, so a cluster of identical timestamps cannot loop forever.

Alarms

BedrockTokenAlarm (morpheus-sam-bedrock-daily-tokens-high) watches AWS/Bedrock InputTokenCount, summed hourly, and fires above 5,000,000 tokens per hour to the SNS topic security-operational-issues. Missing data is treated as not breaching. Tune the threshold in lambdas/template.yaml.

Application Insights is enabled over the SAM stack’s resource group.

Sentry is wired but disabled: SENTRY_DSN is empty in Globals. Populating it turns on unhandled-exception capture plus ERROR-level log forwarding, with send_default_pii=False. See Observability.

Recurring operational notes

Duplicated work is cheap, missed work is not

Every handler is written to be safely re-runnable: upserts key on natural unique columns, "already called today" suppression is recomputed each run, and enrichment/scoring skip rows that already have enrichedAt / payoutPainScore.

Schedules currently run at test cadence

Several rules use rate(…​) values that were set for development (for example the re-engagement segments at rate(6 hours) and the Five9 sync’s commented-out rate(4 minutes)). Confirm the intended production cadence in EventBridge schedules before assuming a handler runs on the cadence its description claims.

The scraper writes to Secrets Manager

enrichment-scraper caches its SmartScout session token back into the shared secret. That is why it holds secretsmanager:PutSecretValue and why a manual put-secret-value that omits SMARTSCOUT_TOKEN simply forces one extra authentication.

Never log payloads

The convention is to log identifiers and counts, not bodies, because messages routinely carry contact PII. One known deviation: the morpheus-hubspot dispatcher includes the full message body in its "Handler failed" ERROR line, so that log group can contain PII after a failure. Preserve the convention when adding diagnostics. See Security and data handling.