Operations

Runbooks

Health checks

Check How

Stream consumer keeps up

IteratorAge of WorkerFunction stays at a few seconds. A growing value means the function is failing whole batches or is too slow.

Rules run

New items in em_event_rule_logs_v2 for recent events; /event_rules_history/event_rule/<uuid>/ for one rule

Watchdog runs

Invocations of the WorkerFunctionWatchdogSchedule EventBridge rule once per period, FailedInvocations at 0, and no Sentry errors tagged trigger:watchdog

CRM events arrive

NumberOfNotificationsFailed of em_events_create_event stays at 0 (the topic carries about 8 messages a day), and no Sentry errors tagged trigger:crm_event

API healthy

5XXError and Latency of RulesApi; an authenticated /constants/statuses answers 200

Errors

Sentry, app:eventlogger-new-rules, filtered by component (api, worker) and trigger

Useful commands

# Stack outputs: API id and URL, function ARNs
aws cloudformation describe-stacks --stack-name eventlogger-rules-prod --query 'Stacks[0].Outputs' --profile prod

# Follow a function's logs
sam logs --stack-name eventlogger-rules-prod --name WorkerFunction --tail --profile prod

# Kinesis mapping state and start position
aws lambda list-event-source-mappings --function-name <WorkerFunctionArn> \
  --query 'EventSourceMappings[].[UUID,State,StartingPosition]' --profile prod

# Consumer lag over the last 15 minutes (macOS date syntax)
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name IteratorAge \
  --dimensions Name=FunctionName,Value=<WorkerFunctionName> --statistics Maximum --period 60 \
  --start-time "$(date -u -v-15M +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" --profile prod

Use dev and eventlogger-rules-dev for staging.