Zappa to SAM cutover

This runbook moves the new-rules app from the Zappa deployments (Python 3.8) to the SAM stacks.

Old (Zappa) New (SAM)

	Old (Zappa)	New (SAM)
Staging (605668041275)	`eventloggerapp-test-rules2`	`eventlogger-rules-dev`, from `develop_new_rules`
Production (625663093496)	`eventloggerapp-prod-rules2`, API `ny63bqmiol` stage `prod_rules2`	`eventlogger-rules-prod`, from `new-rules-P312`

Staging (605668041275)

eventloggerapp-test-rules2

eventlogger-rules-dev, from develop_new_rules

Production (625663093496)

eventloggerapp-prod-rules2, API ny63bqmiol stage prod_rules2

eventlogger-rules-prod, from new-rules-P312

Both deployments consume the same sources:

Source Zappa trigger New stack resource

Source	Zappa trigger	New stack resource
Kinesis `em_events[_staging]`	event source mapping on the Zappa function	`WorkerKinesisMapping`
SNS `em_events_create_event[_staging]`	subscription to the Zappa function	`WorkerCrmSubscription`
Watchdog schedule	EventBridge rule `…-eventlogger_app.handlers.watchdog.handler`	`WorkerFunction` `WatchdogSchedule`
`api.payability.com` `eventlogger_v2`, `track`, `tracking`	base path mappings to `ny63bqmiol`	`RulesApi`, switched by hand

Kinesis em_events[_staging]

event source mapping on the Zappa function

WorkerKinesisMapping

SNS em_events_create_event[_staging]

subscription to the Zappa function

WorkerCrmSubscription

Watchdog schedule

EventBridge rule …-eventlogger_app.handlers.watchdog.handler

WorkerFunction WatchdogSchedule

api.payability.com eventlogger_v2, track, tracking

base path mappings to ny63bqmiol

RulesApi, switched by hand

The old and new deployments must never consume the same source at the same time: every rule action (emails, Slack, tasks, supplier edits) would run twice.

flowchart TB
  subgraph prep ["0. Preparation"]
    P1["Secrets in common/crm"] --> P2["CodeBuild webhook filter"] --> P3["Push branches"]
  end
  subgraph dev ["1. Staging"]
    D1["Stop test_rules2 triggers"] --> D2["Push develop_new_rules"] --> D3["Verify"]
  end
  subgraph dark ["2. Production, dark"]
    K1["Merge to new-rules-P312, EnableTriggers=false"] --> K2["Smoke-test execute-api URL"]
  end
  subgraph cut ["3. Production cutover"]
    C1["Stop Zappa Kinesis mapping, note T"] --> C2["Stop Zappa watchdog"] --> C3["Merge PR: EnableTriggers=true, AT_TIMESTAMP=T"]
    C3 --> C4["Unsubscribe Zappa from CRM topic"] --> C5["Switch base path mappings"] --> C6["Verify"]
  end
  prep --> dev --> dark --> cut

All commands use the AWS CLI profiles dev (605668041275) and prod (625663093496). The IDs below were read from the accounts on 2026-11-10; check them again before running anything.

0. Preparation

Secrets

common/crm already has AUTH_JWT_RSA_PUBLIC_KEY, INTERNAL_JWT, 8001 and SEGMENT_WRITE_KEY in both accounts, which is everything the app reads. The cutover needs no new secret keys.

Deploy credentials and tokens

DEV_AWS_* and PROD_AWS_* need the permissions listed in Build and deployment. _deploy.yml stops if the keys belong to the wrong account.
PYB_GITHUB_TOKEN must read the three private libraries and should not be a personal token that expires.

Legacy CodeBuild webhook

The production CodeBuild project EventLoggerApp builds every push and merged pull request using the pushed commit’s buildspec.yml. The new branches have none, so each push to them would start a failing build. The webhook has one filter group today (EVENT = PUSH, PULL_REQUEST_MERGED). This keeps it and excludes the new branches:

aws codebuild update-webhook --profile prod --region us-east-1 --project-name EventLoggerApp \
  --filter-groups '[
    [{"type": "EVENT", "pattern": "PUSH"},
     {"type": "HEAD_REF", "pattern": "^refs/heads/(develop_new_rules|new-rules-P312)$", "excludeMatchedPattern": true}],
    [{"type": "EVENT", "pattern": "PULL_REQUEST_MERGED"},
     {"type": "BASE_REF", "pattern": "^refs/heads/(develop_new_rules|new-rules-P312)$", "excludeMatchedPattern": true}]
  ]'

aws codebuild batch-get-projects --profile prod --region us-east-1 --names EventLoggerApp \
  --query 'projects[0].webhook.filterGroups'

Feature branches cut from develop_new_rules still trigger it, as every other branch does today. To undo:

aws codebuild update-webhook --profile prod --region us-east-1 --project-name EventLoggerApp \
  --filter-groups '[[{"type": "EVENT", "pattern": "PUSH, PULL_REQUEST_MERGED"}]]'

Branches

develop_new_rules deploys to staging and new-rules-P312 to production. Branch protection and the dev and prod GitHub environments are set up by a repository admin; new-rules-P312 should only change through reviewed pull requests.

1. Staging

[dev] in samconfig.toml deploys with EnableTriggers=true, so the Zappa staging triggers go first.

Stop the Zappa test_rules2 triggers:

export AWS_PROFILE=dev AWS_REGION=us-east-1
aws lambda update-event-source-mapping --uuid 41bcd957-2d81-450b-a653-7885b3c8e4f7 --no-enabled
aws sns unsubscribe --subscription-arn arn:aws:sns:us-east-1:605668041275:em_events_create_event_staging:ac1e4dbd-c507-4026-a960-f0a4d7296dae
aws events disable-rule --name 5272f589a2e03ec2932be4-eventlogger_app.handlers.watchdog.handler
aws events disable-rule --name 5272f589a2e03ec2932be-zappa-keep-warm-handler.keep_warm_callback

em_events_create_event_staging is also subscribed by the old-rules dev stack (eventlogger-app-dev-new-EventLoggerFunction-59Up7u3Sq1Fb), so staging CRM events are already captured twice today. That subscription is outside this migration.

Push develop_new_rules. The Deploy workflow runs CI, then make deploy ENV=dev, and lists the stack outputs in the job summary.

Check the triggers and the API. Staging has no API_BASE_PREFIX, so routes sit directly under the stage:

aws cloudformation describe-stacks --stack-name eventlogger-rules-dev --query 'Stacks[0].Outputs'
aws lambda list-event-source-mappings --function-name <WorkerFunctionArn> --query 'EventSourceMappings[].State'

API=https://<RulesApiId>.execute-api.us-east-1.amazonaws.com/stage
curl -s -H "X-Session-Token: $TOKEN" -H "X-InstanceId: 1" "$API/constants/statuses"
curl -s -H "X-Session-Token: $TOKEN" -H "X-InstanceId: 1" "$API/event_rules/?limit=5"

Compare with what test_rules2 did: rule executions in em_event_rule_logs_v2_staging, a watchdog run every 2 minutes, stream processing and CRM events (sam logs --stack-name eventlogger-rules-dev --name WorkerFunction --profile dev).

To go back in staging, set EnableTriggers=false in [dev], re-enable the mapping and rules from step 1, and subscribe the Zappa function again:

aws sns subscribe --topic-arn arn:aws:sns:us-east-1:605668041275:em_events_create_event_staging \
  --protocol lambda --notification-endpoint arn:aws:lambda:us-east-1:605668041275:function:eventloggerapp-test-rules2

2. Production, dark

Open a pull request from develop_new_rules to new-rules-P312 and merge it after review. [prod] keeps EnableTriggers=false: the stack has no Kinesis mapping or CRM subscription, and the watchdog schedule is DISABLED.

Confirm nothing consumes events yet:

export AWS_PROFILE=prod AWS_REGION=us-east-1
aws cloudformation describe-stacks --stack-name eventlogger-rules-prod --query 'Stacks[0].Outputs'
aws lambda list-event-source-mappings --function-name <WorkerFunctionArn>   # empty
aws sns list-subscriptions-by-topic --topic-arn arn:aws:sns:us-east-1:625663093496:em_events_create_event \
  --query 'Subscriptions[].Endpoint'                                                 # no eventlogger-rules-prod function

Smoke-test RulesApi on its execute-api URL with read-only calls. Production routes are under /eventlogger_v2:

API=https://<RulesApiId>.execute-api.us-east-1.amazonaws.com/prod
curl -s -H "X-Session-Token: $TOKEN" -H "X-InstanceId: 1" "$API/eventlogger_v2/constants/rules"
curl -s -H "X-Session-Token: $TOKEN" -H "X-InstanceId: 1" "$API/eventlogger_v2/event_rules/?limit=5"
curl -s -H "X-Session-Token: $TOKEN" -H "X-InstanceId: 1" "$API/eventlogger_v2/action_templates/"

Don’t call /track, /capture-message, /simulate/rules with real rules, or /trigger/rules: they publish real events or run real actions.

3. Production cutover

Pick a quiet window. The em_events stream keeps 168 hours of data and the CRM topic carries about 8 messages a day, so there’s no time pressure between the steps.

Prepare a pull request to new-rules-P312 in advance that changes:

samconfig.toml, [prod.deploy.parameters]: EnableTriggers=true
template.yaml, EnvironmentSettings.prod: KinesisStartingPosition: AT_TIMESTAMP and KinesisStartingTimestamp: "<T>", filled in at step 2

Then:

Check that Zappa is caught up. IteratorAge of eventloggerapp-prod-rules2 should be a few seconds (it peaked at about 12 s over a day):

export AWS_PROFILE=prod AWS_REGION=us-east-1
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name IteratorAge \
  --dimensions Name=FunctionName,Value=eventloggerapp-prod-rules2 --statistics Maximum --period 60 \
  --start-time "$(date -u -v-15M +%FT%TZ)" --end-time "$(date -u +%FT%TZ)"

Stop the Zappa Kinesis consumer and note the time.

T=$(date -u +%s); echo "T=$T"
aws lambda update-event-source-mapping --uuid 030cc522-988e-4110-a1ef-1229b94b99b6 --no-enabled
aws lambda get-event-source-mapping --uuid 030cc522-988e-4110-a1ef-1229b94b99b6 --query State   # wait for Disabled

The new mapping starts with records that arrived at T or later. A batch Zappa was processing when it was disabled can overlap by a few seconds; a later T would skip events instead.

Stop the Zappa watchdog and keep-warm rules.

aws events disable-rule --name 502c129d7cbb7e4f395e07-eventlogger_app.handlers.watchdog.handler
aws events disable-rule --name 502c129d7cbb7e4f395e0-zappa-keep-warm-handler.keep_warm_callback

Merge the prepared pull request with KinesisStartingTimestamp: "<T>". The Deploy workflow creates the Kinesis mapping at T and the CRM subscription, and enables the watchdog schedule.
Move the CRM subscription off Zappa as soon as the deploy finishes. Unsubscribing only once the new subscription exists means a CRM event can at worst be captured twice in that short window, instead of being lost: SNS doesn’t keep undelivered messages.
```
aws sns unsubscribe --subscription-arn arn:aws:sns:us-east-1:625663093496:em_events_create_event:d1300b55-595d-4a88-90d5-9548a4830043
```

Check the new consumers. The mapping is Enabled and its IteratorAge drops to seconds as it catches up from T, new items appear in em_event_rule_logs_v2, and the watchdog runs every 15 minutes:

aws lambda list-event-source-mappings --function-name <WorkerFunctionArn> \
  --query 'EventSourceMappings[].[State,StartingPosition]'
sam logs --stack-name eventlogger-rules-prod --name WorkerFunction --profile prod --tail

Switch the API. Point the three api.payability.com base path mappings at RulesApi, stage prod:

API_ID=<RulesApiId output>
for bp in eventlogger_v2 track tracking; do
  aws apigateway update-base-path-mapping --domain-name api.payability.com --base-path "$bp" \
    --patch-operations op=replace,path=/restapiId,value="$API_ID" op=replace,path=/stage,value=prod
done
aws apigateway get-base-path-mappings --domain-name api.payability.com \
  --query "items[?basePath=='eventlogger_v2' || basePath=='track' || basePath=='tracking']"

Verify through the custom domain. Repeat the smoke test GETs against https://api.payability.com/eventlogger_v2/…;, and check that /track and /tracking/… clients get 2xx responses (sam logs … --name ApiFunction).

After the cutover, leave the Kinesis start position alone: see Kinesis start position. If anything goes wrong, follow Rollback.