Rollback

Rollback reverses the production cutover. Run the steps in order. Point the API back first; it has no ordering constraint. The two consumers are handled differently, to avoid losing events the way the cutover does:

  • CRM topic: resubscribe Zappa before the new subscription is removed. SNS doesn’t keep undelivered messages, so removing the new subscription first would drop every CRM event until Zappa’s is restored. The overlap mirrors the cutover: a CRM event can at worst be captured twice, never lost.

  • Kinesis stream: stop the new consumer before re-enabling Zappa’s, so the stream is never processed by both at once (that would run every rule action twice). Zappa resumes from its own checkpoint, so nothing is skipped.

    1. Point the API back to Zappa.

      export AWS_PROFILE=prod AWS_REGION=us-east-1
for bp in eventlogger_v2 track tracking; do
  aws apigateway update-base-path-mapping --domain-name api.payability.com --base-path "$bp" \
    --patch-operations op=replace,path=/restapiId,value=ny63bqmiol op=replace,path=/stage,value=prod_rules2
done

    2. Resubscribe Zappa to the CRM topic first, before the new stack’s subscription is removed, so no CRM event is dropped in between. Both subscriptions are briefly live, so a CRM event can at worst be captured twice.

      aws sns subscribe --topic-arn arn:aws:sns:us-east-1:625663093496:em_events_create_event \
  --protocol lambda --notification-endpoint arn:aws:lambda:us-east-1:625663093496:function:eventloggerapp-prod-rules2

    3. Stop the new consumers. Set EnableTriggers=false in [prod] with a revert pull request or, in an emergency, run make deploy ENV=prod AWS_PROFILE=prod from a laptop with that value in samconfig.toml. This deletes the Kinesis mapping and the new stack’s CRM subscription (Zappa’s, added above, stays) and disables the watchdog schedule. Note the time T2 at which the new mapping is gone.

    4. Start Zappa’s Kinesis consumer and schedules again.

      aws lambda update-event-source-mapping --uuid 030cc522-988e-4110-a1ef-1229b94b99b6 --enabled
aws events enable-rule --name 502c129d7cbb7e4f395e07-eventlogger_app.handlers.watchdog.handler
aws events enable-rule --name 502c129d7cbb7e4f395e0-zappa-keep-warm-handler.keep_warm_callback

The Zappa mapping resumes from its own checkpoint, at T, so it processes again everything the new stack handled between T and T2, including rule actions. This repeats actions but skips nothing; it is the safe default.

Only restart Zappa at AT_TIMESTAMP T2 (delete the old mapping and create a new one) if the new worker was fully caught up when you stopped it: IteratorAge at or near zero on every shard at T2. T2 is the wall-clock time the mapping was deleted, not a processed checkpoint, so if the worker was lagging, records that arrived before T2 may still be unprocessed and starting there would skip them. When unsure, resume from the checkpoint and accept the duplicates.

aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name IteratorAge \
  --dimensions Name=FunctionName,Value=<WorkerFunction name> --statistics Maximum --period 60 \
  --start-time "$(date -u -v-15M +%FT%TZ)" --end-time "$(date -u +%FT%TZ)"

To try the cutover again later, start from step 1 of the cutover with a new T. Because the mapping was deleted, the new start position applies.