Zappa → SAM cutover

Until this runbook is finished, both deployments exist side by side. The SAM stacks start with CountersEvents: 'false' in the Env mapping of template.yaml, so the tasks stream and the cleanup schedule keep running on Zappa until step 5. Every step is reversible until step 6.

Zappa (before) SAM (after)

	Zappa (before)	SAM (after)
API	`actionstoolsapp-prod` (+ its API Gateway)	`actionstools-prod-api` + `ActionsToolsApi`
Background	`@task` → `actionstoolsapp-prod`; counters → `actionstoolsapp-prod2`	`actionstools-prod-worker`
Secrets	Lambda env vars, `zappa_settings.json`, code	`actionstools/prod` + `common/crm`
Stage	`actionstoolsapp-stage`	`actionstools-stage`

API

actionstoolsapp-prod (+ its API Gateway)

actionstools-prod-api + ActionsToolsApi

Background

@task → actionstoolsapp-prod; counters → actionstoolsapp-prod2

actionstools-prod-worker

Secrets

Lambda env vars, zappa_settings.json, code

actionstools/prod + common/crm

Stage

actionstoolsapp-stage

actionstools-stage

0. Prerequisites

Before merging the migration branch: add the repository secrets listed in operations/deployment.adoc#ci, create the Secrets Manager secrets (step 1) so the first automatic stage deploy passes its smoke test, and delete the CodeBuild project ActionsToolsApp (or its GitHub webhook) — buildspec.yml is gone, so it would fail on every push.

aws sso login --profile prod && aws sso login --profile dev

Export the live configuration — it is the source of truth for secret values and for any variable that is not in the Env mapping of template.yaml:

for fn in actionstoolsapp-prod actionstoolsapp-prod2; do
  aws lambda get-function-configuration --function-name $fn --profile prod \
    --query 'Environment.Variables' > /tmp/$fn.env.json
done
aws lambda get-function-configuration --function-name actionstoolsapp-stage --profile dev \
  --query 'Environment.Variables' > /tmp/actionstoolsapp-stage.env.json

Compare the non-secret values with the Env mapping in template.yaml and fix any difference before deploying. Delete the /tmp files when done.

1. Secrets

Rotate what was committed to git, then create the secrets (shape: operations/configuration.adoc#the-service-secret):

SLACK_TOKEN — new bot token (Slack).
MESSAGING_API_KEY, EVENT_LOGGER_API_KEY — the instance-8001 key was hardcoded in config.py and the README. Issue a new one in both services, and update common/crm if the same pair is stored there.
AUTH_JWT_RSA_PUBLIC_KEY, SENTRY_DSN — copy from the export.

aws secretsmanager create-secret --name actionstools/stage --profile dev \
  --secret-string file://actionstools-stage.json
aws secretsmanager create-secret --name actionstools/prod --profile prod \
  --secret-string file://actionstools-prod.json

The long-lived X-Session-Token that scripts/ used to contain (login pyb_evt_api, expiry 2045) is also in git history; revoke it in the internal API.

2. Stage

AWS_PROFILE=dev make deploy stage=stage

Smoke test (operations/deployment.adoc#smoke-test-after-a-deploy): Swagger, list notes, create and update a task, add a comment, and check the Worker log shows the notification tasks.

Move the stream to SAM:

STREAM=arn:aws:dynamodb:us-east-1:605668041275:table/em_actions_tasks_staging/stream/2021-01-28T09:15:34.056
aws lambda list-event-source-mappings --event-source-arn $STREAM --profile dev \
  --query 'EventSourceMappings[].[UUID,FunctionArn,State]' --output table
aws lambda update-event-source-mapping --uuid <zappa mapping uuid> --no-enabled --profile dev

Set CountersEvents: 'true' under stage in the Env mapping of template.yaml, deploy again, and confirm a task edit changes a grid count. Point the stage UI at the new ApiUrl.

3. Production, side by side

AWS_PROFILE=prod make deploy stage=prod    # or merge to release and let CI deploy

The new API answers on its execute-api URL only: https://<ApiId>.execute-api.us-east-1.amazonaws.com/prod/actionstools/. Run the smoke test against it. Async tasks queued from it run on the new Worker; counters still come from Zappa.

4. Switch traffic

The custom domain api.payability.com maps the base path actionstools to Zappa’s REST API. Repoint it — a single, instant change:

aws apigateway get-base-path-mappings --domain-name api.payability.com --profile prod
aws apigateway update-base-path-mapping --domain-name api.payability.com \
  --base-path actionstools --profile prod \
  --patch-operations op=replace,path=/restapiId,value=<ApiId output of actionstools-prod>

If get-base-path-mappings shows a different layout (for example a (none) base path, or the domain fronted by something other than API Gateway), keep the layout and only swap the REST API id and stage: the app expects event["path"] to start with /actionstools (request-lifecycle.adoc#paths).

Rollback: the same command with Zappa’s REST API id (recorded from get-base-path-mappings).

Watch Sentry and sam logs --stack-name actionstools-prod --profile prod --tail for an hour.

5. Move the stream and the schedule

Exactly one consumer must update counters. Disable Zappa’s, then enable SAM’s:

STREAM=arn:aws:dynamodb:us-east-1:625663093496:table/em_actions_tasks/stream/2017-09-25T09:09:29.615
aws lambda list-event-source-mappings --event-source-arn $STREAM --profile prod \
  --query 'EventSourceMappings[].[UUID,FunctionArn,State]' --output table
aws lambda update-event-source-mapping --uuid <each zappa uuid> --no-enabled --profile prod

for fn in actionstoolsapp-prod actionstoolsapp-prod2; do
  ARN=$(aws lambda get-function --function-name $fn --query Configuration.FunctionArn --output text --profile prod)
  aws events list-rule-names-by-target --target-arn $ARN --profile prod
done
aws events disable-rule --name <each zappa rule> --profile prod

Set CountersEvents: 'true' under prod in the Env mapping, merge to release. After the deploy, reset counters so drift from the gap is discarded:

aws lambda invoke --function-name actionstools-prod-worker --profile prod \
  --cli-binary-format raw-in-base64-out --payload '{"action": "remove_counters"}' /dev/stdout

Rollback: set the flag back to false, deploy, re-enable the Zappa mapping and rules.

6. Remove Zappa

After a few quiet days:

Delete the functions actionstoolsapp-prod, actionstoolsapp-prod2 (prod) and actionstoolsapp-stage (dev) — this also removes their event source mappings.
Delete the CloudWatch Events rules found in step 5 (and stage’s).
Delete Zappa’s API Gateway stacks: aws cloudformation delete-stack --stack-name actionstoolsapp-prod (and -stage in dev). Check get-base-path-mappings first — nothing may still point at them.
Empty and delete the Zappa buckets app-actionstools—us-east-1 and app-actionstools-staging-dev if nothing else uses them.

Do not delete the IAM role ZappaLambdaExecution: other services (DocumentsApp among them) still use it. The new stack creates its own roles.