Troubleshooting

Cold start fails (every request 502)

Log line Cause → fix

AccessDeniedException … GetSecretValue

The secret is not named actionstools/<EnvName>, or is in another account/region. Fix the secret; the name and the IAM policy are derived from EnvName in template.yaml.

AssertionError: set env variable MESSAGING_API_KEY (or another)

The key is missing from actionstools/<env>. Add it, then recycle (redeploy).

missing AUTH_JWT_RSA_PUBLIC_KEY secret variable

Neither actionstools/<env> nor common/crm holds the public key.

KeyError: 'PAYABILITY_INTERNAL_API_URL' in pyb_flask_utils/auth.py

Something imported pyb_flask_utils before secrets_loader/config. Restore the import order (Configuration & secrets).

Requests fail

  • All requests 400 Unauthorized — the internal API’s token/validate rejected the token (a token from another environment, or expired), PAYABILITY_INTERNAL_API_URL is unreachable from the function, or X-Session-Token is missing.

  • 404 for every path in production — event["path"] does not start with /actionstools: the custom domain base path mapping changed (request-lifecycle.adoc#paths).

  • Timeouts on tasks, settings or notifications — the call to the Payability internal API hangs. Check the function still has its VPC config and the subnets have a route to that API.

  • 400 Unable to parse JWT token with 403 from token/validate — the function is calling staging.dev.payability.com from a public Lambda IP. Stage must use the dev VPC in template.yaml.

  • CORS errors in the browser — preflight is answered by API Gateway; check the Cors block in template.yaml lists the header the UI added.

Background work does not happen

  1. Is WORKER_FUNCTION_NAME set on the API function? Without it, tasks run inline (slow, but they run).

  2. API log AccessDeniedException … lambda:InvokeFunction — the LambdaInvokePolicy is missing.

  3. API log RequestEntityTooLargeException — the task arguments exceed 256 KB; pass ids, not objects.

  4. Worker log Unrecognised worker event or Refusing to run task outside actionstools_app. — the payload was built by hand or the function moved; check task_path.

  5. Worker log RuntimeError: Working outside of request context — the task reads Flask request. Known case: async_create_task (task creation with _mode: async).

  6. Slack only: User … cannot be found on slack — the recipient’s Slack profile email differs from the CRM email.

Grid counts are wrong

  • Make sure exactly one consumer is enabled on the tasks stream (aws lambda list-event-source-mappings --event-source-arn <stream arn>).

  • Reset: invoke the Worker with {"action": "remove_counters"}; counts are recalculated on the next request (Task counters).

Build and local issues

  • UnauthorizedSSOTokenError on import — local runs try to read Secrets Manager. Run aws sso login --profile dev or set AWS_SECRETS_MANAGER=False.

  • sam build: Could not satisfy the requirement: setuptools — pip cannot reach its index. A global pip config pointing at CodeArtifact with an expired token is the usual cause: PIP_INDEX_URL=https://pypi.org/simple make build, or refresh the CodeArtifact login.

  • sam build: Unable to retrieve name/version for package: pyb-flask-utils — SAM ran outside uv run and found no usable python3.12. Use make build.

  • uv sync: authentication failed for github.com/Payability — configure git credentials for the private repos (CI uses the PYB_GITHUB_TOKEN organisation secret in a url.insteadOf rule). In CI, could not read Password for 'https://github.com' means that secret was empty.