Troubleshooting

Attribute errors for environment-specific settings, or unexpectedly default values

ENV_FOR_DYNACONF does not match ENV_NAME. Export both to the same value and regenerate config/settings.json.

ValueError: Environment ENV_NAME value is required

settings_setup.py ran without ENV_NAME.

Any endpoint 500s at construction time, even ones unrelated to Thomson Reuters

ThomsonReutersClient.init resolves the cert/key paths, and several handlers build a full TaskFactory. Run make secrets (dev writes the PEMs directly; production extracts them from the PFX during the build).

pyenv: version '3.9' is not installed in CodeBuild

.python-version is overriding runtime-versions. The install phase already removes it — check it was not re-added inside the build.

ModuleNotFoundError: pkg_resources during install

setuptools ≥ 81. The pins in pyproject.toml (setuptools<81, build-constraint-dependencies) and PIP_CONSTRAINT in buildspec.yml exist for this.

400 "Source is missing from auth token"

The auth API response has no source claim — the token was minted by a service that does not set one.

400 "Supplier key from auth token does not match…"

A supplier-dashboard caller targeted another supplier.

400 {"error_class": "ValidationError", …}

An input validator rejected the payload; error_body.incorrect_fields names each field. Note the task row was already written and is now failed.

400 {"error_class": "CannotRunWorkflow", "error_body": {"reason": "Task already running"}}

An SWF workflow with the same deterministic workflowId is still running for that supplier and process. Wait for it, or use force_new_execution (which does not change the workflow id — it only bypasses the DynamoDB dedup).

403 "Access forbidden."

Either the source is not in the task’s get_available_process_sources(), or a non-internal caller hit an internal-only endpoint (GET /tasks/result, GET /tasks/{id}, DELETE /tasks/result, PUT …/final_statuses).

403 "Forbidden."

The caller has none of the required perms and is not INT_SUPER_USER.

404 on GET /tasks/result

No processed or failed execution exists for that supplier + person + process. A pending run does not count.

409 from JWTAuthorizer

A KeyError reading the auth payload — a required claim (usually login_id) is missing.

409 on an Entrust reprocess

SWF refused the run; the message carries the B2B error.

429

Rate limit: 3 executions per supplier + process (supplier dashboard), or 3 Entrust sessions per person (supplier dashboard / onboarding).

500 "Something went wrong with auth API…"

get_user_perms failed during the permission check.

502 on POST /entrust/sessions

Either the Payability internal API or Onfido failed. A 4xx from Onfido is passed through instead, with the upstream body under entrust_error.

A tool is ERROR with error_message = "Task Timeout"

It sat PENDING for more than 300 s and GET /tools flipped it. Usually the worker is not draining automatic-uw-tool-task-list, or the vendor poll exceeded the window. Check the task row and the SWF run.

A tool stays PENDING after an Entrust run finished

The run status is review — this is by design. The tool resolves on the later terminal completion.

A tool is NOT_RUN although a process ran

GET /tools auto-created the row from SUPPORTED_TOOLS and no execution has written it yet — or, for a background report, note that download_business_background_report has no tool at all.

The task is processed but the verdict looks wrong

Check FinalStatusTable history (versions are kept) and ChangelogTable for a manual override; default_result_status on the detailed result shows what the vendor actually said.

RetryException: The number of retries has been exceeded.

A vendor poll never became ready: Au10tix readiness (10 × 5 s), a Thomson Reuters report stuck on 204 (Retry(total=20), 10 s). Retry the process later.

ResultNotFound

The vendor has no matching records (GroupCount == 0), or a multi-record person search matched no exact first/last name + SSN combination.

MalformedResponse

An expected node was missing. For the business report this can also surface as a raw KeyError, because that task addresses ns2:/ns3:/ns10: prefixes literally.

MissingResponseSection

The personal report lacked one of the seven required sections.

Thomson Reuters not working / ThomsonReutersNotWorkingEvent

A CLEAR call returned an HTTP error. The event lets the platform proceed without the check; the task still fails.

An activity dies at its start-to-close timeout with no failure recorded

Historically caused by an over-long SWF reason; SWFClient now truncates reason to 256 and details to 32768 chars. If you see it again, check for an exception raised before the responder ran.

Nothing is picked up at all

Domain/task-list mismatch (SWF_DOMAIN is the env name outside release), or another worker on the same ENV_NAME is consuming the activities.

Webhook returns 401

Signature mismatch, missing X-SHA2-Signature, or ENTRUST_WEBHOOK_TOKEN unset (logged as a configuration error). Remember the HMAC covers the exact raw bytes.

Webhook returns 200 {"status":"ignored"}

Not a workflow_run.completed event, no object.id, or no TaskExecutionTable row for that run (the session was never created here, or was purged).

Task fails on the document upload

The internal document API rejected the request; the raised HTTPError includes the upstream body (invalid compliance_key, unaccepted file, unknown id). SWF retries; the guards mean the archive/upload is not repeated once it has succeeded.

The run is approved but the documents are still pending in the document store

uploaded_document_ids was empty (nothing was uploaded — check applicant_id and document_actions), or documents_approved was already set by an earlier delivery.

No document was uploaded at all

The session recorded no applicant_id, or Onfido returned no files. Both are logged and treated as "nothing to upload", not as failures.

Re-running produced duplicate documents in the store

Expected: a reprocess archives everything existing and uploads a fresh set. See api/tasks.adoc#entrust-reprocess.

A redelivered webhook did nothing

The row was already processed — the idempotency guard returned the existing output.

HubSpot shows a stale status

HubSpot sync is best effort and swallows errors. Look in Sentry for HubSpotClient exceptions, and verify the contact has a supplierkey matching the supplier.

All executions of a process for a supplier

Query the supplier_key-process_name-index GSI on the task table.

Is this an Entrust or an Au10tix row?

Entrust rows carry workflow_run_id / applicant_id in input; Au10tix rows carry document_id.

The newest finished execution for a person

TaskExecutionTable.get_latest_processed_task (processed or failed), which is what GET /tasks/result uses.

Who changed a verdict

Scan ChangelogTable by supplier_key, sorted by log_ts; type = FINAL_STATUS and action = UPDATE.

The raw vendor evidence

additional_data.full_result_s3_path on the task row, in S3_DATALAKE_BUCKET.