Observability

CloudWatch metrics

Published through utils/metric_utils.py → pyb-sdk CriticalMetricNamespace (PybUserProcess.UNDERWRITING, PybService.INTG, process_name = automatic-uw-tool). Metric publication never fails a request or a task — errors are caught and logged.

Worker (swf_app.py)

Metric Unit Emitted

{process_name}_failure

count

Any exception while building or executing an activity (only when the task object was built).

{process_name}_success_time_taken

seconds

Successful activity, wall-clock from poll to completion.

{process_name} is task.get_process_name(), so both Au10tix and Entrust report under validate_identity_document.

Web API

One timing metric per endpoint, emitted on every call including error paths:

  • execute_endpoint_time_taken

  • get_task_result_endpoint_time_taken

  • get_task_status_endpoint_time_taken

  • delete_task_results_endpoint_time_taken

  • update_final_statuses_endpoint_time_taken

  • get_tools_endpoint_time_taken

  • create_entrust_session_endpoint_time_taken

Because success and failure share a metric name, latency alone will not tell you a call failed — pair these with the Sentry stream and the task statuses.

Logs

Stream Contents

Web API (/ecs/automatic-uw-tool)

WEB REQUEST LOGGER lines — [{login_id}:{source}] {path} ({payload}) — plus handler logs. The Entrust session handler logs a full narrative (authorization, reuse, applicant/run creation, persistence) keyed by the caller’s login id.

SWF worker

Task loggers, one per task, prefixed with the correlation id: {correlation_id} | {task_name} | {LEVEL} | {asctime} - {message}. Five threads interleave, so always filter by correlation id.

Request payloads are logged verbatim, so validate_personal_information inputs put PII (name, date of birth, SSN, address) into the Web API log stream. ThomsonReutersDataNotMatchEvent and ThomsonReutersNotWorkingEvent likewise carry task_input in their Detail. Treat both as sensitive.

Sentry

sentry_sdk.init runs in both entry points:

  • Web API — environment = settings.ENV_NAME, plus SentryAsgiMiddleware for unhandled request errors.

  • Worker — environment = settings.SENTRY_ENV (production in release, the env name otherwise), traces_sample_rate = 1.0, and an explicit capture_exception() for every failed activity.

Also reported deliberately: HubSpot sync failures and supplier-event failures — HubSpotClient and SupplierEventLogger both capture their own exceptions because they swallow them.

Audit trails inside the data

Observability is not only CloudWatch:

Source Answers

TaskExecutionTable

Did the execution finish? status, create_ts/update_ts, requested_by, source, the failure output ({'error': …}), and for Entrust the document_actions counts.

ChangelogTable

Who changed a verdict or a tool, when, and under which app_version — every FinalStatusTable and ToolTable mutation appends a row.

FinalStatusTable

Verdict history: versions are kept per correlation_id, sorted by created_at, with created_by / updated_by (automatic-uw-tool, override, or a login id).

S3 datalake

The raw vendor evidence behind a verdict — for Entrust, including every workflow step’s input and output.

Internal EventLogger

The supplier’s Entrust lifecycle as supplier-timeline events (in_progress → verified / in_review / denied), with workflow_run_id and verdict in extra. Best effort — see Supplier event logger.

SWF console

Activity input, result and failure reason/details per run. The Entrust task’s compact result (status, verdict, document counts) is designed to be readable there.

Health checks

GET /health-check returns {"status": "OK"} unauthenticated — used by the ALB target group. It is a liveness probe only: it does not verify DynamoDB, SWF or vendor connectivity.

The worker exposes no health endpoint. Its liveness is visible as SWF activity throughput and the _success_time_taken metric; a wedged worker shows up as tools stuck PENDING (surfaced as ERROR by GET /tools after 300 s).

Suggested alarms

Not currently defined in this repository, but the useful signals are:

  • {process}_failure count above zero over a short window, per process.

  • {process}_success_time_taken p95 rising — vendor slowdown.

  • Sentry error-rate spikes on ThomsonReutersNotWorking, RetryException, or HTTPError from the internal document API (an Entrust upload rejection).

  • Tools left PENDING beyond the 300 s window, which indicates the worker is not draining the task list.