System architecture

How automatic-uw-tool is structured. For the step-by-step flow of a single request see Request lifecycle.

Two containers, one codebase

Container Entry point Role Scaling

Web API

web_api_app.py

Synchronous orchestrator. Authenticates callers, validates input, persists the request, starts async work, serves status and results.

ECS service behind an ALB

SWF worker

swf_app.py

Asynchronous executor. Long-polls AWS SWF, performs the vendor calls, writes results and emits events.

ECS service, SWF_APP_THREADS (5) polling threads per task

The golden rule: the Web API never calls a vendor API directly. It authenticates the caller, validates input, persists state, and starts an SWF workflow. All external verification I/O happens in handle tasks on the worker. This keeps HTTP requests fast and makes vendor latency and vendor failures a background concern.

Layers

interface/        HTTP layer — FastAPI routers + endpoint handlers   (Web API only)
   └── web_api/
task/             Domain layer — processes & tasks (the business logic)
components/       Integration layer — vendor, AWS and internal-service clients
config/ utils/    Cross-cutting — Dynaconf configuration and helpers

interface/ depends on task/ and components/, and stays thin: parse the request, check auth and permissions, delegate, shape the response. task/ is the heart of the system and depends on components/ for all I/O — no requests or boto3 import belongs under task/. components/ wraps everything external.

Component diagram

flowchart TB
    Client([Client: internal admin, supplier dashboard, onboarding])
    Onfido([Onfido / Entrust webhook])

    subgraph WebAPI["Web API container (web_api_app.py)"]
        Router[FastAPI routers]
        JWT[JWTAuthorizer]
        Handlers[Endpoint handlers]
    end

    subgraph Worker["SWF worker container (swf_app.py)"]
        Loop[poll loop x5 threads]
        TF[TaskFactory]
        Handle[Handle tasks]
    end

    subgraph AWS
        SWF[(AWS SWF)]
        DDB[(DynamoDB)]
        S3[(S3 datalake)]
        EB[(EventBridge)]
        CW[(CloudWatch metrics)]
    end

    subgraph Vendors["External services"]
        AU[Au10tix]
        ENT[Entrust / Onfido]
        TR[Thomson Reuters CLEAR]
        PAY[Payability Internal API]
        DOCS[PYB Documents]
        HS[HubSpot CRM]
        EL[Internal EventLogger]
        B2B[B2B Task Manager]
    end

    Client -->|x-session-token| Router --> JWT --> Handlers
    Onfido -->|X-SHA2-Signature| Router
    JWT --> PAY
    Handlers -->|start workflow| B2B --> SWF
    Handlers --> DDB
    Handlers --> S3
    Handlers --> CW
    Handlers --> ENT
    Handlers --> HS
    Handlers --> EL

    SWF -->|activity task| Loop --> TF --> Handle
    Handle --> AU
    Handle --> ENT
    Handle --> TR
    Handle --> DOCS
    Handle --> PAY
    Handle --> HS
    Handle --> EL
    Handle --> DDB
    Handle --> S3
    Handle --> EB
    Handle -->|success / failure| SWF

The Web API’s only outbound vendor calls are the Entrust session creation (POST /entrust/sessions must mint the Onfido applicant and workflow run before it can return a URL) and the best-effort HubSpot / supplier-event syncs. Both are fast, single-shot calls; everything slow still happens on the worker.

Why SWF, and why through the B2B Task Manager

  • Vendor calls are inherently asynchronous: Au10tix and Thomson Reuters both use submit → poll-for-readiness → fetch-result, and Entrust notifies by webhook minutes later. SWF gives durable, retryable orchestration without the Web API holding a connection open.

  • The Web API does not call SWF directly. It calls the shared B2B Task Manager HTTP service (components/b2b_task_manager.py), which performs StartWorkflowExecution. The workflow definition itself is owned outside this repository; it schedules an activity onto the task list automatic-uw-tool-task-list, which this repository’s worker polls.

SWF coordinates (from settings_setup.py):

Setting Value

SWF_WORKFLOW_TYPE

Update Amazon SC Earnings

SWF_WORKFLOW_TYPE_VERSION

1.1

SWF_DOMAIN

B2B in release, the environment name in dev

SWF_AUTOMATIC_UW_TOOL_TASK_LIST

automatic-uw-tool-task-list

SWF_APP_THREADS

5

Details in SWF orchestration.

Persistence and observability

  • DynamoDB — three purpose-built tables plus an append-only changelog: DynamoDB tables.

  • S3 datalake — raw vendor responses as JSON, partitioned by supplier and person: S3 datalake.

  • EventBridge — downstream notifications: EventBridge events.

  • CloudWatch — per-process and per-endpoint timing/failure metrics via pyb-sdk (utils/metric_utils.py).

  • Sentry — exception capture in both containers (sentry_sdk.init in each entry point).

See Observability for the full metric and log inventory.

Cross-cutting concerns

Concern Where Notes

Authentication

components/auth/jwt_authorizer.py

x-session-token header validated against the Payability auth API; the JWT is then decoded without signature verification to read login_key.

Authorization

components/permissions/permissions.py

Internal callers need a perm (DEV, FINANCE, REST_UW_ACCESS, …); supplier-dashboard callers skip perm checks but are scoped to their own supplier_key.

Webhook trust

interface/web_api/endpoint_handling/handle_entrust_webhook.py

Onfido cannot present a Payability JWT, so the callback is verified by an X-SHA2-Signature HMAC instead.

Configuration

config/config.py (Dynaconf)

Loaded from the generated settings.json + .secrets.json: Configuration & secrets.

Failure handling

utils/task_utils.py — task_failure_handling

Context manager wrapping task logic; on error marks the task failed, flips the tool to ERROR, and re-raises so SWF and Sentry see it.

Retry / polling

components/retry/retry.py

Generic poll-until-ready used for vendor readiness checks. The Entrust client additionally retries 429/5xx at the HTTP-adapter level.

Data deletion

components/s3_purgable/

Handle tasks that archive to S3 implement PurgableS3ObjectProducer.purge() for GDPR-style deletion, driven by DELETE /tasks/result.

The correlation id

One correlation_id is created per process execution and is used as:

  • the value returned to the caller by the request step,

  • the task_id primary key in TaskExecutionTable,

  • the partition key of FinalStatusTable and the correlation_id attribute on ToolTable,

  • the task_correlation_id embedded in the SWF workflow input, so the handle step can reload the request.

For Entrust runs the correlation_id is the Onfido workflow_run_id, which is what makes a redelivered webhook idempotent. Never mint a second correlation id mid-flow.

Directory reference

Path Layer Documented in

web_api_app.py, interface/web_api/

HTTP

API overview

swf_app.py

Worker entry

SWF orchestration

task/

Domain

Domain model: processes & tasks

components/

Integration

Component layer

config/, settings_setup.py, config_setup.py

Configuration

Configuration & secrets

deployment/, artifacts/, buildspec.yml

Delivery

Deployment