Python standards

The reference that reviewers — human and AI — hold plans and diffs against. Goal: every new module models data with the strongest structure the runtime allows, and every design decision survives production load.

Runtime constraint: Python 3.9 (.python-version, requires-python = ">=3.9,<3.10"). That rules out match statements, X | Y union syntax in annotations (use Optional/Union, or from future import annotations), dataclass(slots=True) (3.10+), StrEnum (3.11+) and tomllib. Built-in generics (list[str], dict[str, int]) are fine.

Pick the strongest data structure that fits

Use case Structure Why

API request/response bodies

pydantic BaseModel

Validation at the boundary, self-documenting OpenAPI, no hand-rolled input_validator for new endpoints. Already in the graph via FastAPI 0.110.

Internal domain objects (task inputs/outputs, session records, result mappings)

@dataclass(frozen=True)

Immutable, hashable, typo-proof; replaces raw dicts crossing function boundaries. EntrustFile is the in-repo example.

Closed sets of values (statuses, tool types, decisions)

Enum (class Decision(str, Enum) for JSON-friendly values)

The repo already does this (ToolStatus, ToolType, ProcessSource); never introduce a bare string constant for a status.

Dict-shaped data you cannot restructure yet (DynamoDB items, legacy payloads)

TypedDict at minimum

Documents the shape and lets a type checker catch key errors.

Fixed small records in hot paths

NamedTuple

Cheap, immutable, destructurable.

Duck-typed injection seams (clients passed into TaskFactory)

Protocol

Lets tasks depend on an interface rather than a vendor class, and makes mockito tests honest.

Constants

Final annotations at module level

No magic numbers or strings inline.

Rules of thumb:

  • No raw dict across a boundary. A dict may exist inside a function; the moment it is returned, passed or persisted, give it a dataclass/TypedDict/pydantic shape. Existing code violates this in places — do not add new violations.

  • Parse, don’t validate. Convert external shapes (webhook payloads, DynamoDB items, vendor JSON) into typed objects once, at the edge, then pass the typed object around.

  • Immutable by default. frozen=True dataclasses; mutate with dataclasses.replace.

  • Enums own their mapping logic. Follow ToolType.get_by_process_name — classmethods on the enum, not dict lookups scattered around callers.

Production and scaling patterns

  • DynamoDB: query, never scan, in request paths. ToolTable currently uses scan with filters (get_by_supplier_and_type, get_all_by_supplier) even though supplier_key is the partition key — a known hot spot. New access patterns must use Key(…​) conditions against the PK/sort key or a GSI, and must paginate on LastEvaluatedKey (see FinalStatusTable.get_query_results or TaskExecutionTable.get_query_results for the correct loop).

  • Idempotency is mandatory for webhooks and events. Handlers keyed by correlation_id / workflow_run_id must tolerate redelivery: a repeated payload must not double-write S3, double-emit EventBridge, or flip a terminal status backwards. Use status-transition guards or conditional writes — the Entrust task’s documents_uploaded / documents_approved flags and its processed short-circuit are the reference implementation.

  • correlation_id is the join key across TaskExecutionTable, FinalStatusTable, ToolTable, S3 keys and EventBridge payloads. Preserve it end to end; never mint a second one mid-flow.

  • HTTP out: sessions, timeouts, retries. Reuse a requests.Session per client, set an explicit timeout on every call (no bare requests.post), and use the retry helper or an HTTPAdapter retry for polling. EntrustClient shows the pattern; Au10tixClient and ThomsonReutersClient predate it and lack both session reuse and timeouts — do not copy them.

  • The Web API stays synchronous, thin and vendor-light. Anything slow belongs on the worker. The two exceptions (Entrust session creation, HubSpot sync) are fast single calls, and the webhook hands off rather than processing.

  • Fail loudly, once. Side-effecting task logic goes through task_failure_handling so failures mark the task failed and reach Sentry. Do not swallow exceptions to "keep the loop alive". The one intentional exception is HubSpot, which is explicitly best-effort and documented as such.

  • Configuration through Dynaconf settings only. New settings in settings_setup.py, secrets in Secrets Manager via config_setup.py. Never os.getenv in app code, never a hardcoded table name, URL or account id.

  • S3 layout is part of the contract. Datalake keys follow <vendor>/<process>/supplier_key=<k>/person_id=<p>/<timestamp>.json; downstream consumers parse these paths.

Conventions specific to this repository

  • Tasks live in packages with the exact file layout in Domain model: processes & tasks (implementation.py, input.py, output.py, optional input_validator.py).

  • No direct I/O in task/ — do not import requests or boto3 there. Use the injected components/ clients.

  • Private methods use the double-underscore name-mangled form (self.__do_thing) throughout; match the surrounding style.

  • Endpoint handlers subclass BaseEndpointHandler, log the request, and use StopExecution for early exits rather than returning from nested helpers.

  • When you add a dependency to a task, wire it into every TaskFactory construction site — swf_app.py plus the Web API handlers that build one.

Definition of done

  1. Code follows the conventions above.

  2. Unit tests added or updated under development/unit_tests/ and passing, coverage ≥ 80% (pytest, see Testing).

  3. Any affected page in this documentation component updated (Writing these docs).

  4. No secrets or generated configuration committed.