S3 datalake

Raw vendor responses are archived as JSON in the datalake bucket S3_DATALAKE_BUCKET (payability-datalake in release, dev-payability-datalake in dev) through components/s3_client.py.

def upload_json_data(self, data: dict, bucket: str, key: str):
    with io.BytesIO(json.dumps(data).encode()) as f:
        self.__s3_client.upload_fileobj(
            Fileobj=f, Bucket=bucket, Key=key,
            ExtraArgs={'ACL': 'bucket-owner-full-control'}
        )

The bucket-owner-full-control ACL matters: the datalake bucket lives in a different account from some writers, and downstream consumers must be able to read the objects.

Key conventions

Keys are partitioned by supplier and person and suffixed with a UTC ISO-8601 timestamp, so a supplier’s history is retained rather than overwritten.

Process Key pattern

validate_identity_document (Au10tix)

au10tix/identity_document_validation/supplier_key=<sk>/person_id=<pid>/<timestamp>.json

validate_identity_document (Entrust)

entrust/identity_document_validation/supplier_key=<sk>/person_id=<pid>/<timestamp>.json

validate_personal_information

thomson_reuters/kyc/supplier_key=<sk>/person_id=<pid>/<timestamp>.person_search.json

download_personal_background_report

thomson_reuters/background_reports/supplier_key=<sk>/person_id=<pid>/person/<timestamp>.json

download_business_background_report

thomson_reuters/background_reports/supplier_key=<sk>/person_id=<pid>/business/<timestamp>.json

The layout <vendor>/<process>/supplier_key=<k>/person_id=<p>/<timestamp>.json is part of the platform contract — downstream consumers (for example s3-files-provider) parse these paths. Changing a prefix is a breaking change.

What is stored

Process Payload

Au10tix

The raw IdentityDocuments/Results/{id} response (CompletionStatus == 'Ok').

Entrust

The workflow-run summary merged with its hydrated tasks — each step’s input and output, i.e. the full decision trail. If step hydration failed, the summary alone.

Personal information (KYC)

The parsed EIDV person-search result (xmltodict output of the CLEAR XML).

Personal / business background report

The parsed report response, fetched after the 204 polling loop finished.

The key is written back onto the task row (output.raw_result_s3_path and additional_data.full_result_s3_path) and is what the detailed GET /tasks/result reads via S3Client.get_json_data — a NoSuchKey degrades to an empty result rather than failing the request.

Deletion

DELETE /tasks/result drives the purge flow: ProcessFactory builds both steps of the process, and each step implementing PurgableS3ObjectProducer runs

def purge(self, supplier_key: str, person_id: str, s3_object_purger: S3ObjectPurger) -> List[str]

which, for every processed execution of that person, deletes additional_data.full_result_s3_path and marks the task row purged. The deleted keys are returned to the caller.

Implemented by the three Thomson Reuters handle tasks only. Identity-document archives (Au10tix and Entrust) are not purged by this endpoint — a data-deletion request covering them needs manual work today.

The Entrust flow also pushes the supplier’s captured ID images, selfie and liveness media into Payability’s own document store through the internal API — that is a separate system from the datalake, and the one that feeds compliance review. See Payability internal services and validate_identity_document (Entrust / Onfido).