System architecture

Morpheus is a set of independent, event-driven components joined by three integration points: SQS queues, a shared PostgreSQL database, and one Secrets Manager secret. No component calls another component’s code.

Architectural principles

Single writer to the CRM

Only morpheus-hubspot writes to HubSpot. Every other component that needs a CRM change publishes an event to CrmSyncQueue. This keeps HubSpot rate-limit handling, OAuth, and property mapping in one place.

Registry-based dispatch

Every Lambda is a thin dispatcher over a registry that maps a routing key — an SQS event_type message attribute or an EventBridge detail-type — to a handler class and a Pydantic model. Adding behaviour means adding a handler and one registry entry, never editing the dispatcher. See Handler and registry pattern.

Fat events

Messages carry everything the consumer needs. Seller.CrmSync embeds the company fields and the full contact list so morpheus-hubspot performs no database read just to sync. See Events.

Validate at the boundary

Every inbound payload is parsed by a Pydantic v2 model before a handler sees it. Malformed payloads are logged and dropped rather than retried forever.

Best-effort AI, durable persistence

Database writes commit first; LLM enrichment and scoring run afterwards inside try/except. An AI failure never causes a queue message to be retried and never rolls back a committed row.

Configuration by secret, not by environment

Credentials live in one Secrets Manager secret (agentcore-configs) fetched once per container and cached. Environment variables carry only infrastructure facts (queue URLs, ARNs, tenant id, batch sizes).

No business logic in infrastructure code

Neither the CDK stack nor the SAM template chooses models, prompts, or providers. They provision resources and inject wiring.

Component topology

flowchart TB
  subgraph ext ["External systems"]
    SS(["SmartScout API"])
    HS(["HubSpot CRM"])
    TAV(["Tavily MCP"])
    TW(["Twilio Lookup v2"])
    SL(["Slack"])
    FC(["Files.com\nFive9 CSV drops"])
    RS(["Resend"])
  end

  subgraph aws ["AWS account 625663093496 / us-east-1"]
    subgraph sam ["SAM stack — lambdas/template.yaml"]
      Scraper["enrichment-scraper"]
      Scorer["enrichment-scorer"]
      HubFn["morpheus-hubspot"]
      EmailFn["email-agent"]
      PhoneFn["phone-agent"]
      Q1[["EnrichmentPipelineQueue.fifo"]]
      Q2[["CrmSyncQueue.fifo"]]
      Q3[["EmailAgentQueue"]]
      DLQ[["3 dead-letter queues"]]
      SM(["Secrets Manager\nagentcore-configs"])
    end

    subgraph cdkstack ["CDK stack — AgentCoreStack"]
      RT["Bedrock AgentCore Runtime\nagentcore container"]
      MEM[("AgentCore Memory")]
      ECR[("ECR image")]
      LG[("CloudWatch log group")]
    end

    BR(["Amazon Bedrock"])
  end

  PG[("PostgreSQL — Supabase\nshared with the web app")]
  BQ[("BigQuery\nhubspot4 + outreach_app")]

  SS --> Scraper
  Scraper --> Q1 --> Scorer
  Scraper --> PG
  BQ --> Scraper
  Scorer --> PG
  Scorer --> Q2
  Scorer --> TAV
  Scorer --> BR
  Q2 --> HubFn
  HubFn --> HS
  HubFn --> PG
  BQ --> HubFn
  FC --> HubFn
  HS -. webhook relay .-> Q2
  PhoneFn --> TW
  PhoneFn --> SL
  PhoneFn --> Q2
  PhoneFn --> Q3
  PhoneFn --> PG
  BQ --> PhoneFn
  RS -. inbound relay .-> Q3
  Q3 --> EmailFn --> PG
  EmailFn --> BR
  Q1 -.-> DLQ
  Q2 -.-> DLQ
  Q3 -.-> DLQ
  SM --- Scraper & Scorer & HubFn & EmailFn & PhoneFn & RT
  ECR --> RT
  RT --- MEM
  RT --> BR
  RT --> LG

Data planes

There are two data planes, and it matters which one a given rule reads from.

OLTP — PostgreSQL (Supabase)

The operational database, shared with the Payability web application. Morpheus owns a subset of tables through the morpheus-data-layer ORM: sellers, contacts, target_emails, target_phone_numbers, Call, CampaignTarget, Activity, follow_up_tasks, sales_reps, configs, and the email campaign tables. Schema is owned by the web application’s Prisma schema — see Prisma schema reference.

OLAP — BigQuery

A Stitch export of the HubSpot portal (bigqueryexport-183608.hubspot4) plus a mirror of application tables (bigqueryexport-183608.outreach_app). All re-engagement segmentation and scoring SQL runs here because it needs a full cross-object view of the CRM (deals, engagements, contacts) that PostgreSQL does not hold. See BigQuery warehouse.

Re-engagement handlers read from BigQuery and write to HubSpot. The PostgreSQL row is consulted only to resolve identifiers and ownership. BigQuery data is a periodic export, so a segment can lag reality by the export interval.

Message flows

Discovery to CRM

sequenceDiagram
  participant EB as EventBridge
  participant SC as enrichment-scraper
  participant SS as SmartScout
  participant Q1 as EnrichmentPipelineQueue
  participant SR as enrichment-scorer
  participant DB as PostgreSQL
  participant BR as Bedrock
  participant Q2 as CrmSyncQueue
  participant HF as morpheus-hubspot
  participant HS as HubSpot

  EB->>SC: SellerDiscovery.Amazon
  SC->>DB: read pagination cursor (configs)
  SC->>SS: paginated seller search
  SS-->>SC: seller rows
  SC->>DB: save next_start_row
  SC->>Q1: SellerRecord.Received (batch of 10)
  Q1->>SR: one message per invocation
  SR->>DB: upsert seller (commits first)
  SR->>BR: enrichment agent (Tavily + HubSpot MCP)
  SR->>DB: apply enrichment + upsert contacts
  SR->>BR: payout-pain scoring agent
  SR->>DB: write payoutPainScore
  SR->>Q2: Seller.CrmSync (fat event)
  Q2->>HF: dispatch by event_type
  HF->>HS: upsert Company, upsert Contacts, log Notes
  HF->>DB: write back hubspotCompanyId / hubspotContactId

Daily call-list refresh

sequenceDiagram
  participant EB as EventBridge
  participant HF as morpheus-hubspot
  participant BQ as BigQuery
  participant DB as PostgreSQL
  participant HS as HubSpot

  EB->>HF: Hubspot.EmailClickReEngagement (and 3 sibling segments)
  HF->>BQ: SELECT * FROM v_*_reeng
  BQ-->>HF: candidate rows + priority_score
  HF->>HF: drop rows called today / recently by a human
  HF->>DB: resolve contact, check owner is an active rep
  HF->>DB: round-robin next rep (advisory lock + configs cursor)
  HF->>HS: PATCH call_list_date, call_priority, call_assigned_to, outreach_list_reason
  EB->>HF: Hubspot.SalesRepCallListRefresh (daily)
  HF->>HS: delete yesterday's list, create today's dynamic list per rep
  HF->>DB: store hubspotCallSegmentId

See Daily call lists for the rules and Re-engagement segments for each segment.

Failure and retry model

Queue maxReceiveCount Behaviour on handler failure

EnrichmentPipelineQueue.fifo

1

Straight to EnrichmentPipelineDlq.fifo, retained 14 days. No automatic retry.

CrmSyncQueue.fifo

1

Straight to CrmSyncDlq.fifo, retained 14 days. No automatic retry.

EmailAgentQueue

1

Straight to EmailAgentDlq, retained 14 days.

All three SQS triggers declare ReportBatchItemFailures, so a failure isolates to the offending message rather than the batch. Batch size is 1 everywhere today, which makes the distinction academic but keeps the contract correct if batch size is raised.

Handlers distinguish two failure classes deliberately:

  • Bad data — unknown event_type, subscription mismatch, missing row, malformed UUID. Logged at ERROR and the handler returns normally, so the message is deleted. Retrying cannot help.

  • Transient failure — Bedrock error, HubSpot 5xx, database error. The exception propagates so the message is recorded as a batch item failure.

Scheduled (EventBridge) handlers have no queue behind them. A failed run simply waits for the next tick; most iterate per-row with try/except so one bad row does not abort the batch.

Where to go next