Queues and dead-letter queues
Three queues carry every asynchronous message in the platform. All are declared in
lambdas/template.yaml.
| Queue | Type | Visibility | Retention | Redrive |
|---|---|---|---|---|
|
FIFO |
360s |
4 days |
|
|
FIFO |
180s |
4 days |
|
|
Standard |
180s |
4 days |
|
All three dead-letter queues retain for 14 days — the SQS maximum — so a failure has a fortnight to be noticed and replayed before it is lost.
maxReceiveCount is 1
Every redrive policy sends a message to the dead-letter queue after a single failed receive. There is no in-queue retry.
This is unusual and deliberate. The alternative — the common maxReceiveCount: 3 — assumes
failures are transient, and for these consumers that assumption is mostly false:
-
A malformed or unroutable payload fails identically every time. Retrying it twice more costs three invocations to reach the same destination.
-
A HubSpot 4xx is a rejection, not a blip. The
pyb-hubspottransport already retries 429 responses inside the invocation, so rate limiting never reaches the queue as a failure. -
A message that fails, sits invisible for the visibility timeout, and retries head-of-line blocks its entire FIFO message group for the duration. With a 360-second timeout, three attempts stall a group for eighteen minutes.
The trade is that a genuinely transient failure — a database connection blip — also lands in the DLQ. That is accepted because a DLQ message is visible and replayable, whereas a silently retried-then-expired message is neither. Redriving a DLQ is described in Operations and runbooks.
FIFO ordering
The two FIFO queues both set ContentBasedDeduplication: false, so every producer must supply
an explicit MessageDeduplicationId and MessageGroupId.
Content-based deduplication would hash the body — which breaks the moment a payload carries a
timestamp, since two logically identical events would produce different hashes and both be
delivered. Explicit ids let the producer choose the identity: a seller id, a contact id, an
engagement id. The MessageGroupId then decides the ordering domain — messages about the same
contact are processed in order, messages about different contacts run in parallel.
Neither queue sets DeduplicationScope or FifoThroughputLimit, so both use the defaults:
deduplication across the whole queue, and throughput shared across message groups rather than
per-group high throughput. Full producer conventions are in
Publishing events.
EmailAgentQueue is standard rather than FIFO because reply classification is order-independent
— each message is one email, categorised on its own.
Visibility timeout against function timeout
| Queue | Visibility | Consumer timeout |
|---|---|---|
|
360s |
|
|
180s |
|
|
180s |
|
|
Visibility timeout must always exceed the consuming function’s timeout, with headroom. If it
did not, a slow-but-successful invocation would have its message become visible again
mid-flight and be delivered to a second invocation — the same work done twice, and with
The margins here are 60 seconds on both. Raising a function’s timeout without raising its
queue’s visibility timeout is the most likely way to introduce duplicate processing. The
template says so in a comment above the scorer’s |
Partial batch failure
Every SQS event source declares:
BatchSize: 1
FunctionResponseTypes:
- ReportBatchItemFailuresReportBatchItemFailures lets a handler return {"batchItemFailures": [{"itemIdentifier": id}]}
so only the named messages are retried and the rest are deleted. Without it, one failure in a
batch returns the whole batch to the queue.
With BatchSize: 1 the two are equivalent in effect, but the declaration is kept because
raising the batch size later would otherwise silently reintroduce whole-batch retries. Batch
size 1 is what makes a failure attributable to one message — which is the entire premise of
sending failures straight to a DLQ.
Concurrency
| Consumer | Maximum concurrency | Reserved concurrency |
|---|---|---|
|
10 |
— |
|
5 |
5 |
|
50 (commented out) |
5 |
ScalingConfig.MaximumConcurrency on the event source caps how many invocations SQS starts;
ReservedConcurrentExecutions on the function caps the function overall, including schedule
invocations. Both exist because morpheus-hubspot is driven by a queue and ten schedules —
the event source cap alone would not bound the total.
The HubSpot cap of 10 is set against that API’s rate limits rather than any AWS constraint.
Producers and consumers
| Queue | Produced by | Consumed by |
|---|---|---|
|
|
|
|
|
|
|
|
|
Payload schemas for each queue are in CrmSyncQueue events, EnrichmentPipelineQueue events, and EmailAgentQueue events.
|
|
Monitoring
| Metric | Meaning |
|---|---|
|
Anything above zero is a failure that needs a human. Nothing alarms on this today. |
|
Sustained growth means the consumer is throttled, disabled, or erroring. |
|
The lead indicator. Approaching four days means messages are about to expire silently. |
No CloudWatch alarm is configured on any queue metric. The only alarm in the stack watches Bedrock token consumption — see Observability.