Observability

Observability is thinner than the rest of the infrastructure, and knowing exactly where the gaps are is more useful than a list of features.

Signal Status Notes

Structured logs

Active

CloudWatch, per function, no retention policy

Sentry error reporting

Disabled

SENTRY_DSN is "" in the SAM template

X-Ray tracing

Off

No Tracing: Active; the runtime role has the permissions

CloudWatch alarms

One

Bedrock token consumption

Application Insights

Auto-configured

On the SAM stack resource group

Dashboards

None

—

Sentry

libs/morpheus-common/morpheus_common/sentry.py exposes configure(), and every Lambda handler calls it at cold start before anything else — including before load_secrets(), so a failure to load the secret is itself reportable.

Environment variable Behaviour

SENTRY_DSN

Empty or absent → configure() returns immediately, no initialisation

SENTRY_ENVIRONMENT

"prod" from the template; the code default is "production"

SENTRY_RELEASE

Optional, passed through to sentry_sdk.init(release=…​)

SENTRY_TRACES_SAMPLE_RATE

Documented as 0.1, but the traces_sample_rate argument is commented out

Integrations: AwsLambdaIntegration(timeout_warning=True) and LoggingIntegration(level=logging.INFO, event_level=logging.ERROR). send_default_pii is False.

SENTRY_DSN: "" in Globals means Sentry is off in production. Handlers call configure_sentry(), it finds no DSN, and returns. A real DSN is present in template.yaml as a comment.

Until it is set, an unhandled exception exists only as a CloudWatch log line. Nothing pages, nothing aggregates, and a failure that lands in a dead-letter queue is invisible until someone looks at the queue.

The two configured integrations are worth understanding for when it is turned on: timeout_warning=True reports a Lambda approaching its timeout — the failure mode that otherwise produces no exception at all, just a truncated invocation. And the logging integration’s split levels mean logger.info() calls become breadcrumbs attached to an event, while logger.error() creates the event. That is why the convention is logger.error(…​, exc_info=True) on every caught exception: it is what carries the stack trace into the report.

send_default_pii=False matters here specifically. These handlers process contact names, email addresses, and phone numbers; the default-on setting would attach request data to every event. See PII.

CloudWatch logs

Source Log group Retention

Lambda functions

/aws/lambda/morpheus-sam-<name>

Never expires

AgentCore agent

/aws/bedrock-agentcore/agents/agentcore

One month

AgentCore platform

/aws/bedrock-agentcore/runtimes/*

Platform-managed

The SAM template sets no retention on any Lambda log group, so they default to never expiring and accumulate cost indefinitely. The CDK log group does set ONE_MONTH. The asymmetry is an oversight, not a decision.

Logging conventions the pages elsewhere depend on:

  • logger = logging.getLogger(name), never print().

  • Structured fields as key=value in the message string — "sellerId=%r stage=%r" — so CloudWatch Logs Insights can filter on them.

  • Never log a full invocation payload. They carry PII.

  • logger.debug() for per-record detail, which the LogLevel stack parameter turns on without a code change. See SAM template.

Tailing a function:

aws logs tail /aws/lambda/morpheus-sam-hubspot-sync --follow --region us-east-1

More query recipes in Operations and runbooks.

X-Ray

Not enabled. The SAM Globals block has no Tracing: Active, so no function is traced.

The AgentCore runtime role does carry xray:PutTraceSegments, PutTelemetryRecords, GetSamplingRules, and GetSamplingTargets. Whether traces are produced depends on the AgentCore platform’s own configuration rather than on anything in this repository.

Tracing would be most valuable on the paths that cross services — a CrmSyncQueue message produced by enrichment-scorer and consumed by morpheus-hubspot currently has no correlation identifier tying the two invocations together. Following one seller through the pipeline means grepping two log groups on a shared business id.

The one alarm

Property Value

Name

morpheus-sam-bedrock-daily-tokens-high

Metric

AWS/Bedrock → InputTokenCount

Statistic

Sum over a 3600-second period

Threshold

> 5,000,000 in one hour

Evaluation periods

1

Missing data

notBreaching

Action

SNS arn:aws:sns:us-east-1:625663093496:security-operational-issues

It is a cost guard, not a health check: it fires when something is consuming far more Bedrock input tokens than normal — a runaway retry loop, an agent given an oversized context, a backfill schedule re-enabled at a development cadence.

The SNS topic security-operational-issues is not created by either stack. It is assumed to exist. In a fresh account, the alarm deploys, transitions to ALARM, and its action fails silently.

TreatMissingData: notBreaching is right for this metric — no Bedrock invocations means no data points, and quiet is not a problem here.

Application Insights

ApplicationResourceGroup groups the SAM stack’s resources, and ApplicationInsightsMonitoring enables AutoConfigurationEnabled: true over it. AWS then provisions its own default Lambda and SQS monitors and a dashboard.

Because it is auto-configured, what it watches is not declared anywhere in this repository, and it changes as AWS updates its defaults. Treat it as a convenience layer rather than as the monitoring plan.

Gaps

Ordered by how likely each is to matter:

  1. Sentry disabled. Setting a real SENTRY_DSN is the single highest-value change available — it converts silent failures into alerts.

  2. No DLQ alarms. All three dead-letter queues can fill without notifying anyone. An alarm on ApproximateNumberOfMessagesVisible > 0 for each is straightforward.

  3. No queue-depth or message-age alarms. With enrichment-scorer currently untriggered, EnrichmentPipelineQueue grows until messages expire at four days, and nothing reports it. See Queues and dead-letter queues.

  4. No Lambda error-rate or throttle alarms. ReservedConcurrentExecutions makes throttling a real possibility on phone-agent, which is capped at 2.

  5. No log retention on Lambda groups. Unbounded storage growth.

  6. No schedule-failure signal. A schedule whose detail-type no longer matches a registry entry fires successfully into a handler that does not exist. The observable symptom is a watermark in configs that stops advancing — see Runtime state in configs.

Debugging without alarms

Given the above, the practical order when something looks wrong:

  1. Check the relevant DLQ depth. A message there carries the failing payload.

  2. Check updatedAt on the relevant configs watermark. A stale one names the schedule that stopped completing.

  3. Check ApproximateAgeOfOldestMessage on the main queue. A growing age means the consumer is disabled, throttled, or erroring.

  4. Only then read logs, filtered on the business id you already have from steps 1–3.

The runbooks are in Operations and runbooks.