Seller discovery

Discovery is owned by enrichment-scraper. It queries SmartScout for sellers matching the ideal customer profile and publishes each result to EnrichmentPipelineQueue as a SellerRecord.Received message.

Implementation: lambdas/enrichment_scraper/events/amazon_discovery.py, providers/smartscout/provider.py, services/config_state.py, services/queue.py.

Kill switch — checked first

Before any network call, the handler reads configs.key = 'enrichment.amazon.disabled'. If the value is exactly "true" (case-insensitive, whitespace-trimmed) the run logs and returns. Any other value, or an absent row, means active.

-- stop discovery
INSERT INTO configs (key, value, "subscriptionId")
VALUES ('enrichment.amazon.disabled', 'true', '<subscription-uuid>')
ON CONFLICT (key) DO UPDATE SET value = 'true';

-- resume
UPDATE configs SET value = 'false' WHERE key = 'enrichment.amazon.disabled';

This is a database flag, not an environment variable, so it takes effect on the next scheduled run without a deploy.

Ideal customer profile

Filters are built from Settings and passed to the provider as SellerFilters.

Setting Default Meaning

ICP_REVENUE_MIN

100000

Minimum annual revenue, USD

ICP_REVENUE_MAX

1000000

Maximum annual revenue, USD

ICP_CATEGORIES

empty

Comma-separated category filter; empty means all categories

MAX_PAGES_PER_RUN

1

Pages fetched per invocation; the event’s detail.max_pages overrides it

PAGE_SIZE

10

Rows per SmartScout page

SMARTSCOUT_MARKETPLACE

US

SmartScout marketplace header

ICP_REVENUE_* are annual figures. Settings.monthly_revenue_min and monthly_revenue_max divide by 12 before the values reach SmartScout, which filters on estimated monthly sales. With the defaults the effective band is roughly \$8.3k–\$83k per month.

A model_validator rejects min >= max at import time, so a misconfiguration fails the cold start rather than silently returning nothing.

Pagination across runs

SmartScout results are consumed a page at a time, and the cursor lives in the database so consecutive scheduled runs walk forward through the result set instead of re-reading page one.

sequenceDiagram
  participant EB as EventBridge
  participant H as AmazonDiscoveryHandler
  participant CFG as configs table
  participant SS as SmartScout
  participant Q as EnrichmentPipelineQueue

  EB->>H: SellerDiscovery.Amazon
  H->>CFG: get enrichment.amazon.disabled
  H->>CFG: get enrichment.amazon.next_start_row
  H->>SS: search(filters, start_from_row)
  SS-->>H: (records, next_start_row)
  H->>CFG: save enrichment.amazon.next_start_row
  H->>Q: publish records in batches of 10

fetch_sellers() returns (records, next_start_row). A returned 0 means the full result set has been consumed and the next run restarts from the top. The cursor is saved before the records are published, so a publish failure does not cause the same page to be re-fetched forever.

Reset the cursor to re-walk from the beginning:

UPDATE configs SET value = '0' WHERE key = 'enrichment.amazon.next_start_row';

Field mapping

SmartScoutSeller → SellerRecord (providers/smartscout/provider.py#_to_seller_record):

SmartScout field SellerRecord field Notes

amazonSellerId

seller_id

The external marketplace id and the natural key

businessName or name

business_name

businessName preferred, falls back to name, then empty string

—

storefront_url

Constructed by make_storefront_url(amazonSellerId)

estimateSales

estimated_monthly_revenue

Defaults to 0.0; a validator rejects negatives

numberAsins

asin_count

Defaults to 0; a validator rejects negatives

primarySubCategory, else primaryCategoryId

primary_category

Sub-category preferred

startedSellingDate

seller_since

Parsed with dateutil; unparseable values are logged at DEBUG and dropped

percentFba

percent_fba

moMGrowth

mom_growth

Month-over-month growth percentage

numberWinningBrands

num_brands

state, country

state, country

whole row

raw

Full provider payload retained on the message

SellerRecord is frozen=True with str_strip_whitespace=True. A row that fails validation is logged at ERROR by map_seller() and skipped — one bad row never aborts the page.

Publishing

QueueService.publish() sends send_message_batch calls of 10 to EnrichmentPipelineQueue.fifo:

SQS field Value

MessageBody

record.model_dump_json()

MessageGroupId

record.seller_id

MessageDeduplicationId

sha256(seller_id + marketplace)

MessageAttributes.event_type

SellerRecord.Received

One FIFO message group per seller is deliberate: grouping by marketplace would serialise the entire pipeline behind two groups, whereas one group per seller lets the scorer run many invocations concurrently while still ordering the messages for any single seller.

The deduplication id is content-derived from the seller identity, so the same seller discovered twice inside SQS’s five-minute deduplication window is published once.

Batch failures are logged with the failed entries; the run continues.

SmartScout access

Authentication is username/password against the SmartScout API, producing a session token that is cached in the shared Secrets Manager secret (SMARTSCOUT_TOKEN, SMARTSCOUT_TOKEN_EXPIRES_AT) rather than in memory. This is why enrichment-scraper is the only function with secretsmanager:PutSecretValue.

Requests go through BaseProvider._post_with_retry(), which handles:

  • Timeouts — up to HTTP_MAX_RETRIES attempts with exponential backoff (HTTP_RETRY_BASE_SECONDS * 2^n), then the page is skipped.

  • 401 — one retry after _on_unauthorized() invalidates the cached token so fresh credentials are used; a second 401 skips the page.

  • 429 — sleeps for Retry-After, or the parsed X-Rate-Limit-Reset, then retries without incrementing the attempt counter.

  • 5xx — logged with a truncated body and the page is skipped.

  • Proactive throttling — when X-Rate-Limit-Remaining drops to 1 or 0, sleeps until X-Rate-Limit-Reset before continuing.

Headers include a browser fingerprint (SMARTSCOUT_BROWSER_HEADERS), a per-request Request-Id UUID, and Content-Type: application/json-patch+json.

Other marketplaces

Detail-type Status Notes

SellerDiscovery.Amazon

live

SmartScout, the only implemented path

SellerDiscovery.Walmart

stub

WalmartDiscoveryHandler raises NotImplementedError. Its EventBridge rule is disabled. Implementation notes are in the module docstring: SmartScout supports other marketplaces via the X-SmartScout-Marketplace header.

SellerDiscovery.RunCategory

stub

RunCategoryHandler raises NotImplementedError. Intended for ad-hoc single-category runs that override ICP_CATEGORIES without waiting for the schedule. Already registered, so only the handler body is missing.

sellers.platform supports amazon, walmart, and shopify; only amazon and walmart are mapped from discovery (PLATFORM_MAP in the scorer).

Current schedule state

The AmazonDailyDiscovery rule (cron(0 7 * * ? ) — 2 AM ET) is *commented out in lambdas/template.yaml, as is WalmartDailyDiscovery. The only live schedule on this function is BigQueryContactSync. Discovery therefore runs only when invoked manually:

aws lambda invoke \
  --function-name morpheus-sam-enrichment-scraper \
  --payload '{"detail-type":"SellerDiscovery.Amazon","source":"enrichment.scraper","detail":{"marketplace":"amazon","provider":"smartscout","max_pages":5}}' \
  --cli-binary-format raw-in-base64-out /dev/stdout

The downstream EnrichmentPipelineQueue SQS trigger on enrichment-scorer is also commented out, so published messages accumulate on the queue until it is enabled. See EventBridge schedules.