How sync works

The five-minute tick

A single Trigger.dev scheduled task — sync-all-integrations — runs on a */5 * * * * cron. On each tick it queries for integrations whose next_sync_at is null or in the past and status = 'active', then fans out sync-one-integration jobs for each.

Per-brand queues

Every sync-one-integration is placed on a queue named brand:<slug>. Brand-level rate limits (defined in the manifest or adapter) set max concurrency per queue. This is why a slow Casumo sync can't block a fast Binance sync.

One-at-a-time per integration

Each job uses concurrencyKey: integrationId, guaranteeing only one run per integration at any time. If a second run is triggered while one is still active, it queues behind the first.

Idempotency

idempotencyKey = {integrationId}:{minuteTimestamp}de-duplicates any run scheduled within the same minute. If you click Sync now twice within sixty seconds, only one worker actually fires.

What a single run does

1. Fetch the integration row; resolve the adapter via brand.adapter_key.
2. Decrypt credentials via get_integration_credentials() RPC (service-role only).
3. Build an AdapterContext with proxy credentials, logger, and storage helpers.
4. Insert a sync_runs row with status running.
5. Iterate the adapter's async generator, buffering rows.
6. UPSERT all rows into stats_daily on the unique grain.
7. Update integration: status='active', last_sync_at=now(), next_sync_at=now()+interval, last_error=null.
8. Update the sync_runs row: status='success', rows_ingested, duration_ms.

Failure modes

Soft failure (retryable)

Network error, 5xx from the brand, transient timeout. Trigger.dev retries twice more with exponential backoff (5s → 10s → 20s). If all three attempts fail, the run lands in failed state and the integration is marked error. The next scheduled tick picks it up normally.

Hard failure (not retryable)

Credentials rejected, brand schema changed, adapter throws. The run fails immediately, last_error gets the human-readable message, and the integration detail page shows it. For scraper failures, we capture a screenshot and the full HTML so you can diff against a known-good state.

Late-settling data

Every sync fetches the last 7 days, not just since the last successful run. Affiliate programs routinely back-date FTDs and settle chargebacks 2–5 days later. The UPSERT on (integration_id, date, campaign, source, sub_id)means numbers for a given day keep updating until they stabilise — no deletes, no manual backfill.