Incident Lab · SEV1 · dbt
Retried batches are doubling payments, and the pipeline is still running
RevOps: the hourly collected revenue dashboard is above Stripe and climbing. stripe_payments_sync retried a batch and keeps running every 2 minutes.
[2026-10-07 03:00:41] WARN - Task stripe_payments_sync batch 2 timed out, retrying (1/3) ... SUCCESS
This incident is live. The scheduler delivers a new batch every 2 minutes through load payments , and some batches are retried. The map above shows what's broken right now, and it changes as batches land. Each batch runs INSERT INTO fct payments SELECT FROM load payments . load payments passes on everything in incoming payments , so a retried batch is loaded twice. hourly collected (the collected revenue dashboard) and account statements read fct payments . provider payments is what Stripe reports: the source of truth. Resolve it in three parts. Your script runs after the batches that have already landed: 1. Fix the pipeline: rewrite load payments so a batch only inserts payments that aren't in fct payments yet. 2. Repair the damage: delete the extra copies already in fct payments (keep one row per payment id ). 3. Prove it: Check my work re-runs a batch and delivers a retried new one. Press Run any time to see the map update.