Incident Lab · SEV1 · Airflow
Retried batches are doubling orders, and the pipeline is still running
Finance: the hourly revenue dashboard is above the payment provider and climbing. load_orders retried a batch and keeps running every 2 minutes.
[2026-10-07 03:00:41] WARN - Task load_orders batch 2 timed out, retrying (1/3) ... SUCCESS
This incident is live. The scheduler delivers a new batch every 2 minutes through load orders , and some batches are retried. The map above shows what's broken right now, and it changes as batches land. Each batch runs INSERT INTO fct orders SELECT FROM load orders . load orders passes on everything in incoming orders , so a retried batch is loaded twice. hourly revenue (the revenue dashboard) and customer statements read fct orders . provider orders is what the payment provider reports: the source of truth. Resolve it in three parts. Your script runs after the batches that have already landed: 1. Fix the pipeline: rewrite load orders so a batch only inserts orders that aren't in fct orders yet. 2. Repair the damage: delete the extra copies already in fct orders (keep one row per order id ). 3. Prove it: Check my work re-runs a batch and delivers a retried new one. Press Run any time to see the map update.