Incident Lab · SEV1 · Databricks
Retried batches are doubling trips, and the pipeline is still running
Ops: the hourly gross bookings dashboard is above the driver payout file and climbing. load_trips retried a batch and keeps running every 2 minutes.
[2026-10-07 03:00:41] WARN - Task load_trips batch 2 timed out, retrying (1/3) ... SUCCESS
This incident is live. The scheduler delivers a new batch every 2 minutes through load trips , and some batches are retried. The map above shows what's broken right now, and it changes as batches land. Each batch runs INSERT INTO fct trips SELECT FROM load trips . load trips passes on everything in incoming trips , so a retried batch is loaded twice. hourly bookings (the gross bookings dashboard) and driver statements read fct trips . provider trips is what the driver payout file reports: the source of truth. Resolve it in three parts. Your script runs after the batches that have already landed: 1. Fix the pipeline: rewrite load trips so a batch only inserts trips that aren't in fct trips yet. 2. Repair the damage: delete the extra copies already in fct trips (keep one row per trip id ). 3. Prove it: Check my work re-runs a batch and delivers a retried new one. Press Run any time to see the map update.