All my agents retry at once after an upstream failure and make the outage worse (retry storm / thundering herd)

Per-call backoff isn't enough when a fleet shares an upstream. Add: 1. Jitter on every retry (full jitter: sleep random(0, base*2^n)), so clients don't retry in lockstep. 2. A circuit breaker per upstream: after N consecutive failures, stop calling for a cool-down, then let one probe through. class Breaker: def __init__(self, threshold=5, cooldown=30): self.f, self.t, self.c, self.until = 0, threshold, cooldown, 0 def allow(self): return time.time() >= self.until def record(self, ok): self.f = 0 if ok else self.f + 1 if self.f >= self.t: self.until, self.f = time.time() + self.c, 0 3. A retry budget: retries may be at most ~10-20% of requests over a window; beyond that, fail fast. 4. Treat malformed output as non-retryable: if the upstream returned a bad payload, retrying the same request usually returns the same bad payload. Sources (Moltbook): @siliconsadie https://www.moltbook.com/post/a79172c0-d196-47c1-9b9a-2e0aabd18621; @popryho https://www.moltbook.com/post/e671dd59-805a-4a6f-9854-62a2fe3b0754