Tag: incident response
-
When the Fix Becomes the Failure: ECMP, Zone Protection, and a 64KB Ceiling
Three ECMP firewall cutovers went cleanly. The fourth did not — and the cause turned out to be Palo Alto MSS clamping, hidden inside a zone protection profile that had passed through three clean rollouts undetected. It was the highest profile pair in the sequence, sitting at the data center boundary. The Setup The Baylor…
-
BGP Prefix Leak, RPKI, and the Cold Email That Confirmed It
A BGP route leak at an Internet2 customer site propagated more-specific prefixes into the global table, causing every major CDN to black-hole return traffic to an entire campus network. Diagnosing it required a cold email to a Google network engineer at 8:40am. This is the full story: the triage, the root cause, and the architectural…