An ARP TTL FDB aging mismatch turned a data center into a light show. Around 2005, every top-of-rack switch in the data center was blinking like a light show at a disco rave. I traced the cause back to a server doing exactly what its owner designed it to do. The failure wasn’t a misconfiguration, but two independent aging timers (ARP on the routed interface, FDB on the switch) that had no reason to agree, and one host quiet enough to expose the gap between them.
The Symptom
Sysadmins started calling before I got there, panicked by email notifications reporting lost reachability to critical services on several L2 neighbors across the subnet. On the floor: every rack-top light blinking in perfect sync, 100dB of CRAC white noise, no audible alarm, nothing else visibly wrong. That synchronized blink is the visual signature of a Layer 2 flood, a signal every network engineer soon learns to hate. It’s how the network shows you that every port is pushing traffic at once because the switch has stopped forwarding selectively and started forwarding everywhere.
Classic broadcast storm presentation. Cause unknown.
Finding the Destination, Not the Source
I patched a laptop directly into the server network and started a packet capture. The pattern that stood out wasn’t a chatty source flooding traffic. It was a single destination MAC address absorbing disproportionate volume. That’s a different diagnostic thread than the usual “which host is spamming the network” hunt. Something was trying, repeatedly and unsuccessfully, to reach one address.
I’d been running cron-scheduled ARP and FDB polling into SQL for months by that point. It was not built for this incident, just standing operational hygiene. That corpus turned a MAC address into an IP address into a hostname in minutes instead of requiring a live query against production switches mid-storm. The IP resolved to a syslog server a junior infosec admin had stood up days earlier.
ARP TTL FDB Aging: Two Timers That Never Agreed
The admin had built the server deliberately minimal: a passive UDP receiver, no outbound traffic of any kind, explicitly to shrink its attack surface. From a confidentiality-and-integrity standpoint, that’s a reasonable design. It just collided with a piece of Ethernet mechanics that has nothing to do with security posture.
Two independent timers govern reachability to any host on a routed L2 segment:
- ARP cache TTL, on the routed interface, how long the router trusts its own IP-to-MAC mapping for that host before re-resolving it.
- FDB (forwarding database / CAM table) aging timer, on the switch, how long the switch remembers which port a given MAC address lives behind before it ages the entry out.
These timers are configured independently, on different devices, with no protocol coordinating them. Ethernet switches learn and age out MAC-to-port bindings dynamically (IEEE 802.1Q bridge learning). There’s nothing in the IEEE standard that enforces ARP TTL <= FDB TTL. In this case, they weren’t aligned, and the router’s ARP entry for the syslog server significantly outlived the switch’s FDB entry for the same host.
A host that only receives traffic, without ever originating any, never sends a frame that would let the switch refresh its FDB entry. When that entry aged out, the switch had no record of which port the destination MAC lived behind. Its only remaining option, per standard flood-and-learn behavior, was to flood every frame destined for that MAC out every port on the VLAN, hoping the destination would respond and let the switch relearn its location.
The destination never responded, by design. The flood was never going to resolve itself until the router’s ARP entry itself expired and traffic to that host stopped being generated in the first place, or until someone intervened.
Why “Optimized for Confidentiality” Is the Honest Framing
It would be easy to call this a rookie mistake and leave it there. That’s not quite fair to what actually happened. The admin’s design was internally consistent for the threat model he was optimizing against: minimize the server’s footprint, minimize what it exposes, minimize what it can be used for if compromised. Nothing in that reasoning is wrong.
The failure was an emergent interaction between that design and switching behavior that has nothing to do with security. FDB aging exists to keep forwarding tables from growing unbounded and stale, not to police host behavior. A passive-receiver host is invisible to a mechanism that assumes every host on the network occasionally speaks.
In optimizing for confidentiality, he’d left availability off the checklist he was working from.
The Fix, in Two Layers
He wasn’t willing to change the server’s network posture. I can understand his reasoning, given that the minimal posture was the whole point of the design. I didn’t force the issue. Instead:
Immediate: a cron job pinging the syslog server every three minutes. Each ping forces an ICMP reply, and that reply is enough to refresh the switch’s FDB entry before it ages out. No change to the server. No change to its threat model. Just enough traffic, at a low enough frequency, to keep the switch’s table current.
Durable: aligned ARP TTL on client-facing routed interfaces to be less than or equal to FDB TTL on the corresponding VLAN, campus-wide. If the router re-resolves a host’s MAC no less often than the switch might forget where that MAC lives, the router’s own re-resolution traffic keeps the FDB entry alive as a side effect. With that alignment, any future client that goes silent will not elicit that same failure mode.
The Generalizable Check
If you’re auditing a network for this failure class, the question isn’t “do we have any weird hosts.” It’s a config comparison: what’s the FDB/CAM aging timer on your access switches, and what’s the ARP cache timeout on the routed interfaces serving those VLANs? If the router’s timer is longer, any host quiet enough can trigger the same flood, and it will look identical every time: synchronized port activity, no source anomaly, a destination MAC nobody’s actively chasing. In this case, it was a one-way log receiver, but it could also happen with a passive monitor, or certain appliance management interfaces that don’t often emit packets.
The parallel to an OSPF adjacency stuck on a platform-MTU assumption is direct: two independently-configured systems, each internally consistent, silently disagreeing at a boundary nobody thought to check. The ARP TTL FDB aging comparison takes minutes to run and closes an entire class of failure that otherwise only surfaces when a host happens to be quiet enough, for long enough, on a busy enough segment.
For the full engagement context, see the resume.
Leave a Reply
You must be logged in to post a comment.