1500 != 1500: MTU, OSPF ExStart, and a 14-Byte Blind Spot

In 2013 I designed my first proper redundant border architecture — two IOS-XR ASRs, two NX-OS Nexus 7Ks, OSPF area 0 running the full square, eBGP to two independent ISPs. Clean topology. Replaced a serialized single-point-of-failure chain I’d been living with for years. The adjacency never came up — an OSPF MTU mismatch between IOS-XR and NX-OS kept it stuck in ExStart for two weeks.

The Architecture

The topology was straightforward: two ASRs each terminating an independent ISP handoff (LEARN and Grande Communications), two Nexus 7Ks in core positions, all four devices forming a square with OSPF area 0 stitching it together. iBGP between ASRs. eBGP to the ISPs. Full 10G throughout.

I designed it. A colleague co-executed the build. The intent was a redundant active/active border — each ISP path independently routable, core fully meshed, no single device failure bringing down external connectivity.

The design was sound. The assumption baked into it was not.

The Assumption

MTU is 1500. You set 1500 on both sides. They agree. Adjacency forms.

That’s how I thought about it. That’s how I’d always thought about it, on platforms where it was true.

IOS-XR calculates MTU from a Layer 3 perspective — the IP payload. NX-OS calculates from a Layer 2 Ethernet frame perspective, including the header. Same number in the config. Different number on the wire. The platforms were exchanging database description packets in OSPF ExStart/Exchange with MTU values that didn’t agree, and the adjacency stalled.

The difference is 14 bytes — the Ethernet header that IOS-XR excludes and NX-OS includes. This OSPF MTU mismatch is not a bug — it is a platform model difference that requires explicit configuration to bridge.

Two Weeks on Static Routes

OSPF ExStart adjacency stuck in ExStart is a specific failure mode. The symptom is visible — neighbors appear, reach ExStart, stall, reset, repeat. But the cause isn’t obvious from the symptom alone, especially when the MTU values in the configs match.

Diagnosing it required bridging two internal teams that didn’t normally work together — the NX-OS team and the IOS-XR team — and eventually opening a TAC case to reach the cross-platform conclusion: the configs agreed, but the wire didn’t.

The fix was one line of configuration once we understood the offset. Set equivalent on-the-wire MTU values on each platform. Adjacency formed immediately.

We were on static routes for those two weeks — functional, but fragile and unsustainable. The redundant architecture I’d built was sitting inert behind a config line I didn’t know I needed.

OSPF MTU Mismatch: Why IOS-XR and NX-OS Disagree

The tempting lesson is “always check MTU.” That’s too shallow.

The actual lesson is that adjacent platforms from the same vendor can have fundamentally different models of the same concept. IOS-XR and NX-OS are both Cisco. They both use 1500 as the default MTU value. They count that 1500 differently. That boundary — same vendor, different platform lineage, different internal model — deserves explicit verification before you declare an adjacency design sound.

I didn’t know to ask the question. The assumption that MTU was a universal value was invisible to me because it had always been true on the platforms I’d worked on before. The failure wasn’t a bug, and it wasn’t an execution error. It was an architectural assumption I baked in at design time that I didn’t know to question.

That’s a harder class of failure to catch — and a more useful one to have on record.

Stakeholder Reality

Two weeks is a long time to explain “static routes” to people who expected a redundant architecture to be operational. I owned those conversations. The answer was always honest: we have connectivity, it’s stable, we’re diagnosing a routing protocol issue at the border, here’s what we know today.

The architecture came up intact on the other side. But the more durable output was the discipline of explicit verification at platform boundaries — a practice that’s been part of every cross-vendor design I’ve done since. For the full engagement context, see the resume.