Three ECMP firewall cutovers went cleanly. The fourth did not — and the cause turned out to be Palo Alto MSS clamping, hidden inside a zone protection profile that had passed through three clean rollouts undetected. It was the highest profile pair in the sequence, sitting at the data center boundary.
The Setup
The Baylor core team had spent several years retrofitting campus firewall pairs into an ECMP architecture: active/passive Palo Alto units integrated into the Cisco core via dual uplinks, weighted equally, with near-seamless failover on link loss.
Before the first cutover, a colleague raised a concern that shaped every subsequent deployment: ECMP asymmetry is a silent killer for stateful firewalls. When return traffic can hash to a different uplink than the outbound session, the firewall that receives it has no session state for it. The connection drops, silently, with no obvious routing artifact. The fix for this (“symmetric return”) was only available via a zone protection profile. So we applied one from the start, on all four pairs — with Palo Alto MSS clamping settings we hadn’t fully reviewed.
The first three cutovers generated no performance complaints. The fourth pair was different in one meaningful way: it carried the datacenter boundary traffic, including high-bandwidth connections to a key cloud provider. That distinction didn’t register as a risk factor at the time. It should have.
The Failure Sequence
Cutover triggered immediate complaints. Latency to the cloud provider spiked. Departmental escalations followed within hours. Senior leadership was watching.
Initial triage followed the obvious thread: routing and path symmetry. We shared traceroutes with the ISP, requested inbound traceroutes from the cloud side, and temporarily disabled ECMP by adding higher metrics to one uplink to enforce single-path routing. None of it helped. Performance degraded further.
Palo Alto support engaged. We surfaced the zone protection profile as a potential contributor. The SE was direct: MSS clamping is not default behavior of zone protection profiles. The team accepted that and kept looking. All the diagnostics on routing tables, traceroutes, path symmetry continued to come back clean, and the vendor had just told us the profile wasn’t the issue.
That assurance held the team in the wrong frame for longer than it should have.
What iperf3 Showed
I ran iperf3 independently, working outward from the affected department. This was key. Not between two known endpoints, but hop by hop through the infrastructure stack, testing paths in both directions.
The standard iperf3 output includes a column most people skip past: the receiver window (RWND). On high-performing paths, RWND was reporting 4MB. On degraded paths, it was reporting 64KB: consistently, regardless of endpoint, regardless of direction.
That is not a routing symptom. That is a TCP symptom.
RWND is a function of MSS (Maximum Segment Size) negotiated during the TCP three-way handshake. MSS is fixed for the life of a connection. If the negotiated MSS is small, the receive window is small, and TCP’s congestion control algorithm throttles the sender to match. No amount of available bandwidth compensates for a ceiling on what the receiver will accept per acknowledgment cycle.
The hop-by-hop approach was what surfaced this. Testing between two well-separated endpoints tells you there is a problem. Testing hop by hop tells you where the constraint is being introduced. The 64KB RWND appeared as soon as traffic transited the firewall pair. Everything upstream of it was fine.
The zone protection profile had an MSS clamp. The SE had told us that wasn’t default behavior. What we hadn’t done, and perhaps should have done before accepting that assurance, was read the profile and verify empirically what it was actually configured to do. Reverting the profile to default restored throughput immediately.
Why This Didn’t Surface on the First Three Pairs
The zone protection profile was applied to all four pairs. The MSS clamp was present on all four. It did not generate complaints on the first three.
Three clean cutovers with a misconfigured default in the profile is not evidence the profile was correct. It is evidence the prior environments were not sensitive enough to expose it.
High-bandwidth, long-distance connections (what the networking literature calls Long Fat Networks [LFNs]) are precisely the environment where TCP window scaling matters most. Short connections, low-bandwidth transfers, and local traffic patterns can tolerate a constrained MSS without visible degradation. The datacenter boundary pair was the first in the sequence where the cloud-provider traffic volume made the constraint immediately visible at scale.
The Diagnostic Thread: How Palo Alto MSS Clamping Finally Surfaced
The routing and symmetry frame was not unreasonable. ECMP asymmetry is a real failure mode for stateful firewalls, which is exactly why we had the zone profile in place. But once throughput collapsed to on-campus resources with no cloud-provider involvement, it was clear something at the TCP layer was wrong everywhere. A zone protection profile does not affect routing to a cloud provider.
The SE’s assurance was the point where the diagnosis stalled. The correct response to that assurance was to read the profile contents and run a controlled TCP test against a path that transited the firewall. But that is hindsight. We did not do that. We took the assurance at face value and kept pulling on threads that had no answer.
The RWND column in iperf3 output is not obscure. It is in the default output format. It just does not register as a diagnostic signal until you know what to look for, and the connection from RWND to MSS to the three-way handshake to a profile-level clamp is not a short chain. The team was looking at routing tables and traceroutes. I was looking at iperf3 output, hop by hop, asking what a consistent 64KB ceiling meant and where it was being introduced.
The diagnostic initiative worth naming is not that I found it. It is that I looked at a different layer when the obvious layer had no answers, and that I kept moving when a vendor assurance had stopped everyone else.
On the Zone Protection Profile
The zone protection profile was the right tool for the right problem. Symmetric return for ECMP stateful firewalls is a legitimate requirement, and the profile addressed it correctly on that axis. The Palo Alto MSS clamping option within the profile is configurable, not an inherent design flaw.
The failure was not the profile. The failure was not verifying what the profile contained before applying it across four production pairs, and not testing end-to-end TCP behavior on a representative high-bandwidth path after each cutover. The first three pairs didn’t expose the latent misconfiguration. The fourth did.
I authorized the deployment of the profile on all four pairs. That is the decision I own. The lesson is not “don’t trust vendor recommendations.” It is “read the configuration before you apply it, verify vendor assurances empirically before accepting them, and test TCP behavior specifically (not just routing) after any firewall policy change on a high-bandwidth path.”
What to Check When Throughput Collapses Post-Firewall-Change
If you see throughput degradation immediately following a firewall policy change and routing looks clean:
Run iperf3 hop by hop, not just end to end. End-to-end testing confirms a problem exists. Hop-by-hop testing localizes where the constraint is being introduced. A uniformly small RWND value across paths that transit a specific device is a strong signal that MSS clamping is occurring at that device.
Check the RWND column specifically. A consistent value (such as 64KB, a common Palo Alto MSS clamping signature) across multiple paths and directions points to MSS negotiation being constrained at the handshake. This is distinct from bandwidth or latency problems, which vary by path.
Read every profile applied during the change window. Zone protection profiles, security profiles, and DoS protection profiles can all include MSS-related settings. Intent and contents are not the same thing. Review what was actually configured, not what the profile was meant to do.
Check whether degradation affects new connections only or existing sessions as well. MSS is negotiated at handshake time and fixed for the connection’s life. If pre-change sessions are performing normally and post-change sessions are degraded, the cause is in the handshake path.
Verify vendor assurances empirically before accepting them. A vendor SE telling you a feature is not default behavior is a starting point, not a conclusion. Read the profile. Run a controlled test. The assurance may be correct in general and wrong for your specific configuration.
Revert before adding further changes. We added a metric change to disable ECMP before we understood the problem. That created more variables to unwind and contributed to the delay in finding the actual cause. Revert to a known-good state first, then investigate from there.
For the full incident timeline and engagement context, see the resume.
Leave a Reply
You must be logged in to post a comment.