A Hit Counter Doesn’t Know When It Started Counting

A rule that has matched nothing in nine minutes and a rule that has matched nothing in eighty days look exactly the same in the output. I wrote about that moment on LinkedIn; this post goes deeper into the toolchain around that observation: three small Perl tools written to consolidate a set of independently managed Internet-boundary firewalls into a single Panorama device group.

The consolidation itself was mechanical. The interesting question was not how to lift the rules, but which ones deserved lifting. Answering that meant finding the unused firewall rules in a decade of accumulation, and finding them in a way a reader could trust.

Each of the three tools ended up declaring the boundary of its own validity. Each did it by a different mechanism. There was no standard to guide me as I crafted these tools.

Tool one: the generator, and a scope filter compiled into the code

The first tool reads a standalone firewall’s rule configuration and emits the Panorama set commands that recreate those rules inside the target device group.

It skips any rule whose zone name begins with a fixed internal-facing prefix. That skip was not a shortcut. The audit scope was the Internet-facing boundary, and internal-facing rules were explicitly out of scope for this migration, so the tool refuses to emit them at all rather than emitting them and trusting somebody downstream to filter.

The practical effect is that the output cannot silently exceed its own scope. A reader looking at a set-command file knows what class of rule was never a candidate, because the exclusion lives in the generator rather than in somebody’s memory of the conversation where scope was agreed. That is the first mechanism: a scope filter, stated in code.

Tool two: the expander, and a fault list instead of a fatal error

Set commands are a good input format for a device and a poor one for a human trying to decide what to keep. The second tool reads that same set-format configuration back and flattens it into something auditable.

Flattening meant several normalizations at once. Addresses arrive in four shapes – ip-netmask, ip-range, fqdn, and any – and the tool normalizes all four to CIDR so that two rules covering the same space compare as equal instead of comparing as different strings. Address groups resolve recursively, because a group containing a group containing a host is common in a rulebase nobody has pruned. It preserves rule order by configuration line number, since in a firewall the order is part of the meaning.

The design decision worth naming is the failure handling. Malformed lines do not stop the run. The tool accumulates them in a failure list, and that list is reported at the end of execution alongside the successful output. That is the second mechanism: a fault accumulator.

The difference matters more than it sounds. A tool that dies on the first unparseable line produces nothing, and the operator fixes one line and runs again, repeatedly. A tool that accumulates produces a complete pass plus an explicit inventory of what it could not read. The second form tells you the size of your blind spot in one run. The first form only ever tells you about the next line.

The blind spots that remain are real, and they were authored rather than discovered:

• Addresses and groups were expanded. Services and applications were not. The output tells you which address space a rule covers. It does not tell you which ports or applications it permits, and no part of this toolchain ever claimed to.
• negate-source and negate-destination are captured into columns but never applied to the expanded address list. A negated rule therefore reads, in the flattened output, as though it covers the space it actually excludes. This is a known defect, not a subtlety.
• Only the post-rulebase is read. Pre-rulebase and device-local rules are absent from the output entirely, and nothing in the tool warns about it. This silently matched the shop discipline of implementing policy in post-rulebase only.

I list those here for the same reason the tools list their own boundaries: an audit artifact should say what it failed to examine, else it is worse than no artifact, because it invites the reader to assume coverage that was never there.

Tool three: establishing the observation window before asking the question

The third tool is the one the LinkedIn post is about. It queries the PAN-OS XML API for rules with no hit count: the unused firewall rules the consolidation was trying to identify.

The API answers that question directly. It will tell you which rules are unused. It will not tell you how long it has been watching, and that second number is the one that decides whether the first number means anything.

So the tool asks the second question first, before it reads a single rule:

• On a firewall in a high-availability pair, it retrieves the length of time that node has been in the active state.
• On a standalone firewall, it parses system uptime out of the operational response — the familiar 80 days, 14:04:32 shape — and uses that.
• On a passive node, it refuses to run at all. Hit counters on a device that has not been forwarding traffic are not evidence of anything. Why report on data that has no significance?

Every row it emits carries the start of its own observation window in a column beside the rule. That is the third mechanism: an explicit observation window, attached to the finding rather than held separately.

Two assumptions sit underneath it and should be stated. The tool assumes hit counters reset on a high-availability state transition; if they in fact persist across one, the tool understates the derived window and it is being conservative in a direction it fails to announce. Another assumption it makes is in posting credentials to the management interface with certificate validation disabled. This was a scoped decision at the time, on a small internal estate with self-signed management certificates and no path off-box, but a decision worth naming out loud rather than leaving to be discovered by someone reading the source later.

Why the window column stopped being fastidious

That column looked like over-engineering the day it was written. The audit was a one-off. I knew perfectly well when the firewall had last rebooted, because I had been there.

A descendant of that tool later ran as a Friday morning cron job, mailing a weekly unused-rules report to a distribution list of security analysts. This cron kept going for years. The script that ran in cron was not the file I still have. It was a later version, and I no longer have access to precisely how it differed.

That is the situation the column was accidentally built for. In a one-off audit, a human is holding the context: the reboot on Tuesday, the failover last month, the maintenance window that reset everything. In week ninety-four of a standing report, nobody is holding anything. The only thing standing between a reader and a page of false positives is a column stating how long the counter had been counting.

To be precise about what I know here: The report was produced, it was sent, and it was tolerated for years. Whether anyone acted on it, whether rules were retired because of it, and whether the rulebase ever got smaller … I do not know, and delivery is not consequence.

The same shape, twice

This is the second time a tool I built to answer a question I had once ended up as another team’s standing process. The first was moving threat enforcement off the firewall policy cycle and onto the data plane with BGP, which infosec then routed border-host enforcement through for the rest of my tenure.

Neither was assigned. Neither was announced. In both cases the adoption happened because the output was useful enough that somebody else wired it into their week, which is a slower and more honest signal than a launch.

Of these three mechanisms – scope filter, fault accumulator, observation window – what made the rule audit tool usable after I stopped watching it was its third mechanism, not the first two. If the report’s reader is present, the first two tell you what the tool skipped or what it choked on, while you are sitting there reading the output. For the reader who has no idea where the report comes from or even that it existed before arriving in their inbox, the observation window provides meaningful context. All three tools state what they actually checked; only one of them still says it to somebody who was never told what the tool was for.

Which of your standing reports could state its own observation window, and which are asking the reader to conjure up context out of nowhere?