Monitoring an XDP program in production: BPF stats, counters, and drop rate

Understanding your XDP drop rate is critical when operating high-performance network protection at the kernel level, yet many operators discover too late that their monitoring pipeline is missing the most important counter of all. When packets vanish silently between the NIC driver and your application, the only evidence lives inside BPF map statistics—and if you're not collecting them, you're flying blind through DDoS events, troubleshooting phantom packet loss, and second-guessing every firewall rule you deploy.

XDP programs drop packets with sub-microsecond latency, making them extraordinarily efficient for volumetric attack mitigation. But that same speed creates a visibility gap: unlike traditional iptables DROP counters visible in iptables -nvL, XDP operates before the kernel's network stack allocates an skb, so normal packet accounting never fires. The BPF program updates internal map counters, returns XDP_DROP, and the packet disappears—no netfilter hook, no conntrack entry, no tcpdump trace unless you instrument the XDP program itself.

This article walks through why the XDP drop rate blind spot exists, which BPF statistics solve it, and how to wire those counters into a monitoring stack that gives you real-time visibility into what your kernel protection layer is actually doing. Whether you run a single-server game backend or a fleet of edge nodes, closing this gap transforms XDP from a black box into a transparent, auditable firewall you can trust under attack.

Why Traditional Network Counters Miss XDP Drop Rate

Traditional Linux network observability tools—ifconfig, ip -s link, netstat, ss, and even nftables rule counters—all operate after the kernel has allocated a socket buffer (skb) for the incoming packet. XDP hooks fire at the earliest possible point: immediately after the NIC driver's receive ring, before any skb allocation. When an XDP program returns XDP_DROP, the kernel recycles the DMA buffer in-place and never increments the statistics visible to userspace network utilities.

The result is a counter paradox: your NIC's hardware RX counter shows millions of packets arriving, ip -s link show eth0 shows a much smaller RX packet count, and the delta—your actual drop rate—exists only inside the BPF map that your XDP program updates. If you don't explicitly read that map, you have no idea whether you're dropping 10 packets per second or 10 million.

Common symptoms of the blind spot

• Silent overload: CPU usage spikes during a DDoS, but packet counters in your dashboard barely move because most traffic never reaches the kernel stack.
• False negatives: You deploy a stricter XDP rule, attacks stop reaching your application, yet your monitoring shows "no change" because dropped packets aren't tracked.
• Audit gaps: Compliance or incident-response teams ask "how many malicious packets did you block?"—and you have no data.
• Rule-tuning guesswork: You adjust port ranges or rate-limit thresholds and can't measure the impact, forcing you to rely on indirect signals like application latency or player reports.

For production environments—especially B2B SaaS platforms or managed game-server hosting—this visibility gap is unacceptable. Customers expect transparent metrics, and operators need confidence that rules are working as intended. The fix is straightforward: instrument your XDP program to write per-rule, per-action, and per-interface counters into BPF maps, then read those maps from userspace at regular intervals.

Which BPF Statistics Actually Matter

Not all BPF map counters are equally useful. A naive implementation might update a single global "dropped" counter, but that tells you almost nothing when you're running multiple rules across multiple interfaces. A well-designed XDP statistics schema tracks at least three dimensions: action (drop, pass, rate-limit), rule ID, and source metadata (protocol, port, GeoIP when feasible).

Essential counter types

Counter	Scope	Use case
Per-action totals	Global, per interface	High-level health: "eth0 dropped 2.3M pps, passed 450K pps"
Per-rule totals	Each of your 256 rule slots	Rule effectiveness: "Rule 42 (block UDP/19132) triggered 8.1M times"
Per-protocol breakdown	TCP / UDP / ICMP / other	Attack fingerprinting: "99% UDP, likely amplification"
Rate-limit vs. hard-drop	Separate maps or flags	Distinguish transient bursts (rate-limited) from policy blocks
Packet-size histogram	Buckets: 0–64, 65–512, 513–1500, >1500	Detect fragmentation attacks, jumbo-frame abuse, or SYN floods

PAKKT.io implements all five counter types inside the PAKKT Engine's BPF maps. Each rule slot (up to 256 simultaneous rules) maintains its own hit counter, and a separate per-protocol map aggregates totals for TCP, UDP, ICMP, and "other." The agent reads these maps every 30 seconds via bpf_map_lookup_elem, computes deltas, and streams the metrics over mTLS to TimescaleDB for historical graphing and alerting.

Reading BPF maps from userspace

The kernel exposes BPF maps through the bpf() syscall. In Go (the language PAKKT's agent is written in), the cilium/ebpf library provides a clean API:

import "github.com/cilium/ebpf"

spec, _ := ebpf.LoadCollectionSpec("pakkt_engine.bpf.o")
coll, _ := ebpf.NewCollection(spec)
defer coll.Close()

statsMap := coll.Maps["pakkt_stats"]
var key uint32 = 42  // rule ID
var value uint64

if err := statsMap.Lookup(&key, &value); err == nil {
    fmt.Printf("Rule 42 matched %d packets\n", value)
}

For manual inspection during troubleshooting, bpftool is invaluable:

bpftool map dump name pakkt_stats
bpftool map lookup name pakkt_stats key 42 0 0 0

Combine this with a cron job or systemd timer, and you have a lightweight pull-based metrics exporter. Push the results to Prometheus, InfluxDB, or—if you're running PAKKT—directly to the centralized TimescaleDB backend where they join GeoIP enrichment, audit logs, and per-port drill-downs on a single real-time dashboard.

Filling the Gap: Building a Complete Monitoring Pipeline

Collecting BPF map stats is only half the battle. To close the XDP drop rate blind spot, you need a pipeline that correlates kernel-level counters with higher-layer signals—nftables rule hits, conntrack table size, application-layer request rates—so you can distinguish legitimate traffic shaping from an ongoing attack.

Pipeline architecture

1. Kernel instrumentation: XDP program writes per-rule counters to a BPF hash map; nftables rules in the inet pakkt table use named counters (counter name "port_25565_accept").
2. Agent collector: Userspace daemon (PAKKT agent, Telegraf + BPF plugin, or custom Go service) reads BPF maps and nft list counters every 10–30 seconds.
3. Metric normalization: Compute per-second rates (delta between reads ÷ interval), tag by interface, rule ID, protocol, and action.
4. Time-series storage: Push to Prometheus, InfluxDB, or TimescaleDB. Retention: 15-minute raw granularity for 7 days, 5-minute rollups for 90 days, hourly rollups for 1 year.
5. Visualization & alerting: Grafana dashboards with panels for XDP drop rate, nftables accept rate, conntrack usage, and top source IPs. Alert when drop rate exceeds baseline by 3σ or crosses an absolute threshold (e.g., 100K pps).

This pipeline surfaces the XDP drop rate alongside every other relevant metric, so when an alert fires you instantly see whether the spike is XDP policy enforcement, a new attack vector bypassing XDP and hitting nftables, or legitimate user growth. The PAKKT centralized panel automates steps 2–5: the lightweight Go agent handles collection, the backend ingests metrics into TimescaleDB, and the web UI renders real-time and historical graphs with GeoIP overlays, no Grafana configuration required.

Correlating XDP and nftables layers

Because PAKKT runs XDP and nftables in tandem—stateless L2/L3 filtering in XDP, stateful conntrack and TCP flag validation in the isolated inet pakkt table—you can measure defense-in-depth effectiveness by comparing counters at each layer:

# XDP stats (from BPF map)
Rule 10 (block UDP/19132): 4,200,000 drops/min

# nftables stats (same port, stateful layer)
nft list chain inet pakkt input | grep 19132
tcp dport 19132 ct state new limit rate 100/second accept  # counter: 6,000 packets

# Interpretation:
# 4.2M pps blocked at XDP (no CPU cost), 6K/min passed to nftables,
# 100/sec accepted after rate-limit = 6K/min, zero reached application.
# Attack mitigated entirely in-kernel.

This layered visibility is impossible with a single-technology approach. Pure XDP solutions lack stateful context; pure nftables solutions burn CPU processing millions of packets that XDP would have dropped in nanoseconds. The combination—monitored correctly—gives you both speed and precision.

Interpreting XDP Drop Rate in Production

Raw packet counts mean nothing without context. A game server seeing 500K pps dropped might be under siege; a high-traffic web API might consider that normal scan/probe background noise. Effective interpretation requires baselines, anomaly detection, and an understanding of your application's legitimate traffic profile.

Establishing baselines

• Idle baseline: Measure drop rate during off-peak hours (3–5 AM local time). Anything above this is either user growth or attack traffic.
• Peak legitimate load: Capture metrics during your busiest hour (Saturday evening for games, Monday 9 AM for SaaS). This is your "good traffic" ceiling.
• Protocol ratios: Legitimate game traffic might be 95% UDP; a web API is 98% TCP. Sudden shifts (e.g., UDP spiking to 60% on a web server) flag an attack.
• Packet-size distribution: SYN floods cluster at 40–60 bytes; DNS amplification at 512+ bytes; legitimate game state updates at 100–200 bytes. Histogram shifts reveal attack type.

Once you have baselines, configure alerting thresholds. A simple rule: alert when XDP drop rate exceeds (peak_legitimate × 2) and sustained for >60 seconds. This filters transient bursts (a streamer's audience joins simultaneously) from real DDoS.

Case study: investigating a phantom performance complaint

A PAKKT customer reported "random lag spikes" on their Minecraft server (TCP/25565) every few hours. Application logs showed no errors, CPU and RAM were normal, and nftables counters were steady. The problem was invisible until the operator checked the PAKKT dashboard's XDP drop rate graph and saw sharp 30-second spikes of 1.2M pps dropped on UDP/19132 (Minecraft Bedrock port, which this Java-edition server didn't use). The spikes coincided exactly with the lag reports.

Root cause: a Bedrock-targeted attack was saturating the NIC's interrupt processing, starving the Java server's threads even though XDP dropped every malicious packet before it consumed kernel resources. The fix: the operator added a stricter XDP rule to drop UDP/19132 even earlier in the processing path and enabled NIC RSS (Receive Side Scaling) to distribute interrupts across more CPU cores. Lag spikes vanished. Without XDP drop rate telemetry, this issue would have been unsolvable.

Best Practices for Long-Term XDP Observability

Closing the XDP drop rate blind spot isn't a one-time configuration—it's an ongoing discipline. As your rule set evolves, attack patterns shift, and infrastructure scales, your observability stack must keep pace.

Operational checklist

• Automate map reads: Never rely on manual bpftool checks. Use a systemd timer, cron job, or—better—a dedicated agent that streams metrics continuously.
• Version your BPF programs: When you update the XDP program, map schemas may change. Tag metrics with program version so you can correlate behavior changes with deployments.
• Retain raw data: Store at least 7 days of per-second granularity. Attacks often show pre-cursors (slow ramp-up) hours before the main wave; you need historical resolution to analyze them.
• Correlate with external signals: Join XDP drop rate with GeoIP data (which countries?), ASN data (which networks?), and threat-intel feeds (known botnet IPs?). The PAKKT platform does this automatically, overlaying drop events on a world map and highlighting top source IPs in real time.
• Test your metrics during drills: Schedule a controlled load test (e.g., hping3 --flood from a safe source) quarterly and verify that your XDP drop rate dashboard spikes as expected. If it doesn't, your pipeline is broken.
• Document rule-to-counter mappings: Maintain a table linking each rule ID to its purpose, owner, and creation date. When you see "Rule 73 dropped 900K pps," you should instantly know which policy triggered.

Scaling beyond a single server

If you manage a fleet—game-server hosting, SaaS edge nodes, CDN origins—centralized aggregation is non-negotiable. Each agent must report its local BPF map stats to a central collector, which deduplicates, aggregates, and alerts across the entire fleet. PAKKT's architecture does exactly this: every agent streams metrics over mTLS to a shared TimescaleDB instance, the web UI provides per-agent drill-down and fleet-wide roll-ups, and the audit log records every configuration change so you can trace anomalies back to specific rule edits.

For organizations building their own solution, consider a metrics hierarchy: agent → regional aggregator (Prometheus federation or InfluxDB relay) → global dashboard (Grafana with multiple datasources). Tag every metric with hostname, interface, and rule_id so you can slice by any dimension.

Conclusion

The XDP drop rate blind spot exists because kernel-level packet filtering operates below the visibility layer of traditional network tools. Closing that gap requires deliberate instrumentation—BPF maps that track per-rule, per-protocol, and per-action counters—and a robust pipeline to read, normalize, store, and visualize those metrics alongside stateful firewall data. When implemented correctly, you transform XDP from an opaque performance optimization into a transparent, auditable, and highly effective first line of defense. Operators gain confidence in their rules, incident response teams get the evidence they need, and your infrastructure becomes measurably more resilient under attack.

FAQ

Why doesn't tcpdump show packets that XDP drops?

Tcpdump operates on cloned skbs after the kernel allocates them in the network stack. XDP runs before skb allocation; when it returns XDP_DROP, the packet never reaches the point where tcpdump or any other AF_PACKET consumer can observe it. To trace XDP decisions, use bpf_trace_printk inside your BPF program and read /sys/kernel/debug/tracing/trace_pipe, or better yet, maintain proper BPF map counters and export them to your monitoring stack.

Can I track XDP drop rate per source IP, not just per rule?

Yes, but it requires a larger BPF hash map (source IP as key, counter as value) and careful memory management—tracking millions of unique IPs can exhaust map size limits. A hybrid approach works well: maintain per-rule counters in BPF, then sample top-N source IPs (e.g., update a bounded map with the highest hitters) and export those to userspace for GeoIP enrichment and dashboard display. PAKKT implements this pattern, showing "top source IPs" on the real-time dashboard without storing every unique address in kernel memory.

How do I distinguish between XDP drops due to rate-limiting versus hard policy blocks?

Design your BPF map schema to separate these actions. For example, use distinct map entries for rate_limit_drop and policy_drop, or encode the action type in the counter key. When your XDP program enforces a rate limit (e.g., max_port_pps exceeded), increment the rate_limit counter; when it blocks based on IP blacklist or port policy, increment the policy counter. In userspace, graph these separately so you can see "2M pps rate-limited (transient burst), 300K pps hard-blocked (malicious sources)" and tune thresholds accordingly.