Debugging an XDP Program That Doesn't Match: Complete Checklist

When your XDP program debug session reveals that packets are sailing past your filter unchanged, the frustration is real: you've compiled the eBPF object, attached it to the interface, yet traffic still floods in as if nothing were there. This checklist walks you through the kernel-level verification steps, map inspection commands, and nftables cross-checks that pinpoint why your XDP logic isn't matching—and how to fix it fast.

XDP operates at the earliest possible point in the network stack, which means traditional packet-capture tools and userspace logs often miss the action entirely. A single misconfigured map key, an off-by-one endianness mistake, or a silent program detachment can render hours of careful rule design useless. The good news: systematic debugging uncovers the culprit in minutes once you know where to look.

Verify the XDP Program Is Actually Attached

Before diving into map contents or BPF bytecode, confirm the kernel has your program loaded and bound to the correct interface. Detached programs produce zero log output and zero packet verdicts.

Check with ip link

ip link show dev eth0

Look for the xdp or xdpgeneric flag in the output. If absent, the program never attached—or was silently unloaded by a competing process. Native XDP (xdpoffload or driver mode) delivers the best performance; generic XDP runs in the kernel but bypasses hardware acceleration.

Inspect with bpftool

bpftool net show dev eth0

This command lists the program ID, section name, and attach mode. Cross-reference the program ID against bpftool prog show to ensure it matches your freshly compiled object file. If you see a stale ID from a previous deploy, detach it explicitly:

ip link set dev eth0 xdp off

Then reattach your current binary. On production systems running PAKKT.io, the Go agent handles atomic program replacement and persists the correct object across reboots, eliminating manual reattachment errors.

Kernel Log and Verifier Errors

dmesg | tail -50

The eBPF verifier rejects programs that dereference NULL pointers, exceed complexity budgets, or access packet data beyond data_end. A failed load leaves no program attached and often prints a dense multi-line error. Pay special attention to line numbers; correlate them with your source using llvm-objdump -S on the .o file.

Inspect BPF Map Contents and Key Encoding

XDP programs are stateless by design; all dynamic configuration lives in BPF maps. If packets aren't being dropped or rate-limited, the first suspect is a map lookup miss—either the key doesn't exist or its byte order is wrong.

Dump the Map

bpftool map dump name pakkt_rules

Each entry shows the key (often a composite struct of IP, port, protocol) and value (action, rate-limit counters). Verify that:

• The map is not empty. An empty rules map means every packet hits the default policy (usually XDP_PASS).
• Keys match your intended filter. For example, a rule targeting TCP port 25565 must encode the port in network byte order (big-endian). On little-endian x86, 25565 decimal is 0x63dd, which becomes 0xdd63 in host order—confirm your userspace control plane calls htons() before map update.
• Protocol numbers are correct: TCP = 6, UDP = 17, ICMP = 1. A typo here silently ignores all traffic.

Manual Map Update for Testing

bpftool map update name pakkt_rules key 0x0a 0x00 0x00 0x01 0x00 0x06 0x63 0xdd value 0x01 0x00 0x00 0x00

This example adds a rule for IP 10.0.0.1, protocol 6 (TCP), port 25565 (hex 0x63dd in network order), action 1 (block). If packets from that tuple suddenly drop, your XDP logic is sound—the issue was stale or missing map data. Conversely, if behavior is unchanged, the program's lookup or comparison code needs review.

Per-CPU vs. Global Maps

Per-CPU hash maps store separate values on each core, critical for lock-free rate-limiting. When dumping a per-CPU map, bpftool prints one value array per key. Sum the counters manually to see aggregate packet counts. If you expect rate-limit enforcement but the counter never increments, the map type might be wrong or the program isn't writing to it.

Validate Packet-Matching Logic and Bounds Checks

Even when the program is attached and maps are populated, bugs in the C code itself can cause silent match failures. The eBPF verifier enforces strict pointer arithmetic; any path that might read beyond data_end is rejected at load time, but logical errors still slip through.

Common Pitfalls

• Ethernet + IP header parsing: Failing to check data + sizeof(struct ethhdr) + sizeof(struct iphdr) <= data_end before dereferencing crashes the verifier. Always guard every header access.
• VLAN tags: If the NIC or an upstream switch adds 802.1Q tags, the Ethernet type field shifts by four bytes. Parse ETH_P_8021Q and skip the VLAN header before reading IP.
• IPv6 vs. IPv4: Hardcoding ETH_P_IP ignores IPv6 traffic entirely. Check both htons(ETH_P_IP) and htons(ETH_P_IPV6) if your rules should apply to both families.
• Fragment handling: XDP sees only the first fragment by default. If your rule matches on layer-4 ports, fragmented packets (offset > 0) lack a TCP/UDP header and will miss. Decide policy: drop all fragments, or pass them and rely on conntrack reassembly in nftables.

Use bpf_trace_printk for Live Inspection

bpf_trace_printk("Protocol: %d, Port: %d\n", ip->protocol, ntohs(tcp->dest));

Messages appear in /sys/kernel/debug/tracing/trace_pipe. This low-overhead printf-style debugging confirms which branch your packet takes. Avoid it in production—each call has a small but non-zero cost—but for XDP program debug sessions it's invaluable.

Cross-Check with tcpdump

tcpdump -i eth0 -nn 'tcp port 25565'

If tcpdump sees packets that your XDP program should have dropped, the XDP return code is XDP_PASS instead of XDP_DROP. Review your map lookup: a failed lookup often defaults to pass-through. Conversely, if tcpdump sees nothing, either the packets never arrived or XDP successfully dropped them—correlate with interface RX counters.

Coordinate XDP with nftables Stateful Rules

Modern deployments layer XDP's stateless high-speed filter with nftables' connection-tracking engine. Misalignment between the two causes packets to slip through or legitimate flows to break mid-session.

Check the nftables Ruleset

nft list table inet pakkt

PAKKT organizes rules in an isolated inet pakkt table to avoid collisions with Docker, fail2ban, or legacy iptables-persistent chains. Verify that:

• Base chain priority is set correctly. type filter hook input priority filter; policy accept; runs after conntrack (priority -200) so ct state established,related accept works.
• Port-specific rate-limits use the meter construct: tcp dport 22 meter ssh_limit { ip saddr limit rate 10/minute } accept. A missing meter silently accepts all packets.
• TCP flags validation catches SYN floods: tcp flags & (fin|syn|rst|ack) == syn ct state new limit rate 100/second accept.

Synchronize IP Lists Across Layers

If an IP appears in the XDP blacklist map but not in the nftables pakkt_blacklist set, stateful replies may still egress. The PAKKT Integrations API pushes list updates to both the BPF map and the nft set atomically, preventing this drift. Manual deployments should script both updates together:

bpftool map update name pakkt_blacklist key hex $(printf '%08x' $IP) value 0x01
nft add element inet pakkt pakkt_blacklist { $IP }

Audit nftables Counters

nft list chain inet pakkt input

Each rule can carry a counter keyword. Compare packet counts before and after the suspect rule: if the counter increments but packets still reach the application, a later accept rule is overriding your drop. Reorder rules so specific denies precede broad accepts, or switch the chain policy to drop and whitelist explicitly.

Performance Metrics and Packet Drop Accounting

Even a perfectly written XDP program can appear to "not match" if hardware or driver limitations cause silent packet loss upstream. Rule out capacity bottlenecks before blaming your code.

Interface Statistics

ethtool -S eth0 | grep -E 'rx_drop|rx_missed'

NIC drivers increment rx_missed_errors when the ring buffer overflows. If this counter climbs during your test, packets never reached XDP. Increase ring size:

ethtool -G eth0 rx 4096

Or enable hardware flow steering (ntuple filters) to distribute load across queues.

XDP Program Statistics via Maps

Many XDP programs export a per-action counter map (e.g., pakkt_stats with keys XDP_DROP, XDP_PASS, XDP_TX). Dump it to see how many packets hit each verdict:

bpftool map dump name pakkt_stats

If XDP_DROP is zero but you expect blocks, either no packet matched or the program returns the wrong code. If XDP_PASS dominates, your match conditions are too narrow or the map is empty.

System-Wide eBPF Resource Limits

ulimit -l

Locked memory (used for BPF maps) defaults to 64 KB on many distributions. Large hash maps fail to create, and the control plane silently falls back to an empty map. Raise the limit in /etc/security/limits.conf:

* hard memlock unlimited
* soft memlock unlimited

Then reload your session. The PAKKT agent sets this automatically during installation.

Debugging Multi-Program Scenarios and Conflicts

Kernel 5.x introduced BPF program chaining (via bpf_tail_call) and multiple programs per hook in newer versions. Conflicts arise when two tools try to own the same XDP slot.

List All Loaded Programs

bpftool prog show

Search for programs with type xdp. Note the loaded_at timestamp and tag (a hash of the bytecode). If you see multiple XDP programs, identify which is active on your interface with bpftool net show. Only one native-mode program can attach per interface; a second attach silently replaces the first unless you use XDP_FLAGS_UPDATE_IF_NOEXIST.

Pinned Maps and Persistent State

Programs often pin maps to /sys/fs/bpf/ so they survive program reload. Stale pins from a previous version cause the new program to read outdated rules:

ls -lh /sys/fs/bpf/pakkt_*
rm /sys/fs/bpf/pakkt_rules_old

Recreate the map or update the loader to reuse the existing pin. The PAKKT engine pins maps by a versioned name, allowing zero-downtime upgrades without flushing state.

Coordination with Other Kernel Hooks

XDP runs before tc (traffic control) and netfilter. If a tc bpf program or an early nftables prerouting rule marks or drops packets, XDP never sees the effect. Inspect the full pipeline:

tc filter show dev eth0 ingress
nft list chain netdev pakkt ingress

Netdev family chains (available in nftables 0.9.3+) run at the same XDP-like layer but use nftables syntax. Mixing XDP C code with netdev nftables rules on the same interface is possible but requires careful priority tuning to avoid double-processing.

Automated Debugging Workflow and Continuous Validation

Manual commands are essential for triage, but production environments need programmatic health checks. Integrate these verifications into your deployment pipeline to catch regressions before they reach users.

Scripted Attachment Test

#!/bin/bash
IFACE=eth0
PROG=/opt/pakkt/pakkt_engine.bpf.o
ip link set dev $IFACE xdp obj $PROG sec xdp || exit 1
sleep 1
bpftool net show dev $IFACE | grep -q xdp || { echo "Attach failed"; exit 1; }
echo "XDP program successfully attached to $IFACE"

Run this in CI after each kernel upgrade or agent update. Pair it with a synthetic traffic generator that sends known-bad packets and asserts they are dropped.

Map Integrity Checks

EXPECTED_RULES=42
ACTUAL=$(bpftool map dump name pakkt_rules | grep -c 'key:')
[ "$ACTUAL" -eq "$EXPECTED_RULES" ] || { echo "Rule count mismatch"; exit 1; }

For platforms with dynamic rule provisioning—like PAKKT.io—the central panel exposes a /metrics endpoint that returns active rule count, last sync timestamp, and per-port packet counters in Prometheus format. Alerting on drift triggers immediate investigation.

End-to-End Connectivity Probe

After applying a new XDP program, send a legitimate request from a known-good IP to ensure you haven't accidentally blocked production traffic. A simple HTTP health check or game server query suffices. If the probe times out, rollback the XDP object and review the map before retrying.

Conclusion

Effective XDP program debug hinges on verifying attachment, inspecting map state with the correct byte order, validating packet-parsing bounds checks, and coordinating stateless XDP logic with stateful nftables rules. By systematically walking interface state, BPF map dumps, kernel logs, and nftables counters, you isolate whether the issue lies in configuration, code, or capacity—and apply the precise fix. Automated health checks and centralized telemetry transform ad-hoc troubleshooting into a repeatable, low-latency workflow that keeps your network protection reliable at scale.

FAQ

Why does bpftool show my XDP program attached, yet packets still bypass all rules?

The most common cause is an empty or incorrectly keyed BPF map. Dump the map with bpftool map dump name pakkt_rules and confirm entries exist with keys in network byte order. If the map is empty, your control plane never populated it, or a pin conflict loaded stale data. Additionally, verify the program's default action: many return XDP_PASS when no rule matches, allowing all unmatched packets through.

How can I confirm my XDP program is seeing the packets before nftables drops them?

Insert bpf_trace_printk calls at the top of your XDP function and monitor /sys/kernel/debug/tracing/trace_pipe. If messages appear for every incoming packet, XDP is invoked first. Compare the timestamp of the trace messages against nftables counters (nft list chain inet pakkt input with counter keywords) to see which layer takes action. XDP always executes before netfilter; if nftables counters increment but XDP trace is silent, the program may have detached or the NIC offloaded filtering elsewhere.

What causes the eBPF verifier to reject my XDP program even though it compiles with clang?

The verifier performs runtime safety checks that the compiler cannot: proving all packet data accesses stay within [data, data_end), ensuring loops are bounded, and validating that register state doesn't allow NULL dereferences. Common rejections include missing bounds checks before reading IP or TCP headers, using unbounded loops, or accessing packet bytes after a potential reallocation (e.g., bpf_xdp_adjust_head). Review the verifier log in dmesg, which prints the failing instruction number, then correlate it with your source using llvm-objdump -S on the .o file.