Firewall Logs Centralized Without ELK Stack (2026)

Managing firewall logs centralized infrastructure in 2026 no longer requires deploying heavy stacks like Elasticsearch, Logstash, and Kibana (ELK). Modern kernel-level firewalls—powered by XDP/eBPF and nftables—can stream metrics and events directly to lightweight time-series databases, delivering sub-second query performance, minimal resource overhead, and operationally simple architectures. This article explains how to centralize nftables and XDP logs without ELK, why stateless eBPF programs paired with stateful conntrack rules produce clean audit trails, and how a purpose-built SaaS control plane eliminates the toil of maintaining your own observability cluster.

Traditional centralized logging for firewalls relied on syslog forwarders, intermediate message queues, full-text indexing engines, and visualization layers. Each component consumed CPU, memory, and storage at scale. For teams protecting dozens or hundreds of game servers, edge workloads, or API gateways, ELK's operational complexity often outweighed its benefits. Kernel firewalls have evolved: XDP programs can update BPF maps in nanoseconds, nftables meters track per-connection state efficiently, and time-series databases like TimescaleDB compress firewall counters into hypertables that answer "packets per second by source country" queries in milliseconds—all without Java heap tuning or shard rebalancing.

Why firewall logs centralized architectures moved beyond ELK

Elasticsearch was designed for full-text search across unstructured documents. Firewall events, by contrast, are highly structured: timestamp, source IP, destination port, protocol, action (drop/accept), packet count, byte count. Storing every dropped SYN packet as a JSON document, then building inverted indexes on each field, wastes IOPS and RAM. A time-series database schema—where each metric is a narrow row with indexed timestamp and dimension columns—delivers two orders of magnitude better compression and query speed for numeric aggregations.

Second, ELK requires a dedicated operations team. You must size Elasticsearch data nodes, tune JVM garbage collection, manage index lifecycle policies (hot/warm/cold tiers), rotate indices, monitor cluster health, and secure inter-node transport. Logstash pipelines need grok filters to parse syslog lines, and Kibana dashboards require manual JSON configuration. A three-node ELK cluster can easily consume 16 GB RAM and four CPU cores just for the stack, before ingesting a single log line.

Third, kernel firewalls have grown smarter. XDP programs run before the network stack allocates an sk_buff, dropping malicious packets in single-digit microseconds. Because XDP is stateless, the program itself does not emit per-packet logs—instead, it increments counters in BPF maps (rule hits, bytes, source-IP buckets). A user-space agent reads those maps every few seconds and ships deltas to the central database. Meanwhile, nftables with log targets can selectively write to the kernel ring buffer or a dedicated nflog group, which a collector forwards as structured events. This hybrid design—counters for high-volume drops, discrete events for low-volume accepts or policy violations—keeps log volume manageable without sampling.

Fourth, the threat landscape in 2026 is packet-flood-dominated. Reflection attacks (DNS, NTP, memcached) and TCP SYN floods routinely exceed one million packets per second. Logging every dropped packet to disk, then indexing it in Elasticsearch, guarantees disk saturation and query timeouts. Aggregating drops into per-source-IP counters over five-second windows reduces cardinality by four orders of magnitude while preserving the signal: which /24 netblock just sent 500k packets to UDP/19132?

Kernel-native logging: XDP map updates and nftables counters

XDP programs attached to a network interface can read packet headers (Ethernet, IPv4/IPv6, TCP/UDP/ICMP) and make instantaneous pass/drop decisions. The PAKKT Engine—a single XDP program per interface—consults BPF hash-maps for IP blacklists, port-based rules, and rate-limit state. When a rule triggers, the program atomically increments a per-rule counter map and returns XDP_DROP or XDP_PASS. No syscall, no context switch, no disk write.

// Pseudo-code excerpt (simplified)
struct rule_stats {
    __u64 packets;
    __u64 bytes;
    __u64 last_hit_ns;
};

BPF_HASH(rule_counters, __u32, struct rule_stats, 256);

if (blocked_by_rule_42) {
    __u32 key = 42;
    struct rule_stats *st = rule_counters.lookup(&key);
    if (st) {
        __sync_fetch_and_add(&st->packets, 1);
        __sync_fetch_and_add(&st->bytes, pkt_len);
    }
    return XDP_DROP;
}

A lightweight Go agent on the same host periodically calls bpf_map_lookup_elem via the BPF syscall or bpftool wrapper to read all rule counters, computes deltas since the last read, and sends them over mTLS to the central API. Each sample is ≈100 bytes (timestamp, agent ID, rule ID, packets_delta, bytes_delta). For 256 rules sampled every 30 seconds, bandwidth is ~850 bytes/second per agent—three orders of magnitude less than per-packet syslog.

Stateful firewall rules in nftables use ct state and meter for connection tracking and per-flow rate limits. To centralize these events, you can add a log statement before the final verdict:

table inet pakkt {
    chain input {
        type filter hook input priority filter; policy drop;

        # Established/related: silent accept
        ct state established,related accept

        # New TCP to port 25565: log + rate-limit + accept
        tcp dport 25565 ct state new \\
            log prefix "pakkt[new-conn]: " \\
            limit rate 100/second burst 20 \\
            accept

        # Default drop with log (rate-limited to avoid flood)
        limit rate 10/minute log prefix "pakkt[drop]: "
        drop
    }
}

The kernel writes these log lines to /dev/kmsg or a dedicated nflog group. A collector daemon (systemd-journald, rsyslog, ulogd2, or a custom reader) parses the prefix, extracts five-tuple and verdict, then forwards JSON records to the time-series database. Because the limit directive is in the rule, the kernel itself throttles log volume during attacks, preventing disk exhaustion.

Combining XDP counters (stateless, millions of pps) with nftables logs (stateful, lower volume) gives you complete visibility: XDP telemetry shows which rules blocked the most packets, nftables logs show which legitimate connections were established and from where. Neither layer requires a local Elasticsearch node.

Time-series databases and the PAKKT centralized control plane

TimescaleDB, an extension of PostgreSQL, transforms regular tables into hypertables partitioned by time. Firewall metrics—timestamp, agent_id, rule_id, packets, bytes, src_country—compress naturally because dimensions repeat (same agent, same rule) across thousands of time buckets. Continuous aggregates pre-compute rollups (per-minute, per-hour) in the background, so dashboard queries like "total drops by country, last 24 hours" return in ~50 ms even across a hundred agents.

PAKKT.io runs a managed TimescaleDB cluster that receives samples from all agents every 30 seconds. Each agent authenticates via mTLS client certificate, sends a protobuf or JSON payload with deltas for 256 XDP rules plus nftables counter snapshots, and closes the connection. The API server writes rows in a single bulk INSERT, and hypertable compression kicks in after one hour, achieving ~20:1 compression ratio for counter data.

The web dashboard queries these hypertables to render:

Real-time packet-per-second graph for each agent and rule, overlaying TCP/UDP/ICMP breakdowns.
World map heatmap using MaxMind GeoIP2 lookups on source IPs extracted from XDP map dumps and nftables logs.
Top-N tables: top source IPs, top attacked ports, top countries, sortable and filterable.
Audit log: every configuration change (rule add/delete, IP blacklist update, rate-limit tuning) stored in a separate append-only table with agent ID, user email, timestamp, and diff.
Agent health: last heartbeat, CPU/RAM usage, XDP program attach status, nftables ruleset SHA256.

Because the schema is relational, you can JOIN agent metadata (hostname, datacenter tag, customer account) with metrics in a single SQL query. No need for Elasticsearch's parent-child documents or Kibana's scripted fields. A typical dashboard page executes five SQL queries in parallel and renders in under 200 ms.

For compliance or long-term forensics, you can configure the platform to archive raw nftables logs (not XDP counters, which are aggregates) to object storage (S3-compatible) after 90 days. The time-series database retains one-second granularity for seven days, one-minute rollups for 90 days, and hourly rollups for two years, keeping storage predictable and query performance constant.

Deployment: agent installation and log forwarding in under two minutes

Installing the PAKKT agent on a Linux host (kernel 5.4+, clang/llvm for BPF compilation, nftables ≥0.9.3) takes one command:

curl -sSL https://install.pakkt.io | sudo bash -s -- --token YOUR_API_TOKEN

The installer:

Downloads the Go agent binary (garble-obfuscated, ≈8 MB) and verifies SHA256 signature.
Compiles and attaches pakkt_engine.bpf.o to the primary interface (eth0) in XDP generic or native mode.
Creates the inet pakkt nftables table with a default-drop input chain isolated from Docker's nat table and any existing iptables-persistent rules.
Starts a systemd service that reads BPF maps every 30 seconds, parses journalctl -k for pakkt[ prefixes, and ships both via mTLS to api.pakkt.io.
Registers the agent in the central database with hostname, public IPv4, kernel version, and GeoIP-derived datacenter location.

No local Logstash, no Filebeat, no Fluentd config files. The agent is stateless: if the process crashes, systemd restarts it, re-reads BPF maps (counters are in kernel memory), and resumes forwarding. If the API is unreachable, samples buffer in a 64 KB ring and retry with exponential backoff.

To add custom nftables rules—for example, per-connection TCP rate-limit with meter—edit /etc/nftables.d/pakkt-custom.nft:

table inet pakkt {
    chain input {
        # Allow established
        ct state established,related accept

        # TCP port 25565: max 10 new conn/sec per source IP
        tcp dport 25565 ct state new \\
            meter tcp_new_limit { ip saddr timeout 10s limit rate 10/second } \\
            log prefix "pakkt[accept-25565]: " accept

        # ICMP echo: 5 pps
        icmp type echo-request limit rate 5/second accept

        # Default drop (already logged by base config)
        drop
    }
}

Reload with nft -f /etc/nftables.d/pakkt-custom.nft. The agent detects the new ruleset hash and uploads it to the audit log. The meter's log prefix ensures accepted connections appear in the centralized timeline alongside XDP drops.

For PAKKT Integrations (Pterodactyl, public API), the panel can push IP blacklist updates via webhook whenever a game server admin bans a player. The agent receives the update over the persistent mTLS socket, calls bpf_map_update_elem to add the IPv4 to the XDP blacklist map, and appends an nft add element inet pakkt blacklist_v4 { 203.0.113.42 } command for the stateful layer—both actions logged centrally with microsecond timestamps.

Querying centralized logs: SQL over dashboards, CLI, and API

PAKKT's web dashboard is the primary interface, but power users often need programmatic access. The platform exposes a REST API (bearer-token auth) with endpoints:

GET /api/v1/agents — list all agents, their last heartbeat, XDP program version.
GET /api/v1/metrics?agent_id=X&start=...&end=...&rule_id=Y — time-series data as JSON, CSV, or Prometheus exposition format.
GET /api/v1/events?agent_id=X&type=nftables_log — discrete firewall log events (new connection, drop) with GeoIP enrichment.
POST /api/v1/rules — create or update XDP rules; the agent pulls config changes on next heartbeat.
GET /api/v1/[email protected] — audit trail for compliance (who changed what, when).

Example: fetch dropped packets by source country for agent ag_1a2b3c in the last hour, grouped by five-minute buckets:

curl -H "Authorization: Bearer YOUR_API_KEY" \\
  "https://api.pakkt.io/v1/metrics?agent_id=ag_1a2b3c&start=-1h&interval=5m&group_by=src_country&action=drop" \\
  | jq '.data[] | {country, packets, bytes}'

Response excerpt:

{
  "country": "CN",
  "packets": 1283471,
  "bytes": 78362548
},
{
  "country": "RU",
  "packets": 592038,
  "bytes": 36118324
}

This JSON can feed into Grafana (via JSON API data source), custom alerting scripts, or incident-response automation. No need to learn Elasticsearch Query DSL or Kibana's Lens editor.

For on-call engineers, the PAKKT CLI tool (pakkt-cli) wraps the API into commands like pakkt-cli logs tail --agent ag_1a2b3c --follow, which streams new nftables log lines in real time, similar to journalctl -f but aggregated across all your hosts. Under the hood, it opens a WebSocket to the API and receives server-sent events as rows arrive in TimescaleDB.

Cost and operational comparison: self-hosted ELK vs. centralized SaaS

Dimension	Self-hosted ELK (3-node)	PAKKT centralized (SaaS)
Setup time	2–4 hours (install, tune JVM, configure Logstash pipelines, create Kibana dashboards)	2 minutes (curl install script, agent auto-registers)
Resource overhead (per log source host)	Filebeat: ~100 MB RAM, ~2% CPU; Logstash on central: 2 GB RAM per instance	PAKKT agent: ~5 MB RAM, <1% CPU
Central cluster resources	3× Elasticsearch (each 8 GB RAM, 2 vCPU), 1× Logstash (4 GB, 2 vCPU), 1× Kibana (2 GB, 1 vCPU) ≈ €200–300/month	Managed TimescaleDB + API + UI included in agent price: €3/agent/month
Storage retention (per agent, 90 days)	~50 GB (uncompressed JSON docs), requires hot/warm tier or S3 snapshot lifecycle	~500 MB (hypertable compression), auto-archived to S3 after 90 days
Query performance (24-hour aggregate, 100 agents)	5–30 seconds (depends on shard count, heap size, query cache hits)	50–200 ms (indexed timestamp + continuous aggregates)
Maintenance (monthly effort)	~4 hours (index rotation, cluster health checks, version upgrades, grok debugging)	Zero (SaaS provider handles DB maintenance, agent auto-updates via SHA256-signed binary)

For a 50-agent deployment, ELK's infrastructure cost alone exceeds €250/month, plus ~16 engineer-hours/month for operations. PAKKT's total cost is €150/month (50 × €3) with zero maintenance burden, and the first agent includes a seven-day free trial to validate performance in your environment. The ROI is clear: centralized firewall logs without ELK reduces both CapEx (no dedicated cluster VMs) and OpEx (no specialized Elastic Stack expertise required).

Security and compliance: mTLS, audit trails, and data residency

Every PAKKT agent authenticates to the API using mutual TLS with a unique X.509 client certificate issued at registration. The certificate's Common Name encodes the agent ID; the API validates it against the account database before accepting metrics or configuration updates. This design prevents IP spoofing (an attacker who captures an agent's IP cannot impersonate it without the private key) and ensures end-to-end encryption for log data in transit.

All configuration changes—rule creation, IP blacklist add/remove, rate-limit tuning—are recorded in an immutable audit log table with columns: timestamp, user_id, agent_id, action, old_value, new_value, source_ip. SOC2 and ISO 27001 audits can query this table to prove who changed the firewall policy and when. The nftables ruleset and XDP program bytecode are also hashed (SHA256) and stored per deployment, enabling rollback and forensic comparison.

Data residency: PAKKT offers database regions in EU (Frankfurt), US (Virginia), and APAC (Singapore). Metrics and logs for agents in a given region stay within that region's TimescaleDB cluster, complying with GDPR's data localization requirements. Customers can export raw SQL dumps via the API for offline archive or import into their own Postgres instance.

Sensitive data (source IPs) can be pseudonymized: the agent hashes the last octet of IPv4 addresses with a per-account salt before sending, so the centralized database sees 203.0.113.HASH instead of the real IP. GeoIP lookups still work (first three octets determine country), but PII is protected. This feature is optional and configured per agent.

Retention policies are customer-controlled: delete raw one-second data after 7 days, keep five-minute rollups for 90 days, and hourly rollups for two years. The UI shows projected storage cost in real time as you adjust sliders, preventing bill shock.

Integrations: Pterodactyl, WHMCS, and the public API

Game server hosts using Pterodactyl v1.x can install the PAKKT Integrations (Pterodactyl, public API) plugin, which adds a "Firewall" tab to each server's admin panel. Server owners view real-time attack graphs, add IPs to the blacklist, and adjust per-port rate-limits without SSH access. When a player is banned in-game, the game server plugin (via RCON or log watcher) calls the Pterodactyl API, which forwards the IP to PAKKT's API, which pushes the update to the agent's XDP map—all in under one second.

WHMCS integration (in beta, estimated Q2 2026) will provision PAKKT agents automatically when a customer orders a VPS or dedicated server, include agent cost in the monthly invoice, and display firewall metrics in the client area. The module uses WHMCS hooks to call POST /api/v1/agents with the server's IPv4 and SSH credentials, triggering remote agent installation via Ansible or cloud-init.

The public API also powers a community template marketplace: users share XDP + nftables rule bundles for common scenarios (Minecraft, FiveM, Rust, HTTP/HTTPS, DNS authoritative server). A template is a JSON file specifying port ranges, protocols, rate-limits, and nftables meter expressions. You can import a template into your account, apply it to any agent with one click, and fork it to customize. All templates are versioned and SHA256-signed by the author, preventing tampering.

Advanced use-cases: multi-region deployments and compliance logging

Enterprises running game server fleets across multiple continents use PAKKT's agent tagging (key-value labels: region=eu-west, game=minecraft, env=production) to segment dashboards. A single account can manage 500 agents grouped by tag, with per-group alerting rules (Slack, PagerDuty, webhook) that fire when aggregate PPS exceeds a threshold or a specific rule triggers more than N times in five minutes.

For regulatory compliance (PCI-DSS, HIPAA technical safeguards), the audit log's immutability and the ability to export nftables logs as syslog RFC5424 format satisfy "centralized logging" and "access control review" requirements. Some customers configure the agent to duplicate logs to a local syslog server and PAKKT, providing redundancy and on-premises forensic capture alongside cloud-based real-time analytics.

High-traffic edge workloads (CDN origins, API gateways behind load balancers) deploy PAKKT agents on each origin server. The XDP layer blocks layer-3/4 floods before they reach the application, and centralized metrics show which upstream /24 netblock is sending suspicious traffic. Network engineers export these metrics to their existing Prometheus/Grafana or Datadog via the /metrics endpoint in Prometheus exposition format, unifying firewall telemetry with application metrics in one dashboard.

Conclusion

Centralizing firewall logs in 2026 without ELK is not only feasible—it is the operationally sound choice for kernel-level protection. XDP/eBPF programs provide nanosecond-latency packet filtering with counter-based telemetry, nftables delivers stateful connection tracking and selective logging, and purpose-built time-series databases outperform full-text search engines by orders of magnitude for numeric metrics. A lightweight agent, mTLS transport, and a managed SaaS control plane eliminate the toil of maintaining an Elasticsearch cluster, reduce per-agent resource overhead to negligible levels, and deliver sub-200 ms dashboard queries across hundreds of hosts. Whether you manage a handful of game servers or a continental edge network, consolidating XDP and nftables observability into a single SQL-queryable platform simplifies incident response, satisfies compliance audit requirements, and frees engineering time for higher-value work.

FAQ

Can I keep my existing rsyslog or journald setup and still send logs to a centralized platform?

Yes. The PAKKT agent reads kernel messages (including nftables log lines) from journalctl -k or a specified syslog socket, so your existing rsyslog forwarding rules remain untouched. The agent simply subscribes to messages with the pakkt[ prefix and ships them alongside XDP counter deltas. You can run both local syslog archival and centralized SaaS collection in parallel for redundancy.

How does the agent handle log volume spikes during a DDoS attack without overwhelming the central API?

XDP counters aggregate drops in kernel space, so even a 10 Mpps flood produces only one counter update per 30-second sample (≈100 bytes). Nftables log rules include limit rate directives that throttle discrete log events at the kernel level. The agent also buffers samples in a 64 KB ring and applies exponential backoff if the API returns HTTP 429 or is unreachable, preventing retry storms.

What happens to centralized logs if the agent host loses network connectivity or reboots?

XDP map counters reside in kernel memory and persist across agent restarts (the BPF program remains pinned to the interface). When the agent restarts, it re-reads the current counter values and resumes delta calculation. If connectivity is lost, samples buffer locally until the link recovers, then flush in chronological order. Counter data during the outage window is reconstructed from the next successful read minus the last known baseline. Discrete nftables log events that occurred during the outage are lost unless you also forward them to a local syslog archive.

Centralizing firewall logs from multiple Linux servers without an ELK stack