Understanding conntrack and TCP states for a stateful Linux firewall

Understanding conntrack TCP states is essential for building robust, stateful firewalls on Linux. Connection tracking (conntrack) is the kernel subsystem that monitors TCP state transitions—SYN, SYN-ACK, ESTABLISHED, FIN-WAIT, and more—to enforce security policies that distinguish legitimate traffic from spoofed or malicious packets. In this 2026 guide, we explore how conntrack handles TCP states, how to leverage them in nftables rules, and how platforms like PAKKT.io combine stateless XDP filtering with stateful conntrack firewalling to deliver comprehensive kernel-level protection.

Modern DDoS attacks and exploit attempts often abuse TCP handshake mechanics—SYN floods, ACK floods, out-of-window RST packets—making state-aware filtering a critical layer of defense. Whether you operate game servers, APIs, or high-throughput workloads, mastering conntrack TCP states enables you to drop invalid packets at line rate, enforce per-connection rate limits, and maintain an audit trail of each flow's lifecycle.

What Is Conntrack and Why Does It Matter for TCP States?

Conntrack is the Linux kernel module (nf_conntrack) that tracks the state of network connections—TCP, UDP, ICMP—by maintaining a hash table of flows. Each entry records source/destination IP and port, protocol, current state, and counters. For TCP, conntrack implements a state machine that mirrors the TCP RFC 9293 lifecycle, recognizing when a three-way handshake completes, when data exchange occurs, and when a connection terminates gracefully or abruptly.

Why this matters:

• Stateful filtering: Accept only packets that belong to an ESTABLISHED or RELATED connection, rejecting unsolicited traffic.
• Attack mitigation: Drop SYN packets when a connection is already ESTABLISHED, block RST/FIN outside valid windows, and rate-limit NEW connections per source IP.
• Resource management: Conntrack entries consume kernel memory; tuning nf_conntrack_max and timeouts prevents table exhaustion during floods.
• Integration with nftables: Rules can match ct state {new, established, related, invalid}, enabling rich policies without parsing every TCP flag manually.

For example, a typical nftables policy might look like:

nft add rule inet pakkt input ct state established,related accept
nft add rule inet pakkt input ct state invalid drop
nft add rule inet pakkt input tcp dport 22 ct state new limit rate 5/minute accept

This accepts packets from known flows, drops malformed ones, and rate-limits new SSH connections—all leveraging conntrack TCP states under the hood.

The TCP State Machine in Conntrack

Conntrack's TCP state machine differs slightly from the classic TCP RFC diagram because it tracks both directions of a flow simultaneously. The kernel recognizes the following conntrack states for TCP:

Conntrack State	Description	Typical Scenario
NEW	First packet of a flow (usually SYN)	Client initiates connection
ESTABLISHED	Handshake complete, bidirectional data exchange	After SYN-ACK is ACKed
RELATED	New flow related to an existing one (e.g., FTP data channel)	Connection helpers (nf_conntrack_ftp, etc.)
INVALID	Packet does not match any known flow or violates state machine	Out-of-window SEQ, unexpected RST
UNTRACKED	Explicitly excluded from conntrack (notrack rule)	High-performance bypass for trusted flows

Internal TCP Substates

Within the ESTABLISHED and NEW umbrella states, conntrack maintains finer-grained substates that mirror TCP's own phases:

• SYN_SENT: Client sent SYN, awaiting SYN-ACK.
• SYN_RECV: Server received SYN, sent SYN-ACK, awaiting final ACK.
• ESTABLISHED: Three-way handshake complete, data transfer active.
• FIN_WAIT: One side sent FIN, awaiting ACK or FIN from peer.
• CLOSE_WAIT: Local endpoint received FIN, application still open.
• LAST_ACK: Both sides sent FIN, waiting for final ACK.
• TIME_WAIT: Connection closed, entry retained for 2×MSL to handle retransmissions.
• CLOSE: Connection fully terminated, entry about to be removed.

You can inspect these substates using the conntrack userspace tool:

conntrack -L | grep -E "tcp.*ESTABLISHED|SYN_SENT|FIN_WAIT"

Each substate has an associated timeout; for example, nf_conntrack_tcp_timeout_established defaults to 5 days (432,000 seconds), while nf_conntrack_tcp_timeout_syn_sent is typically 120 seconds. Tuning these values helps reclaim memory during SYN floods or when handling thousands of short-lived connections.

Leveraging Conntrack TCP States in nftables

nftables exposes conntrack state via the ct state match, allowing you to craft policies that respond to TCP lifecycle events. Here are practical examples drawn from production Linux firewall configurations, including those managed by PAKKT.io in the isolated inet pakkt table:

1. Drop INVALID Packets Early

nft add rule inet pakkt input ct state invalid counter drop

This rule discards packets that conntrack cannot associate with any flow—common in spoofed SYN floods, malformed ACKs, or out-of-window RSTs. Dropping INVALID packets reduces CPU cycles spent on deeper inspection.

2. Accept ESTABLISHED and RELATED Flows

nft add rule inet pakkt input ct state established,related counter accept

Once a connection passes the NEW state checks, all subsequent packets bypass further rule evaluation, significantly improving throughput. The RELATED state is useful for protocols with multiple flows (FTP, SIP) when the appropriate nf_conntrack helper is loaded.

3. Rate-Limit NEW Connections per Source IP

nft add rule inet pakkt input tcp dport 25565 ct state new \
  meter minecraft_new { ip saddr limit rate 10/second burst 20 packets } accept
nft add rule inet pakkt input tcp dport 25565 ct state new counter drop

This two-rule chain uses an nftables meter (dynamic set with per-element limits) to allow up to 10 NEW connections per second per source IP, with a burst allowance of 20. Any source exceeding this rate has its NEW packets dropped, mitigating SYN floods without penalizing legitimate clients already ESTABLISHED.

4. Enforce TCP Flag Validation Alongside Conntrack

nft add rule inet pakkt input tcp flags syn ct state new accept
nft add rule inet pakkt input tcp flags \& \(fin\|syn\|rst\|ack\) == syn ct state new counter drop

The first rule accepts SYN-only packets when conntrack sees them as NEW. The second drops packets that claim to be NEW but have conflicting flags (e.g., SYN+FIN, SYN+RST)—indicators of scanning tools or malformed traffic.

5. Bypass Conntrack for Trusted High-Throughput Flows (Advanced)

For ultra-high packet rates (e.g., internal cluster replication), you can exclude traffic from conntrack to save CPU and memory:

nft add rule inet pakkt prerouting ip saddr 10.0.0.0/8 ip daddr 10.0.0.0/8 notrack

Packets matching this rule skip the conntrack table entirely, appearing as UNTRACKED. Use this sparingly, as you lose stateful protections for those flows.

Tuning Conntrack for High-Volume TCP Workloads

Default conntrack parameters are tuned for general-purpose desktops. Servers handling thousands of concurrent TCP connections—game servers, APIs, proxies—require adjustments to prevent table exhaustion and maintain performance.

Increase Maximum Tracked Connections

sysctl -w net.netfilter.nf_conntrack_max=2097152
echo 2097152 > /sys/module/nf_conntrack/parameters/hashsize

The nf_conntrack_max parameter sets the total number of flows the kernel can track. Each conntrack entry consumes roughly 300 bytes, so 2 million entries require ~600 MB of RAM. The hashsize should be set to nf_conntrack_max / 4 for optimal lookup performance.

Reduce TCP Timeout Values

Long-lived ESTABLISHED timeout (5 days) wastes memory for short-lived game sessions or API requests:

sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=3600
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

Setting ESTABLISHED to 1 hour (3600 seconds) and TIME_WAIT to 30 seconds reclaims entries faster. Monitor the nf_conntrack_count metric to ensure you're not prematurely evicting active flows.

Enable TCP Window Scaling and Timestamps

Conntrack validates sequence numbers and window sizes by default. For legitimate high-throughput TCP flows (e.g., bulk transfers), ensure your kernel and remote hosts support window scaling (RFC 7323):

sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.ipv4.tcp_timestamps=1

Disable loose tracking only if you encounter false INVALID drops with legacy or non-RFC-compliant peers:

sysctl -w net.netfilter.nf_conntrack_tcp_loose=0

Monitor Conntrack Table Usage

Check current usage and early-drop statistics:

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
dmesg | grep "nf_conntrack: table full"

If you see "table full" messages, either increase nf_conntrack_max or tighten timeout values. Persistent table exhaustion may indicate a SYN flood or connection leak in an application.

How PAKKT.io Combines XDP and Conntrack TCP States

PAKKT.io deploys a dual-layer architecture: stateless XDP filtering for high-volume, low-latency drops at the NIC driver layer, and stateful nftables rules leveraging conntrack for nuanced TCP state enforcement. This separation ensures that simple attacks (volumetric floods, single-packet exploits) never reach the network stack, while complex multi-packet threats are caught by conntrack logic.

Layer 1: XDP (Stateless, Sub-Microsecond)

The PAKKT Engine—a single XDP program attached per interface—inspects packet headers (IP, TCP/UDP, ICMP) and consults BPF maps to enforce up to 256 simultaneous rules. Example XDP actions:

• Block: Drop packets from blacklisted IPs or prohibited port ranges instantly (XDP_DROP).
• Rate-limit: Track packets-per-second per rule in a BPF map, drop excess (stateless, no per-flow memory).
• Allow-only: Pass packets matching a whitelist, drop everything else.

Because XDP operates before skb allocation and conntrack lookup, it cannot distinguish TCP states—SYN, ACK, FIN all appear identical unless you parse flags manually in eBPF. PAKKT's XDP layer focuses on coarse-grained filters (port, protocol, packet size, global PPS) to shed attack traffic early.

Layer 2: nftables (Stateful, Conntrack-Aware)

Packets that pass XDP enter the inet pakkt nftables table, where conntrack state matching applies. PAKKT synchronizes IP blacklists and whitelists between XDP and nftables, ensuring consistent policy. Example ruleset:

table inet pakkt {
  set blacklist_ips { type ipv4_addr; flags interval; }
  
  chain input {
    type filter hook input priority filter; policy accept;
    
    ip saddr @blacklist_ips counter drop
    ct state invalid counter drop
    ct state established,related counter accept
    
    tcp dport 25565 ct state new limit rate 50/second accept
    tcp dport 25565 tcp flags syn ct state new \
      meter syn_flood { ip saddr limit rate 10/second } accept
    tcp dport 25565 counter drop
  }
}

This chain drops blacklisted IPs, invalid packets, and enforces per-connection NEW limits—all informed by conntrack TCP states. The result: legitimate players experience zero added latency (ESTABLISHED bypass), while attackers exhaust their rate limits before consuming server resources.

Centralized Management and Metrics

Through the PAKKT Integrations panel, administrators monitor per-port packet counters, dropped vs. accepted ratios, and top source IPs in real time (TimescaleDB + GeoIP). The audit log records every rule change, and internal agent logs surface conntrack table warnings or XDP attachment errors. This visibility accelerates troubleshooting when a new attack vector targets a specific TCP state transition.

PAKKT's lightweight Go agent (mTLS, 30s heartbeat, SHA256 self-update) consumes less than 1% CPU and under 5 MB RAM, leaving server resources for your workloads. At €3 per agent per month—with a 7-day free trial on the first agent—you gain enterprise-grade conntrack-aware firewalling without the complexity of manually syncing XDP maps and nftables rules across a fleet.

Common Pitfalls When Working with Conntrack TCP States

1. Forgetting to Load Connection Tracking Modules

If you reference ct state in nftables but nf_conntrack is not loaded, rules fail silently or produce errors. Ensure the module loads at boot:

echo "nf_conntrack" >> /etc/modules-load.d/conntrack.conf
modprobe nf_conntrack

2. Dropping RELATED Without Helpers

If your ruleset accepts RELATED but you haven't loaded the appropriate helper (e.g., nf_conntrack_ftp), secondary flows remain NEW and may be dropped. Verify loaded helpers:

lsmod | grep nf_conntrack

3. INVALID False Positives with Asymmetric Routing

When traffic enters via eth0 but returns via eth1, conntrack may see only one direction and mark packets INVALID. Use nf_conntrack_tcp_loose=1 or create notrack exceptions for asymmetric flows.

4. Ignoring Conntrack Table Exhaustion

SYN floods can fill the conntrack table with half-open entries. Combine XDP rate-limiting (PAKKT Engine) with reduced SYN_SENT timeouts to free slots quickly. Monitor nf_conntrack_count and set alerts at 80% of nf_conntrack_max.

5. Over-Relying on Conntrack for DDoS Mitigation

Conntrack inspection occurs after packet allocation and basic netfilter hooks—too late for multi-million PPS floods. Always pair stateful nftables with a stateless XDP layer (or hardware offload) to drop volumetric attacks before they consume CPU. This is the core design philosophy behind PAKKT's dual-layer approach.

Debugging Conntrack TCP States in Production

When connections behave unexpectedly—timeouts, RST storms, or blocked legitimate traffic—inspect conntrack state in real time:

List All TCP Flows

conntrack -L -p tcp

Filter by Port or IP

conntrack -L -p tcp --dport 25565
conntrack -L -p tcp --src 203.0.113.42

Watch State Transitions Live

conntrack -E -p tcp

This event stream shows each NEW, ESTABLISHED, and DESTROY event as it occurs, invaluable for correlating firewall drops with TCP state machine progression.

Delete a Stuck Entry

conntrack -D -p tcp --src 203.0.113.42 --dport 25565

Use this sparingly; it forces immediate connection reset. Prefer timeout tuning to let conntrack manage lifecycle naturally.

Inspect BPF Maps (PAKKT XDP Layer)

For PAKKT users, verify XDP rule configuration and counters:

bpftool map dump name pakkt_rules
bpftool map dump name pakkt_blacklist_v4

Cross-reference XDP drop counters with nftables conntrack INVALID drops to pinpoint whether an attack was shed at XDP or caught by stateful filtering.

Conclusion

Mastering conntrack TCP states empowers you to build Linux firewalls that intelligently distinguish legitimate connections from attack traffic. By combining stateless XDP filtering for sub-microsecond packet drops with stateful nftables rules that leverage conntrack's TCP state machine—NEW, ESTABLISHED, INVALID—you achieve defense-in-depth at the kernel level. Tuning conntrack parameters for high-volume workloads, monitoring table usage, and integrating centralized management through platforms like PAKKT.io ensure your infrastructure remains resilient, observable, and performant against evolving threats.

FAQ

What happens if conntrack marks a legitimate TCP packet as INVALID?

Legitimate packets may be marked INVALID due to asymmetric routing, out-of-window sequence numbers, or aggressive timeout values. First, verify your network topology does not split inbound and outbound paths across different interfaces. If asymmetric routing is unavoidable, set net.netfilter.nf_conntrack_tcp_loose=1 to relax sequence validation, or add notrack rules for those specific flows. Check conntrack -L to confirm the flow exists and inspect its current state and timeout.

How do I rate-limit NEW TCP connections per source IP using nftables meters?

Use an nftables meter with a per-element limit: nft add rule inet pakkt input tcp dport 22 ct state new meter ssh_new { ip saddr limit rate 5/minute burst 10 packets } accept. This allows each source IP up to 5 NEW connections per minute with a burst of 10, then drops excess NEW packets from that IP. The meter dynamically tracks sources without pre-defining a set, making it ideal for mitigating distributed SYN floods while permitting bursts from legitimate clients.

Can I use conntrack TCP states in XDP programs?

No. XDP executes before the kernel allocates sk_buff and before conntrack lookup occurs. XDP can parse raw TCP flags from packet headers and maintain stateless per-rule counters in BPF maps, but it cannot query conntrack state. To enforce TCP state-aware policies, pass packets to the nftables layer where conntrack integration is available. This is why dual-layer architectures—XDP for volumetric drops, nftables for stateful filtering—deliver optimal protection.