housecarl/docs/flowtable.md

The flowtable: a fast path for forwarded traffic

The flowtable is a shortcut inside the kernel for traffic the firewall forwards (between the internet and your LAN, between zones). It speeds up routing on a machine whose CPU can't keep up with its line. It doesn't make the firewall any safer or less safe, but it changes what some features can see, so read when not to use it before you turn it on.

It is off until you name its devices.

What it does

Without a flowtable, every forwarded packet takes the whole way through the kernel: the prerouting hooks, the routing decision, Housecarl's forward chain (the built-ins, then your rules) and postrouting with NAT. For an established connection almost all of that work only ends in the established built-in accepting it again, packet after packet.

With a flowtable, Housecarl adds one built-in to the forward chain:

meta l4proto { tcp, udp } flow add @ft

A TCP or UDP connection takes the whole way for its first packets, as always: your rules decide whether it is allowed at all. Once the kernel has seen it established (and your rules let it through), it is added to the flowtable. From then on its packets are caught as soon as they arrive on one of the flowtable's devices (the ingress hook) and sent straight out on the other side: NAT is still applied, the TTL still counts down, but every chain in between is skipped.

                 first packets                         every packet after
in ─► prerouting ─► routing ─► forward ─► postrouting ─► out
 │                                                        ▲
 └────────────── flowtable (ingress) ─────────────────────┘

It is the software flowtable, run by the CPU. Offloading to the network card's hardware isn't supported (design.md).

When to use it

Turn it on when the router's CPU is what limits your speed:

• a speed test through the router gets less than your line gives (or less than the same test from the router itself), and
• during it, one CPU core is busy with network work: in top, press 1 for every core, and look at si (softirq) near 100 % on a core.

Typical cases: a small or old machine (Atom, Celeron, a Raspberry Pi, an old Core 2) on a fast line (500 Mbit/s and up), lots of small packets (gaming, VoIP, many downloads at once), or a machine that is busy with other work besides routing. The gain is CPU time per packet: often well over half of the forwarding cost, which shows up as speed only when the CPU was the limit, and otherwise as a cooler, quieter, more responsive machine.

When not to use it

When the CPU keeps up anyway. If the speed test already reaches your line and si stays low, the flowtable buys you nothing you'd notice, and costs you the figures below. Most modern x86 machines route a gigabit line without it.

When you count or limit traffic in the forward chain. The fast path skips the forward chain, so whatever must see every packet of a connection only sees the first few. Housecarl warns about each:

Feature	With the flowtable	Warning
quotas and byte-rate limits on forward rules	only count first packets: a device can go far past its allowance	flowtable_traffic
forward rules with "every packet" (--every-packet)	only see first packets	flowtable_traffic
statistics' traffic per device (stats_per_device)	only count first packets	stats_flowtable
hit counters of the established built-in (forward)	stop rising once a connection is offloaded

Rate limits by packets (--limit 10/second) and connection limits act on new connections, so they work the same.

When other tables must see forwarded traffic. Offloaded packets skip the forward hooks of every table, not only Housecarl's: Docker's, Incus', libvirt's or a table of your own. Those normally only judge new connections, which still pass through them, but a table that counts or shapes established traffic in its forward chain won't see it.

What keeps working as before:

• Your rules decide every new connection: nothing gets offloaded that your rules didn't let through. A rule you add later doesn't stop a connection already open, but that is true without the flowtable too (the established built-in accepts it).
• NAT: port forwards, hairpin NAT, masquerading and SNAT.
• The input and output chains: traffic to and from the firewall itself never uses the flowtable.
• Closing connections (hc connections close, Traffic → Connections) ends their fast path too: the kernel drops a flow with its tracking entry.
• Interface statistics (traffic per zone and interface) come from the interfaces' own counters, which count every packet.
• Traffic shaping with tc on the outgoing interface still applies.
• The log and the packet tracer see what goes through the rules: new connections, as always. The tracer doesn't see the packets of an offloaded connection; trace a new one.
• ICMP, and protocols other than TCP and UDP (GRE, ESP, SCTP), always take the normal way.

How to turn it on

Name the devices whose traffic should take the fast path, both sides: the interface connections come in on and the one their replies come in on. For a router that means the external interface and the inside ones:

hc settings set --flowtable wan --flowtable enp2s0
hc settings                     # Flowtable: wan, enp2s0

In the web UI: Settings → Flowtable, add each device. The fast path only catches packets arriving on a listed device, so a connection to or from an unlisted one gains little or nothing: leave a device out only on purpose (say, a WireGuard interface whose traffic you'd rather keep on the normal way).

• Use exact device names, not wildcards (incusbr* is refused): the kernel attaches the flowtable to each device by name.
• With PPPoE, name the PPP interface (ppp0, or the name your ppp@ unit gives it), the one in your external zone.
• --flowtable replaces the whole list; hc settings set --flowtable none turns the flowtable off.

The change is applied (and on probation, if made over the network) like any other.

Devices that come and go

PPP and WireGuard interfaces don't exist until their service brings them up, and PPPoE makes a new ppp0 each time it reconnects. Housecarl handles that:

• A device that doesn't exist yet is left out of the flowtable, with the warning flowtable_device_missing, instead of failing the whole ruleset.
• housecarld watches the kernel's interface changes: when a flowtable device appears, or comes back as a new interface after a reconnect, it applies the ruleset again with it, a couple of seconds later. The journal says interfaces changed: applied the ruleset again. Applying again keeps the counters, dynamic sets and quotas, as any change does, and adds no history entry.
• The ruleset saved for boot has no flowtable at all: it loads before the network is up, when ppp0 and wg0 can't exist yet, and nftables on Debian 13's kernel refuses a flowtable naming a device that doesn't exist (and with it the whole file). housecarld adds the flowtable once it starts. Until then, traffic takes the normal way, through the same rules.

Checking it works

sudo nft list flowtables          # the devices it is attached to
sudo conntrack -L | grep OFFLOAD  # connections on the fast path

conntrack is in Debian's conntrack package. A long download through the router should show up as [OFFLOAD]. Compare a speed test, and si in top, before and after.

If nothing gets offloaded, check that both of the connection's devices are listed, that flowtable_device_missing isn't in hc status, and that the connection is TCP or UDP and established.

Turning it off

hc settings set --flowtable none

Connections on the fast path go back to the normal way with their next packet.