housecarl/docs/flowtable.md

# The flowtable: a fast path for forwarded traffic

The flowtable is a shortcut inside the kernel for traffic the firewall
**forwards** (between the internet and your LAN, between zones). It
speeds up routing on a machine whose CPU can't keep up with its line.
It doesn't make the firewall any safer or less safe, but it changes what
some features can see, so read [when not to use it](#when-not-to-use-it)
before you turn it on.

It is off until you name its devices.

## What it does

Without a flowtable, every forwarded packet takes the whole way through
the kernel: the prerouting hooks, the routing decision, Housecarl's
forward chain (the built-ins, then your rules) and postrouting with NAT.
For an established connection almost all of that work only ends in the
`established` built-in accepting it again, packet after packet.

With a flowtable, Housecarl adds one built-in to the forward chain:

```
meta l4proto { tcp, udp } flow add @ft
```

A TCP or UDP connection takes the whole way for its first packets, as
always: your rules decide whether it is allowed at all. Once the kernel
has seen it established (and your rules let it through), it is added
to the flowtable. From then on its packets are caught as soon as they
arrive on one of the flowtable's devices (the *ingress* hook) and sent
straight out on the other side: NAT is still applied, the TTL still
counts down, but every chain in between is skipped.

```
                 first packets                         every packet after
in ─► prerouting ─► routing ─► forward ─► postrouting ─► out
 │                                                        ▲
 └────────────── flowtable (ingress) ─────────────────────┘
```

It is the software flowtable, run by the CPU. Offloading to the network
card's hardware isn't supported ([design.md](design.md#not-in-01)).

## When to use it

Turn it on when **the router's CPU is what limits your speed**:

* a speed test through the router gets less than your line gives (or
  less than the same test from the router itself), and
* during it, one CPU core is busy with network work: in `top`, press `1`
  for every core, and look at `si` (softirq) near 100 % on a core.

Typical cases: a small or old machine (Atom, Celeron, a Raspberry Pi, an
old Core 2) on a fast line (500 Mbit/s and up), lots of small packets
(gaming, VoIP, many downloads at once), or a machine that is busy with
other work besides routing. The gain is CPU time per packet: often well
over half of the forwarding cost, which shows up as speed only when the
CPU was the limit, and otherwise as a cooler, quieter, more responsive
machine.

## When not to use it

**When the CPU keeps up anyway.** If the speed test already reaches your
line and `si` stays low, the flowtable buys you nothing you'd notice,
and costs you the figures below. Most modern x86 machines route a
gigabit line without it.

**When you count or limit traffic in the forward chain.** The fast path
skips the forward chain, so whatever must see *every* packet of a
connection only sees the first few. Housecarl warns about each:

| Feature | With the flowtable | Warning |
|---|---|---|
| quotas and byte-rate limits on forward rules | only count first packets: a device can go far past its allowance | `flowtable_traffic` |
| forward rules with "every packet" (`--every-packet`) | only see first packets | `flowtable_traffic` |
| statistics' traffic per device (`stats_per_device`) | only count first packets | `stats_flowtable` |
| hit counters of the `established` built-in (forward) | stop rising once a connection is offloaded | |

Rate limits by packets (`--limit 10/second`) and connection limits act
on new connections, so they work the same.

**When other tables must see forwarded traffic.** Offloaded packets skip
the forward hooks of *every* table, not only Housecarl's: Docker's,
Incus', libvirt's or a table of your own. Those normally only judge new
connections, which still pass through them, but a table that counts or
shapes established traffic in its forward chain won't see it.

What keeps working as before:

* **Your rules** decide every new connection: nothing gets offloaded
  that your rules didn't let through. A rule you add later doesn't stop
  a connection already open, but that is true without the flowtable too
  (the `established` built-in accepts it).
* **NAT**: port forwards, hairpin NAT, masquerading and SNAT.
* **The input and output chains**: traffic to and from the firewall
  itself never uses the flowtable.
* **Closing connections** (`hc connections close`, Traffic →
  Connections) ends their fast path too: the kernel drops a flow with
  its tracking entry.
* **Interface statistics** (traffic per zone and interface) come from the
  interfaces' own counters, which count every packet.
* **Traffic shaping with `tc`** on the outgoing interface still applies.
* **The log** and **the packet tracer** see what goes through the rules:
  new connections, as always. The tracer doesn't see the packets of an
  offloaded connection; trace a new one.
* ICMP, and protocols other than TCP and UDP (GRE, ESP, SCTP), always
  take the normal way.

## How to turn it on

Name the devices whose traffic should take the fast path, **both
sides**: the interface connections come in on and the one their replies
come in on. For a router that means the external interface and the
inside ones:

```sh
hc settings set --flowtable wan --flowtable enp2s0
hc settings                     # Flowtable: wan, enp2s0
```

In the web UI: Settings → Flowtable, add each device. The fast path
only catches packets arriving on a listed device, so a connection to or
from an unlisted one gains little or nothing: leave a device out only on
purpose (say, a WireGuard interface whose traffic you'd rather keep on
the normal way).

* Use **exact device names**, not wildcards (`incusbr*` is refused):
  the kernel attaches the flowtable to each device by name.
* With **PPPoE**, name the PPP interface (`ppp0`, or the name your
  `ppp@` unit gives it), the one in your external zone.
* `--flowtable` replaces the whole list; `hc settings set --flowtable
  none` turns the flowtable off.

The change is applied (and on probation, if made over the network) like
any other.

### Devices that come and go

PPP and WireGuard interfaces don't exist until their service brings
them up, and PPPoE makes a new `ppp0` each time it reconnects. Housecarl
handles that:

* A device that doesn't exist yet is left out of the flowtable, with the
  warning `flowtable_device_missing`, instead of failing the whole
  ruleset.
* housecarld watches the kernel's interface changes: when a flowtable
  device appears, or comes back as a new interface after a reconnect,
  it applies the ruleset again with it, a couple of seconds later. The
  journal says `interfaces changed: applied the ruleset again`.
  Applying again keeps the counters, dynamic sets and quotas, as any
  change does, and adds no history entry.
* The ruleset saved for boot has no flowtable at all: it loads before
  the network is up, when `ppp0` and `wg0` can't exist yet, and
  nftables on Debian 13's kernel refuses a flowtable naming a device
  that doesn't exist (and with it the whole file). housecarld adds the
  flowtable once it starts. Until then, traffic takes the normal way,
  through the same rules.

## Checking it works

```sh
sudo nft list flowtables          # the devices it is attached to
sudo conntrack -L | grep OFFLOAD  # connections on the fast path
```

`conntrack` is in Debian's `conntrack` package. A long download through
the router should show up as `[OFFLOAD]`. Compare a speed test, and `si`
in `top`, before and after.

If nothing gets offloaded, check that both of the connection's devices
are listed, that `flowtable_device_missing` isn't in `hc status`, and
that the connection is TCP or UDP and established.

## Turning it off

```sh
hc settings set --flowtable none
```

Connections on the fast path go back to the normal way with their next
packet.