diff --git a/CHANGELOG.md b/CHANGELOG.md index f8e3985..a8a21a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,7 +66,38 @@ the way Jarl and Steward are. open. - **At boot:** the confirmed ruleset is loaded by `housecarl-ruleset.service` before the network comes up, without - housecarld. An empty configuration applies nothing. + housecarld. An empty configuration applies nothing. It fails closed: + if the saved ruleset doesn't load, a fallback saved beside it + (`ruleset-fallback.nft`) is, until housecarld puts the real one back: + nothing new gets in from external zones, the inside keeps working + (masqueraded), and the unit is marked failed with a critical message. + `hc status`, the dashboard and `/healthz` say while the fallback is in + force; the warning `no_external_zone` says when no zone is external, + which would make the fallback close everything (docs/operations.md). +- **The flowtable and interfaces that come and go:** the boot ruleset + has no flowtable (nftables on Debian 13's kernel refuses one naming a + device that doesn't exist yet, such as `ppp0` before PPPoE is up, and + with it the whole file); housecarld watches the kernel's link messages + and applies the ruleset again when a flowtable device appears or comes + back as a new interface (PPPoE reconnecting). A guide to when the + flowtable helps, and what it costs (docs/flowtable.md). +- **`GET /healthz` for monitoring:** `ok`, `firewall` (applied, + unconfigured, fallback, missing), changes on probation and the + machine's warnings, with HTTP 503 unless the firewall is in force; + no login needed, read at most every five seconds. +- **Import keeps a chain's head in order:** once a would-be built-in at + the head of a chain has to stay a rule (an invalid drop the forward + chain needs after an exception), the built-ins after it stay rules + too, in their place, instead of moving ahead of it: an ICMPv6 accept + no longer jumps ahead of the invalid drop it came after. +- **Renaming a zone, set or service** changes the rules, NAT rules and + port forwards that name it, in the same change (one history entry, one + undo); before, a name in use couldn't change. Dynamic sets keep their + names (their tokens and programs go by them). +- VLAN sub-interfaces (`enp2s0.20`) in zones and the flowtable, tested + with tagged traffic (docs/rules.md). +- `hc settings set --flowtable none` (and `none` for the other lists) + empties the list. - **Warnings about the machine:** while `nftables.service` is enabled (it loads `/etc/nftables.conf` at boot, and flushes every table when stopped), and while the table an import came from still applies. diff --git a/README.md b/README.md index 37337b4..3f3b668 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,8 @@ table of its own. * **The rules every firewall needs**, built in and each one switchable: loopback, established connections, invalid packets, the ICMPv6 that IPv6 needs, antispoofing for your prefixes, MSS clamping for PPPoE, the - DHCPv6 client, and a flowtable fast path. + DHCPv6 client, and a flowtable fast path + ([docs/flowtable.md](docs/flowtable.md)). * **Changes you can trust.** Each change is validated as a whole, checked by nftables itself, and applied in one transaction. As in Jarl, edits can be collected in a changeset, reviewed with the diff of the ruleset, @@ -56,7 +57,8 @@ table of its own. the history of every change, field by field, with undo. * **Its own table only.** Docker, Incus and libvirt keep theirs. The confirmed ruleset is loaded at boot, before the network comes up, even - if housecarld doesn't start. + if housecarld doesn't start; should it fail to load, a fallback keeps + anything new from the internet out until housecarld puts it back. * **Sets other programs can fill.** A dynamic set's entries expire by themselves, and a token can be limited to one set: that is how Steward's captive portal will let a guest in. @@ -102,7 +104,8 @@ The web UI is on `http://127.0.0.1:8070/` (see `listen` in * [rules.md](docs/rules.md): zones, address sets, services, rules and the built-ins * [nat.md](docs/nat.md): port forwards, hairpin NAT, IPv6 pinholes, source NAT, redirects * [import.md](docs/import.md): moving from an existing nftables ruleset -* [operations.md](docs/operations.md): probation, the boot, hits and the log, backups, troubleshooting +* [operations.md](docs/operations.md): probation, the boot and its fallback, hits and the log, monitoring, backups, troubleshooting +* [flowtable.md](docs/flowtable.md): the fast path for forwarded traffic, and when (not) to use it * [certificates.md](docs/certificates.md): HTTPS with a certificate from Let's Encrypt, through Jarl * [jarl.md](docs/jarl.md): connecting Housecarl to Jarl, and what its token may do * [cli.md](docs/cli.md): the `hc` command line client diff --git a/cmd/hc/firewall.go b/cmd/hc/firewall.go index b6fc972..52f4452 100644 --- a/cmd/hc/firewall.go +++ b/cmd/hc/firewall.go @@ -176,6 +176,10 @@ dhcpv6_client, antispoof, mss_clamp. } for flag, v := range map[string][]string{"own-prefix": own, "flowtable": flowtable, "antispoof-v4": spoof4, "antispoof-v6": spoof6} { if fl.Changed(flag) { + // "none" (or "") empties the list. + if len(v) == 1 && (v[0] == "none" || v[0] == "") { + v = []string{} + } p[map[string]string{"own-prefix": "own_prefixes", "flowtable": "flowtable_devices", "antispoof-v4": "antispoof_v4", "antispoof-v6": "antispoof_v6"}[flag]] = v } } diff --git a/cmd/housecarld/main.go b/cmd/housecarld/main.go index 2fa9a16..660acb8 100644 --- a/cmd/housecarld/main.go +++ b/cmd/housecarld/main.go @@ -30,6 +30,7 @@ import ( "housecarl/internal/engine" "housecarl/internal/hostif" "housecarl/internal/jarl" + "housecarl/internal/linkwatch" "housecarl/internal/model" "housecarl/internal/nflog" "housecarl/internal/nft" @@ -196,6 +197,24 @@ func serveCmd() *cobra.Command { a.log.Error("saving the ruleset for the next boot failed", "err", err) } } + // A flowtable device that appears later (ppp0 once PPPoE is up, + // or again after it reconnects) is added by applying again. + go func() { + err := linkwatch.Watch(ctx, func() { + host.Forget() + if !fw.FlowtableStale(eng.State()) { + return + } + if err := eng.ApplyCurrent(ctx); err != nil { + a.log.Error("applying the ruleset again for the flowtable's devices failed", "err", err) + return + } + a.log.Info("interfaces changed: applied the ruleset again, with the flowtable's devices that exist now") + }) + if err != nil { + a.log.Warn("not watching the interfaces: a flowtable device that appears later is only added by the next change", "err", err) + } + }() go nflog.Listen(ctx, render.LogGroup, a.log, srv.LogPacket) go srv.RunQuotas(ctx) srv.SetConntrack(conntrack.Client{}) diff --git a/deploy/systemd/housecarl-ruleset.service b/deploy/systemd/housecarl-ruleset.service index 002b8f8..5d799b3 100644 --- a/deploy/systemd/housecarl-ruleset.service +++ b/deploy/systemd/housecarl-ruleset.service @@ -1,5 +1,8 @@ # Housecarl: loads the last confirmed ruleset at boot, before the network -# comes up and whether or not housecarld starts. +# comes up and whether or not housecarld starts. If it fails to load, the +# fallback housecarld saved beside it is loaded instead: nothing new gets +# in from external zones (docs/operations.md), and the unit is marked +# failed, until housecarld starts and puts the real ruleset back. # Installed as /etc/systemd/system/housecarl-ruleset.service. [Unit] Description=Housecarl firewall ruleset @@ -15,7 +18,8 @@ ConditionPathExists=/var/lib/housecarl/ruleset.nft [Service] Type=oneshot RemainAfterExit=yes -ExecStart=/usr/sbin/nft -f /var/lib/housecarl/ruleset.nft +# "<2>": logged as critical. +ExecStart=/bin/sh -c '/usr/sbin/nft -f /var/lib/housecarl/ruleset.nft && exit 0; echo "<2>the saved ruleset failed to load: loading the fallback (nothing new gets in from external zones) until housecarld puts the real one back" >&2; /usr/sbin/nft -f /var/lib/housecarl/ruleset-fallback.nft; exit 1' [Install] WantedBy=sysinit.target diff --git a/docs/api.md b/docs/api.md index cc6438c..0dc52f0 100644 --- a/docs/api.md +++ b/docs/api.md @@ -74,7 +74,7 @@ Creating answers 201, other changes 200. | | | |---|---| -| `GET /healthz` | `{"ok": true, "version"}`, without authentication | +| `GET /healthz` | `ok`, `version`, `firewall` (`applied`, `unconfigured`, `fallback`, `missing`), `probation` (changes waiting to be confirmed), `warnings` (codes); HTTP 503 unless `ok`. Without authentication; read at most every five seconds ([operations.md](operations.md#monitoring)) | | `GET /status` | `build`, `started`, `applied` (the ruleset is in force), `apply_error`, `problems`, `probation`, `interfaces`, `counts` (objects by kind), `other_tables` (other nftables tables, `"inet main"`) | | `GET /interfaces` | the machine's interfaces: `name`, `up`, `addresses`, `zone` | | `GET /matrix[?changeset=ID]` | the zone matrix: `zones`, `cells` (`from`, `to` (a zone, or `firewall`), `chain`, `verdict` (allowed, blocked), `decided_by`, `rule_id`, `exceptions`: `rule_id`, `chain`, `label`, `verdict` (allowed, blocked, logged, unknown), `what`) ([rules.md](rules.md#the-zone-matrix)) | diff --git a/docs/cli.md b/docs/cli.md index aac9906..6b02443 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -148,7 +148,7 @@ by name. |---|---| | `zone list`, `zone show ZONE` | | | `zone add NAME -i IFACE... [--external] [--comment C]` | | -| `zone update ZONE [-i IFACE]... [--add IFACE] [--remove IFACE] [--external[=false]] [--name N] [--comment C]` | `-i` replaces the interfaces | +| `zone update ZONE [-i IFACE]... [--add IFACE] [--remove IFACE] [--external[=false]] [--name N] [--comment C]` | `-i` replaces the interfaces; `--name` renames it, and the rules naming it follow | | `zone delete ZONE` | refused while rules use it | | `set list`, `set show SET` | | | `set add NAME [ENTRY...] [--dynamic] [--timeout 24h] [--comment C]` | | @@ -208,7 +208,7 @@ See [nat.md](nat.md). | Command | | |---|---| | `settings show` | | -| `settings set [--input-policy\|--forward-policy\|--output-policy accept\|drop] [--log-drops on\|off] [--builtin NAME=on\|off]... [--own-prefix P]... [--antispoof-v4 P]... [--antispoof-v6 P]... [--flowtable DEV]... [--probation 2m] [--auto-confirm on\|off] [--stats-per-device on\|off]` | lists given replace the old ones | +| `settings set [--input-policy\|--forward-policy\|--output-policy accept\|drop] [--log-drops on\|off] [--builtin NAME=on\|off]... [--own-prefix P]... [--antispoof-v4 P]... [--antispoof-v6 P]... [--flowtable DEV]... [--probation 2m] [--auto-confirm on\|off] [--stats-per-device on\|off]` | lists given replace the old ones; `none` empties one (`--flowtable none`: off) | ### probation, history, import diff --git a/docs/configuration.md b/docs/configuration.md index 3b7ca93..cee76eb 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -14,7 +14,7 @@ reads the file when it starts. | `tls.cert_file`, `tls.key_file` | none | serve HTTPS with this certificate; housecarld rereads it when the file changes, or on `systemctl reload housecarl` | | `tls.acme.*` | off | get the certificate from Let's Encrypt, with Jarl publishing the challenge ([certificates.md](certificates.md#settings)) | | `jarl.server`, `jarl.token_file` (or `jarl.token`), `jarl.ca_cert`, `jarl.insecure` | none | the Jarl server for the certificate's challenge, and its token ([jarl.md](jarl.md)) | -| `ruleset` | `/var/lib/housecarl/ruleset.nft` | the confirmed ruleset, which `housecarl-ruleset.service` loads at boot | +| `ruleset` | `/var/lib/housecarl/ruleset.nft` | the confirmed ruleset, which `housecarl-ruleset.service` loads at boot; its fallback is saved beside it as `ruleset-fallback.nft` ([operations.md](operations.md#if-the-ruleset-fails-to-load-at-boot)). Change both paths in the unit too | | `trusted_proxies` | `[]` | addresses or prefixes of reverse proxies whose `X-Forwarded-For` and `X-Forwarded-Proto` are believed | | `socket` | `/run/housecarl/housecarl.sock` | the local API socket; `""` turns it off | | `database` | `/var/lib/housecarl/housecarl.db` | the SQLite database: configuration, history, users, audit log | diff --git a/docs/design.md b/docs/design.md index 1f2b9d4..c42c5f3 100644 --- a/docs/design.md +++ b/docs/design.md @@ -139,6 +139,19 @@ firewall is up even if housecarld can't start. An unconfirmed ruleset is never saved there: a reboot during probation brings back the last confirmed one. +It fails closed. A fallback saved beside it (`ruleset-fallback.nft`, +`render.Fallback`) is loaded if it doesn't: no sets, no flowtable, +interfaces by name only, so there is little in it to fail. It keeps +anything new from the external zones out and lets the inside work, +which is as far as a ruleset can go without knowing the rules; a +fallback that let everything in would leave a router open on the +internet, and one that let nothing in would lock out the person who has +to fix it. The boot ruleset has no flowtable either: nftables refuses a +flowtable naming a device that doesn't exist yet (on Debian 13's +kernel), and before the network is up, `ppp0` doesn't. housecarld +watches the kernel's link messages (internal/linkwatch) and applies +again when a flowtable device appears, or comes back as a new interface. + ## The drop log Rules with logging on, and the policies (Settings), send packets to diff --git a/docs/flowtable.md b/docs/flowtable.md new file mode 100644 index 0000000..97ec151 --- /dev/null +++ b/docs/flowtable.md @@ -0,0 +1,181 @@ +# The flowtable: a fast path for forwarded traffic + +The flowtable is a shortcut inside the kernel for traffic the firewall +**forwards** (between the internet and your LAN, between zones). It +speeds up routing on a machine whose CPU can't keep up with its line. +It doesn't make the firewall any safer or less safe, but it changes what +some features can see, so read [when not to use it](#when-not-to-use-it) +before you turn it on. + +It is off until you name its devices. + +## What it does + +Without a flowtable, every forwarded packet takes the whole way through +the kernel: the prerouting hooks, the routing decision, Housecarl's +forward chain (the built-ins, then your rules) and postrouting with NAT. +For an established connection almost all of that work only ends in the +`established` built-in accepting it again, packet after packet. + +With a flowtable, Housecarl adds one built-in to the forward chain: + +``` +meta l4proto { tcp, udp } flow add @ft +``` + +A TCP or UDP connection takes the whole way for its first packets, as +always: your rules decide whether it is allowed at all. Once the kernel +has seen it established (and your rules let it through), it is added +to the flowtable. From then on its packets are caught as soon as they +arrive on one of the flowtable's devices (the *ingress* hook) and sent +straight out on the other side: NAT is still applied, the TTL still +counts down, but every chain in between is skipped. + +``` + first packets every packet after +in ─► prerouting ─► routing ─► forward ─► postrouting ─► out + │ ▲ + └────────────── flowtable (ingress) ─────────────────────┘ +``` + +It is the software flowtable, run by the CPU. Offloading to the network +card's hardware isn't supported ([design.md](design.md#not-in-01)). + +## When to use it + +Turn it on when **the router's CPU is what limits your speed**: + +* a speed test through the router gets less than your line gives (or + less than the same test from the router itself), and +* during it, one CPU core is busy with network work: in `top`, press `1` + for every core, and look at `si` (softirq) near 100 % on a core. + +Typical cases: a small or old machine (Atom, Celeron, a Raspberry Pi, an +old Core 2) on a fast line (500 Mbit/s and up), lots of small packets +(gaming, VoIP, many downloads at once), or a machine that is busy with +other work besides routing. The gain is CPU time per packet: often well +over half of the forwarding cost, which shows up as speed only when the +CPU was the limit, and otherwise as a cooler, quieter, more responsive +machine. + +## When not to use it + +**When the CPU keeps up anyway.** If the speed test already reaches your +line and `si` stays low, the flowtable buys you nothing you'd notice, +and costs you the figures below. Most modern x86 machines route a +gigabit line without it. + +**When you count or limit traffic in the forward chain.** The fast path +skips the forward chain, so whatever must see *every* packet of a +connection only sees the first few. Housecarl warns about each: + +| Feature | With the flowtable | Warning | +|---|---|---| +| quotas and byte-rate limits on forward rules | only count first packets: a device can go far past its allowance | `flowtable_traffic` | +| forward rules with "every packet" (`--every-packet`) | only see first packets | `flowtable_traffic` | +| statistics' traffic per device (`stats_per_device`) | only count first packets | `stats_flowtable` | +| hit counters of the `established` built-in (forward) | stop rising once a connection is offloaded | | + +Rate limits by packets (`--limit 10/second`) and connection limits act +on new connections, so they work the same. + +**When other tables must see forwarded traffic.** Offloaded packets skip +the forward hooks of *every* table, not only Housecarl's: Docker's, +Incus', libvirt's or a table of your own. Those normally only judge new +connections, which still pass through them, but a table that counts or +shapes established traffic in its forward chain won't see it. + +What keeps working as before: + +* **Your rules** decide every new connection: nothing gets offloaded + that your rules didn't let through. A rule you add later doesn't stop + a connection already open, but that is true without the flowtable too + (the `established` built-in accepts it). +* **NAT**: port forwards, hairpin NAT, masquerading and SNAT. +* **The input and output chains**: traffic to and from the firewall + itself never uses the flowtable. +* **Closing connections** (`hc connections close`, Traffic → + Connections) ends their fast path too: the kernel drops a flow with + its tracking entry. +* **Interface statistics** (traffic per zone and interface) come from the + interfaces' own counters, which count every packet. +* **Traffic shaping with `tc`** on the outgoing interface still applies. +* **The log** and **the packet tracer** see what goes through the rules: + new connections, as always. The tracer doesn't see the packets of an + offloaded connection; trace a new one. +* ICMP, and protocols other than TCP and UDP (GRE, ESP, SCTP), always + take the normal way. + +## How to turn it on + +Name the devices whose traffic should take the fast path, **both +sides**: the interface connections come in on and the one their replies +come in on. For a router that means the external interface and the +inside ones: + +```sh +hc settings set --flowtable wan --flowtable enp2s0 +hc settings # Flowtable: wan, enp2s0 +``` + +In the web UI: Settings → Flowtable, add each device. The fast path +only catches packets arriving on a listed device, so a connection to or +from an unlisted one gains little or nothing: leave a device out only on +purpose (say, a WireGuard interface whose traffic you'd rather keep on +the normal way). + +* Use **exact device names**, not wildcards (`incusbr*` is refused): + the kernel attaches the flowtable to each device by name. +* With **PPPoE**, name the PPP interface (`ppp0`, or the name your + `ppp@` unit gives it), the one in your external zone. +* `--flowtable` replaces the whole list; `hc settings set --flowtable + none` turns the flowtable off. + +The change is applied (and on probation, if made over the network) like +any other. + +### Devices that come and go + +PPP and WireGuard interfaces don't exist until their service brings +them up, and PPPoE makes a new `ppp0` each time it reconnects. Housecarl +handles that: + +* A device that doesn't exist yet is left out of the flowtable, with the + warning `flowtable_device_missing`, instead of failing the whole + ruleset. +* housecarld watches the kernel's interface changes: when a flowtable + device appears, or comes back as a new interface after a reconnect, + it applies the ruleset again with it, a couple of seconds later. The + journal says `interfaces changed: applied the ruleset again`. + Applying again keeps the counters, dynamic sets and quotas, as any + change does, and adds no history entry. +* The ruleset saved for boot has no flowtable at all: it loads before + the network is up, when `ppp0` and `wg0` can't exist yet, and + nftables on Debian 13's kernel refuses a flowtable naming a device + that doesn't exist (and with it the whole file). housecarld adds the + flowtable once it starts. Until then, traffic takes the normal way, + through the same rules. + +## Checking it works + +```sh +sudo nft list flowtables # the devices it is attached to +sudo conntrack -L | grep OFFLOAD # connections on the fast path +``` + +`conntrack` is in Debian's `conntrack` package. A long download through +the router should show up as `[OFFLOAD]`. Compare a speed test, and `si` +in `top`, before and after. + +If nothing gets offloaded, check that both of the connection's devices +are listed, that `flowtable_device_missing` isn't in `hc status`, and +that the connection is TCP or UDP and established. + +## Turning it off + +```sh +hc settings set --flowtable none +``` + +Connections on the fast path go back to the normal way with their next +packet. diff --git a/docs/import.md b/docs/import.md index fdba5f7..dde2772 100644 --- a/docs/import.md +++ b/docs/import.md @@ -50,8 +50,11 @@ Import a ruleset. established, invalid, the ICMPv6 rules, MSS clamping, the DHCPv6 client rule, antispoofing) becomes that built-in only when it sits where the built-in goes, at the head of every chain it covers. Otherwise it stays - a rule where it was, so the ruleset does what it did. Built-ins your - ruleset doesn't have are turned off. + a rule where it was, so the ruleset does what it did. Since built-ins + come before every rule, once one at a chain's head stays a rule, the + ones after it there stay rules too, in order: an invalid drop that + stays a rule keeps the ICMPv6 accepts after it behind it, as they were. + Built-ins your ruleset doesn't have are turned off. * **DNAT** rules become port forwards, TCP and UDP of the same port merged into one. The forward-chain accepts that let them through become part of the port forward, and a masquerade for inside sources diff --git a/docs/operations.md b/docs/operations.md index 4d97590..71415c2 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -71,14 +71,63 @@ sudo hc history undo 42 A confirmed ruleset (or one changed through the socket) is saved to `/var/lib/housecarl/ruleset.nft`, which `housecarl-ruleset.service` -loads before the network comes up. To see what would load: +loads before the network comes up, so before PPPoE, WireGuard or DHCP +bring up their interfaces. To see what would load: ```sh sudo cat /var/lib/housecarl/ruleset.nft sudo nft -c -f /var/lib/housecarl/ruleset.nft # check it ``` -Dynamic sets start empty after a reboot. +Dynamic sets start empty after a reboot. The saved ruleset has no +flowtable, since its devices may not exist yet: housecarld adds it when +it starts, and whenever a flowtable device appears +([flowtable.md](flowtable.md#devices-that-come-and-go)). + +### If the ruleset fails to load at boot + +Should `ruleset.nft` fail to load (a damaged file, an nftables or kernel +upgrade that refuses something in it), the machine would otherwise boot +with no firewall at all until housecarld starts. Instead, +`housecarl-ruleset.service` loads a **fallback** that housecarld saves +beside it, `/var/lib/housecarl/ruleset-fallback.nft`, from the same +confirmed configuration: + +* **Nothing new gets in from the external zones' interfaces**, to the + firewall or through it: no port forwards, no SSH or web UI from the + internet. +* What the firewall and the inside need to keep working is let in: + established connections, the ICMPv6 IPv6 needs (neighbour discovery, + router advertisements, packet too big), and DHCPv6 replies from + link-local addresses. +* **Everything else is let through**: the LAN, WireGuard and the + firewall itself are reachable from the inside, so you can log in and + fix it. +* IPv4 leaving an external zone is masqueraded (when any source NAT is + configured), so the inside still reaches the internet. MSS clamping + stays, if it is on. + +PPPoE itself isn't affected: its discovery and session are Ethernet +frames, and the PPP link's negotiation happens before any IP, neither +of which an `inet` table sees. + +The fallback only knows which interfaces face the internet from the +zones marked **external**. With none, it lets nothing but loopback +through, and the warning `no_external_zone` says so. + +When the fallback loads, the unit is marked failed, and the journal says +why (as a critical message): + +```sh +systemctl status housecarl-ruleset +journalctl -b -u housecarl-ruleset +``` + +housecarld puts the real ruleset back when it starts, and saves a fresh +`ruleset.nft`. If it can't apply it either, the fallback stays in force: +`hc status` and the dashboard say "NOT in force: the boot fallback is in +force", and `/healthz` answers 503 ([Monitoring](#monitoring)). Fix +what `hc status` or the journal reports, then restart housecarld. Keep Debian's `nftables.service` disabled (`systemctl disable nftables`, without `--now`): at boot it loads `/etc/nftables.conf`, and @@ -224,6 +273,30 @@ database is kept next to it as `housecarl.db.before-restore-