Build Housecarl 0.1.0: the nftables firewall manager

Zones, address sets (IPv4 and IPv6 mixed, dynamic sets with timeouts) and
services, ordered rules in the early, input, forward and output chains,
port forwards with hairpin NAT and IPv6 pinholes, source NAT, redirects,
and switchable built-ins (loopback, conntrack, ICMPv6, antispoofing, MSS
clamping, DHCPv6 client, flowtable). Every change is validated, checked
with nft -c and applied in one transaction to Housecarl's own table,
inet housecarl; changes made over the network roll back unless
confirmed, and the confirmed ruleset is loaded at boot by
housecarl-ruleset.service. An empty configuration applies nothing.

Also: hit counters and the drop log (nflog), 'hc import' of an existing
nftables ruleset and 'hc takeover', the hc CLI, the REST API and the web
UI, users, scoped tokens (sets:write:NAME), the audit log, the web UI's
certificate through Jarl, Debian packages, docs, unit tests with real
nft 1.1.3, and an e2e test with real traffic (71 checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/design.md

@@ -8,79 +8,123 @@ talks to that API; SQLite for everything. As Jarl does for BIND, it keeps
 a Linux machine's **nftables** ruleset: the database is the source of
 truth, and the ruleset is built from it.
 
-This page is the starting sketch. Nothing here is built yet; it records
-what was decided and what is still open.
+## The model
+
+Named objects, and ordered rules that refer to them
+([rules.md](rules.md), [nat.md](nat.md)):
+
+* **Zones**: interfaces grouped by what they connect to (`wan`, `lan`,
+  `wg`). External zones face the internet.
+* **Address sets**: IPv4 and IPv6 addresses, prefixes and ranges, mixed.
+  A **dynamic** set takes entries at run time, each with a timeout that
+  nftables enforces by itself.
+* **Services**: ports of a protocol and ICMP types, by name. Many are
+  built in (`ssh`, `dns`, `https`, `wireguard`).
+* **Rules**, in order, in four chains: `input` (to the firewall),
+  `forward` (through it), `output` (from it) and `early` (before
+  connection tracking).
+* **NAT**: port forwards (with hairpin NAT and an optional IPv6 pinhole),
+  source NAT (masquerade or an address), redirects to the firewall.
+* **Settings**: the chains' policies, and the **built-ins**, the rules
+  every firewall needs (loopback, established connections, ICMPv6,
+  antispoofing, MSS clamping), each of which can be turned off.
+
+IPv4 and IPv6 are handled together: a set holds both, and a rule that
+names one becomes one or two nftables rules as needed.
 
 ## How a change is made
 
-As in Jarl, every change goes through one engine:
+Every change goes through one engine (`internal/engine`):
 
 1. the request becomes a change to a copy of the configuration;
-2. the whole result is validated, with errors that say what is wrong and
-   how to fix it;
-3. the ruleset is rendered, and checked with `nft -c -f` (a dry run by
-   nftables itself);
+2. the whole result is validated ([validation.md](validation.md)), with
+   errors that say what is wrong and how to fix it;
+3. the ruleset is rendered (`internal/render`) and checked with
+   `nft -c -f` (a dry run by nftables itself);
 4. it is applied with `nft -f`, which replaces Housecarl's table in one
-   transaction: the old rules stay in force until the new ones are all in;
-5. the change, its history entry and the audit record are saved together.
+   transaction: the old rules stay in force until the new ones are all
+   in;
+5. the objects, the history entry and the audit record are saved
+   together. If saving fails, the previous ruleset is applied again.
 
 Dry runs, history with undo, users, sessions and scoped API tokens come
-from the same machinery, as in Jarl and Steward.
-
-### Not locking yourself out
-
-A firewall managed over the network can cut off its own manager. Housecarl
-should guard against that from the start:
-
-* **Confirm or roll back.** A risky change (one touching the rules for the
-  machine itself) is applied on probation: unless it is confirmed within a
-  minute or two, from the web UI or `hc confirm`, the previous ruleset comes
-  back by itself. Network equipment calls this "commit confirmed".
-* **A management rule that can't be dropped by accident**: the API and SSH
-  stay reachable from the management network unless that rule is removed
-  on purpose.
+from the same machinery as in Jarl and Steward.
 
 ## Its own table only
 
-Housecarl writes one nftables table of its own (`inet housecarl`) and
-leaves every other table alone, such as Docker's or libvirt's. It never
-flushes the whole ruleset.
+Housecarl writes one table, `inet housecarl`, and leaves every other
+table alone (Docker's, Incus's, libvirt's). It never flushes the whole
+ruleset. The table is replaced in one script:
 
-## Concepts (draft)
+```
+table inet housecarl {}
+delete table inet housecarl
+table inet housecarl { ... }
+```
 
-* **Zones**: interfaces or VLANs grouped by trust (`lan`, `iot`, `guest`,
-  `wan`), and the default policy between each pair.
-* **Address sets**: named sets of addresses or prefixes, which rules refer
-  to. An entry may have a **timeout**, which nftables enforces by itself, so
-  an entry expires even if housecarld is down. Other programs can be given
-  a token that may only change one set.
-* **Rules** between zones: allow or drop, by set, port and protocol, with
-  logging and counters.
-* **NAT**: masquerading out of the WAN zone, and port forwards.
-* **Services**: names for ports (`ssh`, `https`, `dns`).
+Two things survive the replacement: the **entries of dynamic sets**
+(read before, written back with their remaining timeouts, in the same
+script), and the **counters** (housecarld adds the old totals to the
+new, so hits don't reset with every change).
+
+Every rule carries a comment `hc:<mark>`, such as `hc:r12` for rule 12,
+`hc:f3` for port forward 3, `hc:b-established` for a built-in. Counters
+and log lines are traced back to their object through it.
+
+## Not locking yourself out
+
+A change made over the network is applied **on probation**: unless it is
+confirmed within two minutes (Settings; `hc confirm`, the banner in the
+web UI, `POST /confirm`), it is rolled back by itself, and the rollback is
+recorded in the history. A change that cuts you off undoes itself.
+
+* Changes made through the local socket (`sudo hc …` on the firewall)
+  need no confirmation: whoever makes them is already on the machine. One
+  made while others are pending joins them.
+* A new change restarts the clock for everything pending.
+* If housecarld stops with changes pending, it rolls them back when it
+  starts.
+* Housecarl doesn't force a rule that keeps SSH or the web UI open. You
+  write the rules that let you in; probation is the safety net.
+
+## At boot
+
+The ruleset is saved to `/var/lib/housecarl/ruleset.nft` whenever it is
+confirmed (or changed through the socket). `housecarl-ruleset.service`
+loads it before the network comes up, without housecarld, so the
+firewall is up even if housecarld can't start. An unconfirmed ruleset is
+never saved there: a reboot during probation brings back the last
+confirmed one.
+
+## The drop log
+
+Rules with logging on, and the policies (Settings), send packets to
+netlink log group 4171. housecarld listens there and keeps the last 2000
+in memory (`hc log`, the Log page). Nothing is written to the kernel log.
+
+## Importing a ruleset
+
+`hc import` turns an existing nftables ruleset into zones, sets,
+services, rules and NAT, and `hc takeover` deletes the old table once
+Housecarl's replaces it ([import.md](import.md)).
 
 ## Working with Steward and Jarl
 
-* **Steward's captive portal** (parked until Housecarl exists): a guest
-  network's zone lets clients that haven't signed in reach only DNS and
-  the portal, and sends their web traffic to it. When a guest signs in,
-  Steward adds its address to a set (`guest-signed-in`) for the session's
-  length, with a token that may only change that set. Addresses, not MAC
+* **Steward's captive portal** (parked until now): a guest network's zone
+  lets clients that haven't signed in reach only DNS and the portal, and
+  sends their web traffic to it. When a guest signs in, Steward adds its
+  address to a dynamic set for the session's length, with a token that
+  may only change that set (`sets:write:guests`). Addresses, not MAC
   addresses: matching by MAC only works on the guests' own network
   segment.
-* **Names instead of addresses**: Steward knows every reservation's and
-  lease's address, so rules could name a device ("the printer") or a
-  subnet; Jarl knows the names in DNS.
-* **Steward's deny lists** could also be enforced here, so a denied device
-  with an address set by hand is still blocked.
+* **Names instead of addresses**: Steward knows every reservation's
+  address and Jarl every name, so rules could name a device or a subnet.
+* **The web UI's certificate** comes through Jarl, as Steward's does
+  ([certificates.md](certificates.md)).
 
-## Open questions
+## Not in 0.1
 
-* IPv6 from the start, or IPv4 first as Steward did?
-* Import an existing nftables (or iptables) ruleset, as Jarl imports a BIND
-  configuration?
-* Connection tracking defaults: established and related traffic allowed
-  everywhere unless a rule says otherwise?
-* Does Housecarl also run the router's interfaces and routes, or only the
-  firewall? (Only the firewall, for now.)
-* Rate limits and flow offloading: later.
+* Interfaces, addresses and routes: Housecarl only runs the firewall.
+* iptables rulesets (only nftables can be imported).
+* Rules by MAC address, by user, or by DNS name.
+* Flow offloading to hardware (the software flowtable is there).