Release 0.1.1-rc.6

Housecarl 0.1.1's sixth release candidate. Since rc.5: the dashboard
lists the rules' quotas. make test passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

AGENTS.md

@@ -30,6 +30,10 @@ Go module `housecarl`, `CGO_ENABLED=0`, Linux only for housecarld.
   on probation and rolled back unless confirmed (docs/design.md). An
   empty configuration applies nothing. But never force a rule that keeps
   SSH or the web UI open: the maintainer doesn't want one.
+* **Shadow mode changes no packet** (docs/shadow.md): anything new the
+  renderer writes that drops, rejects, translates or alters packets needs
+  its shadow form (counted, marked `[would …]`) or must be left out
+  there. `TestShadow` fails on what slips through.
 * **Stay within nftables 1.1.3** (Debian 13). The dev machine's newer nft
   accepts things 1.1.3 refuses; the tests with `HOUSECARL_NFT` and the
   e2e test use 1.1.3.

CHANGELOG.md

@@ -6,6 +6,101 @@ All notable changes to Housecarl are documented here. The format follows
 
 ## [Unreleased]
 
+### Added
+
+- **Quotas on the dashboard.** With any rule holding a quota, the
+  dashboard lists them all: the allowance, what is used (for a
+  per-address quota, how many addresses and the busiest) with a bar, and
+  the next reset. Each opens its rule.
+- **Links to a rule open it.** A rule named in the matrix, the log, the
+  tracer, statistics or the dashboard's busiest rules opens the Rules
+  page on its chain, scrolled to it and highlighted (`#/rules?rule=ID`).
+  The Rules page shows each rule's ID (`#410`, as `hc` and labels name
+  it) beside its order.
+- **A clearer zone matrix.** A cell reads allow, block, or **partly**
+  (some new connections let through, the rest not), with the exceptions
+  counted by what they do (`+3 -1` instead of `+4`), and what they let
+  through: services with the addresses they're limited to (`ssh from
+  trusted`, `5 port forwards`), under the grid in `hc matrix`, in each
+  cell on the Matrix page. A zone to itself is marked (only traffic routed
+  between its interfaces passes the firewall). The details list each
+  rule's hits. API: `summary`, `allowed`, `blocked`, `open`,
+  `same_zone`, `note`, `hits`, and `short` and `hits` per exception.
+  Raw rules that are a match and then accept, drop or reject (without
+  interfaces) are read instead of shown as unknown.
+- **Shadow mode** (`shadow.enabled` in housecarl.yaml; docs/shadow.md):
+  Housecarl's ruleset is loaded beside the firewall you have but blocks,
+  rejects and translates nothing. Drops and rejects are counted and accept
+  instead (an accept only ends Housecarl's chain), policies accept and
+  always log what they would drop, NAT only counts, MSS clamping and the
+  flowtable are left out, and raw rules that do more than accept, drop or
+  reject are left out with a warning (`shadow_raw_left_out`). Comments
+  mark what would happen (`[would drop]`). `shadow.position`: Housecarl's
+  filter chains run after the other firewall's (`filter + 10`, the
+  default: what it would drop is what switching would break) or before
+  (`- 10`); its NAT chains always first. Every change is checked by
+  nftables in both forms, so leaving shadow mode can't fail. No
+  probation; takeovers and closing connections are refused (409). The
+  boot ruleset is the shadow one, and its fallback filters nothing. When
+  a reload (`flush ruleset`) deletes Housecarl's table, housecarld puts it
+  back (it listens to nftables' netlink notifications). `mode` in
+  `/status`, `/healthz` and `/auth/whoami`; a banner in the web UI;
+  would-be drops marked in the log (`shadow`) and the tracer
+  (`would_drop`); `hc ruleset --enforcing` (`GET /ruleset?enforcing=1`)
+  shows the ruleset for after. `shadow_unprotected` warns when no other
+  table filters the input hook; `nftables_service` and `takeover_pending`
+  aren't raised in shadow mode. The e2e test imports in shadow mode
+  before taking over.
+- **Zone names at import:** `hc import --zone enp2s0=lan` (repeat;
+  interfaces given one name share a zone; Zone names on the Import
+  page), and a note listing each zone and where its name came from.
+  Defines holding a VLAN sub-interface (`enp2s0.20`) name their zone,
+  and `_ifname`, `_device` and `_nic` are taken off define names too
+  (docs/import.md, "How things are named").
+- Import: sets with `flags timeout` become dynamic sets; a DNAT of
+  several ports becomes one port forward per port.
+
+### Fixed
+
+- `hc rule add --before/--after/--first` made two changes (create at the
+  end, then move), so two history entries, two probations, and the rule
+  briefly in force at the end of its chain; staged in a changeset, the
+  place was lost. The create now takes the place (`POST
+  /rules?before=ID`, `after`, `first`) and makes one change.
+- `hc matrix` in a terminal: the colours' escape codes threw the columns
+  out of line.
+- `hc counters` (`GET /counters`) kept listing the objects of a
+  configuration replaced since housecarld started (`hc import
+  --replace` gives every object a new ID), beside the current ones; the
+  totals of objects that no longer exist are now forgotten.
+- Import: two base chains on one hook had their rules imported twice,
+  and the second chain's leading built-ins were moved above the first
+  chain's rules; they now follow each other by priority (read in a
+  fixed order), with the stricter policy, and a note.
+- Import: a rule inside a jumped-to chain that stays raw kept only its
+  own text and so applied to everything; the jump's condition now goes
+  in front. A jump and a rule matching the same field are combined only
+  when one's addresses lie within the other's; else the rule stays raw.
+  A jump's ports, ICMP types, connection status and hop limit were
+  dropped; a limit or log on the jump is no longer silently lost.
+- Import: a second match on the same field (`ip saddr A ip saddr != B`)
+  replaced the first; `!=` on the hop limit and TCP flags was read as
+  `==`. Such rules now stay raw, as written.
+- Import: a forward-chain accept to a forwarded host and port without
+  `ct status dnat` was folded into the port forward, which only lets
+  forwarded connections through; it stays a rule now. An IPv6 accept
+  limited to an interface no longer becomes a pinhole open to all.
+- Import: raw rules naming a set of the old table made nftables refuse
+  the ruleset; address sets are now renamed `NAME_v4`/`NAME_v6`, other
+  sets written out with their elements, and what can't be is left out
+  with a note. Rules naming a set that isn't imported stay raw instead
+  of failing validation.
+- Import: a DNAT to an IPv6 address, or with matches a port forward
+  can't hold (an interface excluded, a limit, a log), is left out with a
+  note instead of being imported without them.
+
+## [0.1.0] - 2026-10-19
+
 The first version of Housecarl, a firewall manager for nftables, built
 the way Jarl and Steward are.
 
@@ -35,7 +130,8 @@ the way Jarl and Steward are.
   review shows the problems, nftables' check and the diff of the ruleset.
   The web UI collects edits as pending changes by default (pending rows,
   a bar with Review and Apply now, a Changes page), with a switch for live
-  edits; hc stages with `-C ID` or `HOUSECARL_CHANGESET`, and has `hc
+  edits, and Undo in History is staged like any edit; hc stages with
+  `-C ID` or `HOUSECARL_CHANGESET`, and has `hc
   changeset create/list/show/diff/apply/discard/remove`. An applied
   changeset is on probation like any change. `--dry-run` shows the
   ruleset's diff too (docs/design.md, docs/cli.md).
@@ -55,13 +151,74 @@ the way Jarl and Steward are.
   own table, `inet housecarl`, and nothing else. Dynamic sets' entries
   and the counters carry across.
 - **Probation:** a change made over the network rolls itself back unless
-  confirmed within two minutes (`hc confirm`, a banner in the web UI);
+  confirmed within two minutes. The web UI and `hc` confirm their own
+  changes once they reach Housecarl again over a new connection, from the
+  same address (a change that cuts its client off can't, and rolls back);
+  `--no-confirm` and the "a person confirms each change" setting leave it
+  to `hc confirm` or the banner;
   changes through the local socket are kept at once; a restart rolls
   back what was pending. No rule is forced to keep SSH or the web UI
   open.
 - **At boot:** the confirmed ruleset is loaded by
   `housecarl-ruleset.service` before the network comes up, without
-  housecarld. An empty configuration applies nothing.
+  housecarld. An empty configuration applies nothing. It fails closed:
+  if the saved ruleset doesn't load, a fallback saved beside it
+  (`ruleset-fallback.nft`) is, until housecarld puts the real one back:
+  nothing new gets in from external zones, the inside keeps working
+  (masqueraded), and the unit is marked failed with a critical message.
+  `hc status`, the dashboard and `/healthz` say while the fallback is in
+  force; the warning `no_external_zone` says when no zone is external,
+  which would make the fallback close everything (docs/operations.md).
+- **The flowtable and interfaces that come and go:** the boot ruleset
+  has no flowtable (nftables on Debian 13's kernel refuses one naming a
+  device that doesn't exist yet, such as `ppp0` before PPPoE is up, and
+  with it the whole file); housecarld watches the kernel's link messages
+  and applies the ruleset again when a flowtable device appears or comes
+  back as a new interface (PPPoE reconnecting). A guide to when the
+  flowtable helps, and what it costs (docs/flowtable.md).
+- **`GET /healthz` for monitoring:** `ok`, `firewall` (applied,
+  unconfigured, fallback, missing), changes on probation and the
+  machine's warnings, with HTTP 503 unless the firewall is in force;
+  no login needed, read at most every five seconds.
+- **Import keeps a chain's head in order:** once a would-be built-in at
+  the head of a chain has to stay a rule (an invalid drop the forward
+  chain needs after an exception), the built-ins after it stay rules
+  too, in their place, instead of moving ahead of it: an ICMPv6 accept
+  no longer jumps ahead of the invalid drop it came after.
+- **Renaming a zone, set or service** changes the rules, NAT rules and
+  port forwards that name it, in the same change (one history entry, one
+  undo); before, a name in use couldn't change. Dynamic sets keep their
+  names (their tokens and programs go by them).
+- VLAN sub-interfaces (`enp2s0.20`) in zones and the flowtable, tested
+  with tagged traffic (docs/rules.md).
+- `hc settings set --flowtable none` (and `none` for the other lists)
+  empties the list.
+- **Warnings about the machine:** while `nftables.service` is enabled
+  (it loads `/etc/nftables.conf` at boot, and flushes every table when
+  stopped), and while the table an import came from still applies.
+- **Statistics** (Traffic → Statistics, `hc stats`, `/api/v1/stats/...`):
+  traffic per interface and zone, drops by chain and when they happen,
+  hits per rule and the rules that matched nothing, the sources blocked
+  most and what they tried, connections against the kernel's limit, each
+  quota's use, and housecarld's and the host's memory and load, sampled
+  every minute into a database of their own (`stats.db`, kept 7 days per
+  minute, 90 days per hour, 2 years per day). On by default; traffic per
+  inside device is a setting (`--stats-per-device`) that adds counters
+  to the forward chain. Two charts on the dashboard; a warning when the
+  connection tracking table is nearly full (docs/stats.md).
+- Set names starting with `meter_`, `conn_`, `quota_` or `stats_` are
+  refused (`set_name_reserved`): Housecarl's own sets use them, and a set
+  of yours named so could clash with them.
+- **The zone matrix** (Matrix, `hc matrix`): what the rules do to new
+  connections between zones and to and from the firewall, worked out
+  from the rules in order, with each pair's exceptions; rules about
+  networks connected to other zones are left out of a pair.
+- **Connections and tracing:** the connections the kernel tracks, with
+  their NAT, and closing them so the rules check them again (Traffic →
+  Connections, `hc connections`); a rule option, "every packet", for
+  blocks that stop connections already open; and a packet tracer
+  (Traffic → Trace, `hc trace`) that follows packets through every
+  table's rules and names what accepted or dropped them.
 - **Hits and the log:** packet and byte counters for every rule, NAT rule,
   built-in and policy; packets logged by rules and policies, read from
   netlink log group 4171 (`hc log -f`, the Log page).
@@ -75,7 +232,10 @@ the way Jarl and Steward are.
   verdict maps are expanded into one port forward or rule per element.
 - **The `hc` CLI, the REST API and the web UI** (dashboard, rules per
   chain with hits and reordering, NAT, zones, sets, services, the log,
-  settings, import, history with undo), with users, roles, scoped API
+  settings, import, history with undo; the rule dialog keeps the less
+  used parts, such as rate limits, quotas and time windows, behind "+"
+  buttons; each chain's built-ins are listed, with their hits, and
+  switched in a window), with users, roles, scoped API
   tokens, sessions, the audit log, and the web UI's certificate from
   Let's Encrypt through Jarl, as in Steward.
 - **Packaging:** Debian packages `housecarl` and `housecarl-cli`, a

README.md

@@ -37,7 +37,8 @@ table of its own.
 * **The rules every firewall needs**, built in and each one switchable:
   loopback, established connections, invalid packets, the ICMPv6 that
   IPv6 needs, antispoofing for your prefixes, MSS clamping for PPPoE, the
-  DHCPv6 client, and a flowtable fast path.
+  DHCPv6 client, and a flowtable fast path
+  ([docs/flowtable.md](docs/flowtable.md)).
 * **Changes you can trust.** Each change is validated as a whole, checked
   by nftables itself, and applied in one transaction. As in Jarl, edits
   can be collected in a changeset, reviewed with the diff of the ruleset,
@@ -48,11 +49,20 @@ table of its own.
   ruleset into zones, sets, services, rules and port forwards, keeping the
   rules' order, and keeps what it can't translate as raw rules
   ([docs/import.md](docs/import.md)).
-* **Easy to watch.** Hits per rule, a live log of what was dropped and
-  why, and the history of every change, field by field, with undo.
+* **Try it first, in shadow mode.** Housecarl's ruleset runs beside the
+  firewall you have, blocking nothing: its hits, log and tracer show what
+  it would drop, so you see what switching over would break before you do
+  ([docs/shadow.md](docs/shadow.md)).
+* **Easy to watch.** A zone matrix of what is allowed between zones,
+  statistics over time (traffic per zone and device, drops, the sources
+  blocked most, connections; [docs/stats.md](docs/stats.md)), hits per rule, a live log of what was dropped and
+  why, the connections the firewall tracks (and closing them), a packet
+  tracer that names the rule that dropped something, in any table, and
+  the history of every change, field by field, with undo.
 * **Its own table only.** Docker, Incus and libvirt keep theirs. The
   confirmed ruleset is loaded at boot, before the network comes up, even
-  if housecarld doesn't start.
+  if housecarld doesn't start; should it fail to load, a fallback keeps
+  anything new from the internet out until housecarld puts it back.
 * **Sets other programs can fill.** A dynamic set's entries expire by
   themselves, and a token can be limited to one set: that is how
   Steward's captive portal will let a guest in.
@@ -98,7 +108,9 @@ The web UI is on `http://127.0.0.1:8070/` (see `listen` in
 * [rules.md](docs/rules.md): zones, address sets, services, rules and the built-ins
 * [nat.md](docs/nat.md): port forwards, hairpin NAT, IPv6 pinholes, source NAT, redirects
 * [import.md](docs/import.md): moving from an existing nftables ruleset
-* [operations.md](docs/operations.md): probation, the boot, hits and the log, backups, troubleshooting
+* [shadow.md](docs/shadow.md): shadow mode, Housecarl beside your firewall, changing nothing
+* [operations.md](docs/operations.md): probation, the boot and its fallback, hits and the log, monitoring, backups, troubleshooting
+* [flowtable.md](docs/flowtable.md): the fast path for forwarded traffic, and when (not) to use it
 * [certificates.md](docs/certificates.md): HTTPS with a certificate from Let's Encrypt, through Jarl
 * [jarl.md](docs/jarl.md): connecting Housecarl to Jarl, and what its token may do
 * [cli.md](docs/cli.md): the `hc` command line client
@@ -111,12 +123,14 @@ The web UI is on `http://127.0.0.1:8070/` (see `listen` in
 
 ## Status
 
-Housecarl 0.1.0 is in testing (0.1.0-rc.2): zones, address sets and
+Housecarl 0.1.1 is in testing (0.1.1-rc.6): zones, address sets and
 services with ordered rules, IPv4 and IPv6, port forwards with hairpin NAT
 and IPv6 pinholes, source NAT and redirects, built-ins, probation for
-network changes, the ruleset at boot, hits and the drop log, nftables
+network changes (confirmed by the client that made them), the ruleset
+at boot with a fail-closed fallback, hits and the drop log, nftables
 import, changesets reviewed with the ruleset's diff (as in Jarl),
-bandwidth limits and quotas, a web UI, REST API and CLI.
+bandwidth limits and quotas, shadow mode beside another firewall, a web
+UI, REST API and CLI.
 
 ## Licence
 

VERSION

@@ -1 +1 @@
-0.1.0-rc.2
+0.1.1-rc.6

cmd/hc/firewall.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
+	"net/url"
 	"os"
 	"os/exec"
 	"sort"
@@ -30,6 +31,9 @@ func statusCmd() *cobra.Command {
 			}
 			return output(st, func(w io.Writer) {
 				fmt.Fprintf(w, "Housecarl:\t%s, up since %s\n", st.Build.Label, st.Started.Local().Format("2006-01-02 15:04"))
+				if st.Mode == api.ModeShadow {
+					fmt.Fprintf(w, "Mode:\tSHADOW: the ruleset blocks, rejects and translates nothing; the other firewall decides (Housecarl's chains run %s it)\n", st.ShadowPosition)
+				}
 				if st.Applied {
 					fmt.Fprintln(w, "Ruleset:\tin force (table inet housecarl)")
 				} else if st.Unconfigured {
@@ -125,10 +129,16 @@ func settingsCmd() *cobra.Command {
 				fmt.Fprintf(w, "Antispoof IPv6:\t%s\n", list(s.AntispoofV6))
 				fmt.Fprintf(w, "Flowtable:\t%s\n", list(s.FlowtableDevices))
 				fmt.Fprintf(w, "Probation:\t%s\n", fmtSeconds(s.Probation()))
+				fmt.Fprintf(w, "Traffic per device:\t%s\n", map[bool]string{true: "counted (statistics)", false: "not counted"}[s.StatsPerDevice])
+				if s.ManualConfirm {
+					fmt.Fprintln(w, "Confirmation:\tby a person ('hc confirm', the web UI's banner)")
+				} else {
+					fmt.Fprintln(w, "Confirmation:\tautomatic, by the client that made the change, once it reaches Housecarl again over a new connection")
+				}
 			})
 		},
 	})
-	var input, forward, output, logDrops, probation string
+	var input, forward, output, logDrops, probation, autoConf, perDevice string
 	var builtins, own, flowtable, spoof4, spoof6 []string
 	set := &cobra.Command{
 		Use: "set [flags]", Short: "Change settings (only the flags given)", Args: cobra.NoArgs,
@@ -138,7 +148,8 @@ dhcpv6_client, antispoof, mss_clamp.
 
   hc settings set --builtin mss_clamp=on --own-prefix 2001:db8::/48
   hc settings set --forward-policy drop --log-drops on
-  hc settings set --flowtable wan --flowtable eth1`,
+  hc settings set --flowtable wan --flowtable eth1
+  hc settings set --stats-per-device on`,
 		RunE: func(cmd *cobra.Command, args []string) error {
 			fl := cmd.Flags()
 			p := patch{}
@@ -169,9 +180,27 @@ dhcpv6_client, antispoof, mss_clamp.
 			}
 			for flag, v := range map[string][]string{"own-prefix": own, "flowtable": flowtable, "antispoof-v4": spoof4, "antispoof-v6": spoof6} {
 				if fl.Changed(flag) {
+					// "none" (or "") empties the list.
+					if len(v) == 1 && (v[0] == "none" || v[0] == "") {
+						v = []string{}
+					}
 					p[map[string]string{"own-prefix": "own_prefixes", "flowtable": "flowtable_devices", "antispoof-v4": "antispoof_v4", "antispoof-v6": "antispoof_v6"}[flag]] = v
 				}
 			}
+			if fl.Changed("stats-per-device") {
+				on, err := onOff("stats-per-device", perDevice)
+				if err != nil {
+					return err
+				}
+				p["stats_per_device"] = on
+			}
+			if fl.Changed("auto-confirm") {
+				on, err := onOff("auto-confirm", autoConf)
+				if err != nil {
+					return err
+				}
+				p["manual_confirm"] = !on
+			}
 			if fl.Changed("probation") {
 				n, err := parseSeconds(probation)
 				if err != nil {
@@ -193,6 +222,8 @@ dhcpv6_client, antispoof, mss_clamp.
 	f.StringArrayVar(&spoof4, "antispoof-v4", nil, "IPv4 sources dropped from the internet (replaces)")
 	f.StringArrayVar(&spoof6, "antispoof-v6", nil, "IPv6 sources dropped from the internet (replaces)")
 	f.StringVar(&probation, "probation", "", "how long a network change has to be confirmed (2m)")
+	f.StringVar(&perDevice, "stats-per-device", "", "count traffic per inside device, for the statistics' top devices: on or off")
+	f.StringVar(&autoConf, "auto-confirm", "", "on: the client that made a change confirms it once it reaches Housecarl again over a new connection; off: a person confirms")
 	cmd.AddCommand(set)
 	return cmd
 }
@@ -210,21 +241,28 @@ func onOff(name, v string) (bool, error) {
 // ---- the ruleset, counters, the log
 
 func rulesetCmd() *cobra.Command {
-	return &cobra.Command{
+	var enforcing bool
+	cmd := &cobra.Command{
 		Use: "ruleset", Short: "Print the nftables ruleset Housecarl builds", Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, args []string) error {
 			c, err := newClient()
 			if err != nil {
 				return err
 			}
+			var q url.Values
+			if enforcing {
+				q = url.Values{"enforcing": {"1"}}
+			}
 			var text string
-			if err := c.Do(ctx(), "GET", "/ruleset", nil, nil, &text); err != nil {
+			if err := c.Do(ctx(), "GET", "/ruleset", q, nil, &text); err != nil {
 				return err
 			}
 			fmt.Print(text)
 			return nil
 		},
 	}
+	cmd.Flags().BoolVar(&enforcing, "enforcing", false, "in shadow mode: the ruleset loaded once it is off")
+	return cmd
 }
 
 func countersCmd() *cobra.Command {
@@ -319,6 +357,9 @@ func logCmd() *cobra.Command {
 					if e.TCPFlags != "" {
 						what += " " + e.TCPFlags
 					}
+					if e.Shadow && (e.Verdict == "drop" || e.Verdict == "reject") {
+						what += "  (shadow mode: would " + e.Verdict + ", let through)"
+					}
 					fmt.Printf("%s  %-28s %-6s %-6s %s -> %s  %s\n", e.Time.Local().Format("15:04:05"), strings.TrimSpace(e.Prefix), orDash(e.In), orDash(e.Out), src, dst, what)
 				}
 			}
@@ -407,6 +448,7 @@ func rollbackCmd() *cobra.Command {
 func importCmd() *cobra.Command {
 	var replace bool
 	var table, defines, textFile string
+	var zones []string
 	cmd := &cobra.Command{
 		Use:   "import [FILE|-]",
 		Short: "Import an nftables ruleset (nft -j JSON) as Housecarl's configuration",
@@ -418,7 +460,9 @@ forwards and NAT. The input is nft's JSON; on the firewall:
 Without a FILE, hc runs 'nft -j list ruleset' and 'nft -a list ruleset'
 itself (root on the firewall); from elsewhere, give the JSON as FILE and
 the text with --text. --defines reads the ruleset file's define lines, to
-name the sets as you did (admins, lan_net, …). -n shows the result without
+name the sets as you did (admins, lan_net, …), and the zones; --zone
+enp2s0=lan names one yourself (repeat; interfaces given one name share a
+zone). -n shows the result without
 saving; what can't be expressed becomes raw rules, listed with notes.
 Once it looks right, import without -n, check it with 'hc ruleset', and
 'hc takeover inet main' deletes the old table.`,
@@ -446,7 +490,14 @@ Once it looks right, import without -n, check it with 'hc ruleset', and
 					return err
 				}
 			}
-			in := api.ImportInput{JSON: data, Text: string(text), Table: table, Replace: replace}
+			in := api.ImportInput{JSON: data, Text: string(text), Table: table, Replace: replace, Zones: map[string]string{}}
+			for _, z := range zones {
+				iface, name, ok := strings.Cut(z, "=")
+				if !ok || iface == "" || name == "" {
+					return fmt.Errorf("--zone %q: write it as INTERFACE=NAME (enp2s0=lan)", z)
+				}
+				in.Zones[iface] = name
+			}
 			if defines != "" {
 				b, err := os.ReadFile(defines)
 				if err != nil {
@@ -481,6 +532,7 @@ Once it looks right, import without -n, check it with 'hc ruleset', and
 	cmd.Flags().StringVar(&table, "table", "", "import only this table (\"inet main\"; default: all but Housecarl's)")
 	cmd.Flags().StringVar(&defines, "defines", "", "the ruleset file, whose defines name the sets")
 	cmd.Flags().StringVar(&textFile, "text", "", "'nft -a list ruleset' output, to keep what can't be expressed as raw rules")
+	cmd.Flags().StringArrayVar(&zones, "zone", nil, "name an interface's zone: INTERFACE=NAME (repeat)")
 	return cmd
 }
 
@@ -510,6 +562,13 @@ come back. Don't forget the boot: disable nftables.service, or empty
 				return err
 			}
 			fmt.Println(out["message"])
+			if tok, _ := out["confirm_token"].(string); tok != "" && !flagNoConfirm {
+				if err := autoConfirm(tok); err != nil {
+					fmt.Printf("couldn't confirm it (%v): confirm with 'hc confirm', or the tables come back\n", err)
+				} else {
+					fmt.Println("confirmed: this client reached Housecarl again over a new connection")
+				}
+			}
 			return nil
 		},
 	}

cmd/hc/main.go

@@ -45,6 +45,10 @@ var (
 	flagInsecure  bool
 	flagCACert    string
 	flagQuiet     bool
+	flagNoConfirm bool
+	// lastClient is the client the command made (to confirm a change by
+	// itself after it).
+	lastClient *client.Client
 )
 
 // DefaultSocket is housecarld's local socket.
@@ -135,6 +139,7 @@ func newClient() (*client.Client, error) {
 	if err := setupChangeset(c); err != nil {
 		return nil, err
 	}
+	lastClient = c
 	return c, nil
 }
 
@@ -151,9 +156,10 @@ Changes are made at once; to review several and apply them together,
 stage them in a changeset ('hc changeset --help', -C ID or
 HOUSECARL_CHANGESET).
 
-A change made over the network is on probation: confirm it with 'hc
-confirm' (within 2 minutes by default) or it is rolled back by itself, so a
-change that cuts you off undoes itself. On the firewall itself, root uses
+A change made over the network is on probation: hc confirms it by itself
+once it reaches Housecarl again over a new connection, and a change that
+cuts you off can't be confirmed, so it is rolled back by itself (within 2
+minutes by default). --no-confirm leaves it to 'hc confirm'. On the firewall itself, root uses
 housecarld's local socket: no token, no probation. From elsewhere, set the
 server and an API token once:
 
@@ -175,6 +181,7 @@ Exit codes: 0 ok, 1 error, 3 not authenticated or not allowed, 4 not found,
 	pf.BoolVar(&flagInsecure, "insecure", false, "don't verify the server's TLS certificate")
 	pf.StringVar(&flagCACert, "ca-cert", "", "PEM file with a CA to trust for the server's certificate")
 	pf.BoolVarP(&flagQuiet, "quiet", "q", false, "print only errors (and IDs of new objects)")
+	pf.BoolVar(&flagNoConfirm, "no-confirm", false, "leave a change on probation instead of confirming it once this client reaches Housecarl again")
 	pf.Int64VarP(&flagChangeset, "changeset", "C", 0, "stage configuration changes in this changeset instead of applying them (env HOUSECARL_CHANGESET)")
 	root.AddGroup(&cobra.Group{ID: "fw", Title: "Firewall:"}, &cobra.Group{ID: "changes", Title: "Changes:"}, &cobra.Group{ID: "admin", Title: "Administration:"})
 	add := func(group string, cmds ...*cobra.Command) {
@@ -184,7 +191,7 @@ Exit codes: 0 ok, 1 error, 3 not authenticated or not allowed, 4 not found,
 		}
 	}
 	add("fw", statusCmd(), zoneCmd(), setCmd(), serviceCmd(), ruleCmd(), forwardCmd(), snatCmd(), redirectCmd(), settingsCmd(),
-		rulesetCmd(), countersCmd(), quotasCmd(), logCmd(), interfaceCmd())
+		rulesetCmd(), countersCmd(), quotasCmd(), logCmd(), connectionsCmd(), traceCmd(), matrixCmd(), statsCmd(), interfaceCmd())
 	add("changes", changesetCmd(), confirmCmd(), rollbackCmd(), historyCmd(), importCmd(), takeoverCmd())
 	add("admin", whoamiCmd(), userCmd(), tokenCmd(), passwordCmd(), certCmd(), auditCmd(), configCmd())
 	return root
@@ -299,6 +306,19 @@ func exitCode(err error) int {
 	return 1
 }
 
+// autoConfirm confirms a change with its token: the request goes over a
+// new connection (the server closed the change's), so it gets through
+// only if the new ruleset still lets this client reach Housecarl.
+func autoConfirm(token string) error {
+	if lastClient == nil {
+		return errors.New("no client")
+	}
+	c, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	var out map[string]int
+	return lastClient.Do(c, "POST", "/confirm", nil, api.ConfirmInput{Token: token}, &out)
+}
+
 // report prints the outcome of a change.
 func report(resp *api.MutationResponse) error {
 	switch flagOutput {
@@ -343,6 +363,13 @@ func report(resp *api.MutationResponse) error {
 			fmt.Fprintln(w)
 		}
 		fmt.Fprintln(w, "dry run: nothing was changed")
+	case resp.Applied && resp.ConfirmBy != nil && resp.ConfirmToken != "" && !flagNoConfirm:
+		if err := autoConfirm(resp.ConfirmToken); err != nil {
+			fmt.Fprintf(w, "applied on probation (history %d), but couldn't confirm it: %v\nIt is rolled back within %s unless confirmed ('hc confirm')\n",
+				resp.HistoryID, err, time.Until(*resp.ConfirmBy).Round(time.Second))
+		} else if !flagQuiet {
+			fmt.Fprintf(w, "applied and confirmed (history %d): this client reached Housecarl again over a new connection\n", resp.HistoryID)
+		}
 	case resp.Applied && resp.ConfirmBy != nil:
 		fmt.Fprintf(w, "applied on probation (history %d): confirm with 'hc confirm' within %s, or it is rolled back\n",
 			resp.HistoryID, time.Until(*resp.ConfirmBy).Round(time.Second))

cmd/hc/main_test.go

@@ -1,8 +1,11 @@
 package main
 
 import (
+	"regexp"
+	"strings"
 	"testing"
 
+	"housecarl/internal/matrix"
 	"housecarl/internal/model"
 )
 
@@ -40,3 +43,26 @@ func TestLimitAndDescribe(t *testing.T) {
 		t.Errorf("editList: %v", got)
 	}
 }
+
+// TestMatrixAligned: the grid's columns line up in colour too (the
+// escape codes take no room on screen).
+func TestMatrixAligned(t *testing.T) {
+	st := model.NewState()
+	st.Zones = []*model.Zone{{ID: 1, Name: "lan", Interfaces: []string{"eth1"}}, {ID: 2, Name: "wan", Interfaces: []string{"wan"}, External: true}}
+	st.Rules = []*model.Rule{{ID: 3, Chain: model.ChainForward, Position: 1, Action: model.ActionAccept, Match: model.Match{InZones: []string{"lan"}, Services: []string{"https"}}}}
+	st.Settings.InputPolicy, st.Settings.ForwardPolicy = model.PolicyDrop, model.PolicyDrop
+	var b strings.Builder
+	writeMatrix(&b, matrix.Build(st), true)
+	esc := regexp.MustCompile("\x1b\\[[0-9]+m")
+	lines := strings.Split(esc.ReplaceAllString(b.String(), ""), "\n")
+	col := strings.Index(lines[0], "wan")
+	for _, l := range lines[1:4] {
+		if r := []rune(l); len(r) <= col || r[col-1] != ' ' || r[col] == ' ' {
+			t.Errorf("misaligned at column %d:\n%s", col, strings.Join(lines[:4], "\n"))
+			break
+		}
+	}
+	if !strings.Contains(b.String(), "\x1b[33mpartly +1") || !strings.Contains(b.String(), "lan → wan\thttps") {
+		t.Errorf("no partly cell, or no list of what's open:\n%s", b.String())
+	}
+}

cmd/hc/rules.go

@@ -3,6 +3,7 @@ package main
 import (
 	"fmt"
 	"io"
+	"net/url"
 	"regexp"
 	"strconv"
 	"strings"
@@ -99,7 +100,7 @@ type ruleFlags struct {
 	states, statuses, days                      []string
 	hours, limit, quota                         string
 	connLimit, groupV4, groupV6                 int
-	log, disabled                               bool
+	log, disabled, everyPacket                  bool
 	before, after                               int64
 	first                                       bool
 }
@@ -122,6 +123,7 @@ func (r *ruleFlags) register(f *pflag.FlagSet) {
 	f.StringVar(&r.raw, "raw", "", "an nftables rule written by hand instead of the match flags")
 	f.StringVar(&r.comment, "comment", "", "a comment")
 	f.BoolVar(&r.disabled, "disabled", false, "keep the rule but don't use it")
+	f.BoolVar(&r.everyPacket, "every-packet", false, "check every packet, so a drop or reject also stops connections already open")
 }
 
 var amountRe = regexp.MustCompile(`^(\d+)\s*([a-zA-Z]*)$`)
@@ -244,7 +246,7 @@ func humanBytes(n int64) string {
 
 func (r *ruleFlags) rule(chain string) (model.Rule, error) {
 	rule := model.Rule{Chain: chain, Match: r.match(), Action: r.action, RejectWith: r.rejectWith, CtState: r.states, CtStatus: r.statuses,
-		Days: r.days, ConnLimit: r.connLimit, Log: r.log, LogPrefix: r.logPrefix, Raw: r.raw, Comment: r.comment, Disabled: r.disabled}
+		Days: r.days, ConnLimit: r.connLimit, EveryPacket: r.everyPacket, Log: r.log, LogPrefix: r.logPrefix, Raw: r.raw, Comment: r.comment, Disabled: r.disabled}
 	if r.hours != "" {
 		from, to, ok := strings.Cut(r.hours, "-")
 		if !ok {
@@ -285,6 +287,9 @@ func ruleCmd() *cobra.Command {
 					}
 					extra := ruleExtras(r)
 					action := r.Action
+					if r.Raw != "" {
+						action = "-" // the text's own
+					}
 					if r.Log {
 						action += "+log"
 					}
@@ -298,7 +303,11 @@ func ruleCmd() *cobra.Command {
 	}
 	cmd.AddCommand(ls)
 	cmd.AddCommand(showCmd("rules", "rule", func(w io.Writer, r model.Rule) {
+		if r.Raw == "" {
 			fmt.Fprintf(w, "ID:\t%d\nChain:\t%s\nMatch:\t%s%s\nAction:\t%s\n", r.ID, r.Chain, describe(r.Match), ruleExtras(r), r.Action)
+		} else {
+			fmt.Fprintf(w, "ID:\t%d\nChain:\t%s\n", r.ID, r.Chain)
+		}
 		if r.Raw != "" {
 			fmt.Fprintf(w, "Raw:\t%s\n", r.Raw)
 		}
@@ -332,26 +341,21 @@ output (from it) or early (before connection tracking: cheap drops).
 			if err != nil {
 				return err
 			}
+			// Placed in the same change: one history entry, and never in
+			// force at the end of the chain.
+			q := url.Values{}
+			switch {
+			case rf.first:
+				q.Set("first", "1")
+			case rf.before != 0:
+				q.Set("before", fmt.Sprint(rf.before))
+			case rf.after != 0:
+				q.Set("after", fmt.Sprint(rf.after))
+			}
 			var resp api.MutationResponse
-			if err := c.Do(ctx(), "POST", "/rules", nil, r, &resp); err != nil {
+			if err := c.Do(ctx(), "POST", "/rules", q, r, &resp); err != nil {
 				return err
 			}
-			// Then place it.
-			if resp.Applied && (rf.first || rf.before != 0 || rf.after != 0) {
-				var created model.Rule
-				if err := jsonUnmarshal(resp.Object, &created); err == nil {
-					mv := api.MoveInput{Before: rf.before, After: rf.after}
-					if rf.first {
-						mv.To = "start"
-					}
-					var moved api.MutationResponse
-					if err := c.Do(ctx(), "POST", objPath("rules", fmt.Sprint(created.ID))+"/move", nil, mv, &moved); err != nil {
-						return err
-					}
-					resp.Changes = append(resp.Changes, moved.Changes...)
-					resp.ConfirmBy = moved.ConfirmBy
-				}
-			}
 			return report(&resp)
 		},
 	}
@@ -376,6 +380,7 @@ output (from it) or early (before connection tracking: cheap drops).
 				"status": {"ct_status", r.CtStatus}, "days": {"days", r.Days}, "conn-limit": {"conn_limit", r.ConnLimit},
 				"log": {"log", r.Log}, "log-prefix": {"log_prefix", r.LogPrefix}, "raw": {"raw", r.Raw}, "comment": {"comment", r.Comment},
 				"disabled": {"disabled", r.Disabled}, "group-v4": {"group_v4", r.GroupV4}, "group-v6": {"group_v6", r.GroupV6},
+				"every-packet": {"every_packet", r.EveryPacket},
 			} {
 				if fl.Changed(flag) {
 					p[field[0].(string)] = field[1]
@@ -500,6 +505,9 @@ func ruleExtras(r model.Rule) string {
 	if r.ConnLimit > 0 {
 		x = append(x, fmt.Sprintf("over %d connections", r.ConnLimit))
 	}
+	if r.EveryPacket {
+		x = append(x, "every packet")
+	}
 	if len(x) == 0 {
 		return ""
 	}

cmd/hc/stats.go

@@ -0,0 +1,401 @@
+package main
+
+import (
+	"fmt"
+	"io"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+)
+
+// Statistics: the figures housecarld keeps over time (docs/stats.md). The
+// types mirror internal/stats's JSON, so hc doesn't link the database.
+
+type statEntry struct {
+	Key   string `json:"key"`
+	Count int64  `json:"count"`
+	Name  string `json:"name,omitempty"`
+}
+
+type statTraffic struct {
+	Name     string `json:"name"`
+	Received int64  `json:"received"`
+	Sent     int64  `json:"sent"`
+}
+
+type statSummary struct {
+	Since           string        `json:"since"`
+	From            time.Time     `json:"from"`
+	Zones           []statTraffic `json:"zones"`
+	Interfaces      []statTraffic `json:"interfaces"`
+	Drops           int64         `json:"drops"`
+	DropsByChain    []statEntry   `json:"drops_by_chain"`
+	Rules           []statEntry   `json:"rules"`
+	PeakConnections int64         `json:"peak_connections"`
+	ConntrackMax    int64         `json:"conntrack_max"`
+	BlockedSources  []statEntry   `json:"blocked_sources"`
+	BlockedPorts    []statEntry   `json:"blocked_ports"`
+	DevicesUp       []statEntry   `json:"devices_up"`
+	DevicesDown     []statEntry   `json:"devices_down"`
+}
+
+type statTimeline struct {
+	Step   string  `json:"step"`
+	Times  []int64 `json:"times"`
+	Series []struct {
+		Key    string  `json:"key"`
+		Values []int64 `json:"values"`
+	} `json:"series"`
+}
+
+type statStatus struct {
+	Enabled     bool       `json:"enabled"`
+	Database    string     `json:"database"`
+	Size        int64      `json:"size"`
+	LastSample  *time.Time `json:"last_sample"`
+	SampleError string     `json:"sample_error"`
+	FirstData   *time.Time `json:"first_data"`
+	PerDevice   bool       `json:"per_device"`
+	Retention   struct {
+		Minute, Hour, Day string
+	} `json:"retention"`
+	Latest *struct {
+		Conntrack    int64   `json:"conntrack"`
+		ConntrackMax int64   `json:"conntrack_max"`
+		RSS          int64   `json:"rss"`
+		CPUPermille  int64   `json:"cpu_permille"`
+		MemUsed      int64   `json:"mem_used"`
+		Load         float64 `json:"load"`
+	} `json:"latest"`
+}
+
+func statsCmd() *cobra.Command {
+	var since string
+	var limit int
+	cmd := &cobra.Command{
+		Use: "stats", Short: "Traffic, drops, blocked sources and devices over time",
+		Long: `Sums up what the firewall did over a while (--since: 90m, 24h, 7d, 4w):
+traffic per zone, packets dropped, the busiest rules, the most connections
+at once, the sources blocked most and what they tried, and (with traffic
+per device on) the devices that used the most.
+
+  hc stats --since 7d
+  hc stats traffic --since 24h
+  hc stats top devices --since 30d
+  hc stats rules --unused`,
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := get[statSummary]("/stats/summary", q("since", since, "limit", strconv.Itoa(limit)))
+			if err != nil {
+				return err
+			}
+			return output(s, func(w io.Writer) { printSummary(w, s) })
+		},
+	}
+	cmd.Flags().StringVar(&since, "since", "24h", "the last 90m, 24h, 7d, 4w…")
+	cmd.Flags().IntVar(&limit, "limit", 5, "entries per top list")
+	cmd.AddCommand(statsStatusCmd(), statsTrafficCmd(), statsTopCmd(), statsRulesCmd(), statsConnectionsCmd())
+	return cmd
+}
+
+func printSummary(w io.Writer, s statSummary) {
+	fmt.Fprintf(w, "Since %s (%s)\n\n", s.From.Local().Format("2006-01-02 15:04"), s.Since)
+	if len(s.Zones) > 0 {
+		fmt.Fprintln(w, "ZONE\tRECEIVED\tSENT")
+		for _, t := range s.Zones {
+			fmt.Fprintf(w, "%s\t%s\t%s\n", t.Name, humanBytes(t.Received), humanBytes(t.Sent))
+		}
+		fmt.Fprintln(w)
+	}
+	var chains []string
+	for _, e := range s.DropsByChain {
+		chains = append(chains, fmt.Sprintf("%s %s", e.Key, humanCount(uint64(e.Count))))
+	}
+	drops := humanCount(uint64(s.Drops))
+	if len(chains) > 0 {
+		drops += " (" + strings.Join(chains, ", ") + ")"
+	}
+	fmt.Fprintf(w, "Dropped:\t%s packets\n", drops)
+	if s.ConntrackMax > 0 {
+		fmt.Fprintf(w, "Connections:\tat most %d at once (of %d)\n", s.PeakConnections, s.ConntrackMax)
+	}
+	list := func(title string, es []statEntry, unit func(int64) string) {
+		if len(es) == 0 {
+			return
+		}
+		fmt.Fprintf(w, "\n%s\n", title)
+		for _, e := range es {
+			name := e.Key
+			if e.Name != "" {
+				name = e.Name
+			}
+			fmt.Fprintf(w, "  %s\t%s\n", name, unit(e.Count))
+		}
+	}
+	packets := func(n int64) string { return humanCount(uint64(n)) + " packets" }
+	list("Busiest rules:", s.Rules, packets)
+	list("Blocked most (logged drops):", s.BlockedSources, packets)
+	list("What they tried:", s.BlockedPorts, packets)
+	list("Devices, sent:", s.DevicesUp, humanBytes)
+	list("Devices, received:", s.DevicesDown, humanBytes)
+}
+
+func statsStatusCmd() *cobra.Command {
+	return &cobra.Command{
+		Use: "status", Short: "Whether statistics are collected, since when, and the database", Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			s, err := get[statStatus]("/stats/status", nil)
+			if err != nil {
+				return err
+			}
+			return output(s, func(w io.Writer) {
+				if !s.Enabled {
+					fmt.Fprintln(w, "Statistics are off (stats.enabled in housecarl.yaml).")
+					return
+				}
+				fmt.Fprintf(w, "Collecting:\tyes\n")
+				if s.Database != "" {
+					fmt.Fprintf(w, "Database:\t%s (%s)\n", s.Database, humanBytes(s.Size))
+				} else {
+					fmt.Fprintf(w, "Database size:\t%s\n", humanBytes(s.Size))
+				}
+				if s.FirstData != nil {
+					fmt.Fprintf(w, "Since:\t%s\n", s.FirstData.Local().Format("2006-01-02 15:04"))
+				}
+				if s.LastSample != nil {
+					fmt.Fprintf(w, "Last sample:\t%s\n", s.LastSample.Local().Format("15:04:05"))
+				}
+				if s.SampleError != "" {
+					fmt.Fprintf(w, "Error:\t%s\n", s.SampleError)
+				}
+				fmt.Fprintf(w, "Kept:\tper minute %s, hourly %s, daily %s\n", s.Retention.Minute, s.Retention.Hour, s.Retention.Day)
+				fmt.Fprintf(w, "Per device:\t%s\n", map[bool]string{true: "on", false: "off (hc settings set --stats-per-device on)"}[s.PerDevice])
+				if l := s.Latest; l != nil {
+					if l.ConntrackMax > 0 {
+						fmt.Fprintf(w, "Connections now:\t%d of %d\n", l.Conntrack, l.ConntrackMax)
+					}
+					fmt.Fprintf(w, "housecarld:\t%s, CPU %.1f%%\n", humanBytes(l.RSS), float64(l.CPUPermille)/10)
+					fmt.Fprintf(w, "Host:\t%s in use, load %.2f\n", humanBytes(l.MemUsed), l.Load)
+				}
+			})
+		},
+	}
+}
+
+func statsTrafficCmd() *cobra.Command {
+	var since, step string
+	var ifaces bool
+	cmd := &cobra.Command{
+		Use: "traffic", Short: "Bytes received from and sent to each zone (or interface) over time", Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			rx, tx := "zrx", "ztx"
+			if ifaces {
+				rx, tx = "rx", "tx"
+			}
+			tl, err := get[statTimeline]("/stats/timeline", q("since", since, "step", step, "keys", rx+":*,"+tx+":*"))
+			if err != nil {
+				return err
+			}
+			return output(tl, func(w io.Writer) { printTimeline(w, tl, rx, tx) })
+		},
+	}
+	cmd.Flags().StringVar(&since, "since", "24h", "the last 90m, 24h, 7d…")
+	cmd.Flags().StringVar(&step, "step", "auto", "per 1m, 5m, 15m, 1h, 6h or 1d")
+	cmd.Flags().BoolVar(&ifaces, "interfaces", false, "per interface instead of per zone")
+	return cmd
+}
+
+// printTimeline writes a row per bucket: each name's received and sent.
+func printTimeline(w io.Writer, tl statTimeline, rx, tx string) {
+	var names []string
+	vals := map[string][]int64{}
+	for _, s := range tl.Series {
+		vals[s.Key] = s.Values
+		if n, ok := strings.CutPrefix(s.Key, rx+":"); ok {
+			names = append(names, n)
+		}
+	}
+	for _, s := range tl.Series {
+		if n, ok := strings.CutPrefix(s.Key, tx+":"); ok && vals[rx+":"+n] == nil {
+			names = append(names, n)
+		}
+	}
+	if len(names) == 0 {
+		fmt.Fprintln(w, "no traffic recorded yet")
+		return
+	}
+	fmt.Fprint(w, "TIME")
+	for _, n := range names {
+		fmt.Fprintf(w, "\t%s ↓\t%s ↑", n, n)
+	}
+	fmt.Fprintln(w)
+	layout := "01-02 15:04"
+	if tl.Step == "1d" {
+		layout = "2006-01-02"
+	}
+	at := func(k string, i int) int64 {
+		if v := vals[k]; i < len(v) {
+			return v[i]
+		}
+		return 0
+	}
+	for i, t := range tl.Times {
+		fmt.Fprint(w, time.Unix(t, 0).Local().Format(layout))
+		for _, n := range names {
+			fmt.Fprintf(w, "\t%s\t%s", humanBytes(at(rx+":"+n, i)), humanBytes(at(tx+":"+n, i)))
+		}
+		fmt.Fprintln(w)
+	}
+}
+
+func statsTopCmd() *cobra.Command {
+	var since string
+	var limit int
+	cmd := &cobra.Command{
+		Use: "top devices|blocked|tried|quota RULE-ID", Short: "The largest: devices by traffic, sources blocked, what they tried, a quota's users",
+		Long: `Top lists over the last while:
+
+  devices   inside addresses by traffic to and from external zones (needs
+            hc settings set --stats-per-device on)
+  blocked   sources of logged drops (the policies' drops with 'log drops'
+            on, and rules that log)
+  tried     what those drops tried: tcp/22, udp/53, icmp/echo-request
+  quota ID  who used a per-address quota of rule ID`,
+		Args: cobra.RangeArgs(1, 2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			get1 := func(kind string) ([]statEntry, error) {
+				return get[[]statEntry]("/stats/top", q("kind", kind, "since", since, "limit", strconv.Itoa(limit)))
+			}
+			switch args[0] {
+			case "devices":
+				up, err := get1("device_up")
+				if err != nil {
+					return err
+				}
+				down, err := get1("device_down")
+				if err != nil {
+					return err
+				}
+				out := map[string][2]int64{}
+				var order []string
+				for _, l := range [][]statEntry{down, up} {
+					for _, e := range l {
+						if _, ok := out[e.Key]; !ok {
+							order = append(order, e.Key)
+						}
+					}
+				}
+				for _, e := range down {
+					v := out[e.Key]
+					v[0] = e.Count
+					out[e.Key] = v
+				}
+				for _, e := range up {
+					v := out[e.Key]
+					v[1] = e.Count
+					out[e.Key] = v
+				}
+				return output(map[string]any{"up": up, "down": down}, func(w io.Writer) {
+					fmt.Fprintln(w, "DEVICE\tRECEIVED\tSENT")
+					for _, k := range order {
+						fmt.Fprintf(w, "%s\t%s\t%s\n", k, humanBytes(out[k][0]), humanBytes(out[k][1]))
+					}
+				})
+			case "blocked", "tried":
+				kind := map[string]string{"blocked": "blocked_src", "tried": "blocked_port"}[args[0]]
+				es, err := get1(kind)
+				if err != nil {
+					return err
+				}
+				return output(es, func(w io.Writer) {
+					fmt.Fprintln(w, map[string]string{"blocked": "SOURCE", "tried": "WHAT"}[args[0]]+"\tPACKETS")
+					for _, e := range es {
+						fmt.Fprintf(w, "%s\t%d\n", e.Key, e.Count)
+					}
+				})
+			case "quota":
+				if len(args) != 2 {
+					return fmt.Errorf("give the rule's ID: hc stats top quota 12")
+				}
+				es, err := get1("quota_r" + args[1])
+				if err != nil {
+					return err
+				}
+				return output(es, func(w io.Writer) {
+					fmt.Fprintln(w, "ADDRESS\tUSED")
+					for _, e := range es {
+						fmt.Fprintf(w, "%s\t%s\n", e.Key, humanBytes(e.Count))
+					}
+				})
+			}
+			return fmt.Errorf("unknown list %q: devices, blocked, tried or quota", args[0])
+		},
+	}
+	cmd.Flags().StringVar(&since, "since", "24h", "the last 90m, 24h, 7d…")
+	cmd.Flags().IntVar(&limit, "limit", 20, "at most this many")
+	return cmd
+}
+
+func statsRulesCmd() *cobra.Command {
+	var since string
+	var unused bool
+	cmd := &cobra.Command{
+		Use: "rules", Short: "Packets each rule, built-in and policy counted; --unused: rules that matched nothing", Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			es, err := get[[]statEntry]("/stats/hits", q("since", since))
+			if err != nil {
+				return err
+			}
+			if unused {
+				var keep []statEntry
+				for _, e := range es {
+					if e.Count == 0 && strings.HasPrefix(e.Key, "r") {
+						keep = append(keep, e)
+					}
+				}
+				es = keep
+			}
+			return output(es, func(w io.Writer) {
+				if unused && len(es) == 0 {
+					fmt.Fprintf(w, "every rule matched something in the last %s\n", since)
+					return
+				}
+				fmt.Fprintln(w, "WHAT\tPACKETS")
+				for _, e := range es {
+					fmt.Fprintf(w, "%s\t%d\n", orDash(e.Name), e.Count)
+				}
+			})
+		},
+	}
+	cmd.Flags().StringVar(&since, "since", "30d", "the last 24h, 7d, 30d…")
+	cmd.Flags().BoolVar(&unused, "unused", false, "only the rules that matched nothing")
+	return cmd
+}
+
+func statsConnectionsCmd() *cobra.Command {
+	var since, step string
+	cmd := &cobra.Command{
+		Use: "connections", Short: "How many connections the firewall tracked over time, against the limit", Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			tl, err := get[statTimeline]("/stats/timeline", q("since", since, "step", step, "keys", "g:conntrack,g:conntrack_max"))
+			if err != nil {
+				return err
+			}
+			return output(tl, func(w io.Writer) {
+				fmt.Fprintln(w, "TIME\tCONNECTIONS\tLIMIT")
+				for i, t := range tl.Times {
+					n, m := tl.Series[0].Values[i], tl.Series[1].Values[i]
+					if n < 0 {
+						continue // no sample
+					}
+					fmt.Fprintf(w, "%s\t%d\t%d\n", time.Unix(t, 0).Local().Format("01-02 15:04"), n, m)
+				}
+			})
+		},
+	}
+	cmd.Flags().StringVar(&since, "since", "24h", "the last 90m, 24h, 7d…")
+	cmd.Flags().StringVar(&step, "step", "auto", "per 1m, 5m, 15m, 1h, 6h or 1d")
+	return cmd
+}

cmd/hc/traffic.go

@@ -0,0 +1,361 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"os"
+	"os/signal"
+	"strconv"
+	"strings"
+	"time"
+	"unicode/utf8"
+
+	"github.com/spf13/cobra"
+
+	"housecarl/internal/api"
+)
+
+// Traffic as it happens: connections the kernel tracks, and tracing
+// packets through the ruleset.
+
+func connectionsCmd() *cobra.Command {
+	var search string
+	var limit int
+	cmd := &cobra.Command{
+		Use: "connections", Aliases: []string{"conns"}, Short: "The connections the firewall tracks, and closing them",
+		Long: `Lists the connections the kernel tracks: what the "established" built-in
+lets through without checking the rules again. A rule added later doesn't
+stop them by itself; close them, and their next packet is checked by the
+rules as it stands.
+
+  hc connections --search 192.168.1.50
+  hc connections --search ":443 tcp"
+  hc connections close --search 192.168.1.50`,
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			cs, err := get[[]api.Connection]("/connections", q("q", search, "limit", strconv.Itoa(limit)))
+			if err != nil {
+				return err
+			}
+			return output(cs, func(w io.Writer) {
+				fmt.Fprintln(w, "PROTO\tFROM\tTO\tNAT\tSTATE\tEXPIRES")
+				for _, c := range cs {
+					fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\n", c.Protocol, hostPort(c.Source, c.SourcePort), hostPort(c.Destination, c.DestinationPort), natText(c), c.State, fmtSeconds(c.Timeout))
+				}
+			})
+		},
+	}
+	cmd.Flags().StringVar(&search, "search", "", "only those with all these words: addresses, :port, protocol, state")
+	cmd.Flags().IntVar(&limit, "limit", 500, "at most this many")
+	var csearch string
+	var yes bool
+	closeCmd := &cobra.Command{
+		Use: "close --search TEXT", Short: "Close connections: the rules check them again with their next packet", Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if strings.TrimSpace(csearch) == "" {
+				return fmt.Errorf("say which: --search 192.168.1.50 (see 'hc connections --search' first)")
+			}
+			if !confirm("Close every connection matching "+strconvQuote(csearch)+"?", yes) {
+				return fmt.Errorf("nothing closed")
+			}
+			c, err := newClient()
+			if err != nil {
+				return err
+			}
+			var out map[string]int
+			if err := c.Do(ctx(), "POST", "/connections/close", nil, api.CloseConnections{Search: csearch}, &out); err != nil {
+				return err
+			}
+			if !flagQuiet {
+				fmt.Printf("closed %s\n", plural(out["closed"], "connection"))
+			}
+			return nil
+		},
+	}
+	closeCmd.Flags().StringVar(&csearch, "search", "", "the connections to close, as for 'hc connections --search'")
+	closeCmd.Flags().BoolVarP(&yes, "yes", "y", false, "don't ask")
+	cmd.AddCommand(closeCmd)
+	return cmd
+}
+
+func strconvQuote(s string) string { return strconv.Quote(s) }
+
+func hostPort(addr string, port int) string {
+	if port == 0 {
+		return addr
+	}
+	if strings.Contains(addr, ":") {
+		return fmt.Sprintf("[%s]:%d", addr, port)
+	}
+	return fmt.Sprintf("%s:%d", addr, port)
+}
+
+// natText: "to 192.168.1.10:443" (a port forward), "as 203.0.113.10"
+// (masqueraded).
+func natText(c api.Connection) string {
+	var parts []string
+	if c.DNATTo != "" {
+		port := c.DNATPort
+		if port == 0 {
+			port = c.DestinationPort
+		}
+		parts = append(parts, "to "+hostPort(c.DNATTo, port))
+	}
+	if c.SNATTo != "" {
+		parts = append(parts, "as "+c.SNATTo)
+	}
+	return strings.Join(parts, ", ")
+}
+
+func traceCmd() *cobra.Command {
+	var in api.TraceInput
+	cmd := &cobra.Command{
+		Use: "trace", Short: "Follow packets through every table's rules: what each met, and what decided",
+		Long: `Traces packets from or to an address, on a port (either way), through the
+whole ruleset, Housecarl's and every other table's, for a while: each
+rule and policy they meet, and what accepted or dropped them. Ctrl-C
+stops it.
+
+  hc trace --address 192.168.1.50 --port 443
+  hc trace --port 22 --seconds 120
+  hc trace --address 2001:db8:1::/64 --protocol icmpv6`,
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			c, err := newClient()
+			if err != nil {
+				return err
+			}
+			var st api.TraceStatus
+			if err := c.Do(ctx(), "POST", "/trace", nil, in, &st); err != nil {
+				return err
+			}
+			if !flagQuiet {
+				fmt.Fprintf(os.Stderr, "tracing until %s (Ctrl-C stops)\n", st.Ends.Local().Format("15:04:05"))
+			}
+			sig, stop := signal.NotifyContext(context.Background(), os.Interrupt)
+			defer stop()
+			shown := 0
+			for {
+				select {
+				case <-sig.Done():
+					c.Do(ctx(), "DELETE", "/trace", nil, nil, &st)
+				case <-time.After(time.Second):
+					if err := c.Do(ctx(), "GET", "/trace", nil, nil, &st); err != nil {
+						return err
+					}
+				}
+				for ; shown < len(st.Packets); shown++ {
+					printPacket(os.Stdout, st.Packets[shown])
+				}
+				if !st.Running || sig.Err() != nil {
+					break
+				}
+			}
+			if st.Error != "" {
+				return fmt.Errorf("the trace failed: %s", st.Error)
+			}
+			if !flagQuiet {
+				fmt.Fprintf(os.Stderr, "%d packets traced\n", len(st.Packets))
+			}
+			return nil
+		},
+	}
+	f := cmd.Flags()
+	f.StringVar(&in.Address, "address", "", "packets from or to this address or prefix")
+	f.IntVar(&in.Port, "port", 0, "packets from or to this port")
+	f.StringVar(&in.Protocol, "protocol", "", "tcp, udp, sctp, icmp or icmpv6 (default any)")
+	f.IntVar(&in.Seconds, "seconds", 60, "how long (5 to 600)")
+	return cmd
+}
+
+// printPacket writes a traced packet: its headers, each step, its fate.
+func printPacket(w io.Writer, p api.TracePacket) {
+	what := p.Protocol
+	if p.TCPFlags != "" {
+		what += " " + p.TCPFlags
+	}
+	if p.ICMPType != "" {
+		what += " " + p.ICMPType
+	}
+	where := ""
+	if p.In != "" {
+		where += " in " + p.In
+	}
+	if p.Out != "" {
+		where += " out " + p.Out
+	}
+	fmt.Fprintf(w, "%s  %s -> %s  %s%s\n", p.Time.Local().Format("15:04:05.000"), hostPort(p.Source, p.SourcePort), hostPort(p.Destination, p.DestinationPort), what, where)
+	for _, s := range p.Steps {
+		name := s.Label
+		if name == "" {
+			if s.Kind == "policy" {
+				name = s.Chain + " policy"
+			} else {
+				name = s.Rule
+			}
+		}
+		fmt.Fprintf(w, "    %s %s: %s -> %s\n", s.Table, s.Chain, name, s.Verdict)
+	}
+	decided := p.DecidedBy.Label
+	if decided == "" {
+		decided = strings.TrimSpace(p.DecidedBy.Table + " " + p.DecidedBy.Chain + " " + p.DecidedBy.Kind)
+	}
+	if decided == "" {
+		fmt.Fprintf(w, "    => %s\n", strings.ToUpper(p.Verdict))
+	} else {
+		fmt.Fprintf(w, "    => %s by %s\n", strings.ToUpper(p.Verdict), decided)
+	}
+	if wd := p.WouldDrop; wd != nil {
+		name := wd.Label
+		if name == "" {
+			name = strings.TrimSpace(wd.Chain + " " + wd.Mark)
+		}
+		fmt.Fprintf(w, "    => shadow mode: Housecarl would have dropped it, by %s\n", name)
+	}
+}
+
+func matrixCmd() *cobra.Command {
+	return &cobra.Command{
+		Use: "matrix [FROM TO]", Short: "What the rules do to new connections between zones",
+		Long: `Shows, for each pair of zones (and the firewall itself), what happens to a
+new connection: allowed or blocked, decided by a rule or the policy, and
+how many rules decide for part of the traffic first (+N: exceptions,
+such as some services or addresses). With FROM and TO (a zone, or
+"firewall"), that pair's exceptions in order, and what decides the rest.
+
+  hc matrix
+  hc matrix guest wan
+  hc matrix lan firewall`,
+		Args: cobra.RangeArgs(0, 2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if len(args) == 1 {
+				return fmt.Errorf("give both: FROM and TO")
+			}
+			m, err := get[api.Matrix]("/matrix", nil)
+			if err != nil {
+				return err
+			}
+			if len(args) == 2 {
+				for _, c := range m.Cells {
+					if strings.EqualFold(c.From, args[0]) && strings.EqualFold(c.To, args[1]) {
+						return output(c, func(w io.Writer) { printCell(w, c) })
+					}
+				}
+				return fmt.Errorf("no pair %s → %s (zones: %s, and firewall)", args[0], args[1], strings.Join(m.Zones, ", "))
+			}
+			return output(m, func(w io.Writer) { printMatrix(w, m) })
+		},
+	}
+}
+
+func printMatrix(w io.Writer, m api.Matrix) {
+	writeMatrix(w, m, isTerminal(os.Stdout) && flagOutput == "table")
+}
+
+// writeMatrix writes the grid, in colour on a terminal.
+func writeMatrix(w io.Writer, m api.Matrix, color bool) {
+	cols := append(append([]string{}, m.Zones...), "firewall")
+	cells := map[[2]string]api.MatrixCell{}
+	for _, c := range m.Cells {
+		cells[[2]string{c.From, c.To}] = c
+	}
+	// The grid is padded by hand: tabwriter would count the colours'
+	// escape codes as text, and misalign the columns.
+	grid := [][]string{append([]string{"FROM \\ TO"}, cols...)}
+	kinds := [][]string{make([]string, len(cols)+1)}
+	same := false
+	for _, from := range cols {
+		row, kind := []string{from}, []string{""}
+		for _, to := range cols {
+			c, ok := cells[[2]string{from, to}]
+			if !ok {
+				row, kind = append(row, "-"), append(kind, "")
+				continue
+			}
+			text := map[string]string{"allowed": "allow", "blocked": "block", "partly": "partly"}[c.Summary]
+			if c.Allowed > 0 {
+				text += fmt.Sprintf(" +%d", c.Allowed)
+			}
+			if c.Blocked > 0 {
+				text += fmt.Sprintf(" -%d", c.Blocked)
+			}
+			if c.SameZone {
+				text += " *"
+				same = true
+			}
+			row, kind = append(row, text), append(kind, c.Summary)
+		}
+		grid, kinds = append(grid, row), append(kinds, kind)
+	}
+	width := make([]int, len(cols)+1)
+	for _, row := range grid {
+		for i, t := range row {
+			width[i] = max(width[i], utf8.RuneCountInString(t))
+		}
+	}
+	paint := map[string]string{"allowed": "\x1b[32m", "blocked": "\x1b[31m", "partly": "\x1b[33m"}
+	for r, row := range grid {
+		var b strings.Builder
+		for i, t := range row {
+			pad := strings.Repeat(" ", width[i]-utf8.RuneCountInString(t))
+			if i == len(row)-1 {
+				pad = ""
+			} else {
+				pad += "  "
+			}
+			if p := paint[kinds[r][i]]; color && p != "" {
+				t = p + t + "\x1b[0m"
+			}
+			b.WriteString(t + pad)
+		}
+		fmt.Fprintln(w, b.String())
+	}
+	// What the partly open pairs let through.
+	var open [][2]string
+	for _, from := range cols {
+		for _, to := range cols {
+			c, ok := cells[[2]string{from, to}]
+			if !ok || len(c.Open) == 0 || c.SameZone {
+				continue
+			}
+			items := c.Open
+			more := ""
+			if len(items) > 6 {
+				items, more = items[:6], fmt.Sprintf("; %d more", len(c.Open)-6)
+			}
+			open = append(open, [2]string{from + " → " + to, strings.Join(items, "; ") + more})
+		}
+	}
+	if len(open) > 0 {
+		fmt.Fprintln(w, "\nLet through for part of the traffic:")
+		for _, o := range open {
+			fmt.Fprintf(w, "  %s\t%s\n", o[0], o[1])
+		}
+	}
+	fmt.Fprintln(w, "\nallow, block: every new connection; partly: some are let through, some not.")
+	fmt.Fprintln(w, "+N: rules that let part of the traffic through; -N: rules that block part of it. 'hc matrix FROM TO' lists them.")
+	if same {
+		fmt.Fprintln(w, "*: a zone to itself: only traffic routed between its interfaces passes the firewall.")
+	}
+}
+
+func printCell(w io.Writer, c api.MatrixCell) {
+	summary := map[string]string{"allowed": "all allowed", "blocked": "all blocked", "partly": "partly allowed"}[c.Summary]
+	fmt.Fprintf(w, "%s → %s (the %s chain), new connections: %s\n", c.From, c.To, c.Chain, summary)
+	if c.Note != "" {
+		fmt.Fprintf(w, "  (%s)\n", c.Note)
+	}
+	hits := func(h *uint64) string {
+		if h == nil {
+			return "-"
+		}
+		return fmt.Sprintf("%d", *h)
+	}
+	fmt.Fprintln(w, "  VERDICT\tWHAT\tBY\tHITS")
+	for _, e := range c.Exceptions {
+		fmt.Fprintf(w, "  %s\t%s\t%s\t%s\n", e.Verdict, e.What, e.Label, hits(e.Hits))
+	}
+	fmt.Fprintf(w, "  %s\tthe rest\t%s\t%s\n", c.Verdict, c.DecidedBy, hits(c.Hits))
+	fmt.Fprintln(w, "\nHits: each rule's packets since housecarld started, in all its traffic (a rule can cover several pairs).")
+}

cmd/housecarld/main.go

@@ -26,15 +26,20 @@ import (
 	"housecarl/internal/auth"
 	"housecarl/internal/certs"
 	"housecarl/internal/config"
+	"housecarl/internal/conntrack"
 	"housecarl/internal/engine"
 	"housecarl/internal/hostif"
 	"housecarl/internal/jarl"
+	"housecarl/internal/linkwatch"
 	"housecarl/internal/model"
 	"housecarl/internal/nflog"
 	"housecarl/internal/nft"
 	"housecarl/internal/render"
 	"housecarl/internal/server"
+	"housecarl/internal/stats"
 	"housecarl/internal/store"
+	"housecarl/internal/tablewatch"
+	"housecarl/internal/trace"
 	"housecarl/internal/validate"
 	"housecarl/internal/version"
 	"housecarl/internal/web"
@@ -149,8 +154,12 @@ func serveCmd() *cobra.Command {
 			}
 			host := &hostif.Host{}
 			eng.Env = hostEnv(host)
-			fw := &nft.Firewall{Runner: &nft.Runner{Command: nft.Wrapper()}, Interfaces: host.Exists, BootFile: a.cfg.Ruleset}
+			fw := &nft.Firewall{Runner: &nft.Runner{Command: nft.Wrapper()}, Interfaces: host.Exists, BootFile: a.cfg.Ruleset,
+				Shadow: a.cfg.Shadow.Enabled, ShadowBefore: a.cfg.Shadow.Position == config.ShadowBefore}
 			eng.Firewall = fw
+			if fw.Shadow {
+				a.log.Warn("shadow mode: Housecarl's ruleset is loaded but blocks, rejects and translates nothing; the other firewall decides (docs/shadow.md)", "position", a.cfg.Shadow.Position)
+			}
 			for _, p := range eng.Problems() {
 				a.log.Warn("configuration problem: "+p.Message, "object", p.Object, "code", p.Code)
 			}
@@ -171,6 +180,7 @@ func serveCmd() *cobra.Command {
 				Listen: a.cfg.Listen, TLSCertFile: a.cfg.TLS.CertFile, TLSKeyFile: a.cfg.TLS.KeyFile, TrustedProxies: a.cfg.Proxies(),
 				SessionLifetime: time.Duration(a.cfg.Session.Lifetime), SessionIdle: time.Duration(a.cfg.Session.IdleTimeout),
 				CookieSecure: a.cfg.Session.CookieSecure, Socket: a.cfg.Socket, ACME: acme,
+				Shadow: a.cfg.Shadow.Enabled, ShadowPosition: a.cfg.Shadow.Position,
 			}, eng, fw, host, a.log, web.FS())
 			// Put the ruleset in force: what was on probation when
 			// housecarld stopped is rolled back first. An empty
@@ -178,12 +188,15 @@ func serveCmd() *cobra.Command {
 			// close the machine before any rule lets anyone in.
 			// What the quotas used before housecarld (or the machine)
 			// stopped comes back with the next apply.
+			restoreQuotas := func() {
 				if v, err := a.store.GetMeta(ctx, server.QuotaUsageKey); err == nil && v != "" {
 					var u nft.QuotaUsage
 					if json.Unmarshal([]byte(v), &u) == nil {
 						fw.RestoreQuotas(&u)
 					}
 				}
+			}
+			restoreQuotas()
 			if !srv.ResumeProbation(ctx) {
 				if eng.State().Empty() {
 					a.log.Info("the configuration is empty: no ruleset is applied until the first change (or 'hc import')")
@@ -193,8 +206,53 @@ func serveCmd() *cobra.Command {
 					a.log.Error("saving the ruleset for the next boot failed", "err", err)
 				}
 			}
+			// A flowtable device that appears later (ppp0 once PPPoE is up,
+			// or again after it reconnects) is added by applying again.
+			go func() {
+				err := linkwatch.Watch(ctx, func() {
+					host.Forget()
+					if !fw.FlowtableStale(eng.State()) {
+						return
+					}
+					if err := eng.ApplyCurrent(ctx); err != nil {
+						a.log.Error("applying the ruleset again for the flowtable's devices failed", "err", err)
+						return
+					}
+					a.log.Info("interfaces changed: applied the ruleset again, with the flowtable's devices that exist now")
+				})
+				if err != nil {
+					a.log.Warn("not watching the interfaces: a flowtable device that appears later is only added by the next change", "err", err)
+				}
+			}()
+			// In shadow mode the other firewall's ruleset is reloaded now
+			// and then, and a ruleset file usually starts with flush
+			// ruleset, which deletes Housecarl's table too: put it back.
+			if fw.Shadow {
+				go func() {
+					err := tablewatch.Watch(ctx, func() {
+						if _, err := fw.Table(ctx); !errors.Is(err, nft.ErrNoTable) || eng.State().Empty() {
+							return
+						}
+						// The quotas' usage went with the table: from
+						// what was last saved.
+						restoreQuotas()
+						if err := eng.ApplyCurrent(ctx); err != nil {
+							a.log.Error("Housecarl's table was deleted, and putting it back failed", "err", err)
+							return
+						}
+						a.log.Info("Housecarl's table was deleted (flush ruleset?): put it back")
+					})
+					if err != nil {
+						a.log.Warn("not watching nftables: if Housecarl's table is deleted, only the next change or a restart puts it back", "err", err)
+					}
+				}()
+			}
 			go nflog.Listen(ctx, render.LogGroup, a.log, srv.LogPacket)
 			go srv.RunQuotas(ctx)
+			srv.SetConntrack(conntrack.Client{})
+			statsDone := startStats(ctx, a, fw, srv)
+			// A trace left behind by a crash.
+			fw.DeleteTable(ctx, "inet", trace.Table)
 			hup := make(chan os.Signal, 1)
 			signal.Notify(hup, syscall.SIGHUP)
 			go func() {
@@ -207,12 +265,45 @@ func serveCmd() *cobra.Command {
 			err = srv.Serve(ctx, func() { sdNotify("READY=1") })
 			sdNotify("STOPPING=1")
 			srv.SaveQuotaUsage(context.Background())
+			<-statsDone
 			a.log.Info("stopped")
 			return err
 		},
 	}
 }
 
+// startStats opens the statistics database and runs the collector until
+// ctx ends; the channel closes once it wrote its last figures.
+func startStats(ctx context.Context, a *app, fw *nft.Firewall, srv *server.Server) <-chan struct{} {
+	done := make(chan struct{})
+	c := a.cfg.Stats
+	if !c.Enabled {
+		close(done)
+		return done
+	}
+	st, err := stats.OpenStore(c.Database)
+	if err != nil {
+		a.log.Error("statistics are off: opening their database failed (deleting it is safe)", "err", err)
+		close(done)
+		return done
+	}
+	opts := stats.Options{TopN: c.TopN, RetainMinute: time.Duration(c.Retention.Minute), RetainHour: time.Duration(c.Retention.Hour), RetainDay: time.Duration(c.Retention.Day)}
+	// Tests sample more often than every minute.
+	if v := os.Getenv("HOUSECARL_STATS_INTERVAL"); v != "" {
+		opts.Interval, _ = time.ParseDuration(v)
+	}
+	src := stats.Sources{Table: fw.Table}
+	srv.StatsSources(&src)
+	col := stats.NewCollector(st, opts, src, a.log)
+	srv.SetStats(col)
+	go func() {
+		defer close(done)
+		defer st.Close()
+		col.Run(ctx)
+	}()
+	return done
+}
+
 func sdNotify(state string) {
 	sock := os.Getenv("NOTIFY_SOCKET")
 	if sock == "" {

deploy/systemd/housecarl-ruleset.service

@@ -1,5 +1,8 @@
 # Housecarl: loads the last confirmed ruleset at boot, before the network
-# comes up and whether or not housecarld starts.
+# comes up and whether or not housecarld starts. If it fails to load, the
+# fallback housecarld saved beside it is loaded instead: nothing new gets
+# in from external zones (docs/operations.md), and the unit is marked
+# failed, until housecarld starts and puts the real ruleset back.
 # Installed as /etc/systemd/system/housecarl-ruleset.service.
 [Unit]
 Description=Housecarl firewall ruleset
@@ -15,7 +18,8 @@ ConditionPathExists=/var/lib/housecarl/ruleset.nft
 [Service]
 Type=oneshot
 RemainAfterExit=yes
-ExecStart=/usr/sbin/nft -f /var/lib/housecarl/ruleset.nft
+# "<2>": logged as critical.
+ExecStart=/bin/sh -c '/usr/sbin/nft -f /var/lib/housecarl/ruleset.nft && exit 0; echo "<2>the saved ruleset failed to load: loading the fallback (nothing new gets in from external zones) until housecarld puts the real one back" >&2; /usr/sbin/nft -f /var/lib/housecarl/ruleset-fallback.nft; exit 1'
 
 [Install]
 WantedBy=sysinit.target

docs/api.md

@@ -31,7 +31,7 @@ user name. [permissions.md](permissions.md) has what each call needs.
 | 401 | `unauthenticated`, `session_expired`, `login_failed` | |
 | 403 | `forbidden`, `csrf` | |
 | 404 | `not_found` | |
-| 409 | `conflict` | e.g. undoing a change after the objects changed again |
+| 409 | `conflict` | e.g. undoing a change after the objects changed again; a takeover or closing connections in shadow mode ([shadow.md](shadow.md)) |
 | 422 | `validation` | the change would make the configuration invalid; `problems` lists why ([validation.md](validation.md)) |
 | 422 | `firewall` | nftables refused the ruleset (`nft -c`), or applying it failed; the message has nftables' words. Nothing was changed |
 | 429 | `rate_limited` | |
@@ -50,7 +50,11 @@ Calls that change the configuration answer:
 
 * `confirm_by`: the change is on probation, and is rolled back at that
   time unless `POST /confirm` comes first. Missing for changes made
-  through the socket.
+  through the socket, and in shadow mode ([shadow.md](shadow.md)).
+* `confirm_token`: the client that made the change can confirm it by
+  itself with `POST /confirm {"token"}`, from the same address, as the
+  same user and token, over a new connection (the answer closes this
+  one). Missing when a person must confirm (`manual_confirm`).
 * `?dry_run=1` checks the change, with nftables too, and returns what it
   would do, with `"applied": false, "dry_run": true`, and `diff`: the
   ruleset's unified diff.
@@ -70,13 +74,19 @@ Creating answers 201, other changes 200.
 
 | | |
 |---|---|
-| `GET /healthz` | `{"ok": true, "version"}`, without authentication |
-| `GET /status` | `build`, `started`, `applied` (the ruleset is in force), `apply_error`, `problems`, `probation`, `interfaces`, `counts` (objects by kind), `other_tables` (other nftables tables, `"inet main"`) |
+| `GET /healthz` | `ok`, `version`, `firewall` (`applied`, `unconfigured`, `fallback`, `missing`), `mode` (`enforcing`, or `shadow`: the ruleset blocks nothing, [shadow.md](shadow.md)), `probation` (changes waiting to be confirmed), `warnings` (codes); HTTP 503 unless `ok`. Without authentication; read at most every five seconds ([operations.md](operations.md#monitoring)) |
+| `GET /status` | `build`, `started`, `mode` (`enforcing` or `shadow`), `shadow_position` (`after` or `before`, in shadow mode), `applied` (the ruleset is in force), `apply_error`, `problems`, `probation`, `interfaces`, `counts` (objects by kind), `other_tables` (other nftables tables, `"inet main"`) |
 | `GET /interfaces` | the machine's interfaces: `name`, `up`, `addresses`, `zone` |
-| `GET /ruleset` | the nftables ruleset Housecarl builds, as text |
+| `GET /matrix[?changeset=ID]` | the zone matrix: `zones`, `cells` (`from`, `to` (a zone, or `firewall`), `chain`, `summary` (allowed, blocked, partly), `verdict` (what the rest gets: allowed, blocked), `decided_by`, `rule_id`, `hits`, `allowed` and `blocked` (exceptions by verdict), `open` (what the allowing exceptions let through, in short), `same_zone` and `note`, `exceptions`: `rule_id`, `chain`, `label`, `verdict` (allowed, blocked, logged, unknown), `what`, `short`, `hits`). `hits` are the rule's or policy's packets since housecarld started, all its traffic (missing: no counter) ([rules.md](rules.md#the-zone-matrix)) |
+| `GET /ruleset[?enforcing=1]` | the nftables ruleset Housecarl builds, as text. In shadow mode it is the shadow ruleset; `enforcing=1` gives the one loaded once shadow mode is off |
 | `GET /counters` | `{"since", "counters": {"r12": {"packets", "bytes"}, ...}}`, by mark ([design.md](design.md#its-own-table-only)); totals since housecarld started |
 | `GET /quotas` | each rule's quota: `rule_id`, `rule`, `quota`, `used` (bytes), `addresses` (`[{"address", "used"}]` for a per-address quota), `last_reset`, `next_reset` |
-| `GET /log[?limit=&after=&q=]` | logged packets, newest first: `time`, `prefix`, `in`, `out`, `family`, `protocol`, `source`, `source_port`, `destination`, `destination_port`, `icmp_type`, `tcp_flags`, `length`, `mark` (the rule's). `after` (a time) for those since; `q` searches |
+| `GET /connections[?q=&limit=]` | tracked connections: `family`, `protocol`, `source`, `source_port`, `destination`, `destination_port` (or `icmp_id`, `icmp_type`, `icmp_code`), `dnat_to`, `dnat_port`, `snat_to`, `state`, `assured`, `timeout` (seconds left), `packets` and `bytes` when the kernel counts them. `q`: all of its words in the addresses, `:port`, protocol, state |
+| `POST /connections/close` | `{"connections": [...]}` (as listed) or `{"search": "192.168.1.50"}`: delete their tracking entries (`config:write`, audited); answers `{"closed": N}`. 409 in shadow mode |
+| `POST /trace` | `{"address", "port", "protocol", "seconds"}` (5-600, default 60): start tracing (`config:write`, audited); 409 while one runs |
+| `GET /trace` | the trace running, or the last: `running`, `input`, `started`, `ends`, `error`, `packets`: each with `time`, `in`, `out`, `protocol`, addresses and ports, `tcp_flags`, `icmp_type`, `steps` (`table`, `chain`, `kind` rule or policy, `rule`, `mark`, `label`, `verdict`), `verdict` (accepted, dropped) and `decided_by`; in shadow mode `would_drop`, the step where Housecarl would have dropped it |
+| `DELETE /trace` | stop it |
+| `GET /log[?limit=&after=&q=]` | logged packets, newest first: `time`, `prefix`, `in`, `out`, `family`, `protocol`, `source`, `source_port`, `destination`, `destination_port`, `icmp_type`, `tcp_flags`, `length`, `mark` (the rule's), `verdict`, `shadow` (logged in shadow mode: a drop or reject is what would have been done). `after` (a time) for those since; `q` searches |
 | `GET /catalog` | built-in services, chains, actions, roles, the log group |
 
 ### Settings
@@ -89,7 +99,8 @@ Creating answers 201, other changes 200.
  "builtins": {"loopback": true, "established": true, "drop_invalid": true, "icmpv6": true,
               "ping_internal": true, "dhcpv6_client": true, "antispoof": true, "mss_clamp": false},
  "own_prefixes": ["2001:db8::/48"], "antispoof_v4": ["10.0.0.0/8", "..."], "antispoof_v6": ["fc00::/7", "..."],
- "flowtable_devices": ["wan", "eth1"], "probation_seconds": 120}
+ "flowtable_devices": ["wan", "eth1"], "probation_seconds": 120, "manual_confirm": false,
+ "stats_per_device": false}
 ```
 
 Lists replace the old ones. `probation_seconds` 0 is the default (120).
@@ -103,7 +114,7 @@ share one pattern, `/{kind}` with kind `zones`, `sets`, `services`,
 | | |
 |---|---|
 | `GET /{kind}` | all of them, in order; `GET /rules?chain=input` for one chain; `GET /services` includes the built-in ones |
-| `POST /{kind}` | create; rules, forwards and NAT rules go last unless given a `position` |
+| `POST /{kind}[?before=ID\|after=ID\|first=1]` | create; rules, forwards and NAT rules go last unless given a `position`, or placed before or after another of their chain, or first, in the same change (one history entry; staged in a changeset, the place goes with it) |
 | `GET /{kind}/{ref}` | by ID, or name for zones, sets, services and forwards |
 | `PATCH /{kind}/{ref}` | change the fields given (a JSON merge: `null` clears) |
 | `DELETE /{kind}/{ref}` | refused while something uses it |
@@ -135,6 +146,12 @@ A source NAT rule is a match (as a rule's, without the rule's other
 fields) with `"to"` (empty: masquerade), a redirect a match with
 `"to_port"`. Times are seconds (`timeout`).
 
+### Statistics
+
+`GET /stats/status`, `/stats/summary`, `/stats/timeline`, `/stats/top`,
+`/stats/hits` and `/stats/heatmap` (`log:read`): see
+[stats.md](stats.md#the-api).
+
 ### Changesets
 
 Changes staged to be reviewed and applied together ([design.md](design.md#reviewing-changes-changesets)).
@@ -153,7 +170,10 @@ Changes staged to be reviewed and applied together ([design.md](design.md#review
 Staging (`?changeset=ID` on any change to objects or settings) checks
 the changeset with the new operation, and nftables checks its ruleset;
 an error refuses the operation. Only open changesets take operations, and
-only their owner's (or an administrator's). Undo, `/import/nft` and
+only their owner's (or an administrator's). An undo
+(`POST /history/{id}/revert?changeset=ID`) is staged as one operation,
+which checks when it is replayed that nothing it touches changed since;
+applied, History links it to the entry it undoes. `/import/nft` and
 `/takeover` can't be staged: with a changeset named they answer 400.
 
 Reads of objects, settings and `/ruleset` with `?changeset=ID` show the
@@ -180,7 +200,7 @@ history, but are audited.
 | | |
 |---|---|
 | `GET /probation` | `{"pending": [history IDs], "takeovers": N, "deadline", "seconds_left"}` |
-| `POST /confirm` | keep them: `{"confirmed": N}`; the ruleset is saved for the boot |
+| `POST /confirm` | keep them: `{"confirmed": N}`; the ruleset is saved for the boot. With `{"token"}`, the client confirming its own change (see `confirm_token`): 409 if a newer change is pending, or over the change's own connection; 403 from another user or address |
 | `POST /rollback` | undo them now |
 
 ### History, import, takeover
@@ -190,8 +210,8 @@ history, but are audited.
 | `GET /history[?limit=&before=]` | changes, newest first |
 | `GET /history/{id}` | one, with each object before and after |
 | `POST /history/{id}/revert` | undo it; 409 if an object it touched has changed since, or it was undone already |
-| `POST /import/nft` | `{"json": NFT_JSON, "text": "nft -a list ruleset output", "defines": "ruleset file", "table": "inet main", "replace": false}` ([import.md](import.md)); answers with `state` (the imported configuration), `notes` and `raw` too |
-| `POST /takeover` | `{"tables": ["inet main"]}`: delete these tables (administrators); on probation over the network |
+| `POST /import/nft` | `{"json": NFT_JSON, "text": "nft -a list ruleset output", "defines": "ruleset file", "table": "inet main", "zones": {"enp2s0": "lan"}, "replace": false}` ([import.md](import.md)); answers with `state` (the imported configuration), `notes` and `raw` too |
+| `POST /takeover` | `{"tables": ["inet main"]}`: delete these tables (administrators); on probation over the network; 409 in shadow mode |
 
 ### Certificate
 

docs/cli.md

@@ -28,6 +28,7 @@ command and its flags.
 | | `HOUSECARL_SOCKET` | the local socket's path |
 | `--insecure`, `--ca-cert FILE` | | TLS: skip the check, or trust another CA |
 | `-n, --dry-run` | | check and show a change without making it, with the ruleset's diff |
+| `--no-confirm` | | leave a change on probation instead of confirming it once `hc` reaches Housecarl again |
 | `-C, --changeset ID` | `HOUSECARL_CHANGESET` (checked, see [Changesets](#changesets)) | stage configuration changes in this changeset instead of making them |
 | `-o, --output FORMAT` | | `table` (default), `json`, `yaml` or `id` |
 | `-q, --quiet` | | print only errors (and the IDs of new objects) |
@@ -41,14 +42,14 @@ server error.
 Every change is checked as a whole, then by nftables, before it is put
 in force. A refused change says why, with the problem's code
 ([validation.md](validation.md)). An applied one says its history entry
-and any new warnings. Over the network, it is on probation:
+and any new warnings. Over the network, it is on probation, and `hc`
+confirms it by reaching Housecarl again over a new connection
+([operations.md](operations.md#changes-on-probation)):
 
 ```
 $ hc rule add input --from trusted --service ssh --comment "SSH from home"
 created rule input #31 (from trusted · ssh)
-applied on probation (history 12): confirm with 'hc confirm' within 2m0s, or it is rolled back
-$ hc confirm
-confirmed 1 change; it is kept and loaded at boot
+applied and confirmed (history 12): this client reached Housecarl again over a new connection
 ```
 
 Through the socket the change is kept at once:
@@ -116,9 +117,9 @@ it, because an exported ID easily outlives its changeset:
 `-C ID` and an ID argument are used as given (housecarld still refuses a
 changeset that isn't open, or another user's unless you are an
 administrator). Only configuration changes are staged: dynamic set
-entries, `confirm` and `rollback` are made at once, and `history undo`,
-`import` and `takeover` can't be staged, so they are refused while a
-changeset is named. Changesets staged in the web UI show up here too,
+entries, `confirm` and `rollback` are made at once; `history undo` is
+staged like any change; `import` and `takeover` can't be staged, so they
+are refused while a changeset is named. Changesets staged in the web UI show up here too,
 and the other way round.
 
 ## Commands
@@ -130,12 +131,16 @@ by name.
 
 | Command | |
 |---|---|
-| `status` | the ruleset (in force or not), changes on probation, problems, interfaces, other tables |
+| `status` | the mode (in shadow mode), the ruleset (in force or not), changes on probation, problems, interfaces, other tables |
 | `interfaces` | the machine's interfaces, their addresses and zones |
-| `ruleset` | the nftables ruleset Housecarl builds |
+| `ruleset [--enforcing]` | the nftables ruleset Housecarl builds; in shadow mode, `--enforcing` shows the one loaded once it is off ([shadow.md](shadow.md)) |
 | `counters` | the hits of every rule, NAT rule, built-in and policy, by mark (`r12`, `f3`, `b-established`, `p-input`) |
 | `quotas` | what each rule's quota used (each address's, for per-address quotas), and when it resets |
 | `log [-f] [--search TEXT] [--limit N]` | the packets logged by rules and policies ([operations.md](operations.md#the-log)) |
+| `connections [--search TEXT] [--limit N]`, `connections close --search TEXT [-y]` | tracked connections; closing makes the rules check them again ([operations.md](operations.md#connections)); not in shadow mode |
+| `stats [--since 24h]` | traffic per zone, drops, the busiest rules, blocked sources, devices; `stats status`, `stats traffic`, `stats top devices\|blocked\|tried\|quota ID`, `stats rules [--unused]`, `stats connections` ([stats.md](stats.md#the-cli)) |
+| `matrix [FROM TO]` | what the rules do to new connections between zones (allow, block or partly), and what is let through; with FROM and TO, that pair's rules in order, with their hits ([rules.md](rules.md#the-zone-matrix)) |
+| `trace [--address A] [--port N] [--protocol P] [--seconds S]` | follow packets through every table's rules ([operations.md](operations.md#tracing)) |
 
 ### zone, set, service
 
@@ -143,7 +148,7 @@ by name.
 |---|---|
 | `zone list`, `zone show ZONE` | |
 | `zone add NAME -i IFACE... [--external] [--comment C]` | |
-| `zone update ZONE [-i IFACE]... [--add IFACE] [--remove IFACE] [--external[=false]] [--name N] [--comment C]` | `-i` replaces the interfaces |
+| `zone update ZONE [-i IFACE]... [--add IFACE] [--remove IFACE] [--external[=false]] [--name N] [--comment C]` | `-i` replaces the interfaces; `--name` renames it, and the rules naming it follow |
 | `zone delete ZONE` | refused while rules use it |
 | `set list`, `set show SET` | |
 | `set add NAME [ENTRY...] [--dynamic] [--timeout 24h] [--comment C]` | |
@@ -181,7 +186,7 @@ The match flags (`rule add`, `snat add`, `redirect add`):
 And for rules: `--state`, `--status`, `--days`, `--hours`, `--limit
 10/minute|1mbytes/second[,burst=N][,over][,per-source|,per-destination]`,
 `--quota 50gbytes[,over][,per-source|,per-destination][,daily|,weekly|,monthly=DAY]`,
-`--group-v4 N`, `--group-v6 N`, `--conn-limit N`, `-a
+`--group-v4 N`, `--group-v6 N`, `--conn-limit N`, `--every-packet`, `-a
 accept|drop|reject|log`, `--reject-with tcp-reset`, `--log`,
 `--log-prefix`, `--raw RULE`, `--comment`, `--disabled`
 ([rules.md](rules.md#rules)).
@@ -203,7 +208,7 @@ See [nat.md](nat.md).
 | Command | |
 |---|---|
 | `settings show` | |
-| `settings set [--input-policy\|--forward-policy\|--output-policy accept\|drop] [--log-drops on\|off] [--builtin NAME=on\|off]... [--own-prefix P]... [--antispoof-v4 P]... [--antispoof-v6 P]... [--flowtable DEV]... [--probation 2m]` | lists given replace the old ones |
+| `settings set [--input-policy\|--forward-policy\|--output-policy accept\|drop] [--log-drops on\|off] [--builtin NAME=on\|off]... [--own-prefix P]... [--antispoof-v4 P]... [--antispoof-v6 P]... [--flowtable DEV]... [--probation 2m] [--auto-confirm on\|off] [--stats-per-device on\|off]` | lists given replace the old ones; `none` empties one (`--flowtable none`: off) |
 
 ### probation, history, import
 
@@ -214,8 +219,8 @@ See [nat.md](nat.md).
 | `history list [--limit N]` | changes, newest first |
 | `history show ID` | each object before and after |
 | `history undo ID` | apply the change backwards (refused if something it touched has changed since) |
-| `import [FILE\|-] [--text FILE] [--defines FILE] [--table T] [--replace]` | [import.md](import.md) |
-| `takeover TABLE... [-y]` | delete tables Housecarl's ruleset replaces (administrators) |
+| `import [FILE\|-] [--text FILE] [--defines FILE] [--table T] [--zone IFACE=NAME]... [--replace]` | [import.md](import.md) |
+| `takeover TABLE... [-y]` | delete tables Housecarl's ruleset replaces (administrators); not in shadow mode |
 
 ### cert
 

docs/configuration.md

@@ -14,7 +14,7 @@ reads the file when it starts.
 | `tls.cert_file`, `tls.key_file` | none | serve HTTPS with this certificate; housecarld rereads it when the file changes, or on `systemctl reload housecarl` |
 | `tls.acme.*` | off | get the certificate from Let's Encrypt, with Jarl publishing the challenge ([certificates.md](certificates.md#settings)) |
 | `jarl.server`, `jarl.token_file` (or `jarl.token`), `jarl.ca_cert`, `jarl.insecure` | none | the Jarl server for the certificate's challenge, and its token ([jarl.md](jarl.md)) |
-| `ruleset` | `/var/lib/housecarl/ruleset.nft` | the confirmed ruleset, which `housecarl-ruleset.service` loads at boot |
+| `ruleset` | `/var/lib/housecarl/ruleset.nft` | the confirmed ruleset, which `housecarl-ruleset.service` loads at boot; its fallback is saved beside it as `ruleset-fallback.nft` ([operations.md](operations.md#if-the-ruleset-fails-to-load-at-boot)). Change both paths in the unit too |
 | `trusted_proxies` | `[]` | addresses or prefixes of reverse proxies whose `X-Forwarded-For` and `X-Forwarded-Proto` are believed |
 | `socket` | `/run/housecarl/housecarl.sock` | the local API socket; `""` turns it off |
 | `database` | `/var/lib/housecarl/housecarl.db` | the SQLite database: configuration, history, users, audit log |
@@ -26,11 +26,21 @@ reads the file when it starts.
 | `backup.dir` | `/var/backups/housecarl` | where `housecarld backup` writes |
 | `backup.retention` | `30d` | backups older than this are deleted… |
 | `backup.keep_at_least` | `7` | …keeping at least this many |
+| `stats.enabled` | `true` | keep traffic, drops, hits, blocked sources, connections and quotas over time ([stats.md](stats.md)) |
+| `stats.database` | `/var/lib/housecarl/stats.db` | the statistics' own database; deleting it is safe |
+| `stats.retention.minute`, `.hour`, `.day` | `7d`, `90d`, `730d` | how long per-minute, hourly and daily figures are kept |
+| `stats.top_n` | `100` | entries kept per top list, per hour and per day |
+| `shadow.enabled` | `false` | shadow mode: Housecarl's ruleset is loaded beside the firewall you have, but blocks, rejects and translates nothing; its counters, log and tracer show what it would do. No change is on probation, and takeovers and closing connections are refused ([shadow.md](shadow.md)) |
+| `shadow.position` | `after` | where Housecarl's filter chains run in shadow mode: `after` the other firewall's (`filter + 10`: it sees only what that one lets through, so what it would drop is what switching would break) or `before` them (`filter - 10`: it sees every packet) |
 
 Durations are written `15s`, `12h` or `30d`.
 
+Shadow mode is here, and not a firewall setting, so that no user or API
+token can switch enforcement off; restart housecarld to switch.
+
 The probation's length is a firewall setting, not here: `hc settings set
---probation 5m`.
+--probation 5m`. So is counting traffic per device for the statistics:
+`hc settings set --stats-per-device on`.
 
 ## HTTPS
 

docs/design.md

@@ -75,8 +75,9 @@ to be reviewed and applied together with others:
 | CLI | Immediate: each command is a change of its own. Stage with `-C ID` or `HOUSECARL_CHANGESET`. | Scripts want a clear result per step. |
 | API | Immediate unless a changeset is named (`?changeset=ID`, `X-Housecarl-Changeset`). | As the CLI. |
 
-Undo, imports and takeovers can't be staged; with a changeset named, they
-are refused rather than applied.
+An undo is staged like any edit (in review mode, the web UI's Undo
+collects it for review). Imports and takeovers can't be staged; with a
+changeset named, they are refused rather than applied.
 
 ## Its own table only
 
@@ -99,6 +100,32 @@ Every rule carries a comment `hc:<mark>`, such as `hc:r12` for rule 12,
 `hc:f3` for port forward 3, `hc:b-established` for a built-in. Counters
 and log lines are traced back to their object through it.
 
+## Shadow mode
+
+With `shadow.enabled` in the configuration file ([shadow.md](shadow.md)),
+the same configuration is rendered so that it changes nothing (render's
+`Options.Shadow`): drops and rejects accept instead, policies accept,
+NAT only counts, and what alters packets (MSS clamping, the flowtable)
+and raw rules that do more than accept, drop or reject are left out.
+This rests on how nftables runs base chains: an `accept` only ends its
+own chain, and the packet goes on to the next one on the hook, other
+tables' included; only a drop is final. Another table, the real
+firewall, decides.
+
+What would be dropped, rejected or translated is counted under the same
+marks, and its comment says so (`"hc:r12 [would drop] …"`), which is
+how the tracer tells it apart. The filter chains run at `filter + 10`
+(after the other firewall's: what Housecarl would drop is what switching
+over would break) or `- 10`; the NAT chains always at `- 10`, since the
+kernel asks NAT chains only until one translates.
+
+Every change is checked by nftables in both forms, so leaving shadow
+mode can't fail on the ruleset. There is no probation, since nothing
+can lock anyone out; takeovers and closing connections, which would
+change traffic, are refused. A ruleset reload elsewhere (`flush
+ruleset`) deletes Housecarl's table too: housecarld listens to nftables'
+netlink notifications (internal/tablewatch) and puts it back.
+
 ## Not locking yourself out
 
 A change made over the network is applied **on probation** (a changeset
@@ -107,6 +134,19 @@ confirmed within two minutes (Settings; `hc confirm`, the banner in the
 web UI, `POST /confirm`), it is rolled back by itself, and the rollback is
 recorded in the history. A change that cuts you off undoes itself.
 
+* **The client that made a change confirms it by itself** once it
+  reaches the API again: the change's answer carries a one-time token and
+  closes its connection (HTTP/2: a GOAWAY), and the web UI and `hc` send
+  the token straight back. That proves something only on a new
+  connection, since the built-in that accepts established connections
+  would let the old one through whatever the change did. So the token is
+  accepted only from the same user (and API token), the same address, a
+  different connection, and for the newest change. A change that cuts its
+  client off can't be confirmed by it, and rolls back. This proves the
+  API is still reachable, not that SSH or the LAN still work; with the
+  API, you can fix those. Settings → "a person confirms each change"
+  (`hc settings set --auto-confirm off`) goes back to confirming by hand;
+  `hc --no-confirm` leaves one change waiting.
 * Changes made through the local socket (`sudo hc …` on the firewall)
   need no confirmation: whoever makes them is already on the machine. One
   made while others are pending joins them.
@@ -125,6 +165,19 @@ firewall is up even if housecarld can't start. An unconfirmed ruleset is
 never saved there: a reboot during probation brings back the last
 confirmed one.
 
+It fails closed. A fallback saved beside it (`ruleset-fallback.nft`,
+`render.Fallback`) is loaded if it doesn't: no sets, no flowtable,
+interfaces by name only, so there is little in it to fail. It keeps
+anything new from the external zones out and lets the inside work,
+which is as far as a ruleset can go without knowing the rules; a
+fallback that let everything in would leave a router open on the
+internet, and one that let nothing in would lock out the person who has
+to fix it. The boot ruleset has no flowtable either: nftables refuses a
+flowtable naming a device that doesn't exist yet (on Debian 13's
+kernel), and before the network is up, `ppp0` doesn't. housecarld
+watches the kernel's link messages (internal/linkwatch) and applies
+again when a flowtable device appears, or comes back as a new interface.
+
 ## The drop log
 
 Rules with logging on, and the policies (Settings), send packets to

docs/development.md

@@ -11,6 +11,8 @@ internal/
   render/             State → the nftables ruleset text, with each line's object
   nft/                runs nft: check, apply (keeping dynamic set entries and counters), list, the boot file
   nflog/              reads logged packets from netlink log group 4171
+  linkwatch/          netlink: interfaces that appear or go (the flowtable's devices)
+  tablewatch/         netlink: a table deleted (shadow mode puts Housecarl's back; docs/shadow.md)
   importer/           nft -j list ruleset → State, with notes and raw rules (docs/import.md)
   hostif/             the machine's interfaces
   store/              SQLite (modernc.org/sqlite, no cgo): configuration, history, users, tokens, sessions, audit
@@ -70,8 +72,11 @@ Newer nftables accepts things 1.1.3 doesn't (a meter with a timeout for
   `wan`, `enp2s0` and `wg0` inside it, IPv4 and IPv6;
 * "the internet" on the WAN side, a LAN client, and a "jellyfin" host.
 
-It imports a copy of the maintainer's ruleset (lab addresses), takes
-over the old table, and checks with real traffic: port forwards with
+It imports a copy of the maintainer's ruleset (lab addresses) with
+housecarld in shadow mode beside it (traffic unchanged, a would-be drop
+logged and let through, the table back after the old ruleset is
+reloaded), restarts it enforcing, takes over the old table, and checks
+with real traffic: port forwards with
 hairpin NAT and the IPv6 pinhole, masquerading, antispoofing, the drop
 log, a lockout change that rolls back by itself, a restart, the boot
 ruleset after a flush, and that other tables are left alone. It makes

docs/flowtable.md

@@ -0,0 +1,187 @@
+# The flowtable: a fast path for forwarded traffic
+
+The flowtable is a shortcut inside the kernel for traffic the firewall
+**forwards** (between the internet and your LAN, between zones). It
+speeds up routing on a machine whose CPU can't keep up with its line.
+It doesn't make the firewall any safer or less safe, but it changes what
+some features can see, so read [when not to use it](#when-not-to-use-it)
+before you turn it on.
+
+It is off until you name its devices.
+
+## What it does
+
+Without a flowtable, every forwarded packet takes the whole way through
+the kernel: the prerouting hooks, the routing decision, Housecarl's
+forward chain (the built-ins, then your rules) and postrouting with NAT.
+For an established connection almost all of that work only ends in the
+`established` built-in accepting it again, packet after packet.
+
+With a flowtable, Housecarl adds one built-in to the forward chain:
+
+```
+meta l4proto { tcp, udp } flow add @ft
+```
+
+A TCP or UDP connection takes the whole way for its first packets, as
+always: your rules decide whether it is allowed at all. Once the kernel
+has seen it established (and your rules let it through), it is added
+to the flowtable. From then on its packets are caught as soon as they
+arrive on one of the flowtable's devices (the *ingress* hook) and sent
+straight out on the other side: NAT is still applied, the TTL still
+counts down, but every chain in between is skipped.
+
+```
+                 first packets                         every packet after
+in ─► prerouting ─► routing ─► forward ─► postrouting ─► out
+ │                                                        ▲
+ └────────────── flowtable (ingress) ─────────────────────┘
+```
+
+It is the software flowtable, run by the CPU. Offloading to the network
+card's hardware isn't supported ([design.md](design.md#not-in-01)).
+
+## When to use it
+
+Turn it on when **the router's CPU is what limits your speed**:
+
+* a speed test through the router gets less than your line gives (or
+  less than the same test from the router itself), and
+* during it, one CPU core is busy with network work: in `top`, press `1`
+  for every core, and look at `si` (softirq) near 100 % on a core.
+
+Typical cases: a small or old machine (Atom, Celeron, a Raspberry Pi, an
+old Core 2) on a fast line (500 Mbit/s and up), lots of small packets
+(gaming, VoIP, many downloads at once), or a machine that is busy with
+other work besides routing. The gain is CPU time per packet: often well
+over half of the forwarding cost, which shows up as speed only when the
+CPU was the limit, and otherwise as a cooler, quieter, more responsive
+machine.
+
+In shadow mode ([shadow.md](shadow.md)) Housecarl leaves its flowtable
+out: an offloaded connection skips every table's forward chain, the
+other firewall's too. If that firewall has a flowtable of its own, the
+same goes for Housecarl's forward chain: it only sees each connection's
+first packets.
+
+## When not to use it
+
+**When the CPU keeps up anyway.** If the speed test already reaches your
+line and `si` stays low, the flowtable buys you nothing you'd notice,
+and costs you the figures below. Most modern x86 machines route a
+gigabit line without it.
+
+**When you count or limit traffic in the forward chain.** The fast path
+skips the forward chain, so whatever must see *every* packet of a
+connection only sees the first few. Housecarl warns about each:
+
+| Feature | With the flowtable | Warning |
+|---|---|---|
+| quotas and byte-rate limits on forward rules | only count first packets: a device can go far past its allowance | `flowtable_traffic` |
+| forward rules with "every packet" (`--every-packet`) | only see first packets | `flowtable_traffic` |
+| statistics' traffic per device (`stats_per_device`) | only count first packets | `stats_flowtable` |
+| hit counters of the `established` built-in (forward) | stop rising once a connection is offloaded | |
+
+Rate limits by packets (`--limit 10/second`) and connection limits act
+on new connections, so they work the same.
+
+**When other tables must see forwarded traffic.** Offloaded packets skip
+the forward hooks of *every* table, not only Housecarl's: Docker's,
+Incus', libvirt's or a table of your own. Those normally only judge new
+connections, which still pass through them, but a table that counts or
+shapes established traffic in its forward chain won't see it.
+
+What keeps working as before:
+
+* **Your rules** decide every new connection: nothing gets offloaded
+  that your rules didn't let through. A rule you add later doesn't stop
+  a connection already open, but that is true without the flowtable too
+  (the `established` built-in accepts it).
+* **NAT**: port forwards, hairpin NAT, masquerading and SNAT.
+* **The input and output chains**: traffic to and from the firewall
+  itself never uses the flowtable.
+* **Closing connections** (`hc connections close`, Traffic →
+  Connections) ends their fast path too: the kernel drops a flow with
+  its tracking entry.
+* **Interface statistics** (traffic per zone and interface) come from the
+  interfaces' own counters, which count every packet.
+* **Traffic shaping with `tc`** on the outgoing interface still applies.
+* **The log** and **the packet tracer** see what goes through the rules:
+  new connections, as always. The tracer doesn't see the packets of an
+  offloaded connection; trace a new one.
+* ICMP, and protocols other than TCP and UDP (GRE, ESP, SCTP), always
+  take the normal way.
+
+## How to turn it on
+
+Name the devices whose traffic should take the fast path, **both
+sides**: the interface connections come in on and the one their replies
+come in on. For a router that means the external interface and the
+inside ones:
+
+```sh
+hc settings set --flowtable wan --flowtable enp2s0
+hc settings                     # Flowtable: wan, enp2s0
+```
+
+In the web UI: Settings → Flowtable, add each device. The fast path
+only catches packets arriving on a listed device, so a connection to or
+from an unlisted one gains little or nothing: leave a device out only on
+purpose (say, a WireGuard interface whose traffic you'd rather keep on
+the normal way).
+
+* Use **exact device names**, not wildcards (`incusbr*` is refused):
+  the kernel attaches the flowtable to each device by name.
+* With **PPPoE**, name the PPP interface (`ppp0`, or the name your
+  `ppp@` unit gives it), the one in your external zone.
+* `--flowtable` replaces the whole list; `hc settings set --flowtable
+  none` turns the flowtable off.
+
+The change is applied (and on probation, if made over the network) like
+any other.
+
+### Devices that come and go
+
+PPP and WireGuard interfaces don't exist until their service brings
+them up, and PPPoE makes a new `ppp0` each time it reconnects. Housecarl
+handles that:
+
+* A device that doesn't exist yet is left out of the flowtable, with the
+  warning `flowtable_device_missing`, instead of failing the whole
+  ruleset.
+* housecarld watches the kernel's interface changes: when a flowtable
+  device appears, or comes back as a new interface after a reconnect,
+  it applies the ruleset again with it, a couple of seconds later. The
+  journal says `interfaces changed: applied the ruleset again`.
+  Applying again keeps the counters, dynamic sets and quotas, as any
+  change does, and adds no history entry.
+* The ruleset saved for boot has no flowtable at all: it loads before
+  the network is up, when `ppp0` and `wg0` can't exist yet, and
+  nftables on Debian 13's kernel refuses a flowtable naming a device
+  that doesn't exist (and with it the whole file). housecarld adds the
+  flowtable once it starts. Until then, traffic takes the normal way,
+  through the same rules.
+
+## Checking it works
+
+```sh
+sudo nft list flowtables          # the devices it is attached to
+sudo conntrack -L | grep OFFLOAD  # connections on the fast path
+```
+
+`conntrack` is in Debian's `conntrack` package. A long download through
+the router should show up as `[OFFLOAD]`. Compare a speed test, and `si`
+in `top`, before and after.
+
+If nothing gets offloaded, check that both of the connection's devices
+are listed, that `flowtable_device_missing` isn't in `hc status`, and
+that the connection is TCP or UDP and established.
+
+## Turning it off
+
+```sh
+hc settings set --flowtable none
+```
+
+Connections on the fast path go back to the normal way with their next
+packet.

docs/import.md

@@ -18,11 +18,9 @@ shows what it would make, without saving anything: zones, sets,
 services, rules, port forwards and NAT, then **notes** on every guess it
 made, and the rules it **kept raw**.
 
-`--defines` reads your ruleset file's `define` lines, so sets get your
-names (`define admins = { ... }` becomes the set `admins`).
-Without it, sets are named after what they hold. Interfaces named by a
-define name their zone too (`define wan_interface = "ppp0"` makes the
-zone `wan`); otherwise a zone is named after its interface.
+`--defines` reads your ruleset file's `define` lines, so sets and zones
+get your names ([How things are named](#how-things-are-named)).
+`--zone enp2s0=lan` names a zone yourself.
 
 From another machine, give the JSON as a file, and the text with
 `--text`:
@@ -37,6 +35,46 @@ With several tables, `--table "inet main"` picks one; by default every
 table but Housecarl's is read. The web UI has the same under Server →
 Import a ruleset.
 
+## How things are named
+
+**Zones** are named in this order:
+
+1. **As you say**: `--zone enp2s0=lan` (repeat it; Zone names on the
+   web UI's Import page). Interfaces given the same name share one zone
+   (`--zone enp2s0=lan --zone enp3s0=lan`).
+2. **After a define holding the interface**, with a trailing `_int`,
+   `_if`, `_iface`, `_ifname`, `_interface`, `_dev`, `_device` or `_nic`
+   taken off: `define wan_int = "ppp0"` makes the zone `wan`, `define
+   iot_if = enp2s0.20` (a VLAN) the zone `iot`.
+3. **After the interface itself**: `enp2s0`.
+
+If the name is taken, the zone is called `INTERFACE_zone`. A define
+names what it was *for* in your old ruleset, which isn't always what the
+network is (`define dhcp_int = enp2s0` makes `dhcp`, for your LAN). A
+note lists each zone and where its name came from; rename one with
+`--zone` and import again, or afterwards with `hc zone update dhcp
+--name lan` (its rules follow). Name zones for what the network is for:
+`lan`, `iot`, `guest`, `servers`, `vpn`, `wan`.
+
+**Address sets**: a named set of the table keeps its name; a define
+holding a list of addresses, or one prefix, gives its name to a set
+(`define trusted = { 10.0.10.0/24, 10.1.1.0/24 }` makes `trusted`,
+wherever those addresses appear together). A single address stays an
+address. When two defines hold the same addresses, the first by name
+wins.
+
+**Services**: a define holding several ports names a service (`define
+web_ports = { 80, 443 }` makes `web_ports`); one port stays a port
+(`tcp/22`).
+
+**Port forwards** are named after their comment, without a leading
+"Port forward" (`comment "Port Forward Jellyfin"` makes `jellyfin`);
+several ports in one DNAT become one forward each (`web-80`, `web-443`),
+and a DNAT map's elements are named after the map (`forwards-443`).
+
+Names that aren't valid are made so: other characters become `_` (`-`
+in forward names), and a name must start with a letter.
+
 ## What becomes what
 
 * **Interfaces** become zones. Zones are marked external when the rules
@@ -45,18 +83,30 @@ Import a ruleset.
 * **Sets and defines** of addresses become address sets; port lists
   become services.
 * **Jumps and gotos** to your own chains are flattened: the jump's
-  condition is added to each rule of the chain, in place.
+  condition is added to each rule of the chain, in place. When the jump
+  and a rule both match sources (or destinations), the narrower is kept
+  if one lies within the other; otherwise, and for any rule there that
+  stays raw, the jump's condition is written in front of the raw rule's
+  text, so it still only applies where the jump did.
 * **Rules** keep their order. A rule that is a built-in (loopback,
   established, invalid, the ICMPv6 rules, MSS clamping, the DHCPv6 client
   rule, antispoofing) becomes that built-in only when it sits where the
   built-in goes, at the head of every chain it covers. Otherwise it stays
-  a rule where it was, so the ruleset does what it did. Built-ins your
-  ruleset doesn't have are turned off.
+  a rule where it was, so the ruleset does what it did. Since built-ins
+  come before every rule, once one at a chain's head stays a rule, the
+  ones after it there stay rules too, in order: an invalid drop that
+  stays a rule keeps the ICMPv6 accepts after it behind it, as they were.
+  Built-ins your ruleset doesn't have are turned off.
 * **DNAT** rules become port forwards, TCP and UDP of the same port
-  merged into one. The forward-chain accepts that let them through
-  become part of the port forward, and a masquerade for inside sources
-  to the same host becomes its hairpin. An IPv6 accept to the same port
-  becomes the forward's IPv6 pinhole when there is only one candidate.
+  merged into one, one per port when a rule has several. The
+  forward-chain accepts that let them through (`ct status dnat`) become
+  part of the port forward; an accept to the same host and port without
+  `ct status dnat` lets in more than forwarded connections, and stays a
+  rule. A masquerade for inside sources to the same host becomes its
+  hairpin. An IPv6 accept to the same port, from everywhere, becomes the
+  forward's IPv6 pinhole when there is only one candidate. A DNAT to an
+  IPv6 address, or with matches a port forward has no field for, is left
+  out with a note.
 * **Maps are expanded**, one element at a time. A DNAT map
   (`dnat ip to tcp dport map @forwards`, named or written in the rule)
   becomes one port forward per element; forwards from a named map are
@@ -75,13 +125,27 @@ Import a ruleset.
   Byte rates and quotas are imported too; a quota imported resets only by
   hand until you give it a period. A rule with a quota or a byte rate that
   accepts stays a raw rule ([rules.md](rules.md#bandwidth-and-quotas)).
+* **Sets with `flags timeout`** (filled by a program or a rule) become
+  dynamic sets, without their entries of the moment: whatever fills them
+  must fill Housecarl's from then on, as a note says.
 * **Policies** become the chains' policies.
-* **Anything else** becomes a raw rule with its original text, in place.
-  A raw rule naming a set of the old table needs the new set name:
-  Housecarl's are `NAME_v4` and `NAME_v6`, and a note says so.
+* **Anything else** becomes a raw rule with its original text, in place:
+  a match Housecarl has no field for, two matches on the same field
+  (`ip saddr 10.0.0.0/8 ip saddr != 10.0.10.0/24`), `!=` where Housecarl
+  only has `==`, a set of interface names. Its set references are made
+  to work in Housecarl's table: an imported address set becomes
+  `@NAME_v4` or `@NAME_v6`, any other set (interface names, ports) is
+  written out with its elements. A raw rule naming a set that can't be
+  (a meter, an empty set) is left out with a note.
+* **Logging rules** log to Housecarl's log (`hc log`), not the kernel's.
 
-Chains on hooks Housecarl doesn't manage (ingress, a bridge table) are
-left out, with a note.
+Several base chains on one hook (`input` at priority -10 and at 0) are
+put one after the other, by priority, into Housecarl's one chain, with
+the stricter policy; a note says to check it, since a packet had to get
+past each, and an accept in the first no longer ends there. A prerouting
+filter chain after connection tracking becomes the early chain, which
+runs before it: a note says so. Chains on hooks Housecarl doesn't manage
+(ingress, a bridge table) are left out, with a note.
 
 ## 2. Import
 
@@ -95,6 +159,11 @@ Housecarl's table is now in force **alongside** the old one: a packet
 must be accepted by both. Nothing should change for your traffic. If
 Housecarl already has a configuration, `--replace` replaces it.
 
+To be sure first, import in **shadow mode** ([shadow.md](shadow.md)):
+Housecarl's ruleset is then loaded but drops nothing, and its log shows
+each packet the old ruleset lets through that Housecarl would drop. When
+there are none you mind, leave shadow mode and take over.
+
 Imported over the network (not through the local socket), the import is
 on probation like any change: confirm it.
 
@@ -107,7 +176,10 @@ the old table:
 sudo hc takeover "inet main"
 ```
 
-From now on only Housecarl's rules apply. Over the network, this is on
+From now on only Housecarl's rules apply. Until you do, the dashboard
+and `hc status` warn that the imported table still applies
+(`takeover_pending`); they also warn while `nftables.service` is enabled
+(`nftables_service`). Over the network, this is on
 probation too: unless confirmed, the old table comes back.
 
 ## 4. The boot
@@ -122,6 +194,11 @@ sudo systemctl disable nftables.service
 sudo mv /etc/nftables.conf /etc/nftables.conf.before-housecarl
 ```
 
+Disable it, but **don't stop it** (no `systemctl stop nftables`, no
+`disable --now`): Debian's unit stops by running `nft flush ruleset`,
+which deletes every table, Housecarl's and Docker's included, at once.
+Disabled, it simply doesn't run at the next boot.
+
 `housecarl-ruleset.service` runs after `nftables.service` in any case,
 so a leftover `flush ruleset` can't undo it. Docker, Incus and libvirt
 keep adding their own tables as before.

docs/install.md

@@ -85,6 +85,10 @@ you are changing the firewall you are connected through.
    sudo systemctl disable nftables.service
    ```
 
+   To try Housecarl beside your firewall first, turn on **shadow mode**
+   before the import: it loads Housecarl's ruleset but blocks nothing,
+   and shows what it would do ([shadow.md](shadow.md)).
+
    **Or start from scratch**, through the local socket (no probation).
    The first change puts the ruleset in force, and the input and forward
    policies drop what no rule accepts, so let yourself in first:

docs/nat.md

@@ -7,6 +7,10 @@ public port are sent to the host, and let through the forward chain.
 One object does what takes three rules by hand (the DNAT, the forward
 accept, and the hairpin masquerade).
 
+In shadow mode ([shadow.md](shadow.md)) NAT rules translate nothing:
+they count the connections they would translate, and the other
+firewall's NAT does the work.
+
 ```sh
 hc forward add 8443 --to 192.168.1.10:443 --public 203.0.113.10 --hairpin --name web
 hc forward add 27015-27030 --proto tcp,udp --to 192.168.1.20 --public 203.0.113.10 --hairpin --name game

docs/operations.md

@@ -28,7 +28,26 @@ edits) applies each edit at once instead. With hc, see
 ## Changes on probation
 
 Changes made over the network roll back unless confirmed
-([design.md](design.md#not-locking-yourself-out)):
+([design.md](design.md#not-locking-yourself-out)); in shadow mode, where
+a change can't cut anyone off, none do ([shadow.md](shadow.md)). The web UI and `hc`
+confirm their own changes once they reach Housecarl again over a new
+connection, so normally nothing waits:
+
+```
+$ hc rule add input --from admins --service ssh
+created rule input #31
+applied and confirmed (history 12): this client reached Housecarl again over a new connection
+```
+
+A change that cuts the client off can't be confirmed that way:
+
+```
+applied on probation (history 13), but couldn't confirm it: … context deadline exceeded
+It is rolled back within 1m50s unless confirmed ('hc confirm')
+```
+
+By hand (with `--no-confirm`, or with Settings → "a person confirms each
+change"):
 
 ```sh
 hc status              # Probation: 2 changes roll back in 97s unless confirmed
@@ -53,26 +72,91 @@ sudo hc history undo 42
 
 A confirmed ruleset (or one changed through the socket) is saved to
 `/var/lib/housecarl/ruleset.nft`, which `housecarl-ruleset.service`
-loads before the network comes up. To see what would load:
+loads before the network comes up, so before PPPoE, WireGuard or DHCP
+bring up their interfaces. To see what would load:
 
 ```sh
 sudo cat /var/lib/housecarl/ruleset.nft
 sudo nft -c -f /var/lib/housecarl/ruleset.nft     # check it
 ```
 
-Dynamic sets start empty after a reboot.
+In shadow mode both files are shadow mode's, and the fallback filters
+nothing ([shadow.md](shadow.md#what-works-differently)).
 
-Keep Debian's `nftables.service` disabled, or `/etc/nftables.conf`
-without `flush ruleset`: otherwise it deletes Housecarl's table at
-boot. `housecarl-ruleset.service` runs after it, which covers the
-ordinary case.
+Dynamic sets start empty after a reboot. The saved ruleset has no
+flowtable, since its devices may not exist yet: housecarld adds it when
+it starts, and whenever a flowtable device appears
+([flowtable.md](flowtable.md#devices-that-come-and-go)).
+
+### If the ruleset fails to load at boot
+
+Should `ruleset.nft` fail to load (a damaged file, an nftables or kernel
+upgrade that refuses something in it), the machine would otherwise boot
+with no firewall at all until housecarld starts. Instead,
+`housecarl-ruleset.service` loads a **fallback** that housecarld saves
+beside it, `/var/lib/housecarl/ruleset-fallback.nft`, from the same
+confirmed configuration:
+
+* **Nothing new gets in from the external zones' interfaces**, to the
+  firewall or through it: no port forwards, no SSH or web UI from the
+  internet.
+* What the firewall and the inside need to keep working is let in:
+  established connections, the ICMPv6 IPv6 needs (neighbour discovery,
+  router advertisements, packet too big), and DHCPv6 replies from
+  link-local addresses.
+* **Everything else is let through**: the LAN, WireGuard and the
+  firewall itself are reachable from the inside, so you can log in and
+  fix it.
+* IPv4 leaving an external zone is masqueraded (when any source NAT is
+  configured), so the inside still reaches the internet. MSS clamping
+  stays, if it is on.
+
+PPPoE itself isn't affected: its discovery and session are Ethernet
+frames, and the PPP link's negotiation happens before any IP, neither
+of which an `inet` table sees.
+
+The fallback only knows which interfaces face the internet from the
+zones marked **external**. With none, it lets nothing but loopback
+through, and the warning `no_external_zone` says so.
+
+When the fallback loads, the unit is marked failed, and the journal says
+why (as a critical message):
+
+```sh
+systemctl status housecarl-ruleset
+journalctl -b -u housecarl-ruleset
+```
+
+housecarld puts the real ruleset back when it starts, and saves a fresh
+`ruleset.nft`. If it can't apply it either, the fallback stays in force:
+`hc status` and the dashboard say "NOT in force: the boot fallback is in
+force", and `/healthz` answers 503 ([Monitoring](#monitoring)). Fix
+what `hc status` or the journal reports, then restart housecarld.
+
+Keep Debian's `nftables.service` disabled (`systemctl disable
+nftables`, without `--now`): at boot it loads `/etc/nftables.conf`, and
+stopping or restarting it runs `nft flush ruleset`, which deletes every
+table, Housecarl's included. `housecarl-ruleset.service` runs after it
+at boot, which covers the ordinary case, but not a restart of
+`nftables.service` later. Housecarl never reads or writes
+`/etc/nftables.conf`; its ruleset is saved in
+`/var/lib/housecarl/ruleset.nft`. The dashboard and `hc status` warn
+while `nftables.service` is enabled.
+
+Other programs' tables are left alone: fail2ban's (`inet f2b-table`),
+Docker's, Incus's. Housecarl replaces only its own table, so they keep
+their rules through every Housecarl change and restart, and nothing
+needs restarting in a particular order. A packet must get past every
+table on its hook, so a drop in any of them wins.
 
 ## Hits
 
 Every rule, NAT rule, built-in and policy has a counter. `hc rule list`,
 `hc forward list` and the web UI show packets; `hc counters` shows them
 all, with bytes. They count from housecarld's start, and carry across
-changes to the ruleset.
+changes to the ruleset. A deleted object's counter goes with it; an
+import with `--replace` gives every object a new ID, so its counters
+start again. A disabled rule keeps its count for when it is enabled.
 
 ## Quotas
 
@@ -99,21 +183,71 @@ hc log --search 198.51.100.7
 10:02:15  hc forward policy drop: wan  eth1  198.51.100.9:443 -> 192.168.1.20:51432  tcp
 ```
 
+In shadow mode ([shadow.md](shadow.md)) what the policies would drop is
+always logged, in all three chains, and drops are marked as what would
+have been done: the packets went on.
+
 The packets go to netlink log group 4171, where housecarld reads them;
 the last 2000 are kept in memory. They aren't written to the kernel log
 or the journal. A rule's log isn't rate-limited: on a busy rule, give it
 a limit, or log only new connections (`--state new`).
 
+## Connections
+
+The built-in that accepts established connections lets their packets
+through without checking the rules again: a rule added later doesn't
+stop a connection already open. Traffic → Connections (`hc connections`)
+lists what the kernel tracks, with NAT (`to 192.168.1.10:443`: a port
+forward; `as 203.0.113.10`: masqueraded) and the time left:
+
+```sh
+hc connections --search 192.168.1.50
+hc connections --search ":443 tcp"
+hc connections close --search 192.168.1.50
+```
+
+Closing a connection deletes its tracking entry: its next packet is
+checked by the rules as they are now. One they block is stopped; one they
+allow carries on (TCP connections are picked up again mid-stream).
+
+To make a block stop open connections by itself, tick "every packet" on
+the rule (`--every-packet`): it is checked before established connections
+are accepted, like rules with a quota ([rules.md](rules.md#rules)).
+
+## Tracing
+
+Traffic → Trace (`hc trace`) follows packets through the whole ruleset:
+Housecarl's rules and every other table (Docker's, fail2ban's, an old
+`inet main`), each rule and policy they meet, and what accepted or
+dropped them:
+
+```
+$ hc trace --address 192.168.1.50 --port 443 --seconds 60
+14:02:13.120  192.168.1.50:51432 -> 198.51.100.9:443  tcp syn in eth1 out wan
+    inet housecarl forward: built-in: mss-clamp -> continue
+    inet housecarl forward: rule forward #12 (kids at night) -> drop
+    => DROPPED by rule forward #12 (kids at night)
+```
+
+It traces packets from or to the address, on the port (either way), for
+5 to 600 seconds, at most 200 packets and 20 a second. It works with
+nftables' own tracing: a table of its own (`inet housecarl-trace`,
+hooked before everything else) marks the packets, and `nft monitor
+trace` reports them; the table goes when the trace ends. One trace runs
+at a time.
+
 ## When something doesn't get through
 
-1. `hc log -f` with `--log-drops on`: is it dropped by a policy (no rule
+1. Trace it (above): it names the rule or policy that dropped it, in
+   whichever table.
+2. `hc log -f` with `--log-drops on`: is it dropped by a policy (no rule
    accepted it), or by a rule (its number is in the line)?
-2. `hc rule list forward` (or `input`): hits on the rule you expect?
-3. `hc ruleset`: the rule as nftables sees it, with the zones' interfaces
+3. `hc rule list forward` (or `input`): hits on the rule you expect?
+4. `hc ruleset`: the rule as nftables sees it, with the zones' interfaces
    and the sets' families.
-4. `hc status`: other tables. A Docker or Incus table, or an old one,
+5. `hc status`: other tables. A Docker or Incus table, or an old one,
    can drop what Housecarl accepts.
-5. Port forwards: `hc forward list` hits, and the inside host's own
+6. Port forwards: `hc forward list` hits, and the inside host's own
    firewall and default route (answers must go back through Housecarl).
 
 ## Backups
@@ -149,6 +283,31 @@ database is kept next to it as `housecarl.db.before-restore-<time>`.
 database when it starts, and an older one refuses a database a newer one
 has upgraded: restore a backup to go back.
 
+## Monitoring
+
+`GET /healthz` needs no login, and answers within a few seconds of any
+change (it reads the ruleset at most every five seconds, however often
+it is asked):
+
+```sh
+$ curl -s https://firewall.example.net:8070/healthz
+{"ok":true,"version":"0.1.0","firewall":"applied","mode":"enforcing","probation":0,"warnings":[]}
+```
+
+| Field | |
+|---|---|
+| `ok` | `true` with HTTP 200 when the firewall is as it should be; `false` with HTTP **503** otherwise |
+| `firewall` | `applied` (Housecarl's ruleset is in force), `unconfigured` (nothing configured yet, so nothing to apply), `fallback` (the [boot fallback](#if-the-ruleset-fails-to-load-at-boot) is in force), `missing` (no Housecarl table: flushed, or never applied) |
+| `mode` | `enforcing`, or `shadow`: Housecarl's ruleset is loaded but blocks nothing, and another firewall decides ([shadow.md](shadow.md)). `ok` doesn't depend on it; to be told when a machine isn't enforcing, also expect `"mode":"enforcing"` |
+| `probation` | changes waiting to be confirmed, rolled back unless they are |
+| `warnings` | codes of problems with the machine, as `hc status` shows them (`nftables_service`) |
+
+For Checkmk, an HTTP check on the URL that expects status 200 (and,
+if you like, the string `"firewall":"applied"`) covers housecarld being
+down, the fallback and a missing ruleset. Reach it from where the input
+rules let the monitor in (a rule for the Checkmk server's address and
+port 8070).
+
 ## Checks
 
 `housecarld check` validates the configuration in the database against

docs/permissions.md

@@ -7,9 +7,9 @@ else gets **grants**, `resource:access`:
 
 | Resource | read | write |
 |---|---|---|
-| `config` | settings, zones, sets, services, rules, NAT, the ruleset, history, interfaces, probation, changesets (the web UI shows the forms locked) | change them, stage and apply changesets (your own, or anyone's for administrators), undo, import, confirm and roll back |
+| `config` | settings, zones, sets, services, rules, NAT, the ruleset, history, interfaces, probation, changesets (the web UI shows the forms locked) | change them, stage and apply changesets (your own, or anyone's for administrators), undo, import, confirm and roll back; close connections; start and stop traces |
 | `sets` | dynamic sets' entries | add and remove entries |
-| `log` | hit counters and the log | – |
+| `log` | hit counters, the log, connections, traces | – |
 | `audit` | the audit log | – |
 | `*` | all of the above | |
 

docs/rules.md

@@ -21,6 +21,32 @@ A zone may name an interface that doesn't exist yet (a WireGuard
 interface that comes up later): its rules match once it appears, and
 `hc status` warns until then.
 
+**VLANs.** A VLAN sub-interface is an interface like any other, by its
+name: `enp2s0.20` (or whatever your network configuration calls it,
+`vlan20`, `br0.20`). Tagged traffic arrives on the sub-interface, and
+untagged traffic on the parent, so `enp2s0` alone doesn't cover
+`enp2s0.20`. Give each VLAN its own zone to keep them apart:
+
+```sh
+hc zone add lan --interface enp2s0          # untagged
+hc zone add iot --interface enp2s0.20
+hc zone add guest --interface enp2s0.30
+hc rule add forward --in iot --out wan
+```
+
+or one zone for the parent and all of its VLANs, when they should be
+treated alike: `--interface enp2s0 --interface 'enp2s0.*'`. Interface
+names are at most 15 characters (the kernel's limit). The flowtable
+takes VLAN sub-interfaces too, by exact name
+([flowtable.md](flowtable.md)).
+
+**Renaming** a zone (`hc zone update dhcp --name lan`, or Edit on the
+Zones page) changes every rule and NAT rule that names it, in the same
+change: the history shows them together, and undoing it puts them all
+back. Sets and services rename the same way, except dynamic sets: their
+entries, and the API tokens and programs that fill them, go by their
+name.
+
 ## Address sets
 
 A set is a named list of IPv4 and IPv6 addresses, prefixes and ranges,
@@ -38,7 +64,10 @@ directly (`--from 192.168.1.3`); a set is better when several rules share
 them.
 
 A **dynamic** set's entries are added at run time, each with a timeout,
-and nftables removes them when it runs out, even with housecarld stopped:
+and nftables removes them when it runs out, even with housecarld stopped.
+It has no configured entries (validation refuses them); its timeout is
+the default for entries added without one, and without a timeout they
+stay until removed:
 
 ```sh
 hc set add guests --dynamic --timeout 24h
@@ -116,6 +145,10 @@ The action is `accept`, `drop`, `reject` (an ICMP error, or with
 `--reject-with tcp-reset` a TCP reset) or `log` (log and go on).
 `--log` logs the packets of any rule ([operations.md](operations.md#the-log)).
 `--disabled` (or `hc rule disable ID`) keeps a rule without using it.
+`--every-packet` ("every packet" in the web UI) checks a drop or reject
+against every packet, before established connections are accepted, so it
+also stops connections already open, not just new ones; it can't accept
+([operations.md](operations.md#connections)).
 
 Examples:
 
@@ -155,6 +188,10 @@ hc quotas                      # what each used, and the next reset
 hc rule reset-quota 31         # start again now
 ```
 
+In the web UI each quota's use is on its rule (with **Reset**), all of
+them together on the dashboard, and their history under **Traffic →
+Statistics**.
+
 | | |
 |---|---|
 | `per-source`, `per-destination` | each address its own allowance; without, one allowance for everything the rule matches |
@@ -176,7 +213,8 @@ first, marked "every packet"). So they can drop, reject or log, but not
 accept: an accept there would let packets past every later rule. Let the
 traffic in with a rule of its own. The flowtable's fast path skips the
 forward chain for established connections, so with it on, forward
-quotas and byte rates only count first packets (a warning says so).
+quotas and byte rates only count first packets (a warning says so;
+[flowtable.md](flowtable.md)).
 
 **Per-address limits group addresses**: each IPv4 address on its own, and
 IPv6 addresses by /64, since one device uses several (privacy addresses
@@ -193,11 +231,19 @@ hand, placed like any other:
 hc rule add forward --raw 'iifname != "wan" ct state invalid ip daddr @trusted_v4 accept'
 ```
 
-It is checked by nftables with the rest of the ruleset, and can use the
-sets' nftables names (`@trusted_v4`). The importer makes raw rules from
+It is used exactly as written: its text replaces the rule's zones,
+addresses, services, action, limits and log, so a raw rule with any of
+those set is refused (`raw_conflict`); write them in the text. It is
+checked by nftables with the rest of the ruleset, and can use the sets'
+nftables names (`@trusted_v4`). The importer makes raw rules from
 what it can't translate. It goes into the ruleset as written: add
 `counter` to it to count its packets.
 
+In shadow mode ([shadow.md](shadow.md)) a raw rule's final `drop` or
+`reject` becomes `accept`; a raw rule that does anything else to packets
+(NAT, `meta mark set`, `jump`, `notrack`, a verdict map…) is left out,
+with a warning (`shadow_raw_left_out`).
+
 ## Built-ins
 
 Housecarl adds the rules every firewall needs before yours. Each can be
@@ -222,8 +268,87 @@ them. The antispoof lists can be changed with `--antispoof-v4` and
 
 The **flowtable** (`--flowtable wan --flowtable eth1`) lets established
 TCP and UDP connections forwarded between those devices skip the rules,
-which helps on a slow CPU. It needs exact device names. Counters of
-forward rules then only count each connection's first packets.
+which helps when the CPU can't keep up with the line. It needs exact
+device names, and costs quotas, byte rates and traffic per device in the
+forward chain their accuracy: see [flowtable.md](flowtable.md) for when
+to use it, and when not.
+
+## The zone matrix
+
+The Matrix page (`hc matrix`) sums up what the rules do to **new
+connections** between zones: a row per zone the traffic comes from, a
+column per zone it goes to, and the firewall itself (to it: the input
+chain; from it: output; zone to zone: forward).
+
+```
+$ hc matrix
+FROM \ TO  guest         lan           wan         firewall
+guest      block *       block         partly +1   partly +4
+lan        block         block *       allow       partly +4
+wan        partly +1 -1  partly +1 -1  block -1 *  partly +3 -1
+firewall   allow         allow         allow       -
+
+Let through for part of the traffic:
+  guest → wan       http, https
+  guest → firewall  ICMPv6; ping; ssh from admins; dns
+  lan → firewall    ICMPv6; ping; ssh from admins; dns
+  wan → guest       1 port forward
+  wan → lan         1 port forward
+  wan → firewall    ICMPv6; DHCPv6; ssh from admins
+
+allow, block: every new connection; partly: some are let through, some not.
++N: rules that let part of the traffic through; -N: rules that block part of it. 'hc matrix FROM TO' lists them.
+*: a zone to itself: only traffic routed between its interfaces passes the firewall.
+```
+
+Each cell walks that pair's rules in order, as nftables would, with the
+built-ins that act on new connections (ping from internal zones,
+ICMPv6, the DHCPv6 client, antispoofing, port forwards):
+
+* a rule that applies to the whole pair decides it;
+* a rule that applies to part of its traffic (some addresses, services,
+  times, rates, a quota) is an **exception**, and the walk goes on;
+* what's left falls to the deciding rule or the chain's policy.
+
+A cell reads **allow** or **block** when every new connection gets the
+same, and **partly** when exceptions let some through while the rest is
+blocked (or block some while the rest is allowed). **+N** counts the
+exceptions that let traffic through, **-N** those that block it. Under
+the grid (in the web UI, in each cell) is what the exceptions let
+through: their services, with the addresses they are limited to.
+
+A zone to itself (`*`, faded in the web UI) is mostly traffic that never
+reaches the firewall: within one interface's network it goes straight
+from host to host. Only what is routed between the zone's interfaces (or
+back out of the one it came in on) passes.
+
+`hc matrix guest wan` (or clicking the cell) lists the exceptions in
+order, with each rule's hits:
+
+```
+guest → wan (the forward chain), new connections: partly allowed
+  VERDICT  WHAT         BY                HITS
+  allowed  http, https  rule forward #14  1520
+  blocked  the rest     forward policy    37
+```
+
+The hits are the rule's packets since housecarld started, in all of its
+traffic: a rule without zones counts for every pair it covers. They are
+how you tell an exception that is used from one that isn't.
+
+A raw rule is read when it is a match followed by `accept`, `drop` or
+`reject` (with `counter` and `log` if you like): it counts as an exception
+for every pair of its chain, shown as `raw: …`. One that matches
+interfaces (the matrix can't tell which zones), jumps, translates, sets
+something or uses a verdict map shows as **unknown**: check it yourself.
+Raw rules have hits only if they say `counter`.
+
+A rule whose addresses all lie in networks directly connected to other
+zones is left out of a pair: traffic between those zones can't have them
+(`from wg_net` doesn't concern traffic coming in on the LAN). Routed
+networks, mixed lists and dynamic sets might, so they stay. Replies and
+established connections aren't shown: the built-in lets them through. In
+review mode, the matrix shows your pending changes.
 
 ## The ruleset
 

docs/shadow.md

@@ -0,0 +1,171 @@
+# Shadow mode
+
+In shadow mode Housecarl runs beside the firewall you have now and
+changes nothing. Its ruleset is loaded, so its counters, log, tracer and
+statistics show what it *would* do with your real traffic, but it blocks,
+rejects and translates nothing. Your current firewall keeps deciding.
+
+Use it to:
+
+* **try Housecarl** on the real machine before you trust it with the
+  firewall: the web UI, the API, `hc`, changesets, history;
+* **check an import**: import your ruleset ([import.md](import.md)) and
+  watch whether Housecarl would drop anything your firewall lets through;
+* **develop against the API**: Steward, scripts or fail2ban can fill
+  dynamic sets and see them used, with nothing at stake.
+
+## Turning it on
+
+Shadow mode is set in `/etc/housecarl/housecarl.yaml`
+([configuration.md](configuration.md)), not through the API, so no user
+or token can switch enforcement off:
+
+```yaml
+shadow:
+  enabled: true
+  position: after     # or before; see below
+```
+
+housecarld reads the file when it starts:
+
+```sh
+sudo systemctl restart housecarl
+sudo hc status        # Mode: SHADOW …
+```
+
+The web UI shows a banner on every page, `hc status` a `Mode:` line, and
+`/healthz` `"mode": "shadow"`.
+
+## Why it can't let anything through
+
+In nftables, `accept` only ends the chain it is in. The packet then
+goes on to the next base chain on the same hook, and other tables' chains
+are among those. Only a drop is final. Housecarl's table in shadow mode
+never drops, so it can't let through anything your firewall drops, and it
+can't block anything your firewall accepts.
+
+## What changes in the ruleset
+
+The same configuration is rendered, with everything that acts on packets
+turned into counting:
+
+| | Enforcing | Shadow mode |
+|---|---|---|
+| Rules that drop or reject | `drop`, `reject` | counted, then `accept` (Housecarl's chain ends as it would have). Their comment says `[would drop]` or `[would reject]` |
+| The built-ins that drop (invalid packets, antispoofing) | `drop` | counted, `[would drop]` |
+| Policies | as configured | `accept`. What a drop policy would drop is counted and **always logged**, rate-limited, in all three chains |
+| Port forwards, source NAT, redirects | translate | counted only (`[would dnat]`, `[would masquerade]`, `[would snat]`, `[would redirect]`) |
+| MSS clamping | changes SYN packets | left out |
+| The flowtable | offloads connections | left out: offloaded connections skip *every* table's forward chain, your firewall's too |
+| Raw rules | as written | as written if they only match, count, log or accept. A final `drop` or `reject` becomes `accept`. Anything else that acts on packets (NAT, `set`, `jump`, `notrack`, a verdict map…) is left out, with a warning (`shadow_raw_left_out`) |
+| Everything else | | the same: accepts, rate limits, connection limits, quotas, dynamic sets, per-device counters, logging |
+
+`hc ruleset` prints the shadow ruleset, the one loaded now.
+`hc ruleset --enforcing` prints the ruleset that will be loaded once
+shadow mode is off.
+
+**Every change is checked by nftables twice**: the shadow ruleset, and the
+enforcing one. A change that nftables would refuse once shadow mode is off
+is refused now, so leaving shadow mode can't fail on that.
+
+## Where Housecarl's chains run
+
+Your firewall's chains are usually at priority `filter` (0), its raw
+chains at `raw`. `shadow.position` puts Housecarl's filter chains just
+next to them:
+
+* **`after`** (the default) runs them at `filter + 10` (and `raw + 10`).
+  Housecarl only sees what your firewall let through. So **every would-drop
+  is traffic that would break if you switched over**: the list to work
+  through before you do. What your firewall drops and Housecarl would
+  accept (a hole) doesn't show; compare the rules for that.
+* **`before`** runs them at `filter - 10` (and `raw - 10`). Housecarl sees
+  every packet, as it would when enforcing, and its hits and drops are the
+  numbers you would get. Its would-drops include everything your firewall
+  drops too (the internet's background noise).
+
+If your firewall uses other priorities, say `filter + 20`, "after" is
+relative to `filter`, not to it: check with `nft list ruleset`.
+
+NAT chains are different: the kernel asks NAT chains in turn only until
+one translates. Housecarl's NAT chains always run first (`dstnat - 10`,
+`srcnat - 10`), whatever the position, or they would never see a
+connection your firewall translates. They translate nothing, so your
+firewall's NAT still does the work.
+
+## What you see
+
+* **Hits** (`hc counters`, the Rules page): a drop rule's hits are what it
+  would have dropped. NAT counters count connections (only a connection's
+  first packet reaches the NAT chains).
+* **The log** (`hc log`, the Log page): every packet a drop policy would
+  have dropped, and what rules with logging would have dropped, marked
+  *would drop*. `hc log` adds "(shadow mode: would drop, let through)".
+* **The tracer** (`hc trace`, the Trace page): the verdict is your
+  firewall's, the real one. When Housecarl would have dropped the packet,
+  the trace says by which rule ("shadow mode: Housecarl would have
+  dropped it, by …").
+* **Statistics** ([stats.md](stats.md)): drops, drops per chain and the
+  blocked sources are what Housecarl would have dropped.
+
+## What works differently
+
+* **No probation.** A change can't cut anyone off, so changes from the
+  network are applied and saved at once, with no confirmation. Changesets
+  work as usual; their ruleset diff is of the shadow ruleset.
+* **No takeover, no closing connections.** `hc takeover` (deleting your
+  firewall's tables) and `hc connections close` are refused (409,
+  `hc` exits with 5): both would change your traffic.
+* **The boot.** `housecarl-ruleset.service` loads the shadow ruleset at
+  boot. Its fallback ([operations.md](operations.md#if-the-ruleset-fails-to-load-at-boot))
+  is an empty table that filters nothing, since your firewall protects the
+  machine.
+* **Your firewall reloading.** A ruleset file such as `/etc/nftables.conf`
+  usually starts with `flush ruleset`, and Debian's `nftables.service`
+  runs `nft flush ruleset` when it stops or restarts. Both delete
+  Housecarl's table too. housecarld watches nftables' notifications and
+  puts its table back within a few seconds (the journal says "Housecarl's
+  table was deleted … put it back"). Dynamic sets lose their entries, and
+  quotas go back to what was last saved (every five minutes).
+* **Warnings.** `nftables_service` and `takeover_pending` aren't raised:
+  another firewall beside Housecarl is the point. Instead:
+  `shadow_unprotected` when no other table has a filter chain on the input
+  hook (nothing protects the machine), and `shadow_raw_left_out` for each
+  raw rule left out ([validation.md](validation.md#the-machine)).
+* **Quotas count** what passes in shadow mode, and keep counting when
+  you switch over.
+
+## A trial, step by step
+
+From a machine whose firewall is `/etc/nftables.conf` with `table inet main`:
+
+1. Install Housecarl and create the administrator
+   ([install.md](install.md)).
+2. Turn shadow mode on in `/etc/housecarl/housecarl.yaml`, and start
+   housecarld: `sudo systemctl enable --now housecarl`.
+3. Import your ruleset: `sudo hc import --defines /etc/nftables.conf`
+   ([import.md](import.md)). Leave `nftables.service` as it is.
+4. Use your network for a while, and watch `hc log -f`, `hc counters` and
+   the dashboard. With `position: after`, each would-drop is something
+   your firewall allows and Housecarl wouldn't: fix the rule in Housecarl,
+   or decide it doesn't matter. `hc status` lists the raw rules left out.
+5. When nothing you need would be dropped, leave shadow mode.
+
+## Leaving shadow mode
+
+1. Look at what will be loaded: `sudo hc ruleset --enforcing`.
+2. Set `shadow.enabled: false` (or remove the section), and
+   `sudo systemctl restart housecarl`. Housecarl's ruleset is now in
+   force **alongside** your firewall: a packet must be accepted by both.
+   The restart applies it at once, not on probation, but your old firewall
+   is still there and still decides too.
+3. Take over: `sudo hc takeover "inet main"` deletes the old table (on
+   probation over the network), and `sudo systemctl disable nftables`
+   (without `--now`) retires it ([import.md](import.md#3-take-over)).
+
+## Going back
+
+Set `shadow.enabled: true` and restart housecarld. If Housecarl was your
+only firewall, **the machine is now unprotected**: load your other
+firewall first. Until one filters the input hook, `hc status` and
+`/healthz` warn (`shadow_unprotected`).

docs/stats.md

@@ -0,0 +1,144 @@
+# Statistics
+
+Housecarl keeps what the firewall did over time:
+
+- traffic per interface and zone, received and sent;
+- packets dropped, by chain, and when drops happen (a weekday × hour
+  heatmap);
+- hits per rule, built-in and policy, and the rules that matched nothing;
+- the sources blocked most and what they tried (from the log);
+- traffic per inside device (a setting, below);
+- connections tracked, against the kernel's limit;
+- what each quota counted;
+- housecarld's and the machine's memory, CPU and load.
+
+All of it is in the web UI (**Traffic → Statistics**, and two charts on
+the dashboard), the CLI (`hc stats`) and the API (`/api/v1/stats/...`),
+for anyone with `log:read`.
+
+Statistics are **on by default**. They cost nothing on the packets'
+path: housecarld reads counters the kernel keeps anyway, once a minute.
+Only traffic per device adds to the ruleset, so it is a setting you turn
+on. The figures are kept as aggregates in a database of their own,
+`/var/lib/housecarl/stats.db`, separate from the configuration. Deleting
+it is always safe (stop housecarl first), and backups leave it out.
+
+## Settings
+
+In `/etc/housecarl/housecarl.yaml` (restart housecarl after a change):
+
+```yaml
+stats:
+  enabled: true
+  database: /var/lib/housecarl/stats.db
+  retention:
+    minute: 7d     # per-minute figures
+    hour: 90d      # hourly figures and top lists
+    day: 730d      # daily figures and top lists
+  top_n: 100       # entries kept per top list, per hour and day
+```
+
+Old data is removed once an hour. A range picks the finest data that
+covers it: per minute up to 2 days, hourly up to 90 days, then daily.
+
+## Traffic per device
+
+```sh
+hc settings set --stats-per-device on
+```
+
+(or **Settings → Statistics** in the web UI). This is a firewall setting,
+so it goes through the usual review and probation. Housecarl then adds
+four sets to its table, and four rules at the top of the forward chain
+(shown as the built-in `stats`):
+
+```
+iifname != "wan" oifname "wan" update @stats_up_v4 { ip saddr }
+iifname != "wan" oifname "wan" update @stats_up_v6 { ip6 saddr and ffff:ffff:ffff:ffff:: }
+iifname "wan" oifname != "wan" update @stats_down_v4 { ip daddr }
+iifname "wan" oifname != "wan" update @stats_down_v6 { ip6 daddr and ffff:ffff:ffff:ffff:: }
+```
+
+Each inside address talking to an external zone gets a counter: what it
+sent out, and what came back to it. IPv6 addresses are counted per /64,
+since one device uses many addresses. Counting is done in the kernel, a
+set lookup per forwarded packet. An address quiet for an hour drops out
+of the sets, and a reload keeps the counts.
+
+With the flowtable on, established connections skip the forward chain,
+so only their first packets are counted (warning `stats_flowtable`). Set
+names starting with `stats_` (and `meter_`, `conn_`, `quota_`) are
+Housecarl's own and can't be used for your sets.
+
+## What is collected
+
+Every minute housecarld samples its sources and stores what changed:
+
+| Figure | Source |
+|---|---|
+| `rx:IF`, `tx:IF` (bytes), `rxp:IF`, `txp:IF` (packets) | the interfaces' counters (`/sys/class/net`); the loopback is left out |
+| `zrx:ZONE`, `ztx:ZONE` | the same, summed per zone: received from and sent to the zone's interfaces |
+| `hits:MARK`, `hitb:MARK` | each rule's (`r12`), built-in's (`b-established`), port forward's (`f3`) and policy's (`p-input`) counter, in packets and bytes |
+| `drop:CHAIN`, `drops` | the packets counted by what drops them: rules that drop or reject, the drop policies, antispoofing (`early`) and invalid packets (`invalid`) |
+| `quota:r12` | bytes counted against a rule's quota |
+| `g:conntrack`, `g:conntrack_max` | connections tracked, and the limit (`nf_conntrack_max`) |
+| `g:rss`, `g:cpu` | housecarld's memory (bytes) and CPU (tenths of a percent of one CPU) |
+| `g:mem`, `g:load` | the host's memory in use, and its 1-minute load average × 100 |
+
+`g:` figures are gauges: the API averages them over a bucket.
+
+Top lists, per hour and per day, the `top_n` largest:
+
+| List | |
+|---|---|
+| `blocked_src` | sources of logged drops |
+| `blocked_port` | what they tried: `tcp/22`, `udp/53`, `icmp/echo-request` |
+| `device_up`, `device_down` | bytes per inside address, sent and received (traffic per device) |
+| `quota_r12` | a per-address quota's use, per address |
+
+The blocked lists only see what is **logged**: the input and forward
+policies' drops with `hc settings set --log-drops on`, and rules with
+logging. The drop counts above see everything.
+
+In shadow mode ([shadow.md](shadow.md)) the drops, and the blocked
+lists, are what Housecarl would have dropped: the packets went on. What
+the policies would drop is always logged there.
+
+A counter that goes down (an interface that came back, a quota reset)
+counts again from zero. What happens while housecarld is stopped isn't
+counted.
+
+## The CLI
+
+```sh
+hc stats --since 7d              # summary: traffic per zone, drops, busiest rules, top lists
+hc stats status                  # collecting? since when, the database, figures now
+hc stats traffic --since 24h     # per zone over time (--interfaces: per interface)
+hc stats top devices --since 30d # devices by traffic (blocked, tried, quota RULE-ID)
+hc stats rules --unused          # rules that matched nothing in 30 days
+hc stats connections --since 24h
+```
+
+Ranges are written `90m`, `24h`, `7d` or `4w`. `-o json` gives the API's
+answer.
+
+## The API
+
+| Endpoint | |
+|---|---|
+| `GET /stats/status` | `enabled`, `database`, `size`, `first_data`, `last_sample`, `sample_error`, `per_device`, `retention`, `latest` (the newest gauges) |
+| `GET /stats/summary?since=24h&limit=10` | `zones` and `interfaces` (`name`, `received`, `sent`), `drops`, `drops_by_chain`, `rules` (busiest, named), `peak_connections`, `conntrack_max`, `blocked_sources`, `blocked_ports`, `devices_up`, `devices_down` |
+| `GET /stats/timeline?keys=zrx:*,drops&since=24h&step=auto` | `step`, `times` (bucket starts), `series` (`key`, `values`); `GROUP:*` means every key of a group; `step` is `auto`, `1m`, `5m`, `15m`, `1h`, `6h` or `1d`; a gauge without a sample is -1 |
+| `GET /stats/top?kind=device_down&since=7d&limit=20` | a top list (`key`, `count`) |
+| `GET /stats/hits?since=30d` | packets per rule, built-in and policy (`key`, `count`, `name`), with the enabled rules that counted none |
+| `GET /stats/heatmap?key=drops&since=28d` | `avg[weekday][hour]` (Monday first, local time; -1 where there is no data) |
+
+With statistics off, `/stats/status` says `enabled: false` and the others
+answer 503.
+
+## Warnings
+
+When the connection tracking table is over 80% full, the status shows
+the warning `conntrack_full`: once it is full, the kernel drops new
+connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens
+so many on the Connections page.

docs/validation.md

@@ -34,6 +34,8 @@ are unique per kind, ignoring case.
 | `flowtable_device` | error | a flowtable device with a wildcard: name each device |
 | `flowtable_device_missing` | warning, *host* | a flowtable device doesn't exist; the flowtable is left out until it does |
 | `probation_range` | error | the probation isn't 10 to 3600 seconds |
+| `stats_no_external` | warning | traffic per device is on, but no zone is external: it counts what goes to and comes from external zones |
+| `stats_flowtable` | warning | traffic per device with the flowtable on: its fast path skips the forward chain, so only each connection's first packets are counted |
 
 ## Zones
 
@@ -43,14 +45,16 @@ are unique per kind, ignoring case.
 | `zone_interface` | error | not an interface name (up to 15 characters; a trailing `*` matches many) |
 | `zone_interface_taken` | error | the interface is in another zone already |
 | `zone_interface_missing` | warning, *host* | the interface doesn't exist here (yet); its rules match once it does |
-| `no_external_zone` | warning | no zone is external: antispoofing, MSS clamping, the DHCPv6 client rule and port forwards without public addresses need one |
+| `no_external_zone` | warning | no zone is external: antispoofing, MSS clamping, the DHCPv6 client rule and port forwards without public addresses need one, and the boot fallback lets nothing but loopback through without one ([operations.md](operations.md#if-the-ruleset-fails-to-load-at-boot)) |
 
 ## Sets and services
 
 | Code | | When |
 |---|---|---|
+| `set_name_reserved` | error | a set name starting with `meter_`, `conn_`, `quota_` or `stats_`: Housecarl's own sets (rate limits, quotas, statistics) use those |
 | `set_entry` | error | an entry that isn't an address, prefix (without host bits) or range |
 | `set_empty` | warning | a set (not dynamic) with no entries: rules using it match nothing |
+| `set_dynamic_entries` | error | configured entries on a dynamic set: its entries come at run time |
 | `set_timeout` | error | a timeout on a set that isn't dynamic, or under a second |
 | `service_empty` | error | a service with no entries |
 | `service_entry` | error | an entry that isn't `PROTO/PORTS` (`tcp`, `udp`, `sctp`; a port or range) or `icmp/TYPE`, `icmpv6/TYPE` with a type nftables knows |
@@ -77,6 +81,7 @@ These apply to rules, source NAT and redirects.
 | `rule_chain` | error | a chain that isn't `input`, `forward`, `output` or `early` |
 | `rule_action` | error | an action that isn't `accept`, `drop`, `reject` or `log` |
 | `raw_rule` | error | a raw rule over several lines, or with `;` |
+| `raw_conflict` | error | a raw rule with other fields set (zones, addresses, services, limits, log…): it is used as written, so they would be ignored |
 | `early_conntrack` | error | connection states, limits or a connection limit in the early chain, which runs before connection tracking |
 | `early_reject` | error | reject in the early chain: drop instead |
 | `ct_state`, `ct_status` | error | an unknown connection state or status |
@@ -91,7 +96,8 @@ These apply to rules, source NAT and redirects.
 | `rule_quota_reset` | error | a reset that isn't daily, weekly or monthly; a day with another reset than monthly; a day outside 1-31 |
 | `rule_group` | error | a grouping prefix outside 1-32 (IPv4) or 1-128 (IPv6) |
 | `rule_traffic_accept` | error | a rule with a quota or a byte rate that accepts: it is checked before established connections are accepted, so it can only drop, reject or log |
-| `flowtable_traffic` | warning | a forward rule with a quota or a byte rate while the flowtable is on: it only counts each connection's first packets |
+| `rule_every_packet_accept` | error | a rule checked against every packet (`every_packet`) that accepts: it comes before established connections are accepted, so it can only drop, reject or log |
+| `flowtable_traffic` | warning | a forward rule checked against every packet (a quota, a byte rate, `every_packet`) while the flowtable is on: it only sees each connection's first packets |
 | `reject_with` | error | `tcp-reset` on a rule that matches more than TCP, or with an action other than reject |
 | `log_prefix` | error | a log prefix over 100 characters |
 
@@ -115,3 +121,19 @@ These apply to rules, source NAT and redirects.
 | `snat_everything` | warning | no out zones and no sources: everything leaving the firewall is translated |
 | `redirect_service` | error | no service, or one without TCP, UDP or SCTP ports |
 | `redirect_to_port` | error | the local port isn't one port |
+
+## The machine
+
+These come from housecarld looking at the machine, not at the
+configuration: the dashboard and `hc status` show them.
+
+| Code | | When |
+|---|---|---|
+| `nftables_service` | warning, *host* | Debian's `nftables.service` is enabled: at boot it loads `/etc/nftables.conf`, and stopping or restarting it runs `nft flush ruleset`, deleting Housecarl's table too. Disable it with `systemctl disable nftables` (without `--now`) |
+| `conntrack_full` | warning, *host* | the connection tracking table is over 80% full; once it is full, the kernel drops new connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens so many ([stats.md](stats.md#warnings)) |
+| `takeover_pending` | warning, *host* | the table you imported from is still in force beside Housecarl's, so a packet must get past both; `hc takeover TABLE` deletes it once Housecarl's rules are right |
+| `shadow_unprotected` | warning, *host* | shadow mode, and no other table has a filter chain on the input hook: Housecarl's ruleset blocks nothing, and nothing else protects the machine. Load your firewall, or leave shadow mode ([shadow.md](shadow.md)) |
+| `shadow_raw_left_out` | warning, *host* | shadow mode leaves this raw rule out: it does more to packets than accept, drop or reject (NAT, setting a mark or field, a jump, a verdict map…). Its hits aren't counted, and the rules after it see what it would have taken ([shadow.md](shadow.md#what-changes-in-the-ruleset)) |
+
+In shadow mode `nftables_service` and `takeover_pending` aren't raised:
+another firewall beside Housecarl is the point there.

internal/api/types.go

@@ -40,8 +40,18 @@ type Whoami struct {
 	TokenID   int64         `json:"token_id,omitempty"`
 	CSRFToken string        `json:"csrf_token,omitempty"`
 	Build     version.Build `json:"build"`
+	// Mode is enforcing, or shadow (docs/shadow.md).
+	Mode string `json:"mode"`
 }
 
+// Modes: housecarld's ruleset decides, or (shadow mode, the
+// configuration file's shadow.enabled) it only shows what it would do,
+// beside another firewall.
+const (
+	ModeEnforcing = "enforcing"
+	ModeShadow    = "shadow"
+)
+
 // Login is the login request.
 type Login struct {
 	Username string `json:"username"`
@@ -66,6 +76,11 @@ type MutationResponse struct {
 	// ConfirmBy is set when the change is on probation: unless confirmed
 	// (POST /confirm) by then, it is rolled back.
 	ConfirmBy *time.Time `json:"confirm_by,omitempty"`
+	// ConfirmToken lets the client that made the change confirm it by
+	// itself (POST /confirm {"token"}): from the same address, over a new
+	// connection, which proves it can still reach the API. The answer
+	// closes its connection, so the next request opens a new one.
+	ConfirmToken string `json:"confirm_token,omitempty"`
 	// Diff is the ruleset's unified diff, for a dry run.
 	Diff string `json:"diff,omitempty"`
 	// Staged: the change was added to a changeset (Changeset), not
@@ -150,6 +165,9 @@ type LogEntry struct {
 	TCPFlags string    `json:"tcp_flags,omitempty"`
 	RuleMark string    `json:"mark,omitempty"`
 	Verdict  string    `json:"verdict,omitempty"`
+	// Shadow: logged in shadow mode, where a drop or reject is what would
+	// have been done; the packet went on.
+	Shadow bool `json:"shadow,omitempty"`
 }
 
 // Status is the dashboard's overview.
@@ -157,6 +175,11 @@ type Status struct {
 	Build    version.Build `json:"build"`
 	Started  time.Time     `json:"started"`
 	Database string        `json:"database,omitempty"`
+	// Mode is enforcing or shadow; ShadowPosition (shadow mode) after or
+	// before: where Housecarl's filter chains run beside the other
+	// firewall's.
+	Mode           string `json:"mode"`
+	ShadowPosition string `json:"shadow_position,omitempty"`
 	Applied        bool   `json:"applied"`
 	ApplyError     string `json:"apply_error,omitempty"`
 	// Unconfigured: the configuration is empty, and no ruleset was
@@ -193,6 +216,9 @@ type ImportInput struct {
 	// Replace replaces Housecarl's configuration; without it, the import
 	// is refused when there is one.
 	Replace bool `json:"replace,omitempty"`
+	// Zones names the zones by interface ({"enp2s0": "lan"}); interfaces
+	// given the same name share a zone.
+	Zones map[string]string `json:"zones,omitempty"`
 }
 
 // ImportResult reports an import.
@@ -318,3 +344,168 @@ type QuotaAddress struct {
 	Address string `json:"address"`
 	Used    int64  `json:"used"`
 }
+
+// ConfirmInput confirms the changes on probation. With Token (a change's
+// confirm_token), it is the client that made the change confirming it by
+// itself; without, a person's confirmation.
+type ConfirmInput struct {
+	Token string `json:"token,omitempty"`
+}
+
+// Connection is a connection the kernel tracks (conntrack), as it was
+// first seen: Source:SourcePort → Destination:DestinationPort. NAT shows
+// as DNATTo (where it was really sent: a port forward) and SNATTo (the
+// source it went out with: masquerading).
+type Connection struct {
+	ID              uint32 `json:"id,omitempty"`
+	Family          string `json:"family"`
+	Protocol        string `json:"protocol"`
+	Source          string `json:"source"`
+	SourcePort      int    `json:"source_port,omitempty"`
+	Destination     string `json:"destination"`
+	DestinationPort int    `json:"destination_port,omitempty"`
+	ICMPID          int    `json:"icmp_id,omitempty"`
+	ICMPType        int    `json:"icmp_type,omitempty"`
+	ICMPCode        int    `json:"icmp_code,omitempty"`
+	DNATTo          string `json:"dnat_to,omitempty"`
+	DNATPort        int    `json:"dnat_port,omitempty"`
+	SNATTo          string `json:"snat_to,omitempty"`
+	// State is TCP's (established, time-wait, …); Assured: traffic went
+	// both ways.
+	State   string `json:"state,omitempty"`
+	Assured bool   `json:"assured,omitempty"`
+	// Timeout is the seconds left until it is forgotten unless more
+	// traffic comes.
+	Timeout int    `json:"timeout"`
+	Mark    uint32 `json:"mark,omitempty"`
+	// Packets and Bytes, both ways, when the kernel counts them
+	// (net.netfilter.nf_conntrack_acct).
+	Packets uint64 `json:"packets,omitempty"`
+	Bytes   uint64 `json:"bytes,omitempty"`
+}
+
+// CloseConnections closes connections: these, or every one Search finds
+// (as GET /connections?q= does).
+type CloseConnections struct {
+	Connections []Connection `json:"connections,omitempty"`
+	Search      string       `json:"search,omitempty"`
+}
+
+// TraceInput says what to trace: packets from or to Address (an address
+// or prefix), on Port (either way), of Protocol, for Seconds (0: 60).
+type TraceInput struct {
+	Address  string `json:"address,omitempty"`
+	Port     int    `json:"port,omitempty"`
+	Protocol string `json:"protocol,omitempty"`
+	Seconds  int    `json:"seconds,omitempty"`
+}
+
+// TraceStep is one rule or policy a traced packet met, in any table.
+type TraceStep struct {
+	Table string `json:"table"` // "inet housecarl", "ip nat"…
+	Chain string `json:"chain"`
+	Kind  string `json:"kind"` // rule or policy
+	// Rule is the rule as nftables writes it; Mark its Housecarl mark
+	// ("r12", "b-established"), if it is one of Housecarl's.
+	Rule    string `json:"rule,omitempty"`
+	Mark    string `json:"mark,omitempty"`
+	Verdict string `json:"verdict"` // accept, drop, continue, jump X…
+	// Label names Housecarl's object ("rule input #12 (ssh)").
+	Label string `json:"label,omitempty"`
+}
+
+// TracePacket is one traced packet: its headers as it arrived, every
+// step, and its fate.
+type TracePacket struct {
+	ID              string      `json:"id"`
+	Time            time.Time   `json:"time"`
+	In              string      `json:"in,omitempty"`
+	Out             string      `json:"out,omitempty"`
+	Protocol        string      `json:"protocol,omitempty"`
+	Source          string      `json:"source,omitempty"`
+	SourcePort      int         `json:"source_port,omitempty"`
+	Destination     string      `json:"destination,omitempty"`
+	DestinationPort int         `json:"destination_port,omitempty"`
+	TCPFlags        string      `json:"tcp_flags,omitempty"`
+	ICMPType        string      `json:"icmp_type,omitempty"`
+	Steps           []TraceStep `json:"steps"`
+	// Verdict is accepted or dropped; DecidedBy the step that decided.
+	Verdict   string    `json:"verdict"`
+	DecidedBy TraceStep `json:"decided_by"`
+	// WouldDrop, in shadow mode, is the step of Housecarl's where it
+	// would have dropped (or rejected) the packet, had it been enforcing.
+	WouldDrop *TraceStep `json:"would_drop,omitempty"`
+}
+
+// TraceStatus is the trace running (or the last one).
+type TraceStatus struct {
+	Running bool          `json:"running"`
+	Input   TraceInput    `json:"input"`
+	Started time.Time     `json:"started"`
+	Ends    time.Time     `json:"ends"`
+	Packets []TracePacket `json:"packets"`
+	Error   string        `json:"error,omitempty"`
+}
+
+// Matrix is what the rules do to new connections between zones:
+// zone → zone (the forward chain), zone → the firewall (input) and the
+// firewall → zone (output). "firewall" stands for the machine itself.
+type Matrix struct {
+	Zones []string     `json:"zones"`
+	Cells []MatrixCell `json:"cells"`
+}
+
+// MatrixCell is one pair: what happens to new connections from From to
+// To that no exception covers (Verdict, decided by DecidedBy), and the
+// exceptions: rules and built-ins that decide for some of them (some
+// addresses, services, times…), in the order they are checked.
+type MatrixCell struct {
+	From       string            `json:"from"`
+	To         string            `json:"to"`
+	Chain      string            `json:"chain"`
+	Verdict    string            `json:"verdict"` // allowed or blocked
+	DecidedBy  string            `json:"decided_by"`
+	RuleID     int64             `json:"rule_id,omitempty"` // the deciding rule
+	Exceptions []MatrixException `json:"exceptions"`
+	// Summary is the pair at a glance: allowed or blocked when every new
+	// connection gets the same, partly when some are let through and
+	// some not.
+	Summary string `json:"summary"`
+	// Allowed and Blocked count the exceptions that let part of the
+	// traffic through, and that block part of it.
+	Allowed int `json:"allowed"`
+	Blocked int `json:"blocked"`
+	// Open sums up what the allowing exceptions let through, shortest
+	// first in their order: "ssh", "dns", "5 port forwards".
+	Open []string `json:"open"`
+	// SameZone: from a zone to itself. Note says what of that traffic
+	// passes the firewall at all.
+	SameZone bool   `json:"same_zone,omitempty"`
+	Note     string `json:"note,omitempty"`
+	// Hits are the deciding rule's (or policy's) packets, all of its
+	// traffic since housecarld started (nil: no counter).
+	Hits *uint64 `json:"hits,omitempty"`
+	// Marks are the counters behind Hits.
+	Marks []string `json:"-"`
+}
+
+// MatrixException is a rule (or built-in) that decides for part of a
+// pair's traffic.
+type MatrixException struct {
+	RuleID int64  `json:"rule_id,omitempty"` // 0: a built-in, or port forwards
+	Chain  string `json:"chain"`
+	Label  string `json:"label"`
+	// Verdict is allowed, blocked, logged (and goes on), or unknown (a raw
+	// rule Housecarl can't read).
+	Verdict string `json:"verdict"`
+	// What it covers: "from trusted · ssh", "Monday 08:00–17:00".
+	What string `json:"what"`
+	// Short is What in a word or two, for the overview: its services,
+	// or else its first condition.
+	Short string `json:"short"`
+	// Hits are its packets, all of its traffic since housecarld started
+	// (a rule may apply to several pairs; nil: no counter).
+	Hits *uint64 `json:"hits,omitempty"`
+	// Marks are the counters behind Hits ("r12", "b-ping", "f3").
+	Marks []string `json:"-"`
+}

internal/client/client.go

@@ -39,9 +39,9 @@ type Client struct {
 }
 
 // Staged reports whether a request is a configuration change that a
-// changeset takes: objects and settings, and the changes that can't be
-// staged (undo, import, takeover), which the server then refuses rather
-// than apply them. Changesets' own calls and dynamic set entries aren't.
+// changeset takes: objects, settings and undo, and the changes that can't
+// be staged (import, takeover), which the server then refuses rather than
+// apply them. Changesets' own calls and dynamic set entries aren't.
 func Staged(method, path string) bool {
 	if method == http.MethodGet || strings.Contains(path, "/elements") {
 		return false

internal/config/config.go

@@ -39,6 +39,44 @@ type Config struct {
 	// Ruleset is where the confirmed ruleset is saved for the boot unit
 	// (housecarl-ruleset.service loads it before the network comes up).
 	Ruleset string `yaml:"ruleset"`
+	// Stats keeps traffic, hits and drops over time (docs/stats.md).
+	Stats Stats `yaml:"stats"`
+	// Shadow loads the ruleset without letting it block, reject or
+	// translate anything, beside the firewall in force (docs/shadow.md).
+	Shadow Shadow `yaml:"shadow"`
+}
+
+// Shadow is shadow mode: Housecarl's ruleset is loaded and counts, logs
+// and traces what it would do, while another firewall decides.
+type Shadow struct {
+	Enabled bool `yaml:"enabled"`
+	// Position is where Housecarl's filter chains run: after the other
+	// firewall's (it sees what that one lets through, so what it would
+	// drop is what switching would break) or before (it sees everything).
+	Position string `yaml:"position"`
+}
+
+// Shadow positions.
+const (
+	ShadowAfter  = "after"
+	ShadowBefore = "before"
+)
+
+// Stats configures statistics: a database of their own, aggregates
+// only, pruned by age.
+type Stats struct {
+	Enabled   bool           `yaml:"enabled"`
+	Database  string         `yaml:"database"`
+	Retention StatsRetention `yaml:"retention"`
+	// TopN is how many entries the top lists keep per hour and day.
+	TopN int `yaml:"top_n"`
+}
+
+// StatsRetention is how long each resolution is kept.
+type StatsRetention struct {
+	Minute Duration `yaml:"minute"`
+	Hour   Duration `yaml:"hour"`
+	Day    Duration `yaml:"day"`
 }
 
 // TLS is the web UI's certificate: from files, or obtained through ACME
@@ -183,6 +221,9 @@ func Default() *Config {
 		Log:            Log{Level: "info", Format: "text"},
 		Session:        Session{Lifetime: Duration(12 * time.Hour), IdleTimeout: Duration(2 * time.Hour), CookieSecure: "auto"},
 		Backup:         Backup{Dir: "/var/backups/housecarl", Retention: Duration(30 * 24 * time.Hour), KeepAtLeast: 7},
+		Stats: Stats{Enabled: true, Database: "/var/lib/housecarl/stats.db", TopN: 100,
+			Retention: StatsRetention{Minute: Duration(7 * 24 * time.Hour), Hour: Duration(90 * 24 * time.Hour), Day: Duration(730 * 24 * time.Hour)}},
+		Shadow: Shadow{Position: ShadowAfter},
 		TLS:    TLS{ACME: ACME{Directory: "letsencrypt-staging", Resolvers: []string{"1.1.1.1:53", "9.9.9.9:53", "8.8.8.8:53"}, CheckPropagation: true}},
 	}
 }
@@ -254,6 +295,9 @@ func (c *Config) check() error {
 	default:
 		return fmt.Errorf("log.level must be debug, info, warn or error")
 	}
+	if c.Shadow.Position != ShadowAfter && c.Shadow.Position != ShadowBefore {
+		return fmt.Errorf("shadow.position must be after or before")
+	}
 	return nil
 }
 
@@ -393,4 +437,29 @@ backup:
   dir: /var/backups/housecarl
   retention: 30d
   keep_at_least: 7
+
+# Statistics (docs/stats.md): traffic per interface and zone, hits and
+# drops, blocked sources, connections, quotas, kept over time in a database
+# of their own (aggregates only). Deleting it is always safe. Traffic per
+# device is a firewall setting: hc settings set --stats-per-device on.
+stats:
+  enabled: true
+  database: /var/lib/housecarl/stats.db
+  retention:
+    minute: 7d     # per-minute figures
+    hour: 90d      # hourly figures and top lists
+    day: 730d      # daily figures and top lists
+  top_n: 100       # entries kept per top list, per hour and day
+
+# Shadow mode (docs/shadow.md): Housecarl's ruleset is loaded beside the
+# firewall you have now, but blocks, rejects and translates nothing. Its
+# counters, log and tracer show what it would do. No change is on
+# probation, and takeovers and closing connections are refused. housecarld
+# reads this when it starts: restart it to switch.
+shadow:
+  enabled: false
+  # after: Housecarl's filter chains run after the other firewall's, and
+  # see only what it lets through: what Housecarl would drop is what
+  # switching over would break. before: they see every packet.
+  position: after
 `

internal/config/config_test.go

@@ -35,6 +35,7 @@ func TestErrors(t *testing.T) {
 		"{tls: {acme: {domain: '*.example.com'}}, jarl: {server: https://j, token: x}}":                            "full name",
 		"{tls: {acme: {domain: d.example.com, directory: http://x}}, jarl: {server: https://j, token: x}}":         "https://",
 		"ruleset: relative.nft": "ruleset",
+		"shadow: {position: x}": "shadow.position",
 	}
 	for yml, want := range cases {
 		path := filepath.Join(t.TempDir(), "s.yaml")

internal/conntrack/conntrack.go

@@ -0,0 +1,351 @@
+// Package conntrack lists and closes the kernel's tracked connections,
+// through ctnetlink (the netfilter netlink subsystem for connection
+// tracking): what the "established" built-in lets through, and what a
+// change of rules doesn't stop by itself.
+//
+// Closing a connection deletes its tracking entry. Its next packet is
+// then checked by the rules again, as a new connection would be: one the
+// rules now block is stopped, one they allow carries on (TCP connections
+// are picked up again mid-stream).
+package conntrack
+
+import (
+	"context"
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"net/netip"
+	"sync/atomic"
+	"syscall"
+	"time"
+
+	"golang.org/x/sys/unix"
+
+	"housecarl/internal/api"
+)
+
+const (
+	subsysCtnetlink = 1
+	msgNew          = 0
+	msgGet          = 1
+	msgDelete       = 2
+
+	ctaTupleOrig     = 1
+	ctaTupleReply    = 2
+	ctaStatus        = 3
+	ctaProtoinfo     = 4
+	ctaTimeout       = 7
+	ctaMark          = 8
+	ctaCountersOrig  = 9
+	ctaCountersReply = 10
+	ctaID            = 12
+
+	ctaTupleIP    = 1
+	ctaTupleProto = 2
+
+	ctaIPv4Src = 1
+	ctaIPv4Dst = 2
+	ctaIPv6Src = 3
+	ctaIPv6Dst = 4
+
+	ctaProtoNum      = 1
+	ctaProtoSrcPort  = 2
+	ctaProtoDstPort  = 3
+	ctaProtoICMPID   = 4
+	ctaProtoICMPType = 5
+	ctaProtoICMPCode = 6
+	ctaProtoICMP6ID  = 7
+	ctaProtoICMP6Typ = 8
+	ctaProtoICMP6Cod = 9
+
+	ctaProtoinfoTCP      = 1
+	ctaProtoinfoTCPState = 1
+
+	ctaCountersPackets = 1
+	ctaCountersBytes   = 2
+
+	statusAssured = 1 << 2
+	statusSrcNAT  = 1 << 4
+	statusDstNAT  = 1 << 5
+
+	nlaNested   = 0x8000
+	nlaTypeMask = 0x3fff
+)
+
+var tcpStates = []string{"none", "syn-sent", "syn-recv", "established", "fin-wait", "close-wait", "last-ack", "time-wait", "close", "syn-sent2"}
+
+var protoNames = map[uint8]string{1: "icmp", 6: "tcp", 17: "udp", 58: "icmpv6", 132: "sctp", 136: "udplite", 33: "dccp", 47: "gre"}
+
+// Client talks to ctnetlink. The zero value works.
+type Client struct{}
+
+var seq atomic.Uint32
+
+// List returns every tracked connection, IPv4 and IPv6.
+func (Client) List(ctx context.Context) ([]api.Connection, error) {
+	msgs, err := request(ctx, msgGet, unix.NLM_F_REQUEST|unix.NLM_F_DUMP, unix.AF_UNSPEC, nil)
+	if err != nil {
+		return nil, err
+	}
+	out := []api.Connection{}
+	for _, m := range msgs {
+		if c, ok := parse(m); ok {
+			out = append(out, c)
+		}
+	}
+	return out, nil
+}
+
+// Close deletes a connection's tracking entry, found by its original
+// direction (addresses, protocol, ports or ICMP id). Gone already is fine.
+func (Client) Close(ctx context.Context, c api.Connection) error {
+	src, err1 := netip.ParseAddr(c.Source)
+	dst, err2 := netip.ParseAddr(c.Destination)
+	if err1 != nil || err2 != nil || src.Is4() != dst.Is4() {
+		return fmt.Errorf("connection %s -> %s: not a pair of addresses", c.Source, c.Destination)
+	}
+	num := protoNumber(c.Protocol)
+	if num == 0 {
+		return fmt.Errorf("connection %s -> %s: unknown protocol %q", c.Source, c.Destination, c.Protocol)
+	}
+	family, ipSrc, ipDst := uint8(unix.AF_INET), uint16(ctaIPv4Src), uint16(ctaIPv4Dst)
+	if src.Is6() {
+		family, ipSrc, ipDst = unix.AF_INET6, ctaIPv6Src, ctaIPv6Dst
+	}
+	ip := attr(ipSrc, src.AsSlice())
+	ip = append(ip, attr(ipDst, dst.AsSlice())...)
+	proto := attr(ctaProtoNum, []byte{num})
+	switch num {
+	case 1:
+		proto = append(proto, attr(ctaProtoICMPID, be16(uint16(c.ICMPID)))...)
+		proto = append(proto, attr(ctaProtoICMPType, []byte{uint8(c.ICMPType)})...)
+		proto = append(proto, attr(ctaProtoICMPCode, []byte{uint8(c.ICMPCode)})...)
+	case 58:
+		proto = append(proto, attr(ctaProtoICMP6ID, be16(uint16(c.ICMPID)))...)
+		proto = append(proto, attr(ctaProtoICMP6Typ, []byte{uint8(c.ICMPType)})...)
+		proto = append(proto, attr(ctaProtoICMP6Cod, []byte{uint8(c.ICMPCode)})...)
+	default:
+		proto = append(proto, attr(ctaProtoSrcPort, be16(uint16(c.SourcePort)))...)
+		proto = append(proto, attr(ctaProtoDstPort, be16(uint16(c.DestinationPort)))...)
+	}
+	tuple := attr(ctaTupleIP|nlaNested, ip)
+	tuple = append(tuple, attr(ctaTupleProto|nlaNested, proto)...)
+	_, err := request(ctx, msgDelete, unix.NLM_F_REQUEST|unix.NLM_F_ACK, family, attr(ctaTupleOrig|nlaNested, tuple))
+	if errors.Is(err, unix.ENOENT) {
+		return nil
+	}
+	return err
+}
+
+func protoNumber(name string) uint8 {
+	for n, s := range protoNames {
+		if s == name {
+			return n
+		}
+	}
+	return 0
+}
+
+// ---- netlink
+
+func request(ctx context.Context, msg uint16, flags uint16, family uint8, payload []byte) ([][]byte, error) {
+	fd, err := unix.Socket(unix.AF_NETLINK, unix.SOCK_RAW|unix.SOCK_CLOEXEC, unix.NETLINK_NETFILTER)
+	if err != nil {
+		return nil, err
+	}
+	defer unix.Close(fd)
+	if err := unix.Bind(fd, &unix.SockaddrNetlink{Family: unix.AF_NETLINK}); err != nil {
+		return nil, err
+	}
+	deadline := time.Now().Add(10 * time.Second)
+	if d, ok := ctx.Deadline(); ok && d.Before(deadline) {
+		deadline = d
+	}
+	tv := unix.NsecToTimeval(time.Until(deadline).Nanoseconds())
+	unix.SetsockoptTimeval(fd, unix.SOL_SOCKET, unix.SO_RCVTIMEO, &tv)
+	unix.SetsockoptInt(fd, unix.SOL_SOCKET, unix.SO_RCVBUF, 1<<22)
+
+	s := seq.Add(1)
+	b := make([]byte, unix.SizeofNlMsghdr+4, unix.SizeofNlMsghdr+4+len(payload))
+	binary.NativeEndian.PutUint16(b[4:], subsysCtnetlink<<8|msg)
+	binary.NativeEndian.PutUint16(b[6:], flags)
+	binary.NativeEndian.PutUint32(b[8:], s)
+	b[16] = family // nfgenmsg: family, version 0, res_id 0
+	b = append(b, payload...)
+	binary.NativeEndian.PutUint32(b[0:], uint32(len(b)))
+	if err := unix.Sendto(fd, b, 0, &unix.SockaddrNetlink{Family: unix.AF_NETLINK}); err != nil {
+		return nil, err
+	}
+	var out [][]byte
+	buf := make([]byte, 1<<16)
+	for {
+		n, _, err := unix.Recvfrom(fd, buf, 0)
+		if err != nil {
+			if errors.Is(err, unix.EAGAIN) {
+				return nil, errors.New("no answer from connection tracking (is nf_conntrack loaded?)")
+			}
+			return nil, err
+		}
+		for off := 0; off+unix.SizeofNlMsghdr <= n; {
+			l := int(binary.NativeEndian.Uint32(buf[off:]))
+			typ := binary.NativeEndian.Uint16(buf[off+4:])
+			if l < unix.SizeofNlMsghdr || off+l > n {
+				return nil, errors.New("connection tracking: a malformed answer")
+			}
+			body := buf[off+unix.SizeofNlMsghdr : off+l]
+			switch typ {
+			case unix.NLMSG_DONE:
+				return out, nil
+			case unix.NLMSG_ERROR:
+				if len(body) >= 4 {
+					if code := int32(binary.NativeEndian.Uint32(body)); code != 0 {
+						return nil, syscall.Errno(-code)
+					}
+				}
+				return out, nil // the acknowledgement
+			default:
+				out = append(out, append([]byte(nil), body...))
+			}
+			off += (l + 3) &^ 3
+		}
+		if flags&unix.NLM_F_DUMP == 0 && flags&unix.NLM_F_ACK == 0 {
+			return out, nil
+		}
+	}
+}
+
+func attr(typ uint16, data []byte) []byte {
+	l := 4 + len(data)
+	b := make([]byte, (l+3)&^3)
+	binary.NativeEndian.PutUint16(b[0:], uint16(l))
+	binary.NativeEndian.PutUint16(b[2:], typ)
+	copy(b[4:], data)
+	return b
+}
+
+func be16(v uint16) []byte {
+	b := make([]byte, 2)
+	binary.BigEndian.PutUint16(b, v)
+	return b
+}
+
+// attrs splits netlink attributes: type (without the nested flag) → data.
+func attrs(b []byte) map[uint16][]byte {
+	m := map[uint16][]byte{}
+	for len(b) >= 4 {
+		l := int(binary.NativeEndian.Uint16(b[0:]))
+		typ := binary.NativeEndian.Uint16(b[2:]) & nlaTypeMask
+		if l < 4 || l > len(b) {
+			break
+		}
+		m[typ] = b[4:l]
+		b = b[min((l+3)&^3, len(b)):]
+	}
+	return m
+}
+
+// ---- parsing
+
+type tuple struct {
+	src, dst         netip.Addr
+	proto            uint8
+	sport, dport     uint16
+	icmpID           uint16
+	icmpType, icmpCd uint8
+}
+
+func parseTuple(b []byte) (tuple, bool) {
+	var t tuple
+	a := attrs(b)
+	ip := attrs(a[ctaTupleIP])
+	if v, ok := ip[ctaIPv4Src]; ok {
+		t.src, _ = netip.AddrFromSlice(v)
+		t.dst, _ = netip.AddrFromSlice(ip[ctaIPv4Dst])
+	} else if v, ok := ip[ctaIPv6Src]; ok {
+		t.src, _ = netip.AddrFromSlice(v)
+		t.dst, _ = netip.AddrFromSlice(ip[ctaIPv6Dst])
+	}
+	p := attrs(a[ctaTupleProto])
+	if v := p[ctaProtoNum]; len(v) == 1 {
+		t.proto = v[0]
+	}
+	if v := p[ctaProtoSrcPort]; len(v) == 2 {
+		t.sport = binary.BigEndian.Uint16(v)
+	}
+	if v := p[ctaProtoDstPort]; len(v) == 2 {
+		t.dport = binary.BigEndian.Uint16(v)
+	}
+	for _, k := range [][3]uint16{{ctaProtoICMPID, ctaProtoICMPType, ctaProtoICMPCode}, {ctaProtoICMP6ID, ctaProtoICMP6Typ, ctaProtoICMP6Cod}} {
+		if v := p[k[0]]; len(v) == 2 {
+			t.icmpID = binary.BigEndian.Uint16(v)
+		}
+		if v := p[k[1]]; len(v) == 1 {
+			t.icmpType = v[0]
+		}
+		if v := p[k[2]]; len(v) == 1 {
+			t.icmpCd = v[0]
+		}
+	}
+	return t, t.src.IsValid() && t.dst.IsValid()
+}
+
+// parse reads one connection (an nfgenmsg and its attributes).
+func parse(m []byte) (api.Connection, bool) {
+	if len(m) < 4 {
+		return api.Connection{}, false
+	}
+	a := attrs(m[4:])
+	orig, ok1 := parseTuple(a[ctaTupleOrig])
+	reply, ok2 := parseTuple(a[ctaTupleReply])
+	if !ok1 || !ok2 {
+		return api.Connection{}, false
+	}
+	c := api.Connection{
+		Family: "ipv4", Protocol: protoNames[orig.proto],
+		Source: orig.src.String(), SourcePort: int(orig.sport), Destination: orig.dst.String(), DestinationPort: int(orig.dport),
+		ICMPID: int(orig.icmpID), ICMPType: int(orig.icmpType), ICMPCode: int(orig.icmpCd),
+	}
+	if orig.src.Is6() {
+		c.Family = "ipv6"
+	}
+	if c.Protocol == "" {
+		c.Protocol = fmt.Sprint(orig.proto)
+	}
+	// NAT shows in the reply direction: it comes from where the packets
+	// really went, to where they really came from.
+	if reply.src != orig.dst || reply.sport != orig.dport {
+		c.DNATTo = reply.src.String()
+		if reply.sport != orig.dport {
+			c.DNATPort = int(reply.sport)
+		}
+	}
+	if reply.dst != orig.src || reply.dport != orig.sport {
+		c.SNATTo = reply.dst.String()
+	}
+	if v := a[ctaStatus]; len(v) == 4 {
+		st := binary.BigEndian.Uint32(v)
+		c.Assured = st&statusAssured != 0
+	}
+	if v := a[ctaTimeout]; len(v) == 4 {
+		c.Timeout = int(binary.BigEndian.Uint32(v))
+	}
+	if v := a[ctaMark]; len(v) == 4 {
+		c.Mark = binary.BigEndian.Uint32(v)
+	}
+	if v := a[ctaID]; len(v) == 4 {
+		c.ID = binary.BigEndian.Uint32(v)
+	}
+	if v := attrs(attrs(a[ctaProtoinfo])[ctaProtoinfoTCP])[ctaProtoinfoTCPState]; len(v) == 1 && int(v[0]) < len(tcpStates) {
+		c.State = tcpStates[v[0]]
+	}
+	for _, k := range []uint16{ctaCountersOrig, ctaCountersReply} {
+		ct := attrs(a[k])
+		if v := ct[ctaCountersPackets]; len(v) == 8 {
+			c.Packets += binary.BigEndian.Uint64(v)
+		}
+		if v := ct[ctaCountersBytes]; len(v) == 8 {
+			c.Bytes += binary.BigEndian.Uint64(v)
+		}
+	}
+	return c, true
+}

internal/conntrack/conntrack_test.go

@@ -0,0 +1,51 @@
+package conntrack
+
+import (
+	"encoding/binary"
+	"net/netip"
+	"testing"
+)
+
+func nested(typ uint16, parts ...[]byte) []byte {
+	var b []byte
+	for _, p := range parts {
+		b = append(b, p...)
+	}
+	return attr(typ|nlaNested, b)
+}
+
+func tupleAttr(typ uint16, src, dst string, proto uint8, sport, dport uint16) []byte {
+	s, d := netip.MustParseAddr(src), netip.MustParseAddr(dst)
+	return nested(typ,
+		nested(ctaTupleIP, attr(ctaIPv4Src, s.AsSlice()), attr(ctaIPv4Dst, d.AsSlice())),
+		nested(ctaTupleProto, attr(ctaProtoNum, []byte{proto}), attr(ctaProtoSrcPort, be16(sport)), attr(ctaProtoDstPort, be16(dport))))
+}
+
+// A port-forwarded TCP connection from the internet, as ctnetlink
+// describes it: the reply comes from the inside host.
+func TestParse(t *testing.T) {
+	status, timeout := make([]byte, 4), make([]byte, 4)
+	binary.BigEndian.PutUint32(status, statusAssured|statusDstNAT)
+	binary.BigEndian.PutUint32(timeout, 431999)
+	msg := []byte{2, 0, 0, 0} // nfgenmsg: AF_INET
+	msg = append(msg, tupleAttr(ctaTupleOrig, "198.51.100.7", "203.0.113.10", 6, 51432, 8443)...)
+	msg = append(msg, tupleAttr(ctaTupleReply, "192.168.1.10", "198.51.100.7", 6, 443, 51432)...)
+	msg = append(msg, attr(ctaStatus, status)...)
+	msg = append(msg, attr(ctaTimeout, timeout)...)
+	msg = append(msg, nested(ctaProtoinfo, nested(ctaProtoinfoTCP, attr(ctaProtoinfoTCPState, []byte{3})))...)
+	c, ok := parse(msg)
+	if !ok {
+		t.Fatal("not parsed")
+	}
+	if c.Protocol != "tcp" || c.Source != "198.51.100.7" || c.SourcePort != 51432 || c.Destination != "203.0.113.10" || c.DestinationPort != 8443 ||
+		c.DNATTo != "192.168.1.10" || c.DNATPort != 443 || c.SNATTo != "" || c.State != "established" || !c.Assured || c.Timeout != 431999 {
+		t.Errorf("parsed %+v", c)
+	}
+	// Masqueraded on the way out: the reply comes back to the WAN address.
+	msg = []byte{2, 0, 0, 0}
+	msg = append(msg, tupleAttr(ctaTupleOrig, "192.168.1.20", "198.51.100.9", 17, 5353, 53)...)
+	msg = append(msg, tupleAttr(ctaTupleReply, "198.51.100.9", "203.0.113.10", 17, 53, 5353)...)
+	if c, _ := parse(msg); c.SNATTo != "203.0.113.10" || c.DNATTo != "" || c.Protocol != "udp" {
+		t.Errorf("masqueraded: %+v", c)
+	}
+}

internal/engine/engine.go

@@ -137,6 +137,9 @@ type Edit func(st *model.State, newID func() int64) error
 type Options struct {
 	DryRun bool
 	Via    string // web, cli, api, import
+	// RevertOf is the history entry the change undoes (a changeset with a
+	// staged undo), so History links the two.
+	RevertOf int64
 }
 
 // Apply makes a change: edit runs on a copy of the configuration, the
@@ -161,7 +164,7 @@ func (e *Engine) Apply(ctx context.Context, p *auth.Principal, description strin
 	if idErr != nil {
 		return nil, idErr
 	}
-	return e.commit(ctx, p, description, opts, next, 0)
+	return e.commit(ctx, p, description, opts, next, opts.RevertOf)
 }
 
 // commit validates next and saves it as the new state.
@@ -241,12 +244,16 @@ func newProblems(before, after []validate.Problem) []validate.Problem {
 	return out
 }
 
-// ApplyCurrent puts the current configuration in force (at start).
+// ApplyCurrent puts the current configuration in force again: at start,
+// and when a flowtable device appears. It holds the engine's lock, so a
+// change can't be applied in between.
 func (e *Engine) ApplyCurrent(ctx context.Context) error {
 	if e.Firewall == nil {
 		return nil
 	}
-	return e.Firewall.Apply(ctx, e.State())
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	return e.Firewall.Apply(ctx, e.state.Clone())
 }
 
 // Revert undoes a history entry, if the objects it changed are still as
@@ -264,8 +271,22 @@ func (e *Engine) Revert(ctx context.Context, p *auth.Principal, id int64, opts O
 	}
 	e.mu.Lock()
 	defer e.mu.Unlock()
-	cur := objects(e.state)
 	next := e.state.Clone()
+	if err := Undo(next, h); err != nil {
+		return nil, err
+	}
+	return e.commit(ctx, p, UndoDescription(h), opts, next, h.ID)
+}
+
+// UndoDescription is an undo's description: "undo 12: create rule …".
+func UndoDescription(h *model.HistoryEntry) string {
+	return fmt.Sprintf("undo %d: %s", h.ID, h.Description)
+}
+
+// Undo applies a history entry backwards to st, if every object it
+// changed is still as it left it (a change made since would be lost).
+func Undo(st *model.State, h *model.HistoryEntry) error {
+	cur := objects(st)
 	for i := len(h.Changes) - 1; i >= 0; i-- {
 		c := h.Changes[i]
 		now := cur[key{c.Kind, c.ID}]
@@ -274,13 +295,13 @@ func (e *Engine) Revert(ctx context.Context, p *auth.Principal, id int64, opts O
 			if what == "" {
 				what = fmt.Sprintf("%s %d", c.Kind, c.ID)
 			}
-			return nil, Errorf(KindConflict, "%s %s has changed since entry %d; undo the later changes first, or change it by hand", c.Kind, what, id)
+			return Errorf(KindConflict, "%s %s has changed since entry %d; undo the later changes first, or change it by hand", c.Kind, what, h.ID)
 		}
-		if err := put(next, c.Kind, c.ID, c.Before); err != nil {
-			return nil, err
+		if err := put(st, c.Kind, c.ID, c.Before); err != nil {
+			return err
 		}
 	}
-	return e.commit(ctx, p, fmt.Sprintf("undo %d: %s", h.ID, h.Description), opts, next, h.ID)
+	return nil
 }
 
 // Replace makes the configuration next as a whole (imports).

internal/hostif/hostif.go

@@ -63,6 +63,14 @@ func (h *Host) Up(name string) bool {
 	return h.up[name]
 }
 
+// Forget drops what was read, so the next question reads the interfaces
+// again (after the kernel said one appeared or went).
+func (h *Host) Forget() {
+	h.mu.Lock()
+	h.at = time.Time{}
+	h.mu.Unlock()
+}
+
 // Exists returns the set of interface names (nil if they can't be read).
 func (h *Host) Exists() map[string]bool {
 	h.mu.Lock()

internal/importer/audit_test.go

@@ -0,0 +1,98 @@
+package importer
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"strings"
+	"testing"
+
+	"housecarl/internal/render"
+	"housecarl/internal/validate"
+)
+
+// TestImportAudit: rulesets the importer must not change the meaning of
+// (testdata/audit.nft, as nft 1.1.3 lists it).
+func TestImportAudit(t *testing.T) {
+	js, _ := os.ReadFile("testdata/audit.json")
+	txt, _ := os.ReadFile("testdata/audit.txt")
+	defs, _ := os.ReadFile("testdata/audit.nft")
+	res, err := Import(Input{JSON: js, Text: string(txt), Defines: string(defs), Zones: map[string]string{"enp2s0.20": "iot", "eth9": "x"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	st := res.State
+	notes := strings.Join(res.Notes, "\n")
+	text := render.Render(st, render.Options{}).Text
+	has := func(what, s string) {
+		t.Helper()
+		if !strings.Contains(text, s) {
+			t.Errorf("%s: no %q in\n%s", what, s, text)
+		}
+	}
+	// Two chains on the input hook: their rules once each, in priority
+	// order; established stays a rule after the first chain's rule.
+	if n := strings.Count(text, "192.0.2.66"); n != 1 {
+		t.Errorf("the early input chain's rule %d times", n)
+	}
+	if strings.Index(text, "192.0.2.66") > strings.Index(text, "ct state established,related counter accept comment \"hc:r") ||
+		!strings.Contains(notes, "both hook input") {
+		t.Errorf("chains on one hook:\n%s", text)
+	}
+	// A second match on a field, and != on the hop limit: raw, as written.
+	has("two source matches", "ip saddr 10.0.0.0/8 ip saddr != 10.0.10.0/24 tcp dport 22 drop")
+	has("hop limit", "ip6 hoplimit != 255 drop")
+	// A jump: a rule within it is the rule's own; one outside it, or raw,
+	// keeps the jump's match in front.
+	has("within the jump", `ip saddr 10.9.0.5 tcp dport 22 counter accept comment "hc:r`)
+	has("outside the jump", "ip saddr 10.9.0.0/24 ip saddr 10.8.0.1 tcp dport 23 accept")
+	has("raw under the jump", "ip saddr 10.9.0.0/24 meta mark 0x00000001 accept")
+	// Sets: a timeout set is dynamic; an interface set is written out in
+	// the raw rule that names it.
+	if s := st.Set("blocklist"); s == nil || !s.Dynamic || s.Timeout != 3600 {
+		t.Errorf("blocklist: %+v", s)
+	}
+	has("dynamic set", "ip saddr @blocklist_v4 counter drop")
+	has("interface set", `iifname { "enp2s0", "enp3s0" } accept`)
+	// An accept to a forwarded host that isn't only for forwarded
+	// connections stays a rule; a DNAT of two ports, two forwards.
+	has("not only forwarded", `ip daddr 10.0.10.37 tcp dport 443 counter accept comment "hc:r`)
+	if len(st.Forwards) != 2 || st.Forwards[0].Name != "web-80" || st.Forwards[1].Name != "web-443" {
+		t.Errorf("forwards %+v", st.Forwards)
+	}
+	// Zones: as given (a VLAN), from defines, and why.
+	if z := st.Zone("iot"); z == nil || z.Interfaces[0] != "enp2s0.20" {
+		t.Errorf("zone iot: %+v", st.Zones)
+	}
+	if st.Zone("lan") == nil || st.Zone("wan") == nil {
+		t.Errorf("zones %+v", st.Zones)
+	}
+	for _, want := range []string{"iot (enp2s0.20: as given)", "lan (enp2s0: the define lan_if)", "--zone eth9=x: no rule names"} {
+		if !strings.Contains(notes, want) {
+			t.Errorf("no note %q in\n%s", want, notes)
+		}
+	}
+	if ps := validate.Check(st, nil); validate.HasErrors(ps) {
+		t.Errorf("problems: %+v", validate.Errors(ps))
+	}
+	// And nftables takes it.
+	cmd := strings.Fields(os.Getenv("HOUSECARL_NFT"))
+	if len(cmd) == 0 {
+		return
+	}
+	c := exec.CommandContext(context.Background(), cmd[0], append(cmd[1:], "-c", "-f", "/dev/stdin")...)
+	c.Stdin = strings.NewReader(text)
+	if out, err := c.CombinedOutput(); err != nil {
+		t.Errorf("nft -c: %v\n%s\n%s", err, out, text)
+	}
+}
+
+// TestValidIface: interface names a define may hold, VLANs too.
+func TestValidIface(t *testing.T) {
+	for s, want := range map[string]bool{"enp2s0": true, "wan": true, "enp2s0.20": true, "br0.4094": true, "veth*": true,
+		"10.0.10.5": false, "example.com": false, "a.b.c": false, "enp2s0.20.30": false, "averyveryverylongname": false} {
+		if validIface(s) != want {
+			t.Errorf("validIface(%q) = %v", s, !want)
+		}
+	}
+}

internal/importer/importer.go

@@ -20,6 +20,7 @@ import (
 	"strings"
 
 	"housecarl/internal/model"
+	"housecarl/internal/render"
 )
 
 // Input is what to import.
@@ -34,11 +35,16 @@ type Input struct {
 	// Table is the table to import ("inet main"); empty: the only table
 	// that isn't Housecarl's.
 	Table string
+	// Zones names zones by interface ("enp2s0": "lan"), over the names
+	// the importer would choose; interfaces given one name share a zone.
+	Zones map[string]string
 }
 
 // Result is the imported configuration and what to know about it.
 type Result struct {
 	State *model.State
+	// Table is the table imported ("inet main").
+	Table string
 	Notes []string
 	// Raw lists the rules that became raw rules.
 	Raw []string
@@ -79,6 +85,8 @@ type nftSet struct {
 	Table   string            `json:"table"`
 	Name    string            `json:"name"`
 	Type    json.RawMessage   `json:"type"`
+	Flags   []string          `json:"flags"`
+	Timeout int               `json:"timeout"`
 	Elem    []json.RawMessage `json:"elem"`
 }
 
@@ -119,6 +127,8 @@ type importer struct {
 	meters    map[string]bool   // sets used as meters
 	noted     map[string]bool
 	positions map[string]float64
+	// zoneFrom says where each zone's name came from, for the note.
+	zoneFrom map[string]string
 }
 
 func (im *importer) note(format string, args ...any) {
@@ -134,7 +144,7 @@ func Import(in Input) (*Result, error) {
 		return nil, fmt.Errorf("the input isn't nft's JSON (nft -j list ruleset): %v", err)
 	}
 	im := &importer{in: in, chains: map[string]nftChain{}, rules: map[string][]nftRule{}, sets: map[string]nftSet{}, maps: map[string]nftMap{},
-		text: map[int]string{}, st: model.NewState(), used: map[string]bool{}, noted: map[string]bool{}, positions: map[string]float64{}}
+		text: map[int]string{}, st: model.NewState(), used: map[string]bool{}, noted: map[string]bool{}, positions: map[string]float64{}, zoneFrom: map[string]string{}}
 	var tables []nftTable
 	for _, item := range doc.Nftables {
 		if raw, ok := item["table"]; ok {
@@ -205,8 +215,9 @@ func Import(in Input) (*Result, error) {
 	im.importSets()
 	im.run()
 	im.nameForwards()
+	im.noteZones()
 	im.st.Renumber()
-	return &Result{State: im.st, Notes: im.notes, Raw: im.raw}, nil
+	return &Result{State: im.st, Table: im.table.Family + " " + im.table.Name, Notes: im.notes, Raw: im.raw}, nil
 }
 
 func tableList(ts []nftTable) string {
@@ -307,10 +318,13 @@ func key(vals []string) string {
 	return strings.Join(s, ",")
 }
 
-var ifaceRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_.@-]{0,14}\*?$`)
+var ifaceRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_@-]{0,14}\*?$`)
+
+// vlanRe is a VLAN sub-interface: enp2s0.20.
+var vlanRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_@-]*\.[0-9]{1,4}$`)
 
 func validIface(s string) bool {
-	return ifaceRe.MatchString(s) && !strings.ContainsAny(s, ".:") || s == "lo"
+	return len(s) <= 15 && (ifaceRe.MatchString(s) || vlanRe.MatchString(s))
 }
 
 // ---- sets
@@ -358,12 +372,22 @@ func (im *importer) importSets() {
 		for _, e := range s.Elem {
 			entries = append(entries, values(e)...)
 		}
-		if existing := im.st.Set(n); existing != nil {
+		name := setName(n)
+		if contains(s.Flags, "timeout") || contains(s.Flags, "dynamic") {
+			// Filled at run time (by a program, or a rule): a dynamic set.
+			// Its entries now are passing state, not configuration.
+			im.st.Sets = append(im.st.Sets, &model.AddressSet{ID: im.newID(), Name: name, Dynamic: true, Timeout: s.Timeout})
+			im.used[name] = true
+			im.note("set %s has entries that come and go (flags %s): it became a dynamic set, without its %d entries now; whatever fills it must now fill Housecarl's (hc set entries add %s ADDRESS, the API, or nft add element inet housecarl %s_v4 / %s_v6)",
+				n, strings.Join(s.Flags, ","), len(entries), name, name, name)
+			continue
+		}
+		if existing := im.st.Set(name); existing != nil {
 			existing.Entries = append(existing.Entries, entries...)
 			continue
 		}
-		im.st.Sets = append(im.st.Sets, &model.AddressSet{ID: im.newID(), Name: setName(n), Entries: entries})
-		im.used[n] = true
+		im.st.Sets = append(im.st.Sets, &model.AddressSet{ID: im.newID(), Name: name, Entries: entries})
+		im.used[name] = true
 	}
 }
 
@@ -395,7 +419,10 @@ func (im *importer) namedSet(entries []string) string {
 	return name
 }
 
-// zone returns the zone of an interface, making it on first use.
+// zone returns the zone of an interface, making it on first use. Its
+// name: the one given for it (Input.Zones); else the name of a define
+// holding the interface ("define lan_if = enp2s0": lan); else the
+// interface's own.
 func (im *importer) zone(iface string) string {
 	for _, z := range im.st.Zones {
 		for _, i := range z.Interfaces {
@@ -404,21 +431,78 @@ func (im *importer) zone(iface string) string {
 			}
 		}
 	}
-	name := iface
+	if given, ok := im.in.Zones[iface]; ok {
+		name := setName(given)
+		// Interfaces given one name share its zone.
+		if z := im.st.Zone(name); z != nil && im.zoneFrom[name] == "given" {
+			z.Interfaces = append(z.Interfaces, iface)
+			return name
+		}
+		if im.st.Zone(name) == nil {
+			return im.newZone(name, iface, "given")
+		}
+		im.note("--zone %s=%s: the zone %s is taken (by %s), so %s is named as below", iface, given, name, strings.Join(im.st.Zone(name).Interfaces, ", "), iface)
+	}
+	name, from := iface, "its interface's name"
 	if d, ok := im.ifNames[iface]; ok {
-		name = d
-		for _, suf := range []string{"_interface", "_iface", "_int", "_if", "_dev"} {
+		name, from = d, "the define "+d
+		for _, suf := range []string{"_interface", "_ifname", "_iface", "_device", "_int", "_nic", "_if", "_dev"} {
 			name = strings.TrimSuffix(name, suf)
 		}
 	}
 	name = setName(strings.TrimSuffix(name, "*"))
 	if im.st.Zone(name) != nil {
-		name = setName(iface + "_zone")
+		name, from = setName(iface+"_zone"), "its interface's name (the other was taken)"
 	}
+	return im.newZone(name, iface, from)
+}
+
+// zone2 is the zone an interface is in, or "" (making none).
+func (im *importer) zoneOfIface(iface string) string {
+	for _, z := range im.st.Zones {
+		if contains(z.Interfaces, iface) {
+			return z.Name
+		}
+	}
+	return ""
+}
+
+func (im *importer) newZone(name, iface, from string) string {
 	im.st.Zones = append(im.st.Zones, &model.Zone{ID: im.newID(), Name: name, Interfaces: []string{iface}})
+	im.zoneFrom[name] = from
 	return name
 }
 
+// noteZones says how each zone was named, and how to change it.
+func (im *importer) noteZones() {
+	ifaces := make([]string, 0, len(im.in.Zones))
+	for i := range im.in.Zones {
+		ifaces = append(ifaces, i)
+	}
+	sort.Strings(ifaces)
+	for _, i := range ifaces {
+		if im.zoneOfIface(i) == "" {
+			im.note("--zone %s=%s: no rule names the interface %s, so there is no zone for it", i, im.in.Zones[i], i)
+		}
+	}
+	if len(im.st.Zones) == 0 {
+		return
+	}
+	var parts []string
+	for _, z := range im.st.Zones {
+		from := im.zoneFrom[z.Name]
+		if from == "given" {
+			from = "as given"
+		}
+		ext := ""
+		if z.External {
+			ext = ", external"
+		}
+		parts = append(parts, fmt.Sprintf("%s (%s%s: %s)", z.Name, strings.Join(z.Interfaces, ", "), ext, from))
+	}
+	im.note("zones: %s; name them yourself with --zone INTERFACE=NAME (lan, iot, guest: what the network is for), or later with hc zone update NAME --name NEW", strings.Join(parts, "; "))
+}
+
 func (im *importer) external(iface string) {
 	if z := im.st.Zone(im.zone(iface)); z != nil && !z.External {
 		z.External = true
@@ -520,6 +604,11 @@ type parsed struct {
 	// name: a port forward's name, when the rule's comment won't do
 	// (one of a map's elements).
 	name string
+	// inJump: the rule came from a chain jumped to; jumpText is what the
+	// jumps to it match, as nftables text (none when it isn't known), put
+	// in front of the rule's own text if it stays raw.
+	inJump   bool
+	jumpText string
 }
 
 // mapUse is a lookup in a map: "dnat ip to tcp dport map @forwards" or
@@ -548,7 +637,11 @@ func parse(r nftRule) *parsed {
 	return p
 }
 
-func (p *parsed) bad(what string) { p.unsupported = append(p.unsupported, what) }
+func (p *parsed) bad(what string) {
+	if !contains(p.unsupported, what) {
+		p.unsupported = append(p.unsupported, what)
+	}
+}
 
 func (p *parsed) expr(k string, v json.RawMessage) {
 	switch k {
@@ -837,19 +930,24 @@ func (p *parsed) match(m match) {
 	if meta, ok := left["meta"]; ok {
 		var x struct{ Key string }
 		json.Unmarshal(meta, &x)
-		switch x.Key {
-		case "iifname":
+		switch {
+		case (x.Key == "iifname" || x.Key == "oifname") && anySet(right):
+			p.bad("meta " + x.Key + " in a named set")
+		case x.Key == "iifname" && len(p.inIf) > 0, x.Key == "oifname" && len(p.outIf) > 0:
+			// Both must hold; Housecarl's rule has one list: kept raw.
+			p.bad("two matches on meta " + x.Key)
+		case x.Key == "iifname":
 			p.inIf, p.notIn = right, not
-		case "oifname":
+		case x.Key == "oifname":
 			p.outIf, p.notOut = right, not
-		case "nfproto":
-			if len(right) == 1 && !not {
+		case x.Key == "nfproto":
+			if len(right) == 1 && !not && (p.family == "" || p.family == right[0]) {
 				p.family = right[0]
 			} else {
 				p.bad("meta nfproto")
 			}
-		case "l4proto":
-			if len(right) == 1 && !not {
+		case x.Key == "l4proto":
+			if len(right) == 1 && !not && (p.proto == "" || p.proto == right[0]) {
 				p.proto = right[0]
 			} else {
 				p.bad("meta l4proto with several protocols")
@@ -862,23 +960,29 @@ func (p *parsed) match(m match) {
 	if pl, ok := left["payload"]; ok {
 		var x struct{ Protocol, Field string }
 		json.Unmarshal(pl, &x)
+		fam := map[string]string{"ip": "ipv4", "ip6": "ipv6"}[x.Protocol]
 		switch {
+		// A second match on a field Housecarl keeps one list for (both
+		// must hold): the rule stays raw, as written.
+		case x.Field == "saddr" && len(p.src) > 0, x.Field == "daddr" && len(p.dst) > 0,
+			x.Field == "dport" && len(p.dports) > 0, x.Field == "type" && len(p.icmp) > 0:
+			p.bad("two matches on " + x.Protocol + " " + x.Field)
+		case fam != "" && p.family != "" && p.family != fam && (x.Field == "saddr" || x.Field == "daddr"):
+			p.bad("both IPv4 and IPv6 addresses")
 		case (x.Protocol == "ip" || x.Protocol == "ip6") && (x.Field == "saddr" || x.Field == "daddr"):
-			if p.family == "" {
-				p.family = map[string]string{"ip": "ipv4", "ip6": "ipv6"}[x.Protocol]
-			}
+			p.family = fam
 			if x.Field == "saddr" {
 				p.src, p.notSrc = right, not
 			} else {
 				p.dst, p.notDst = right, not
 			}
-		case (x.Protocol == "tcp" || x.Protocol == "udp" || x.Protocol == "sctp") && x.Field == "dport" && !not:
+		case (x.Protocol == "tcp" || x.Protocol == "udp" || x.Protocol == "sctp") && x.Field == "dport" && !not && (p.proto == "" || p.proto == x.Protocol):
 			p.proto, p.dports = x.Protocol, right
-		case x.Protocol == "tcp" && x.Field == "flags" && len(right) == 1 && right[0] == "syn":
+		case x.Protocol == "tcp" && x.Field == "flags" && len(right) == 1 && right[0] == "syn" && !not:
 			p.tcpSyn = true
 		case (x.Protocol == "icmp" || x.Protocol == "icmpv6") && x.Field == "type" && !not:
 			p.icmpProto, p.icmp = x.Protocol, right
-		case x.Protocol == "ip6" && x.Field == "hoplimit" && len(right) == 1 && right[0] == "255":
+		case x.Protocol == "ip6" && x.Field == "hoplimit" && len(right) == 1 && right[0] == "255" && !not:
 			p.hop255 = true
 		default:
 			p.bad(x.Protocol + " " + x.Field)
@@ -889,9 +993,9 @@ func (p *parsed) match(m match) {
 		var x struct{ Key string }
 		json.Unmarshal(ct, &x)
 		switch {
-		case x.Key == "state" && !not:
+		case x.Key == "state" && !not && len(p.ctState) == 0:
 			p.ctState = right
-		case x.Key == "status" && !not:
+		case x.Key == "status" && !not && len(p.ctStatus) == 0:
 			p.ctStatus = right
 		default:
 			p.bad("ct " + x.Key)
@@ -903,43 +1007,85 @@ func (p *parsed) match(m match) {
 	}
 }
 
+// anySet reports whether a match's values name a set ("@name").
+func anySet(vals []string) bool {
+	for _, v := range vals {
+		if strings.HasPrefix(v, "@") {
+			return true
+		}
+	}
+	return false
+}
+
 // ---- converting
 
 func (im *importer) run() {
 	type base struct {
 		name, kind string
+		prio       int
 	}
 	var bases []base
-	for name, c := range im.chains {
+	names := make([]string, 0, len(im.chains))
+	for name := range im.chains {
+		names = append(names, name)
+	}
+	sort.Strings(names)
+	policies := map[string][]string{}
+	for _, name := range names {
+		c := im.chains[name]
 		switch {
 		case c.Type == "filter" && c.Hook == "prerouting":
-			bases = append(bases, base{name, model.ChainEarly})
+			bases = append(bases, base{name, model.ChainEarly, c.Prio})
+			if c.Prio > -200 {
+				im.note("chain %s hooks prerouting at priority %d, after connection tracking; Housecarl's early chain runs before it (raw), where connection states aren't known yet: check its rules", name, c.Prio)
+			}
 		case c.Type == "filter" && (c.Hook == "input" || c.Hook == "forward" || c.Hook == "output"):
-			bases = append(bases, base{name, c.Hook})
+			bases = append(bases, base{name, c.Hook, c.Prio})
+			policies[c.Hook] = append(policies[c.Hook], c.Policy)
 		case c.Type == "nat" && c.Hook == "prerouting":
-			bases = append(bases, base{name, "dstnat"})
+			bases = append(bases, base{name, "dstnat", c.Prio})
 		case c.Type == "nat" && c.Hook == "postrouting":
-			bases = append(bases, base{name, "srcnat"})
+			bases = append(bases, base{name, "srcnat", c.Prio})
 		case c.Hook != "":
 			im.note("chain %s (%s %s) isn't imported: Housecarl doesn't manage that hook", name, c.Type, c.Hook)
 		}
+	}
+	// Several chains on one hook: a packet must get past each (an accept
+	// ends only its own chain), and any policy drop drops it.
 	set := &im.st.Settings
-		switch {
-		case c.Hook == "input" && c.Type == "filter":
-			set.InputPolicy = c.Policy
-		case c.Hook == "forward" && c.Type == "filter":
-			set.ForwardPolicy = c.Policy
-		case c.Hook == "output" && c.Type == "filter":
-			set.OutputPolicy = c.Policy
+	for hook, ps := range policies {
+		policy := model.PolicyAccept
+		if contains(ps, model.PolicyDrop) {
+			policy = model.PolicyDrop
+		}
+		switch hook {
+		case "input":
+			set.InputPolicy = policy
+		case "forward":
+			set.ForwardPolicy = policy
+		case "output":
+			set.OutputPolicy = policy
 		}
 	}
 	// NAT first: the filter chains' port-forward accepts are folded into
-	// the port forwards.
+	// the port forwards. Chains on the same hook in priority order.
 	order := map[string]int{"dstnat": 0, "srcnat": 1, model.ChainEarly: 2, model.ChainInput: 3, model.ChainForward: 4, model.ChainOutput: 5}
-	sort.SliceStable(bases, func(i, j int) bool { return order[bases[i].kind] < order[bases[j].kind] })
+	sort.SliceStable(bases, func(i, j int) bool {
+		if bases[i].kind != bases[j].kind {
+			return order[bases[i].kind] < order[bases[j].kind]
+		}
+		return bases[i].prio < bases[j].prio
+	})
+	for i := 1; i < len(bases); i++ {
+		if b, a := bases[i], bases[i-1]; b.kind == a.kind && b.kind != "dstnat" && b.kind != "srcnat" {
+			im.note("chains %s and %s both hook %s: a packet must get past each, but Housecarl has one %s chain, so their rules follow each other (by priority); that only does the same if what %s accepts, %s would accept too: check it", a.name, b.name, b.kind, b.kind, a.name, b.name)
+		}
+	}
 	im.st.Settings.Builtins = model.Builtins{}
-	found := map[string]map[string]string{} // builtin → chain → "prefix" or "later"
-	var deferred []func()
+	found := map[string]map[string]string{} // builtin → chain → "prefix", "later" or "demoted"
+	heads := map[string][]headRule{}        // chain → the built-ins its rules start with, in order
+	var chains []string
+	rest := map[string][]*parsed{} // chain → the rules after its head
 	for _, b := range bases {
 		rules := im.flatten(b.name, nil, 0)
 		switch b.kind {
@@ -952,9 +1098,13 @@ func (im *importer) run() {
 				im.srcnat(p)
 			}
 		default:
-			prefix := true
+			if !contains(chains, b.kind) {
+				chains = append(chains, b.kind)
+			}
+			// Built-ins come first only at the head of the hook's first
+			// chain (or of one after a chain of nothing but built-ins).
+			prefix := len(rest[b.kind]) == 0
 			for _, p := range rules {
-				p := p
 				if bi := im.builtinOf(b.kind, p); bi != "" {
 					if found[bi] == nil {
 						found[bi] = map[string]string{}
@@ -965,18 +1115,25 @@ func (im *importer) run() {
 					}
 					found[bi][b.kind] = where
 					if prefix {
+						heads[b.kind] = append(heads[b.kind], headRule{bi, p})
 						continue
 					}
 				}
 				prefix = false
-				kind := b.kind
-				deferred = append(deferred, func() { im.rule(kind, p) })
+				rest[b.kind] = append(rest[b.kind], p)
 			}
 		}
 	}
-	im.settleBuiltins(found)
-	for _, f := range deferred {
-		f()
+	im.settleBuiltins(found, heads)
+	for _, chain := range chains {
+		for _, h := range heads[chain] {
+			if found[h.builtin][chain] == "demoted" {
+				im.rule(chain, h.p)
+			}
+		}
+		for _, p := range rest[chain] {
+			im.rule(chain, p)
+		}
 	}
 	if len(im.flow) > 0 {
 		im.st.Settings.FlowtableDevices = im.flow[0].Dev
@@ -1005,6 +1162,7 @@ func (im *importer) flatten(chain string, prefix *parsed, depth int) []*parsed {
 			}
 			jumpMatch := *p
 			jumpMatch.verdict, jumpMatch.target = "", ""
+			jumpMatch.jumpText = joinText(p.jumpText, im.condText(p))
 			out = append(out, im.flatten(p.target, &jumpMatch, depth+1)...)
 			if p.verdict == "goto" {
 				im.note("the goto to %s became the same rules followed by the rest of %s: check the order", p.target, chain)
@@ -1136,36 +1294,153 @@ func hasMatch(p, prefix *parsed) bool {
 		len(p.inIf) != len(prefix.inIf) || len(p.outIf) != len(prefix.outIf) || len(p.ctState) != len(prefix.ctState) || len(p.icmp) != len(prefix.icmp)
 }
 
-// merge puts a jump's matches in front of a jumped-to rule's. When both
-// match the same field, the rule's own (narrower, usually) is kept and
-// noted.
+// condText is a jump rule's text without its verdict, counter and
+// comment: what it matches ("ip saddr 10.50.50.0/24"); "" when there is
+// no text for it.
+var verdictRe = regexp.MustCompile(`\s*\b(jump|goto)\s+\S+\s*$`)
+
+func (im *importer) condText(p *parsed) string {
+	text, ok := im.text[p.handle]
+	if !ok {
+		return ""
+	}
+	if i := strings.LastIndex(text, ` comment "`); i >= 0 {
+		text = text[:i]
+	}
+	text = verdictRe.ReplaceAllString(strings.TrimSpace(text), "")
+	return strings.Join(strings.Fields(strings.ReplaceAll(" "+text+" ", " counter ", " ")), " ")
+}
+
+func joinText(a, b string) string {
+	return strings.TrimSpace(a + " " + b)
+}
+
+// merge puts a jump's matches in front of a jumped-to rule's: the rule
+// applies where both match. When both match the same field, that holds as
+// one list only when one's values lie within the other's (the narrower is
+// kept); otherwise the rule is kept raw, after the jump's text.
 func (im *importer) merge(prefix, p *parsed) *parsed {
 	q := *p
+	q.inJump, q.jumpText = true, prefix.jumpText
 	q.unsupported = append(append([]string{}, prefix.unsupported...), p.unsupported...)
-	fill := func(what string, dst *[]string, not *bool, src []string, srcNot bool) {
-		if len(*dst) == 0 && len(src) > 0 {
+	both := func(what string) { q.bad("both the jump to it and the rule match " + what) }
+	addrs := func(what string, dst *[]string, not *bool, src []string, srcNot bool) {
+		switch {
+		case len(src) == 0:
+		case len(*dst) == 0:
 			*dst, *not = src, srcNot
-		} else if len(*dst) > 0 && len(src) > 0 && key(*dst) != key(src) {
-			im.note("rule handle %d: both the jump to it and the rule match %s; the rule's own (%s) are kept, the jump's (%s) dropped: check it",
-				p.handle, what, strings.Join(*dst, ", "), strings.Join(src, ", "))
+		case *not || srcNot:
+			if key(*dst) != key(src) || *not != srcNot {
+				both(what)
+			}
+		case im.within(*dst, src):
+			// The rule's own are narrower: they say it all.
+		case im.within(src, *dst):
+			*dst = src
+		default:
+			both(what)
 		}
 	}
-	fill("interfaces in", &q.inIf, &q.notIn, prefix.inIf, prefix.notIn)
-	fill("interfaces out", &q.outIf, &q.notOut, prefix.outIf, prefix.notOut)
-	fill("sources", &q.src, &q.notSrc, prefix.src, prefix.notSrc)
-	fill("destinations", &q.dst, &q.notDst, prefix.dst, prefix.notDst)
-	if q.family == "" {
+	same := func(what string, dst *[]string, not *bool, src []string, srcNot bool) {
+		switch {
+		case len(src) == 0:
+		case len(*dst) == 0:
+			*dst, *not = src, srcNot
+		case key(*dst) != key(src) || *not != srcNot:
+			both(what)
+		}
+	}
+	var noNot bool
+	same("interfaces in", &q.inIf, &q.notIn, prefix.inIf, prefix.notIn)
+	same("interfaces out", &q.outIf, &q.notOut, prefix.outIf, prefix.notOut)
+	addrs("sources", &q.src, &q.notSrc, prefix.src, prefix.notSrc)
+	addrs("destinations", &q.dst, &q.notDst, prefix.dst, prefix.notDst)
+	same("connection states", &q.ctState, &noNot, prefix.ctState, false)
+	same("connection status", &q.ctStatus, &noNot, prefix.ctStatus, false)
+	switch {
+	case prefix.family == "":
+	case q.family == "":
 		q.family = prefix.family
+	case q.family != prefix.family:
+		both("an address family")
 	}
-	if len(q.ctState) == 0 {
-		q.ctState = prefix.ctState
-	}
-	if q.proto == "" {
+	switch {
+	case prefix.proto == "":
+	case q.proto == "":
 		q.proto, q.dports = prefix.proto, prefix.dports
+	case q.proto != prefix.proto:
+		both("a protocol")
+	default:
+		same("ports", &q.dports, &noNot, prefix.dports, false)
+	}
+	switch {
+	case prefix.icmpProto == "":
+	case q.icmpProto == "":
+		q.icmpProto, q.icmp = prefix.icmpProto, prefix.icmp
+	case q.icmpProto != prefix.icmpProto || key(q.icmp) != key(prefix.icmp):
+		both("ICMP types")
+	}
+	q.hop255 = q.hop255 || prefix.hop255
+	q.tcpSyn = q.tcpSyn || prefix.tcpSyn
+	if prefix.limit != nil || prefix.quota != nil || prefix.connLimit > 0 || prefix.log {
+		// Once in the jump, these count or log each packet once; on every
+		// rule they would act once per rule.
+		q.bad("a jump with a limit, quota or log")
 	}
 	return &q
 }
 
+// within reports whether every address of inner lies within outer's
+// (literal addresses and prefixes, or sets of the table).
+func (im *importer) within(inner, outer []string) bool {
+	resolve := func(list []string) ([]netip.Prefix, bool) {
+		var out []netip.Prefix
+		for _, e := range list {
+			vals := []string{e}
+			if strings.HasPrefix(e, "@") {
+				s, ok := im.sets[e[1:]]
+				if !ok {
+					return nil, false
+				}
+				vals = nil
+				for _, el := range s.Elem {
+					vals = append(vals, values(el)...)
+				}
+			}
+			for _, v := range vals {
+				pf, err := netip.ParsePrefix(v)
+				if err != nil {
+					a, err := netip.ParseAddr(v)
+					if err != nil {
+						return nil, false // a range, or something else
+					}
+					pf = netip.PrefixFrom(a, a.BitLen())
+				}
+				out = append(out, pf.Masked())
+			}
+		}
+		return out, len(out) > 0
+	}
+	in, ok1 := resolve(inner)
+	out, ok2 := resolve(outer)
+	if !ok1 || !ok2 {
+		return false
+	}
+	for _, a := range in {
+		found := false
+		for _, b := range out {
+			if b.Bits() <= a.Bits() && b.Contains(a.Addr()) {
+				found = true
+				break
+			}
+		}
+		if !found {
+			return false
+		}
+	}
+	return true
+}
+
 // ---- built-ins
 
 var essentials6 = map[string]bool{"destination-unreachable": true, "packet-too-big": true, "time-exceeded": true, "parameter-problem": true, "echo-request": true, "echo-reply": true}
@@ -1237,10 +1512,20 @@ func (im *importer) builtinOf(chain string, p *parsed) string {
 	return ""
 }
 
+// headRule is a rule at the head of a chain that is a built-in.
+type headRule struct {
+	builtin string
+	p       *parsed
+}
+
 // settleBuiltins turns on the built-ins found where they go, in every
 // chain they apply to; the others stay off, and the rules that looked
-// like them are kept as rules.
-func (im *importer) settleBuiltins(found map[string]map[string]string) {
+// like them are kept as rules. Built-ins come before every rule, so once
+// one at a chain's head stays a rule, the ones after it there stay rules
+// too ("demoted"), in their order: an accept moved ahead of a drop (ICMPv6
+// ahead of dropping invalid packets) would let through what the drop
+// stopped.
+func (im *importer) settleBuiltins(found map[string]map[string]string, heads map[string][]headRule) {
 	b := &im.st.Settings.Builtins
 	on := func(name string, chains ...string) bool {
 		for _, c := range chains {
@@ -1250,6 +1535,7 @@ func (im *importer) settleBuiltins(found map[string]map[string]string) {
 		}
 		return true
 	}
+	for {
 		b.Loopback = on("loopback", model.ChainInput)
 		b.Established = on("established", model.ChainInput, model.ChainForward)
 		b.DropInvalid = on("invalid", model.ChainInput, model.ChainForward)
@@ -1258,6 +1544,27 @@ func (im *importer) settleBuiltins(found map[string]map[string]string) {
 		b.DHCPv6Client = on("dhcpv6-client", model.ChainInput)
 		b.MSSClamp = on("mss-clamp", model.ChainForward)
 		b.Antispoof = on("antispoof", model.ChainEarly)
+		kept := map[string]bool{"loopback": b.Loopback, "established": b.Established, "invalid": b.DropInvalid, "icmpv6": b.ICMPv6, "icmpv6-nd": b.ICMPv6,
+			"ping": b.PingInternal, "dhcpv6-client": b.DHCPv6Client, "mss-clamp": b.MSSClamp, "antispoof": b.Antispoof, "flowtable": true}
+		changed := false
+		for chain, hs := range heads {
+			cut := false
+			for _, h := range hs {
+				// mss-clamp and flowtable decide nothing: their place
+				// doesn't matter.
+				if !kept[h.builtin] && h.builtin != "mss-clamp" && h.builtin != "flowtable" {
+					cut = true
+				}
+				if cut && found[h.builtin][chain] == "prefix" {
+					found[h.builtin][chain] = "demoted"
+					changed = true
+				}
+			}
+		}
+		if !changed {
+			break
+		}
+	}
 	if !b.Antispoof {
 		im.st.Settings.AntispoofV4, im.st.Settings.AntispoofV6, im.st.Settings.OwnPrefixes = nil, nil, nil
 	} else if im.st.Settings.AntispoofV6 == nil {
@@ -1266,40 +1573,16 @@ func (im *importer) settleBuiltins(found map[string]map[string]string) {
 	if b.ICMPv6 {
 		im.note("the ICMPv6 built-in also accepts MLD from link-local addresses, and lets ICMPv6 errors through the forward chain (RFC 4890)")
 	}
-	// The ones found in one chain's head but not another's: explicit rules
-	// in the chains where they were found.
-	for name, chains := range found {
-		for chain, where := range chains {
-			if where != "prefix" {
-				continue
-			}
-			kept := map[string]bool{"loopback": b.Loopback, "established": b.Established, "invalid": b.DropInvalid, "icmpv6": b.ICMPv6, "icmpv6-nd": b.ICMPv6,
-				"ping": b.PingInternal, "dhcpv6-client": b.DHCPv6Client, "mss-clamp": b.MSSClamp, "antispoof": b.Antispoof, "flowtable": true}[name]
-			if !kept {
-				im.note("%s in the %s chain stays a rule (it isn't in every chain the built-in covers, or comes after other rules)", name, chain)
-				im.builtinRule(name, chain)
+	// The ones that stay rules, in their place: say which.
+	for _, chain := range []string{model.ChainEarly, model.ChainInput, model.ChainForward, model.ChainOutput} {
+		for _, h := range heads[chain] {
+			if found[h.builtin][chain] == "demoted" {
+				im.note("%s in the %s chain stays a rule, in its place (the built-in isn't in every chain it covers, or a rule before it stays a rule)", h.builtin, chain)
 			}
 		}
 	}
 }
 
-// builtinRule writes a built-in that stays off as a rule of its chain.
-func (im *importer) builtinRule(name, chain string) {
-	r := &model.Rule{ID: im.newID(), Chain: chain, Position: im.nextPos(chain) - 1000}
-	switch name {
-	case "established":
-		r.CtState, r.Action = []string{"established", "related"}, model.ActionAccept
-	case "invalid":
-		r.CtState, r.Action = []string{"invalid"}, model.ActionDrop
-	case "loopback":
-		r.Raw, r.Action = `iifname "lo" accept`, model.ActionAccept
-	default:
-		im.note("built-in %s of the %s chain couldn't be kept; add it by hand", name, chain)
-		return
-	}
-	im.st.Rules = append(im.st.Rules, r)
-}
-
 func (im *importer) nextPos(chain string) float64 {
 	im.positions[chain]++
 	return im.positions[chain]
@@ -1383,8 +1666,19 @@ func (im *importer) zones(ifaces []string) []string {
 	return out
 }
 
+// checkSets marks a rule naming a set that wasn't imported (not of
+// addresses, or a meter) unsupported: it stays raw.
+func (im *importer) checkSets(p *parsed) {
+	for _, e := range append(append([]string{}, p.src...), p.dst...) {
+		if n, ok := strings.CutPrefix(e, "@"); ok && (im.meters[n] || im.st.Set(setName(n)) == nil) {
+			p.bad("the set @" + n + ", which isn't an address set Housecarl imported")
+		}
+	}
+}
+
 // toMatch builds a Match; ok is false when something can't be said.
 func (im *importer) toMatch(p *parsed) (model.Match, bool) {
+	im.checkSets(p)
 	svcs, ok := im.services(p)
 	if !ok || len(p.unsupported) > 0 {
 		return model.Match{}, false
@@ -1422,6 +1716,10 @@ func (im *importer) rule(chain string, p *parsed) {
 	default:
 		p.bad(p.verdict)
 	}
+	if p.log && !im.noted["log"] {
+		im.noted["log"] = true
+		im.note("rules that log now log to Housecarl's log (hc log, Traffic → Log), not the kernel log or the journal")
+	}
 	m, ok := im.toMatch(p)
 	if r.CountsTraffic() && r.Action == model.ActionAccept {
 		// Housecarl checks these before established connections are
@@ -1452,6 +1750,14 @@ func (im *importer) rawRule(r *model.Rule, p *parsed) {
 	if i := strings.LastIndex(text, ` comment "`); i >= 0 {
 		text = strings.TrimSpace(text[:i])
 	}
+	// From a chain jumped to: only where the jumps matched.
+	if p.inJump {
+		if p.jumpText == "" {
+			im.note("rule handle %d of chain %s couldn't be expressed (%s), and the jump to it has no text to keep it raw with: add it by hand: %s", p.handle, r.Chain, strings.Join(p.unsupported, ", "), text)
+			return
+		}
+		text = p.jumpText + " " + text
+	}
 	// A named map isn't carried over: a raw rule naming it would make
 	// nftables refuse the whole ruleset.
 	for _, ref := range mapRefRe.FindAllStringSubmatch(text, -1) {
@@ -1461,51 +1767,103 @@ func (im *importer) rawRule(r *model.Rule, p *parsed) {
 			return
 		}
 	}
+	text, bad := im.setRefs(text)
+	if bad != "" {
+		im.note("rule handle %d of chain %s uses the set @%s, which Housecarl's table doesn't have (%s): left out, add it by hand: %s",
+			p.handle, r.Chain, bad, strings.Join(p.unsupported, ", "), text)
+		return
+	}
 	if r.Action == "" {
 		r.Action = model.ActionAccept
 	}
-	r.Match, r.Raw = model.Match{}, text
+	// Only the text counts: what was parsed of it goes.
+	*r = model.Rule{ID: r.ID, Chain: r.Chain, Position: r.Position, Comment: r.Comment, Action: r.Action, Raw: text}
 	im.st.Rules = append(im.st.Rules, r)
 	why := strings.Join(p.unsupported, ", ")
 	if why == "" {
 		why = "a match Housecarl has no field for"
 	}
 	im.raw = append(im.raw, fmt.Sprintf("%s: %s (%s)", r.Chain, text, why))
-	if strings.Contains(text, "@") {
-		im.note("raw rule %q names a set of the old table: Housecarl's sets are NAME_v4 and NAME_v6", text)
+}
+
+var addrSetRefRe = regexp.MustCompile(`\b(ip6?) (saddr|daddr) (!= )?@([A-Za-z0-9_]+)`)
+
+// setRefs makes a raw rule's set references work in Housecarl's table:
+// an imported address set is NAME_v4 or NAME_v6 there, by the family the
+// rule matches; another set of the old table (interface names, ports) is
+// written out as an anonymous set of its elements. It returns the name of
+// a set it couldn't resolve (a meter, an empty or unknown one).
+func (im *importer) setRefs(text string) (string, string) {
+	text = addrSetRefRe.ReplaceAllStringFunc(text, func(m string) string {
+		sm := addrSetRefRe.FindStringSubmatch(m)
+		name := setName(sm[4])
+		if im.meters[sm[4]] || im.st.Set(name) == nil {
+			return m
+		}
+		return fmt.Sprintf("%s %s %s@%s", sm[1], sm[2], sm[3], render.SetName(name, sm[1] == "ip6"))
+	})
+	bad := ""
+	text = mapRefRe.ReplaceAllStringFunc(text, func(m string) string {
+		n := m[1:]
+		if strings.HasSuffix(n, "_v4") || strings.HasSuffix(n, "_v6") {
+			if im.st.Set(strings.TrimSuffix(strings.TrimSuffix(n, "_v4"), "_v6")) != nil {
+				return m // rewritten above
 			}
 		}
+		set, ok := im.sets[n]
+		if !ok || im.meters[n] || len(set.Elem) == 0 || contains(set.Flags, "timeout") || contains(set.Flags, "dynamic") {
+			bad = n
+			return m
+		}
+		var typ string
+		json.Unmarshal(set.Type, &typ)
+		var elems []string
+		for _, e := range set.Elem {
+			for _, v := range values(e) {
+				if typ == "ifname" || typ == "string" {
+					v = quote(v)
+				}
+				elems = append(elems, v)
+			}
+		}
+		return "{ " + strings.Join(elems, ", ") + " }"
+	})
+	return text, bad
+}
+
+func quote(s string) string { return `"` + strings.ReplaceAll(s, `"`, "") + `"` }
 
 // ---- NAT
 
 func (im *importer) dstnat(p *parsed) {
+	im.checkSets(p)
+	if a, err := netip.ParseAddr(p.dnatAddr); err == nil && a.Is6() {
+		p.bad("a DNAT to an IPv6 address (Housecarl's port forwards translate IPv4; IPv6 gets a pinhole)")
+	}
+	if p.notIn || len(p.outIf) > 0 || p.limit != nil || p.quota != nil || p.connLimit > 0 || p.log || len(p.ctState) > 0 && key(p.ctState) != "new" || len(p.icmp) > 0 {
+		p.bad("a DNAT with matches a port forward has no field for")
+	}
 	switch {
-	case p.dnatAddr != "" && len(p.unsupported) == 0 && p.proto != "" && len(p.dports) == 1 && !p.notDst && !p.notSrc:
-		ports, err := model.NormalizePorts(p.dports[0])
+	case p.dnatAddr != "" && len(p.unsupported) == 0 && p.proto != "" && len(p.dports) > 0 && !p.notDst && !p.notSrc:
+		var ports []string
+		for _, d := range p.dports {
+			n, err := model.NormalizePorts(d)
 			if err != nil {
+				p.bad("the port " + d)
 				break
 			}
-		f := &model.PortForward{Protocols: []string{p.proto}, Ports: ports, To: p.dnatAddr, ToPort: p.dnatPort, Comment: p.comment, Name: forwardName(p.comment)}
-		if p.name != "" {
-			f.Name = strings.ToLower(nameRe.ReplaceAllString(p.name, "-"))
+			ports = append(ports, n)
 		}
-		if f.ToPort == ports {
-			f.ToPort = ""
+		if len(p.unsupported) > 0 {
+			break
+		}
+		// "tcp dport { 80, 443 } dnat to X": a forward for each port.
+		for _, port := range ports {
+			im.portForward(p, port, len(ports) > 1)
 		}
-		f.PublicAddresses = im.addrRefs(p.dst)
-		f.Sources = im.addrRefs(p.src)
 		for _, i := range p.inIf {
 			im.external(i)
 		}
-		// The same forward for another protocol: one forward, both.
-		for _, o := range im.st.Forwards {
-			if o.Ports == f.Ports && o.To == f.To && o.ToPort == f.ToPort && key(o.PublicAddresses) == key(f.PublicAddresses) && key(o.Sources) == key(f.Sources) {
-				o.Protocols = append(o.Protocols, p.proto)
-				return
-			}
-		}
-		f.ID, f.Position = im.newID(), float64(len(im.st.Forwards)+1)
-		im.st.Forwards = append(im.st.Forwards, f)
 		return
 	case p.redirect != "" && p.redirect != "?" && len(p.unsupported) == 0:
 		m, ok := im.toMatch(p)
@@ -1514,7 +1872,35 @@ func (im *importer) dstnat(p *parsed) {
 			return
 		}
 	}
-	im.note("the dstnat rule handle %d couldn't be imported (%s): %s", p.handle, strings.Join(p.unsupported, ", "), im.text[p.handle])
+	im.note("the dstnat rule handle %d couldn't be imported (%s): add it by hand: %s", p.handle, strings.Join(p.unsupported, ", "), im.text[p.handle])
+}
+
+// portForward makes the port forward for one public port of a DNAT rule
+// (several: the port goes into its name).
+func (im *importer) portForward(p *parsed, ports string, several bool) {
+	f := &model.PortForward{Protocols: []string{p.proto}, Ports: ports, To: p.dnatAddr, ToPort: p.dnatPort, Comment: p.comment, Name: forwardName(p.comment)}
+	if p.name != "" {
+		f.Name = strings.ToLower(nameRe.ReplaceAllString(p.name, "-"))
+	}
+	if several && f.Name != "" {
+		f.Name += "-" + ports
+	}
+	if f.ToPort == ports {
+		f.ToPort = ""
+	}
+	f.PublicAddresses = im.addrRefs(p.dst)
+	f.Sources = im.addrRefs(p.src)
+	// The same forward for another protocol: one forward, both.
+	for _, o := range im.st.Forwards {
+		if o.Ports == f.Ports && o.To == f.To && o.ToPort == f.ToPort && key(o.PublicAddresses) == key(f.PublicAddresses) && key(o.Sources) == key(f.Sources) {
+			if !contains(o.Protocols, p.proto) {
+				o.Protocols = append(o.Protocols, p.proto)
+			}
+			return
+		}
+	}
+	f.ID, f.Position = im.newID(), float64(len(im.st.Forwards)+1)
+	im.st.Forwards = append(im.st.Forwards, f)
 }
 
 var forwardWords = regexp.MustCompile(`(?i)^\s*port\s*forward(ing)?\s*[:-]?\s*`)
@@ -1550,6 +1936,7 @@ func (im *importer) nameForwards() {
 }
 
 func (im *importer) srcnat(p *parsed) {
+	im.checkSets(p)
 	if len(p.unsupported) > 0 {
 		im.note("the srcnat rule handle %d couldn't be imported (%s): %s", p.handle, strings.Join(p.unsupported, ", "), im.text[p.handle])
 		return
@@ -1598,7 +1985,8 @@ func (im *importer) srcnat(p *parsed) {
 // absorbForward folds a port forward's own forward-chain accept (and an
 // IPv6 pinhole on its port) into the port forward.
 func (im *importer) absorbForward(p *parsed) bool {
-	if p.verdict != "accept" || len(p.unsupported) > 0 || len(p.dst) != 1 || p.notDst || len(p.dports) != 1 || p.proto == "" || len(p.src) > 0 || len(p.outIf) > 0 {
+	if p.verdict != "accept" || len(p.unsupported) > 0 || len(p.dst) != 1 || p.notDst || len(p.dports) != 1 || p.proto == "" || len(p.src) > 0 || len(p.outIf) > 0 ||
+		p.notIn || p.limit != nil || p.quota != nil || p.connLimit > 0 || p.log || len(p.ctState) > 0 && key(p.ctState) != "new" {
 		return false
 	}
 	port, err := model.NormalizePorts(p.dports[0])
@@ -1607,6 +1995,12 @@ func (im *importer) absorbForward(p *parsed) bool {
 	}
 	dst := p.dst[0]
 	if p.family == "ipv4" || !strings.Contains(dst, ":") {
+		// The forward's own accept takes only what was port forwarded: an
+		// accept of anything to that host and port stays a rule (inside
+		// clients reach the host directly through it).
+		if key(p.ctStatus) != "dnat" {
+			return false
+		}
 		for _, f := range im.st.Forwards {
 			inside := f.ToPort
 			if inside == "" {
@@ -1633,7 +2027,8 @@ func (im *importer) absorbForward(p *parsed) bool {
 			cands = append(cands, f)
 		}
 	}
-	if len(cands) == 1 && len(p.ctStatus) == 0 {
+	// The pinhole lets in from everywhere: only an accept that did too.
+	if len(cands) == 1 && len(p.ctStatus) == 0 && len(p.inIf) == 0 && len(p.ctState) == 0 {
 		cands[0].To6 = dst
 		im.note("the IPv6 rule to %s port %s became %s's IPv6 pinhole", dst, port, cands[0].Label())
 		return true

internal/importer/importer_test.go

@@ -50,11 +50,24 @@ func TestImportRouter(t *testing.T) {
 	if s := st.Set("masq_to_internet"); s == nil {
 		t.Errorf("no set masq_to_internet (sets %v)", setNames(st))
 	}
-	// Built-ins: the ones at the head of their chains.
+	// Built-ins: the ones at the head of their chains, up to the invalid
+	// drop, which stays a rule (below): the ICMP, ICMPv6 and DHCPv6
+	// accepts after it stay rules too, in their order, so nothing the
+	// drop stopped gets through ahead of it.
 	b := st.Settings.Builtins
-	if !b.Loopback || !b.Established || !b.ICMPv6 || !b.PingInternal || !b.DHCPv6Client || !b.MSSClamp || !b.Antispoof {
+	if !b.Loopback || !b.Established || b.ICMPv6 || b.PingInternal || b.DHCPv6Client || !b.MSSClamp || !b.Antispoof {
 		t.Errorf("built-ins %+v", b)
 	}
+	var input []string
+	for _, r := range st.Rules {
+		if r.Chain == model.ChainInput {
+			input = append(input, strings.Join(r.CtState, ",")+"|"+strings.Join(r.Services, ",")+"|"+r.Raw)
+		}
+	}
+	if len(input) < 5 || input[0] != "invalid||" || !strings.HasPrefix(input[1], "|icmp/echo-reply,icmp/echo-request|") ||
+		!strings.Contains(input[2], "icmpv6/packet-too-big") || !strings.Contains(input[3], "ip6 hoplimit 255") || !strings.Contains(input[4], "udp/546") {
+		t.Errorf("input rules start %q", input)
+	}
 	// The forward chain's asymmetric-routing rule sits between the
 	// established and invalid rules: invalid stays rules, in order.
 	if b.DropInvalid {
@@ -102,9 +115,16 @@ func TestImportRouter(t *testing.T) {
 	if !wgDNS {
 		t.Errorf("wg0_input not inlined")
 	}
-	// The jump's and the rule's sources both: the rule's are kept, noted.
-	if !strings.Contains(strings.Join(res.Notes, "\n"), "the rule's own (10.50.50.3, 10.50.50.6) are kept") {
-		t.Errorf("no note on the overlapping sources: %q", res.Notes)
+	// The jump's and the rule's sources both: the rule's lie within the
+	// jump's, so they say it all, without a note.
+	var wgSSH bool
+	for _, r := range st.Rules {
+		if r.Chain == model.ChainInput && strings.Join(r.Services, ",") == "tcp/22" && strings.Join(r.Sources, ",") == "10.50.50.3,10.50.50.6" {
+			wgSSH = true
+		}
+	}
+	if !wgSSH || strings.Contains(strings.Join(res.Notes, "\n"), "jump") {
+		t.Errorf("the WireGuard SSH rule: found %v, notes %q", wgSSH, res.Notes)
 	}
 	// The imported configuration is valid, and renders.
 	if ps := validate.Check(st, nil); validate.HasErrors(ps) {

internal/importer/testdata/audit.json

@@ -0,0 +1 @@
+{"nftables": [{"metainfo": {"version": "1.1.3", "release_name": "Commodore Bullmoose #4", "json_schema_version": 1}}, {"table": {"family": "inet", "name": "audit", "handle": 1}}, {"chain": {"family": "inet", "table": "audit", "name": "early", "handle": 1, "type": "filter", "hook": "input", "prio": -10, "policy": "accept"}}, {"chain": {"family": "inet", "table": "audit", "name": "input", "handle": 2, "type": "filter", "hook": "input", "prio": 0, "policy": "drop"}}, {"chain": {"family": "inet", "table": "audit", "name": "admin", "handle": 3}}, {"chain": {"family": "inet", "table": "audit", "name": "forward", "handle": 4, "type": "filter", "hook": "forward", "prio": 0, "policy": "drop"}}, {"chain": {"family": "inet", "table": "audit", "name": "prerouting", "handle": 5, "type": "nat", "hook": "prerouting", "prio": -100, "policy": "accept"}}, {"set": {"family": "inet", "name": "blocklist", "table": "audit", "type": "ipv4_addr", "handle": 6, "flags": ["timeout"], "timeout": 3600}}, {"set": {"family": "inet", "name": "lans", "table": "audit", "type": "ifname", "handle": 7, "elem": ["enp2s0", "enp3s0"]}}, {"rule": {"family": "inet", "table": "audit", "chain": "early", "handle": 8, "comment": "an early input chain", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": "192.0.2.66"}}, {"drop": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 9, "expr": [{"match": {"op": "in", "left": {"ct": {"key": "state"}}, "right": ["established", "related"]}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 10, "comment": "dynamic", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": "@blocklist"}}, {"drop": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 11, "comment": "an interface set", "expr": [{"match": {"op": "==", "left": {"meta": {"key": "iifname"}}, "right": "@lans"}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 12, "comment": "two source matches", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": {"prefix": {"addr": "10.0.0.0", "len": 8}}}}, {"match": {"op": "!=", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": {"prefix": {"addr": "10.0.10.0", "len": 24}}}}, {"match": {"op": "==", "left": {"payload": {"protocol": "tcp", "field": "dport"}}, "right": 22}}, {"drop": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 13, "comment": "hop limit not 255", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "icmpv6", "field": "type"}}, "right": "nd-neighbor-solicit"}}, {"match": {"op": "!=", "left": {"payload": {"protocol": "ip6", "field": "hoplimit"}}, "right": 255}}, {"drop": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 14, "comment": "DNS from the IoT VLAN", "expr": [{"match": {"op": "==", "left": {"meta": {"key": "iifname"}}, "right": "enp2s0.20"}}, {"match": {"op": "==", "left": {"payload": {"protocol": "udp", "field": "dport"}}, "right": 53}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 15, "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": {"prefix": {"addr": "10.9.0.0", "len": 24}}}}, {"jump": {"target": "admin"}}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "admin", "handle": 16, "comment": "within the jump", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": "10.9.0.5"}}, {"match": {"op": "==", "left": {"payload": {"protocol": "tcp", "field": "dport"}}, "right": 22}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "admin", "handle": 17, "comment": "outside the jump", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": "10.8.0.1"}}, {"match": {"op": "==", "left": {"payload": {"protocol": "tcp", "field": "dport"}}, "right": 23}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "admin", "handle": 18, "comment": "raw under the jump", "expr": [{"match": {"op": "==", "left": {"meta": {"key": "mark"}}, "right": 1}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "forward", "handle": 19, "expr": [{"match": {"op": "in", "left": {"ct": {"key": "state"}}, "right": ["established", "related"]}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "forward", "handle": 20, "comment": "not only forwarded", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "daddr"}}, "right": "10.0.10.37"}}, {"match": {"op": "==", "left": {"payload": {"protocol": "tcp", "field": "dport"}}, "right": 443}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "forward", "hanLine truncated

internal/importer/testdata/audit.nft

@@ -0,0 +1,53 @@
+#! /sbin/nft -f
+# Cases the importer must not get wrong (importer_test.go TestImportAudit).
+
+define wan_if = "wan"
+define lan_if = "enp2s0"
+define iot_if = "enp2s0.20"
+define admins = { 10.9.0.0/24 }
+
+table inet audit {
+	set blocklist {
+		type ipv4_addr
+		flags timeout
+		timeout 1h
+	}
+	set lans {
+		type ifname
+		elements = { "enp2s0", "enp3s0" }
+	}
+
+	chain early {
+		type filter hook input priority -10; policy accept;
+		ip saddr 192.0.2.66 drop comment "an early input chain"
+	}
+
+	chain input {
+		type filter hook input priority filter; policy drop;
+		ct state established,related accept
+		ip saddr @blocklist drop comment "dynamic"
+		iifname @lans accept comment "an interface set"
+		ip saddr 10.0.0.0/8 ip saddr != 10.0.10.0/24 tcp dport 22 drop comment "two source matches"
+		icmpv6 type nd-neighbor-solicit ip6 hoplimit != 255 drop comment "hop limit not 255"
+		iifname $iot_if udp dport 53 accept comment "DNS from the IoT VLAN"
+		ip saddr $admins jump admin
+	}
+
+	chain admin {
+		ip saddr 10.9.0.5 tcp dport 22 accept comment "within the jump"
+		ip saddr 10.8.0.1 tcp dport 23 accept comment "outside the jump"
+		meta mark 0x00000001 accept comment "raw under the jump"
+	}
+
+	chain forward {
+		type filter hook forward priority filter; policy drop;
+		ct state established,related accept
+		ip daddr 10.0.10.37 tcp dport 443 accept comment "not only forwarded"
+		iifname $lan_if oifname $wan_if accept
+	}
+
+	chain prerouting {
+		type nat hook prerouting priority dstnat; policy accept;
+		ip daddr 203.0.113.1 tcp dport { 80, 443 } dnat ip to 10.0.10.37 comment "web"
+	}
+}

internal/importer/testdata/audit.txt

@@ -0,0 +1,46 @@
+table inet audit { # handle 1
+	set blocklist { # handle 6
+		type ipv4_addr
+		timeout 1h
+	}
+
+	set lans { # handle 7
+		type ifname
+		elements = { "enp2s0",
+			     "enp3s0" }
+	}
+
+	chain early { # handle 1
+		type filter hook input priority filter - 10; policy accept;
+		ip saddr 192.0.2.66 drop comment "an early input chain" # handle 8
+	}
+
+	chain input { # handle 2
+		type filter hook input priority filter; policy drop;
+		ct state established,related accept # handle 9
+		ip saddr @blocklist drop comment "dynamic" # handle 10
+		iifname @lans accept comment "an interface set" # handle 11
+		ip saddr 10.0.0.0/8 ip saddr != 10.0.10.0/24 tcp dport 22 drop comment "two source matches" # handle 12
+		icmpv6 type nd-neighbor-solicit ip6 hoplimit != 255 drop comment "hop limit not 255" # handle 13
+		iifname "enp2s0.20" udp dport 53 accept comment "DNS from the IoT VLAN" # handle 14
+		ip saddr 10.9.0.0/24 jump admin # handle 15
+	}
+
+	chain admin { # handle 3
+		ip saddr 10.9.0.5 tcp dport 22 accept comment "within the jump" # handle 16
+		ip saddr 10.8.0.1 tcp dport 23 accept comment "outside the jump" # handle 17
+		meta mark 0x00000001 accept comment "raw under the jump" # handle 18
+	}
+
+	chain forward { # handle 4
+		type filter hook forward priority filter; policy drop;
+		ct state established,related accept # handle 19
+		ip daddr 10.0.10.37 tcp dport 443 accept comment "not only forwarded" # handle 20
+		iifname "enp2s0" oifname "wan" accept # handle 21
+	}
+
+	chain prerouting { # handle 5
+		type nat hook prerouting priority dstnat; policy accept;
+		ip daddr 203.0.113.1 tcp dport { 80, 443 } dnat ip to 10.0.10.37 comment "web" # handle 23
+	}
+}

internal/linkwatch/linkwatch.go

@@ -0,0 +1,96 @@
+// Package linkwatch tells when the machine's network interfaces change:
+// one appears, goes, or comes back (ppp0 when PPPoE reconnects). It
+// listens to the kernel's rtnetlink link messages and calls back once
+// they have been quiet for a moment, since one change brings several.
+package linkwatch
+
+import (
+	"context"
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"os"
+	"time"
+
+	"golang.org/x/sys/unix"
+)
+
+// Settle is how long the link messages must be quiet before the callback.
+const Settle = 2 * time.Second
+
+// Watch calls changed after interfaces changed, until ctx ends. It returns
+// an error only if it can't listen at all.
+func Watch(ctx context.Context, changed func()) error {
+	fd, err := unix.Socket(unix.AF_NETLINK, unix.SOCK_RAW|unix.SOCK_CLOEXEC|unix.SOCK_NONBLOCK, unix.NETLINK_ROUTE)
+	if err != nil {
+		return fmt.Errorf("opening a netlink socket: %w", err)
+	}
+	if err := unix.Bind(fd, &unix.SockaddrNetlink{Family: unix.AF_NETLINK, Groups: unix.RTMGRP_LINK}); err != nil {
+		unix.Close(fd)
+		return fmt.Errorf("listening to link changes: %w", err)
+	}
+	// A non-blocking descriptor goes through Go's poller, so closing the
+	// file ends a Read in progress.
+	f := os.NewFile(uintptr(fd), "rtnetlink")
+	go func() {
+		<-ctx.Done()
+		f.Close()
+	}()
+	events := make(chan struct{}, 1)
+	go func() {
+		buf := make([]byte, 64<<10)
+		for {
+			n, err := f.Read(buf)
+			switch {
+			case err != nil && ctx.Err() != nil:
+				close(events)
+				return
+			case errors.Is(err, unix.ENOBUFS):
+				// Messages were lost: something changed anyway.
+			case err != nil:
+				time.Sleep(time.Second)
+				continue
+			case !links(buf[:n]):
+				continue
+			}
+			select {
+			case events <- struct{}{}:
+			default:
+			}
+		}
+	}()
+	var timer <-chan time.Time
+	for {
+		select {
+		case _, ok := <-events:
+			if !ok {
+				return nil
+			}
+			timer = time.After(Settle)
+		case <-timer:
+			timer = nil
+			changed()
+		}
+	}
+}
+
+// links reports whether a netlink datagram holds a link being added,
+// changed or deleted.
+func links(b []byte) bool {
+	for len(b) >= unix.NLMSG_HDRLEN {
+		size := int(binary.NativeEndian.Uint32(b[0:4]))
+		typ := binary.NativeEndian.Uint16(b[4:6])
+		if size < unix.NLMSG_HDRLEN || size > len(b) {
+			return false
+		}
+		if typ == unix.RTM_NEWLINK || typ == unix.RTM_DELLINK {
+			return true
+		}
+		size = (size + unix.NLMSG_ALIGNTO - 1) &^ (unix.NLMSG_ALIGNTO - 1)
+		if size >= len(b) {
+			return false
+		}
+		b = b[size:]
+	}
+	return false
+}

internal/linkwatch/linkwatch_test.go

@@ -0,0 +1,36 @@
+package linkwatch
+
+import (
+	"encoding/binary"
+	"testing"
+
+	"golang.org/x/sys/unix"
+)
+
+// msg is a netlink message of a type, with a payload of n bytes.
+func msg(typ uint16, n int) []byte {
+	b := make([]byte, unix.NLMSG_HDRLEN+n)
+	binary.NativeEndian.PutUint32(b[0:4], uint32(len(b)))
+	binary.NativeEndian.PutUint16(b[4:6], typ)
+	return b
+}
+
+func TestLinks(t *testing.T) {
+	for _, c := range []struct {
+		name string
+		b    []byte
+		want bool
+	}{
+		{"a new link", msg(unix.RTM_NEWLINK, 16), true},
+		{"a link going", msg(unix.RTM_DELLINK, 16), true},
+		{"an address", msg(unix.RTM_NEWADDR, 16), false},
+		{"a link after an address (padded)", append(msg(unix.RTM_NEWADDR, 13), append(make([]byte, 3), msg(unix.RTM_NEWLINK, 8)...)...), true},
+		{"truncated", msg(unix.RTM_NEWLINK, 16)[:10], false},
+		{"a length beyond the datagram", msg(unix.RTM_NEWLINK, 16)[:unix.NLMSG_HDRLEN+4], false},
+		{"empty", nil, false},
+	} {
+		if got := links(c.b); got != c.want {
+			t.Errorf("%s: %v", c.name, got)
+		}
+	}
+}

internal/matrix/matrix.go

@@ -0,0 +1,546 @@
+// Package matrix works out what the rules do to new connections between
+// zones, for an overview: for each pair it walks the rules as nftables
+// would, in order. A rule that applies to the whole pair decides it; one
+// that applies to some of its traffic (some addresses, services, times,
+// rates…) is an exception, and the walk goes on; what is left falls to
+// the chain's policy.
+package matrix
+
+import (
+	"fmt"
+	"net/netip"
+	"regexp"
+	"strings"
+
+	"housecarl/internal/api"
+	"housecarl/internal/model"
+)
+
+// Firewall stands for the machine itself in the matrix.
+const Firewall = "firewall"
+
+// Options tell Build about the machine.
+type Options struct {
+	// Connected maps the networks directly connected to the firewall's
+	// interfaces to their zones. A rule whose addresses all lie in other
+	// zones' networks can't apply to a pair, and is left out of it.
+	Connected map[netip.Prefix]string
+}
+
+// Build works out the matrix of a configuration.
+func Build(st *model.State, opts ...Options) api.Matrix {
+	var o Options
+	if len(opts) > 0 {
+		o = opts[0]
+	}
+	m := api.Matrix{Zones: []string{}, Cells: []api.MatrixCell{}}
+	for _, z := range st.Zones {
+		m.Zones = append(m.Zones, z.Name)
+	}
+	for _, from := range m.Zones {
+		for _, to := range m.Zones {
+			m.Cells = append(m.Cells, cell(st, o, model.ChainForward, from, to))
+		}
+		m.Cells = append(m.Cells, cell(st, o, model.ChainInput, from, Firewall))
+	}
+	for _, to := range m.Zones {
+		m.Cells = append(m.Cells, cell(st, o, model.ChainOutput, Firewall, to))
+	}
+	return m
+}
+
+// Cell returns one pair (from or to may be Firewall).
+func Cell(st *model.State, o Options, from, to string) (api.MatrixCell, bool) {
+	known := func(z string) bool { return st.Zone(z) != nil }
+	switch {
+	case from == Firewall && known(to):
+		return cell(st, o, model.ChainOutput, from, to), true
+	case to == Firewall && known(from):
+		return cell(st, o, model.ChainInput, from, to), true
+	case known(from) && known(to):
+		return cell(st, o, model.ChainForward, from, to), true
+	}
+	return api.MatrixCell{}, false
+}
+
+func cell(st *model.State, o Options, chain, from, to string) api.MatrixCell {
+	c := walkCell(st, o, chain, from, to)
+	sum(&c)
+	if from == to && chain == model.ChainForward {
+		c.SameZone, c.Note = true, sameZoneNote(st.Zone(from))
+	}
+	return c
+}
+
+// sum fills in the overview: the summary, the exceptions counted by
+// verdict, and what the allowing ones let through.
+func sum(c *api.MatrixCell) {
+	c.Open = []string{}
+	seen := map[string]bool{}
+	for _, e := range c.Exceptions {
+		switch e.Verdict {
+		case "allowed":
+			c.Allowed++
+			if !seen[e.Short] {
+				seen[e.Short] = true
+				c.Open = append(c.Open, e.Short)
+			}
+		case "blocked":
+			c.Blocked++
+		}
+	}
+	c.Summary = c.Verdict
+	if c.Verdict == "blocked" && c.Allowed > 0 || c.Verdict == "allowed" && c.Blocked > 0 {
+		c.Summary = "partly"
+	}
+}
+
+// sameZoneNote says what of a zone's traffic to itself the firewall sees:
+// only what it routes between the zone's interfaces.
+func sameZoneNote(z *model.Zone) string {
+	if z == nil {
+		return ""
+	}
+	if len(z.Interfaces) == 1 && !strings.HasSuffix(z.Interfaces[0], "*") {
+		return fmt.Sprintf("traffic within %s stays on its own network: it passes the firewall only if routed back out of %s", z.Name, z.Interfaces[0])
+	}
+	return fmt.Sprintf("only traffic routed between %s's interfaces (%s) passes the firewall; within one of them it stays on its own network", z.Name, strings.Join(z.Interfaces, ", "))
+}
+
+func walkCell(st *model.State, o Options, chain, from, to string) api.MatrixCell {
+	c := api.MatrixCell{From: from, To: to, Chain: chain, Exceptions: []api.MatrixException{}}
+	b := st.Settings.Builtins
+	fromZone := st.Zone(from)
+	external := fromZone != nil && fromZone.External
+	// What comes in passes the early chain first.
+	if chain != model.ChainOutput {
+		if b.Antispoof && external {
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: model.ChainEarly, Label: "built-in: antispoof", Verdict: "blocked",
+				What: "sources that can't come from outside (private ranges, your own prefixes)", Short: "spoofed sources", Marks: []string{"b-antispoof", "b-antispoof6"}})
+		}
+		if decided := walk(st, o, &c, model.ChainEarly, from, ""); decided {
+			return c
+		}
+	}
+	// The built-ins that let some new connections in.
+	switch chain {
+	case model.ChainInput:
+		if b.ICMPv6 {
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: icmpv6", Verdict: "allowed", What: "ICMPv6 errors, echo, neighbour discovery, MLD",
+				Short: "ICMPv6", Marks: []string{"b-icmpv6", "b-icmpv6-nd", "b-icmpv6-mld"}})
+		}
+		if b.PingInternal && !external {
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: ping", Verdict: "allowed", What: "ping (ICMP echo)", Short: "ping", Marks: []string{"b-ping"}})
+		}
+		if b.DHCPv6Client && external {
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: dhcpv6-client", Verdict: "allowed", What: "DHCPv6 replies to the firewall",
+				Short: "DHCPv6", Marks: []string{"b-dhcpv6-client"}})
+		}
+	case model.ChainForward:
+		if marks := forwards(st); len(marks) > 0 && external && !st.Zone(to).External {
+			n := fmt.Sprintf("%d port forward%s", len(marks), map[bool]string{true: "", false: "s"}[len(marks) == 1])
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "port forwards", Verdict: "allowed",
+				What: n + " to inside hosts", Short: n, Marks: marks})
+		}
+	case model.ChainOutput:
+		if st.Settings.OutputPolicy == model.PolicyDrop && b.ICMPv6 {
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: icmpv6", Verdict: "allowed", What: "ICMPv6 errors, echo, neighbour discovery, MLD",
+				Short: "ICMPv6", Marks: []string{"b-icmpv6"}})
+		}
+	}
+	if walk(st, o, &c, chain, from, to) {
+		return c
+	}
+	policy := map[string]string{model.ChainInput: st.Settings.InputPolicy, model.ChainForward: st.Settings.ForwardPolicy, model.ChainOutput: st.Settings.OutputPolicy}[chain]
+	c.Verdict, c.DecidedBy = verdictOf(policy), chain+" policy"
+	if policy == model.PolicyDrop {
+		c.Marks = []string{"p-" + chain} // only a drop policy is counted
+	}
+	return c
+}
+
+// forwards returns the counters' marks of the port forwards that send
+// connections inside.
+func forwards(st *model.State) []string {
+	var marks []string
+	for _, f := range st.Forwards {
+		if !f.Disabled && f.To != "" {
+			marks = append(marks, fmt.Sprintf("f%d", f.ID))
+		}
+	}
+	return marks
+}
+
+// walk goes through a chain's rules for the pair, as nftables checks
+// them (those that check every packet first). It reports whether a rule
+// decided the whole pair.
+func walk(st *model.State, o Options, c *api.MatrixCell, chain, from, to string) bool {
+	var rules []*model.Rule
+	for _, every := range []bool{true, false} {
+		for _, r := range st.Rules {
+			if r.Chain == chain && !r.Disabled && r.ChecksEveryPacket() == every {
+				rules = append(rules, r)
+			}
+		}
+	}
+	for _, r := range rules {
+		mark := []string{fmt.Sprintf("r%d", r.ID)}
+		if r.Raw != "" {
+			rr := readRaw(r.Raw)
+			switch {
+			case !rr.ok:
+				c.Exceptions = append(c.Exceptions, api.MatrixException{RuleID: r.ID, Chain: chain, Label: "rule " + r.Label(), Verdict: "unknown", What: "a raw rule: " + r.Raw,
+					Short: "a raw rule", Marks: mark})
+			case !rr.forNew || chain == model.ChainEarly && rr.verdict == "allowed":
+				// Not for new connections; an early accept decides nothing.
+			case rr.match == "":
+				c.Verdict, c.DecidedBy, c.RuleID, c.Marks = rr.verdict, "rule "+r.Label(), r.ID, mark
+				return true
+			default:
+				c.Exceptions = append(c.Exceptions, api.MatrixException{RuleID: r.ID, Chain: chain, Label: "rule " + r.Label(), Verdict: rr.verdict, What: "raw: " + rr.match,
+					Short: rr.short, Marks: mark})
+			}
+			continue
+		}
+		if !zones(r, from, to) || !forNew(r) {
+			continue
+		}
+		// Addresses that all live behind other zones can't come from (or
+		// go to) this pair's.
+		if !r.NotSources && elsewhere(st, o, r.Sources, from) || !r.NotDests && to != Firewall && to != "" && elsewhere(st, o, r.Destinations, to) {
+			continue
+		}
+		if chain == model.ChainEarly && r.Action != model.ActionDrop && r.Action != model.ActionReject {
+			continue // an early accept decides nothing: input or forward still check
+		}
+		v := verdictOf(r.Action)
+		what := conditions(st, r)
+		if r.Action == model.ActionLog || what != "" {
+			if what == "" {
+				what = "everything"
+			}
+			c.Exceptions = append(c.Exceptions, api.MatrixException{RuleID: r.ID, Chain: chain, Label: "rule " + r.Label(), Verdict: v, What: what,
+				Short: short(r, what), Marks: mark})
+			continue
+		}
+		c.Verdict, c.DecidedBy, c.RuleID, c.Marks = v, "rule "+r.Label(), r.ID, mark
+		return true
+	}
+	return false
+}
+
+func verdictOf(action string) string {
+	switch action {
+	case model.ActionAccept: // and the accept policy
+		return "allowed"
+	case model.ActionLog:
+		return "logged"
+	}
+	return "blocked"
+}
+
+// elsewhere reports whether every address of refs (addresses, sets) lies
+// in networks connected to zones other than zone: then traffic of zone
+// can't have them. An empty list, a dynamic set, or an address the
+// firewall isn't directly connected to (a routed network) might.
+func elsewhere(st *model.State, o Options, refs []string, zone string) bool {
+	if len(refs) == 0 || len(o.Connected) == 0 || zone == Firewall {
+		return false
+	}
+	var entries []string
+	for _, r := range refs {
+		if s := st.Set(r); s != nil {
+			if s.Dynamic {
+				return false
+			}
+			entries = append(entries, s.Entries...)
+		} else {
+			entries = append(entries, r)
+		}
+	}
+	if len(entries) == 0 {
+		return false
+	}
+	for _, e := range entries {
+		a, err := model.ParseAddress(e)
+		if err != nil {
+			return false
+		}
+		first, last := a.Prefix.Masked().Addr(), lastAddr(a.Prefix)
+		if a.First.IsValid() {
+			first, last = a.First, a.End
+		}
+		owner := ""
+		for net, z := range o.Connected {
+			if net.Contains(first) && net.Contains(last) {
+				owner = z
+				break
+			}
+		}
+		if owner == "" || strings.EqualFold(owner, zone) {
+			return false
+		}
+	}
+	return true
+}
+
+// lastAddr is a prefix's last address.
+func lastAddr(p netip.Prefix) netip.Addr {
+	b := p.Masked().Addr().AsSlice()
+	for i := p.Bits(); i < len(b)*8; i++ {
+		b[i/8] |= 0x80 >> (i % 8)
+	}
+	a, _ := netip.AddrFromSlice(b)
+	return a
+}
+
+// zones reports whether a rule's zones take in the pair.
+func zones(r *model.Rule, from, to string) bool {
+	in := func(list []string, not bool, z string) bool {
+		if len(list) == 0 {
+			return true
+		}
+		for _, x := range list {
+			if strings.EqualFold(x, z) {
+				return !not
+			}
+		}
+		return not
+	}
+	if from == Firewall {
+		if len(r.InZones) > 0 {
+			return false
+		}
+	} else if !in(r.InZones, r.NotIn, from) {
+		return false
+	}
+	if to == "" || to == Firewall {
+		return len(r.OutZones) == 0
+	}
+	return in(r.OutZones, r.NotOut, to)
+}
+
+// forNew reports whether a rule can match a new connection's first
+// packet (its connection states allow "new").
+func forNew(r *model.Rule) bool {
+	if len(r.CtState) == 0 {
+		return true
+	}
+	for _, s := range r.CtState {
+		if s == "new" {
+			return true
+		}
+	}
+	return false
+}
+
+// raw is what the matrix can read of a raw rule.
+type raw struct {
+	ok      bool   // read: it applies to every pair of its chain
+	forNew  bool   // it can match a new connection
+	verdict string // allowed or blocked
+	match   string // what it matches ("" for everything)
+	short   string
+}
+
+// rawStops are words that make a raw rule's reach or effect unclear to
+// the matrix: interfaces (which zones?), jumps, NAT, changes, verdict maps.
+var rawStops = map[string]bool{
+	"iifname": true, "oifname": true, "iif": true, "oif": true, "iiftype": true, "oiftype": true,
+	"jump": true, "goto": true, "return": true, "vmap": true, "queue": true, "continue": true,
+	"dnat": true, "snat": true, "masquerade": true, "redirect": true, "tproxy": true,
+	"set": true, "notrack": true, "dup": true, "fwd": true, "synproxy": true, "flow": true,
+}
+
+// readRaw reads a raw rule that is a match and then accept, drop or
+// reject: the verdict is the last word (but a reject's "with …"); counter
+// and log only count and log.
+func readRaw(text string) raw {
+	toks := fields(text)
+	for i, t := range toks {
+		if t == "comment" {
+			toks = toks[:i]
+			break
+		}
+	}
+	verdict := -1
+	for i, t := range toks {
+		if strings.HasPrefix(t, `"`) {
+			continue
+		}
+		w := strings.Trim(t, ",{}")
+		if rawStops[w] {
+			return raw{}
+		}
+		if w == "accept" || w == "drop" || w == "reject" {
+			verdict = i
+		}
+	}
+	if verdict < 0 || verdict != len(toks)-1 && !(toks[verdict] == "reject" && toks[verdict+1] == "with") {
+		return raw{}
+	}
+	rr := raw{ok: true, forNew: true, verdict: verdictOf(toks[verdict])}
+	var match []string
+	for i := 0; i < verdict; i++ {
+		switch toks[i] {
+		case "counter":
+			continue
+		case "log":
+			// Its options run up to the next match.
+			for i+1 < verdict && map[string]bool{"prefix": true, "level": true, "group": true, "flags": true}[toks[i+1]] {
+				i += 2
+			}
+			continue
+		}
+		match = append(match, toks[i])
+	}
+	rr.match = strings.Join(match, " ")
+	// ct state without new: not for new connections. The states are one
+	// word ("established,related") or a set ("{ established, related }").
+	for i := 0; i+2 < len(match); i++ {
+		if match[i] != "ct" || match[i+1] != "state" {
+			continue
+		}
+		list := match[i+2]
+		if list == "{" || strings.HasPrefix(list, "{") {
+			var b []string
+			for j := i + 2; j < len(match); j++ {
+				b = append(b, match[j])
+				if strings.HasSuffix(match[j], "}") {
+					break
+				}
+			}
+			list = strings.Join(b, " ")
+		}
+		rr.forNew = false
+		for _, st := range strings.FieldsFunc(list, func(c rune) bool { return c == ',' || c == ' ' || c == '{' || c == '}' }) {
+			rr.forNew = rr.forNew || st == "new"
+		}
+	}
+	rr.short = rr.match
+	if m := icmpRe.FindStringSubmatch(rr.match); m != nil {
+		rr.short = m[1]
+	}
+	if len(rr.short) > 40 {
+		rr.short = rr.short[:39] + "…"
+	}
+	return rr
+}
+
+// fields splits a rule into words, a quoted string being one.
+func fields(s string) []string {
+	var out []string
+	var cur strings.Builder
+	quoted := false
+	for _, c := range s {
+		switch {
+		case c == '"':
+			quoted = !quoted
+			cur.WriteRune(c)
+		case !quoted && (c == ' ' || c == '\t'):
+			if cur.Len() > 0 {
+				out = append(out, cur.String())
+				cur.Reset()
+			}
+		default:
+			cur.WriteRune(c)
+		}
+	}
+	if cur.Len() > 0 {
+		out = append(out, cur.String())
+	}
+	return out
+}
+
+// icmpRe finds an ICMP type match in a raw rule: shown as just icmp or
+// icmpv6.
+var icmpRe = regexp.MustCompile(`\b(icmpv6|icmp) type\b`)
+
+// short sums up an exception for the overview: its services (ICMP types
+// as just icmp or icmpv6), with the addresses it is limited to, which
+// matter as much ("tcp/22 from trusted"); times and rates are left to
+// the details.
+func short(r *model.Rule, what string) string {
+	var parts []string
+	if len(r.Services) > 0 {
+		var svcs []string
+		seen := map[string]bool{}
+		for _, s := range r.Services {
+			if p, _, ok := strings.Cut(s, "/"); ok && (p == "icmp" || p == "icmpv6") {
+				s = p
+			}
+			if !seen[s] {
+				seen[s] = true
+				svcs = append(svcs, s)
+			}
+		}
+		parts = append(parts, strings.Join(svcs, ", "))
+	}
+	for _, c := range strings.Split(what, " · ") {
+		if strings.HasPrefix(c, "from ") || strings.HasPrefix(c, "to ") {
+			parts = append(parts, c)
+		}
+	}
+	if len(parts) == 0 {
+		first, _, _ := strings.Cut(what, " · ")
+		return first
+	}
+	return strings.Join(parts, " ")
+}
+
+// conditions says what limits a rule to part of a pair's traffic ("" for
+// none: it decides the pair).
+func conditions(st *model.State, r *model.Rule) string {
+	var parts []string
+	list := func(not bool, xs []string) string {
+		s := strings.Join(xs, ", ")
+		if not {
+			return "any but " + s
+		}
+		return s
+	}
+	if len(r.Sources) > 0 {
+		parts = append(parts, "from "+list(r.NotSources, r.Sources))
+	}
+	if len(r.Destinations) > 0 {
+		parts = append(parts, "to "+list(r.NotDests, r.Destinations))
+	}
+	if len(r.Services) > 0 {
+		parts = append(parts, strings.Join(r.Services, ", "))
+	}
+	if r.Family != "" {
+		parts = append(parts, r.Family+" only")
+	}
+	if len(r.CtStatus) > 0 {
+		parts = append(parts, "connections that were "+strings.Join(r.CtStatus, ", ")+"'d")
+	}
+	if len(r.Days) > 0 {
+		parts = append(parts, strings.Join(r.Days, ", "))
+	}
+	if r.HourFrom != "" {
+		parts = append(parts, r.HourFrom+"–"+r.HourTo)
+	}
+	if l := r.Limit; l != nil {
+		unit := ""
+		if l.Unit != "" && l.Unit != "packets" {
+			unit = " " + l.Unit
+		}
+		rate := fmt.Sprintf("%d%s/%s", l.Rate, unit, l.Per)
+		if l.Over {
+			parts = append(parts, "beyond "+rate)
+		} else {
+			parts = append(parts, "within "+rate)
+		}
+	}
+	if r.ConnLimit > 0 {
+		parts = append(parts, fmt.Sprintf("sources with over %d connections", r.ConnLimit))
+	}
+	if q := r.Quota; q != nil {
+		if q.Over {
+			parts = append(parts, "once its quota is used up")
+		} else {
+			parts = append(parts, "while its quota lasts")
+		}
+	}
+	return strings.Join(parts, " · ")
+}

internal/matrix/matrix_test.go

@@ -0,0 +1,199 @@
+package matrix
+
+import (
+	"fmt"
+	"net/netip"
+	"strings"
+	"testing"
+
+	"housecarl/internal/api"
+	"housecarl/internal/model"
+)
+
+func state() *model.State {
+	st := model.NewState()
+	st.Zones = []*model.Zone{
+		{ID: 1, Name: "guest", Interfaces: []string{"eth2"}},
+		{ID: 2, Name: "lan", Interfaces: []string{"eth1"}},
+		{ID: 3, Name: "wan", Interfaces: []string{"wan"}, External: true},
+	}
+	st.Sets = []*model.AddressSet{{ID: 4, Name: "admins", Entries: []string{"192.168.1.0/28"}}}
+	pos := 0.0
+	rule := func(r model.Rule) {
+		pos++
+		r.ID, r.Position = int64(10+pos), pos
+		st.Rules = append(st.Rules, &r)
+	}
+	rule(model.Rule{Chain: "input", Match: model.Match{Sources: []string{"admins"}, Services: []string{"ssh"}}, Action: "accept", Comment: "SSH"})
+	rule(model.Rule{Chain: "input", Match: model.Match{InZones: []string{"lan", "guest"}, Services: []string{"dns"}}, Action: "accept"})
+	rule(model.Rule{Chain: "forward", Match: model.Match{InZones: []string{"lan"}, OutZones: []string{"wan"}}, Action: "accept", Comment: "LAN out"})
+	rule(model.Rule{Chain: "forward", Match: model.Match{InZones: []string{"guest"}, OutZones: []string{"wan"}, Services: []string{"http", "https"}}, Action: "accept"})
+	rule(model.Rule{Chain: "forward", Match: model.Match{InZones: []string{"wan"}}, Action: "drop", Comment: "nothing in"})
+	rule(model.Rule{Chain: "forward", Match: model.Match{InZones: []string{"lan"}}, CtState: []string{"established"}, Action: "drop"}) // not for new
+	rule(model.Rule{Chain: "output", Match: model.Match{OutZones: []string{"wan"}}, Action: "drop", Disabled: true})
+	st.Forwards = []*model.PortForward{{ID: 30, Name: "web", Protocols: []string{"tcp"}, Ports: "443", To: "192.168.1.10"}}
+	st.Settings.Builtins.MSSClamp = true
+	return st
+}
+
+func find(m api.Matrix, from, to string) api.MatrixCell {
+	for _, c := range m.Cells {
+		if c.From == from && c.To == to {
+			return c
+		}
+	}
+	return api.MatrixCell{}
+}
+
+func TestMatrix(t *testing.T) {
+	m := Build(state())
+	if strings.Join(m.Zones, ",") != "guest,lan,wan" || len(m.Cells) != 3*3+3+3 {
+		t.Fatalf("zones %v, %d cells", m.Zones, len(m.Cells))
+	}
+	for _, c := range []struct {
+		from, to, verdict, decided string
+		exceptions                 int
+	}{
+		{"lan", "wan", "allowed", "rule forward #13 (LAN out)", 0},
+		{"guest", "wan", "blocked", "forward policy", 1},              // only web
+		{"wan", "lan", "blocked", "rule forward #15 (nothing in)", 2}, // antispoof, port forwards come first
+		{"lan", "guest", "blocked", "forward policy", 0},              // the established-only rule is no exception
+		{"lan", Firewall, "blocked", "input policy", 4},               // icmpv6, ping, ssh from admins, dns
+		{"wan", Firewall, "blocked", "input policy", 4},               // antispoof, icmpv6, dhcpv6, ssh from admins
+		{Firewall, "wan", "allowed", "output policy", 0},              // the drop is disabled
+	} {
+		got := find(m, c.from, c.to)
+		if got.Verdict != c.verdict || got.DecidedBy != c.decided || len(got.Exceptions) != c.exceptions {
+			t.Errorf("%s → %s: %s by %s, %d exceptions %+v; want %s by %s, %d", c.from, c.to, got.Verdict, got.DecidedBy, len(got.Exceptions), got.Exceptions, c.verdict, c.decided, c.exceptions)
+		}
+	}
+	g := find(m, "guest", "wan").Exceptions[0]
+	if g.Verdict != "allowed" || g.What != "http, https" || g.RuleID != 14 {
+		t.Errorf("the guest exception: %+v", g)
+	}
+	if e := find(m, "wan", "lan").Exceptions; e[0].Label != "built-in: antispoof" || e[1].Label != "port forwards" {
+		t.Errorf("wan → lan exceptions: %+v", e)
+	}
+	if _, ok := Cell(state(), Options{}, "lan", "nowhere"); ok {
+		t.Error("an unknown zone")
+	}
+}
+
+// Rules whose addresses all live behind another zone are left out of a
+// pair; routed (unknown) networks, and mixed lists, stay.
+func TestConnected(t *testing.T) {
+	st := state()
+	st.Sets = append(st.Sets, &model.AddressSet{ID: 5, Name: "guest_net", Entries: []string{"192.168.2.0/24"}},
+		&model.AddressSet{ID: 6, Name: "mixed", Entries: []string{"192.168.2.0/24", "10.20.30.0/24"}})
+	st.Rules = append(st.Rules,
+		&model.Rule{ID: 40, Chain: "forward", Position: 10, Match: model.Match{Sources: []string{"guest_net"}}, Action: "drop", Comment: "guests"},
+		&model.Rule{ID: 41, Chain: "forward", Position: 11, Match: model.Match{Destinations: []string{"192.168.2.5"}}, Action: "accept", Comment: "to a guest"},
+		&model.Rule{ID: 42, Chain: "forward", Position: 12, Match: model.Match{Sources: []string{"mixed"}}, Action: "drop", Comment: "mixed"})
+	o := Options{Connected: map[netip.Prefix]string{netip.MustParsePrefix("192.168.1.0/24"): "lan", netip.MustParsePrefix("192.168.2.0/24"): "guest"}}
+	ids := func(c api.MatrixCell) []int64 {
+		var out []int64
+		for _, e := range c.Exceptions {
+			out = append(out, e.RuleID)
+		}
+		return out
+	}
+	lanWan, _ := Cell(st, o, "lan", "wan")
+	if fmt.Sprint(ids(lanWan)) != "[]" {
+		t.Errorf("lan → wan: %v", ids(lanWan)) // decided by "LAN out" before the others anyway
+	}
+	lanGuest, _ := Cell(st, o, "lan", "guest")
+	if fmt.Sprint(ids(lanGuest)) != "[41 42]" {
+		t.Errorf("lan → guest: %v", ids(lanGuest)) // not "guests": their addresses are guest's
+	}
+	guestLan, _ := Cell(st, o, "guest", "lan")
+	if fmt.Sprint(ids(guestLan)) != "[40 42]" {
+		t.Errorf("guest → lan: %v", ids(guestLan)) // not "to a guest": it is in guest, not lan
+	}
+}
+
+// TestSummary: each pair at a glance, what its exceptions let through,
+// and the counters behind each line.
+func TestSummary(t *testing.T) {
+	st := state()
+	st.Settings.InputPolicy = model.PolicyDrop
+	m := Build(st)
+	for _, c := range []struct {
+		from, to, summary string
+		allowed, blocked  int
+		open              string
+	}{
+		{"lan", "wan", "allowed", 0, 0, ""},
+		{"guest", "wan", "partly", 1, 0, "http, https"},
+		{"wan", "lan", "partly", 1, 1, "1 port forward"},
+		{"lan", "guest", "blocked", 0, 0, ""},
+		{"lan", Firewall, "partly", 4, 0, "ICMPv6|ping|ssh from admins|dns"},
+	} {
+		got := find(m, c.from, c.to)
+		if got.Summary != c.summary || got.Allowed != c.allowed || got.Blocked != c.blocked || strings.Join(got.Open, "|") != c.open {
+			t.Errorf("%s → %s: %s +%d -%d %q; want %s +%d -%d %q", c.from, c.to, got.Summary, got.Allowed, got.Blocked, got.Open, c.summary, c.allowed, c.blocked, c.open)
+		}
+	}
+	if c := find(m, "lan", "lan"); !c.SameZone || !strings.Contains(c.Note, "out of eth1") {
+		t.Errorf("same zone: %+v", c)
+	}
+	if c := find(m, "lan", "wan"); c.SameZone || c.Note != "" {
+		t.Errorf("not the same zone: %+v", c)
+	}
+	wl := find(m, "wan", "lan")
+	if strings.Join(wl.Exceptions[0].Marks, ",") != "b-antispoof,b-antispoof6" || strings.Join(wl.Exceptions[1].Marks, ",") != "f30" || strings.Join(wl.Marks, ",") != "r15" {
+		t.Errorf("marks: %+v", wl)
+	}
+	icmp := &model.Rule{Match: model.Match{Services: []string{"icmp/echo-request", "icmp/echo-reply", "icmpv6/echo-request", "ssh"}}}
+	if s := short(icmp, "x"); s != "icmp, icmpv6, ssh" {
+		t.Errorf("short: %q", s)
+	}
+	ssh := &model.Rule{Match: model.Match{Sources: []string{"trusted"}, Services: []string{"ssh"}}, Days: []string{"mon"}}
+	if s := short(ssh, conditions(st, ssh)); s != "ssh from trusted" {
+		t.Errorf("short with a source: %q", s)
+	}
+	if s := short(&model.Rule{Days: []string{"mon"}}, "mon"); s != "mon" {
+		t.Errorf("short of a time: %q", s)
+	}
+	if c := find(m, "lan", Firewall); strings.Join(c.Marks, ",") != "p-input" {
+		t.Errorf("a drop policy's mark: %v", c.Marks)
+	}
+}
+
+// TestRaw: raw rules that are a match and a verdict are read; others stay
+// unknown.
+func TestRaw(t *testing.T) {
+	for _, c := range []struct {
+		raw, verdict, short string
+		ok, forNew          bool
+	}{
+		{"icmpv6 type { nd-router-solicit, nd-neighbor-advert } ip6 hoplimit 255 accept", "allowed", "icmpv6", true, true},
+		{`tcp dport 22 counter log prefix "ssh: " drop comment "x"`, "blocked", "tcp dport 22", true, true},
+		{"ip saddr 10.0.0.1 reject with icmpx type admin-prohibited", "blocked", "ip saddr 10.0.0.1", true, true},
+		{"ct state established,related accept", "allowed", "ct state established,related", true, false},
+		{"ct state { new, untracked } tcp dport 80 accept", "allowed", "ct state { new, untracked } tcp dport 80", true, true},
+		{"counter accept", "allowed", "", true, true},
+		{`iifname "wan" tcp dport 22 accept`, "", "", false, false},
+		{"tcp dport vmap { 22 : accept }", "", "", false, false},
+		{"meta mark set 1 accept", "", "", false, false},
+		{"tcp dport 22 counter", "", "", false, false},
+	} {
+		rr := readRaw(c.raw)
+		if rr.ok != c.ok || c.ok && (rr.verdict != c.verdict || rr.short != c.short || rr.forNew != c.forNew) {
+			t.Errorf("%q: %+v", c.raw, rr)
+		}
+	}
+	// In a pair: an exception that lets ND in, not "unknown".
+	st := state()
+	st.Rules = append(st.Rules, &model.Rule{ID: 90, Chain: "input", Position: 0.5, Raw: "icmpv6 type { nd-router-solicit } ip6 hoplimit 255 accept"})
+	st.Sort()
+	e := find(Build(st), "wan", Firewall).Exceptions
+	found := false
+	for _, x := range e {
+		if x.RuleID == 90 {
+			found = x.Verdict == "allowed" && x.Short == "icmpv6" && strings.HasPrefix(x.What, "raw: icmpv6 type")
+		}
+	}
+	if !found {
+		t.Errorf("the raw ND rule: %+v", e)
+	}
+}

internal/model/changeset.go

@@ -35,13 +35,16 @@ const (
 	OpDelete   = "delete"
 	OpMove     = "move"
 	OpSettings = "settings"
+	// OpRevert undoes a history entry: Data is the entry (its changes),
+	// ID its number.
+	OpRevert = "revert"
 )
 
 // Op is one change, as the API received it, so it can be stored and
 // replayed on whatever the configuration is when it is applied.
 type Op struct {
 	Seq  int    `json:"seq"`
-	Op   string `json:"op"` // create, update, delete, move, settings
+	Op   string `json:"op"` // create, update, delete, move, settings, revert
 	Kind string `json:"kind,omitempty"`
 	// Ref names the object (ID, or name) for update, delete and move.
 	Ref string `json:"ref,omitempty"`
@@ -51,6 +54,9 @@ type Op struct {
 	// Data is the object (create), the fields to change (update,
 	// settings) or where to move it (move).
 	Data json.RawMessage `json:"data,omitempty"`
+	// Place is where a create puts the object in its order (a move's
+	// data); none: at the end.
+	Place   json.RawMessage `json:"place,omitempty"`
 	Summary string          `json:"summary"`
 	AddedAt time.Time       `json:"added_at"`
 }

internal/model/objects.go

@@ -143,6 +143,10 @@ type Rule struct {
 	// IPv4 address alone, IPv6 by /64, since one device has several).
 	GroupV4 int `json:"group_v4,omitempty"`
 	GroupV6 int `json:"group_v6,omitempty"`
+	// EveryPacket checks the rule against every packet, before established
+	// connections are accepted, so a drop or reject also stops connections
+	// already open, not just new ones.
+	EveryPacket bool   `json:"every_packet,omitempty"`
 	Action      string `json:"action"`
 	// RejectWith: "" (ICMP port/admin unreachable as fits) or tcp-reset.
 	RejectWith string `json:"reject_with,omitempty"`
@@ -252,6 +256,12 @@ func (r *Rule) CountsTraffic() bool {
 	return bytes
 }
 
+// ChecksEveryPacket reports whether a rule is checked before the built-in
+// that accepts established connections: it counts traffic, or asks to.
+func (r *Rule) ChecksEveryPacket() bool {
+	return r.Raw == "" && (r.EveryPacket || r.CountsTraffic())
+}
+
 // Groups returns a rule's address grouping, defaults applied.
 func (r *Rule) Groups() (v4, v6 int) {
 	v4, v6 = r.GroupV4, r.GroupV6

internal/model/settings.go

@@ -31,6 +31,15 @@ type Settings struct {
 	// ProbationSeconds is how long a change made over the network has to
 	// be confirmed before it is rolled back (0: the default).
 	ProbationSeconds int `json:"probation_seconds,omitempty"`
+	// ManualConfirm: changes made over the network wait for a person to
+	// confirm them. Otherwise the client that made a change confirms it by
+	// itself once it reaches the API again over a new connection, from the
+	// same address: proof that it can still manage the firewall.
+	ManualConfirm bool `json:"manual_confirm,omitempty"`
+
+	// StatsPerDevice counts forwarded traffic to and from external zones
+	// per inside address (IPv6 per /64), for the statistics' top devices.
+	StatsPerDevice bool `json:"stats_per_device,omitempty"`
 }
 
 // Builtins are rules Housecarl adds itself, shown as "added by Housecarl".

internal/nft/nft.go

@@ -8,11 +8,13 @@
 package nft
 
 import (
+	"bufio"
 	"bytes"
 	"context"
 	"encoding/json"
 	"errors"
 	"fmt"
+	"net"
 	"os"
 	"os/exec"
 	"path/filepath"
@@ -24,6 +26,7 @@ import (
 
 	"housecarl/internal/model"
 	"housecarl/internal/render"
+	"housecarl/internal/trace"
 )
 
 // Runner runs nft.
@@ -199,11 +202,17 @@ type Table struct {
 	// Quotas: the rules' quotas' used bytes (quota_r12), and the
 	// per-address quotas' entries by set (quota_r12_v4).
 	QuotaUsage
+	// DeviceSets are the per-device traffic counters by set
+	// (stats_up_v4, stats_down_v6 …): each address's packets and bytes.
+	DeviceSets map[string][]render.Element
 	// Since is when the counters started (housecarld's first apply).
 	Since time.Time
 	// Tables are all the machine's tables ("inet main"), when the whole
 	// ruleset was read.
 	Tables []string
+	// Fallback: the table is the boot fallback (render.Fallback), not a
+	// ruleset housecarld applied.
+	Fallback bool
 }
 
 // QuotaUsage is what the quotas used: kept across reloads, and saved in
@@ -285,7 +294,7 @@ func (q jsonQuota) bytes() int64 {
 
 // ParseTable reads `nft -j list table inet housecarl`.
 func ParseTable(data []byte) (*Table, error) {
-	t := &Table{Counters: map[string]Counter{}, Elements: map[string][]render.Element{},
+	t := &Table{Counters: map[string]Counter{}, Elements: map[string][]render.Element{}, DeviceSets: map[string][]render.Element{},
 		QuotaUsage: QuotaUsage{Quotas: map[string]int64{}, QuotaSets: map[string][]render.Element{}}}
 	if len(data) == 0 {
 		return t, nil
@@ -301,6 +310,9 @@ func ParseTable(data []byte) (*Table, error) {
 				continue
 			}
 			mark, _, _ := strings.Cut(strings.TrimPrefix(r.Comment, "hc:"), " ")
+			if strings.HasPrefix(mark, render.FallbackMark+"-") {
+				t.Fallback = true
+			}
 			for _, e := range r.Expr {
 				var c struct {
 					Counter *Counter `json:"counter"`
@@ -338,6 +350,21 @@ func ParseTable(data []byte) (*Table, error) {
 				}
 				continue
 			}
+			if render.IsDeviceSet(s.Name) {
+				for _, e := range s.Elem {
+					var w struct {
+						Elem struct {
+							Val     string   `json:"val"`
+							Expires int      `json:"expires"`
+							Counter *Counter `json:"counter"`
+						} `json:"elem"`
+					}
+					if json.Unmarshal(e, &w) == nil && w.Elem.Counter != nil {
+						t.DeviceSets[s.Name] = append(t.DeviceSets[s.Name], render.Element{Addr: w.Elem.Val, Timeout: w.Elem.Expires, Packets: w.Elem.Counter.Packets, Bytes: w.Elem.Counter.Bytes})
+					}
+				}
+				continue
+			}
 			name := strings.TrimSuffix(strings.TrimSuffix(s.Name, "_v4"), "_v6")
 			if name == s.Name || strings.HasPrefix(s.Name, "meter_") || strings.HasPrefix(s.Name, "conn_") {
 				continue
@@ -382,6 +409,10 @@ type Firewall struct {
 	// BootFile is where the confirmed ruleset is saved for the boot unit
 	// (housecarl-ruleset.service; "" for none).
 	BootFile string
+	// Shadow is shadow mode: the ruleset changes nothing, beside another
+	// firewall (render.Options; docs/shadow.md). ShadowBefore runs its
+	// filter chains before that one's.
+	Shadow, ShadowBefore bool
 
 	// Replacing the table resets its counters: base keeps the totals of
 	// the tables replaced since housecarld started.
@@ -391,6 +422,35 @@ type Firewall struct {
 	// restore is the quota usage saved before housecarld (or the machine)
 	// stopped, put back by the next apply.
 	restore *QuotaUsage
+	// flowDevs are the flowtable's devices in the ruleset in force, by
+	// name, with their interface index then (0: left out, missing).
+	flowDevs map[string]int
+}
+
+// ifIndex is an interface's index, 0 if it doesn't exist (tests change it).
+var ifIndex = func(name string) int {
+	if i, err := net.InterfaceByName(name); err == nil {
+		return i.Index
+	}
+	return 0
+}
+
+// FlowtableStale reports whether the flowtable in force no longer matches
+// the machine: a device of st's flowtable appeared, went, or came back as
+// a new interface (ppp0 after PPPoE reconnects; the kernel drops a
+// device from the flowtable when it goes). Applying again fixes it.
+func (f *Firewall) FlowtableStale(st *model.State) bool {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if f.flowDevs == nil {
+		return false
+	}
+	for _, d := range st.Settings.FlowtableDevices {
+		if ifIndex(d) != f.flowDevs[d] {
+			return true
+		}
+	}
+	return false
 }
 
 // RestoreQuotas has the next apply put this saved quota usage back where
@@ -422,7 +482,7 @@ func (f *Firewall) ResetQuota(ctx context.Context, ruleID int64, perAddress bool
 }
 
 func (f *Firewall) options() render.Options {
-	var o render.Options
+	o := render.Options{Shadow: f.Shadow, ShadowBefore: f.ShadowBefore}
 	if f.Interfaces != nil {
 		o.Interfaces = f.Interfaces()
 	}
@@ -432,9 +492,26 @@ func (f *Firewall) options() render.Options {
 // Render builds the ruleset for st as Apply would.
 func (f *Firewall) Render(st *model.State) *render.Ruleset { return render.Render(st, f.options()) }
 
-// Check implements engine.Firewall.
+// RenderEnforcing builds the ruleset for st as Apply would outside shadow
+// mode.
+func (f *Firewall) RenderEnforcing(st *model.State) *render.Ruleset {
+	o := f.options()
+	o.Shadow, o.ShadowBefore = false, false
+	return render.Render(st, o)
+}
+
+// Check implements engine.Firewall. In shadow mode it checks the
+// enforcing ruleset too, so that leaving shadow mode can't fail.
 func (f *Firewall) Check(ctx context.Context, st *model.State) error {
-	return f.Runner.Check(ctx, render.Render(st, f.options()))
+	if err := f.Runner.Check(ctx, render.Render(st, f.options())); err != nil {
+		return err
+	}
+	if f.Shadow {
+		if err := f.Runner.Check(ctx, f.RenderEnforcing(st)); err != nil {
+			return fmt.Errorf("the ruleset for when shadow mode is off: %w", err)
+		}
+	}
+	return nil
 }
 
 // Apply implements engine.Firewall.
@@ -455,10 +532,19 @@ func (f *Firewall) Apply(ctx context.Context, st *model.State) error {
 	}
 	usage.merge(f.restore)
 	opt.Quotas, opt.QuotaSets = usage.Quotas, usage.QuotaSets
+	if old != nil {
+		opt.DeviceSets = old.DeviceSets
+	}
 	if err := f.Runner.Apply(ctx, render.Render(st, opt)); err != nil {
 		return err
 	}
 	f.restore = nil
+	f.flowDevs = map[string]int{}
+	for _, d := range st.Settings.FlowtableDevices {
+		if opt.Interfaces == nil || opt.Interfaces[d] {
+			f.flowDevs[d] = ifIndex(d)
+		}
+	}
 	if f.base == nil {
 		f.base, f.since = map[string]Counter{}, time.Now()
 	}
@@ -470,23 +556,75 @@ func (f *Firewall) Apply(ctx context.Context, st *model.State) error {
 			f.base[k] = b
 		}
 	}
+	pruneCounters(f.base, st)
 	return nil
 }
 
-// SaveBoot writes the ruleset of st for the boot unit, atomically.
+// pruneCounters forgets the totals of objects st no longer has (deleted,
+// or replaced by an import with new IDs), so they aren't listed beside
+// the current ones. A disabled object keeps its totals for when it comes
+// back; built-ins and policies are always kept.
+func pruneCounters(base map[string]Counter, st *model.State) {
+	ids := map[string]bool{}
+	for _, r := range st.Rules {
+		ids[fmt.Sprintf("r%d", r.ID)] = true
+	}
+	for _, x := range st.Forwards {
+		ids[fmt.Sprintf("f%d", x.ID)] = true
+	}
+	for _, x := range st.SNATs {
+		ids[fmt.Sprintf("s%d", x.ID)] = true
+	}
+	for _, x := range st.Redirects {
+		ids[fmt.Sprintf("d%d", x.ID)] = true
+	}
+	for mark := range base {
+		obj, _, _ := strings.Cut(mark, "-") // f3-hairpin, f3-6 are f3's
+		if len(obj) < 2 || !strings.ContainsRune("rfsd", rune(obj[0])) || strings.Trim(obj[1:], "0123456789") != "" {
+			continue // b-established, p-input
+		}
+		if !ids[obj] {
+			delete(base, mark)
+		}
+	}
+}
+
+// SaveBoot writes the ruleset of st for the boot unit, and the fallback
+// it loads if that fails (FallbackFile), atomically. The boot ruleset has
+// no flowtable: it loads before the network comes up, when its devices
+// (ppp0, wg0) don't exist yet and nftables would refuse the whole file;
+// housecarld adds it once they do. In shadow mode both are shadow mode's:
+// the fallback then filters nothing.
 func (f *Firewall) SaveBoot(st *model.State) error {
 	if f.BootFile == "" {
 		return nil
 	}
-	rs := render.Render(st, f.options())
-	tmp := f.BootFile + ".tmp"
 	if err := os.MkdirAll(filepath.Dir(f.BootFile), 0o750); err != nil {
 		return err
 	}
-	if err := os.WriteFile(tmp, []byte(rs.Text), 0o640); err != nil {
+	fallback := render.Fallback(st)
+	if f.Shadow {
+		fallback = render.ShadowFallback()
+	}
+	if err := writeAtomic(FallbackFile(f.BootFile), fallback); err != nil {
 		return err
 	}
-	return os.Rename(tmp, f.BootFile)
+	rs := render.Render(st, render.Options{Interfaces: map[string]bool{}, Shadow: f.Shadow, ShadowBefore: f.ShadowBefore})
+	return writeAtomic(f.BootFile, rs.Text)
+}
+
+// FallbackFile is where the fallback for a boot ruleset is saved:
+// ruleset.nft's is ruleset-fallback.nft beside it.
+func FallbackFile(boot string) string {
+	return strings.TrimSuffix(boot, ".nft") + "-fallback.nft"
+}
+
+func writeAtomic(path, text string) error {
+	tmp := path + ".tmp"
+	if err := os.WriteFile(tmp, []byte(text), 0o640); err != nil {
+		return err
+	}
+	return os.Rename(tmp, path)
 }
 
 // Table reads Housecarl's table back, with counters totalled since
@@ -541,6 +679,41 @@ func (f *Firewall) ApplyScript(ctx context.Context, script string) error {
 	return f.Runner.ApplyScript(ctx, script)
 }
 
+// Trace follows packets: it starts `nft monitor trace`, puts the tracing
+// table (script) in, and passes each line to line until ctx ends; then
+// it takes the table out again, whatever happened.
+func (f *Firewall) Trace(ctx context.Context, script string, line func(string)) error {
+	c := f.Runner.cmd(ctx, "monitor", "trace")
+	out, err := c.StdoutPipe()
+	if err != nil {
+		return err
+	}
+	var errb bytes.Buffer
+	c.Stderr = &errb
+	if err := c.Start(); err != nil {
+		return err
+	}
+	defer f.Runner.run(context.Background(), "", "delete", "table", "inet", trace.Table)
+	if err := f.Runner.ApplyScript(ctx, script); err != nil {
+		c.Process.Kill()
+		c.Wait()
+		return fmt.Errorf("starting the trace: %w", err)
+	}
+	sc := bufio.NewScanner(out)
+	sc.Buffer(make([]byte, 64<<10), 1<<20)
+	for sc.Scan() {
+		line(sc.Text())
+	}
+	err = c.Wait()
+	if ctx.Err() != nil {
+		return nil // stopped, or its time was up
+	}
+	if err != nil && errb.Len() > 0 {
+		return fmt.Errorf("nft monitor trace: %s", strings.TrimSpace(errb.String()))
+	}
+	return err
+}
+
 // DeleteTable deletes another table (takeover).
 func (f *Firewall) DeleteTable(ctx context.Context, family, name string) error {
 	if name == render.Table && family == "inet" {

internal/nft/nft_test.go

@@ -2,8 +2,11 @@ package nft
 
 import (
 	"context"
+	"fmt"
 	"os"
 	"os/exec"
+	"path/filepath"
+	"sort"
 	"strings"
 	"testing"
 
@@ -208,3 +211,253 @@ func TestQuotasInContainer(t *testing.T) {
 		t.Errorf("after a reboot: total %d, per destination %d (saved %+v)", t4, d4, saved)
 	}
 }
+
+// TestDeviceCountsInContainer: the per-device counters are read back and
+// survive a reload.
+func TestDeviceCountsInContainer(t *testing.T) {
+	image := os.Getenv("HOUSECARL_NFT_IMAGE")
+	if image == "" {
+		t.Skip("HOUSECARL_NFT_IMAGE not set")
+	}
+	name := "housecarl-nft-stats-test"
+	exec.Command("docker", "rm", "-f", name).Run()
+	if out, err := exec.Command("docker", "run", "-d", "--name", name, "--cap-add", "NET_ADMIN", image, "sleep", "300").CombinedOutput(); err != nil {
+		t.Fatalf("starting the container: %v %s", err, out)
+	}
+	t.Cleanup(func() { exec.Command("docker", "rm", "-f", name).Run() })
+	fw := &Firewall{Runner: &Runner{Command: []string{"docker", "exec", "-i", name, "nft"}}}
+	ctx := context.Background()
+	st := model.NewState()
+	st.Zones = []*model.Zone{{ID: 1, Name: "wan", Interfaces: []string{"eth0"}, External: true}}
+	st.Settings.StatsPerDevice = true
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatalf("apply: %v", err)
+	}
+	if out, err := exec.Command("docker", "exec", name, "nft", "add", "element", "inet", "housecarl", "stats_up_v4", "{ 10.0.10.5 counter packets 7 bytes 7000 }").CombinedOutput(); err != nil {
+		t.Fatalf("adding an element: %v %s", err, out)
+	}
+	read := func() render.Element {
+		tb, err := fw.Table(ctx)
+		if err != nil {
+			t.Fatal(err)
+		}
+		for _, e := range tb.DeviceSets["stats_up_v4"] {
+			if e.Addr == "10.0.10.5" {
+				return e
+			}
+		}
+		t.Fatalf("no 10.0.10.5 in %+v", tb.DeviceSets)
+		return render.Element{}
+	}
+	if e := read(); e.Packets != 7 || e.Bytes != 7000 || e.Timeout <= 0 {
+		t.Fatalf("read back %+v", e)
+	}
+	st.Settings.InputPolicy = model.PolicyAccept // a change: a reload
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatal(err)
+	}
+	if e := read(); e.Packets != 7 || e.Bytes != 7000 {
+		t.Errorf("after a reload: %+v", e)
+	}
+}
+
+// TestFlowtableStale: the flowtable in force goes stale when one of its
+// devices appears, goes, or comes back as a new interface.
+func TestFlowtableStale(t *testing.T) {
+	idx := map[string]int{"eth1": 3}
+	old := ifIndex
+	ifIndex = func(name string) int { return idx[name] }
+	t.Cleanup(func() { ifIndex = old })
+	exists := func() map[string]bool {
+		out := map[string]bool{}
+		for n, i := range idx {
+			out[n] = i > 0
+		}
+		return out
+	}
+	// "true" stands in for nft: every apply succeeds, and lists nothing.
+	fw := &Firewall{Runner: &Runner{Command: []string{"true"}}, Interfaces: exists}
+	st := model.NewState()
+	st.Settings.FlowtableDevices = []string{"ppp0", "eth1"}
+	if fw.FlowtableStale(st) {
+		t.Error("stale before anything was applied")
+	}
+	if err := fw.Apply(context.Background(), st); err != nil {
+		t.Fatal(err)
+	}
+	if fw.FlowtableStale(st) {
+		t.Error("stale right after applying")
+	}
+	idx["ppp0"] = 7 // PPPoE came up
+	if !fw.FlowtableStale(st) {
+		t.Error("ppp0 appeared, yet not stale")
+	}
+	fw.Apply(context.Background(), st)
+	idx["ppp0"] = 9 // and reconnected: a new interface by the same name
+	if !fw.FlowtableStale(st) {
+		t.Error("ppp0 came back as a new interface, yet not stale")
+	}
+	fw.Apply(context.Background(), st)
+	idx["eth9"] = 4 // not a flowtable device
+	if fw.FlowtableStale(st) {
+		t.Error("stale over an interface the flowtable doesn't use")
+	}
+}
+
+// TestBootInContainer: the boot ruleset loads although its flowtable
+// devices don't exist yet, the fallback loads and is recognised, and
+// housecarld's apply replaces it.
+func TestBootInContainer(t *testing.T) {
+	image := os.Getenv("HOUSECARL_NFT_IMAGE")
+	if image == "" {
+		t.Skip("HOUSECARL_NFT_IMAGE not set")
+	}
+	name := "housecarl-nft-boot-test"
+	exec.Command("docker", "rm", "-f", name).Run()
+	if out, err := exec.Command("docker", "run", "-d", "--name", name, "--cap-add", "NET_ADMIN", image, "sleep", "300").CombinedOutput(); err != nil {
+		t.Fatalf("starting the container: %v %s", err, out)
+	}
+	t.Cleanup(func() { exec.Command("docker", "rm", "-f", name).Run() })
+	nft := func(script string) error {
+		c := exec.Command("docker", "exec", "-i", name, "nft", "-f", "/dev/stdin")
+		c.Stdin = strings.NewReader(script)
+		if out, err := c.CombinedOutput(); err != nil {
+			return fmt.Errorf("%v: %s", err, out)
+		}
+		return nil
+	}
+	dir := t.TempDir()
+	// ppp0 exists when housecarld saves, but not at boot.
+	fw := &Firewall{Runner: &Runner{Command: []string{"docker", "exec", "-i", name, "nft"}}, BootFile: filepath.Join(dir, "ruleset.nft"),
+		Interfaces: func() map[string]bool { return map[string]bool{"ppp0": true, "eth0": true} }}
+	st := model.NewState()
+	st.Zones = []*model.Zone{{ID: 1, Name: "wan", Interfaces: []string{"ppp0"}, External: true}, {ID: 2, Name: "lan", Interfaces: []string{"eth0"}}}
+	st.SNATs = []*model.SourceNAT{{ID: 3, Match: model.Match{OutZones: []string{"wan"}}}}
+	st.Settings.FlowtableDevices = []string{"ppp0", "eth0"}
+	if err := fw.SaveBoot(st); err != nil {
+		t.Fatal(err)
+	}
+	boot, _ := os.ReadFile(fw.BootFile)
+	if strings.Contains(string(boot), "flowtable") {
+		t.Errorf("the boot ruleset has a flowtable:\n%s", boot)
+	}
+	fallback, err := os.ReadFile(filepath.Join(dir, "ruleset-fallback.nft"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := nft(string(boot)); err != nil {
+		t.Fatalf("the boot ruleset doesn't load: %v", err)
+	}
+	ctx := context.Background()
+	if tb, err := fw.Table(ctx); err != nil || tb.Fallback {
+		t.Errorf("after the boot ruleset: %+v %v", tb, err)
+	}
+	if err := nft(string(fallback)); err != nil {
+		t.Fatalf("the fallback doesn't load: %v", err)
+	}
+	if tb, err := fw.Table(ctx); err != nil || !tb.Fallback {
+		t.Errorf("the fallback isn't recognised: %+v %v", tb, err)
+	}
+	fw.Interfaces = func() map[string]bool { return map[string]bool{"eth0": true} }
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatalf("apply over the fallback: %v", err)
+	}
+	if tb, err := fw.Table(ctx); err != nil || tb.Fallback {
+		t.Errorf("the fallback stayed after an apply: %+v %v", tb, err)
+	}
+}
+
+// TestShadowInContainer: in shadow mode a drop counts but doesn't drop,
+// the ruleset for leaving shadow mode is checked too, and the boot files
+// are shadow mode's.
+func TestShadowInContainer(t *testing.T) {
+	image := os.Getenv("HOUSECARL_NFT_IMAGE")
+	if image == "" {
+		t.Skip("HOUSECARL_NFT_IMAGE not set")
+	}
+	name := "housecarl-nft-shadow-test"
+	exec.Command("docker", "rm", "-f", name).Run()
+	if out, err := exec.Command("docker", "run", "-d", "--name", name, "--cap-add", "NET_ADMIN", image, "sleep", "300").CombinedOutput(); err != nil {
+		t.Fatalf("starting the container: %v %s", err, out)
+	}
+	t.Cleanup(func() { exec.Command("docker", "rm", "-f", name).Run() })
+	dir := t.TempDir()
+	fw := &Firewall{Runner: &Runner{Command: []string{"docker", "exec", "-i", name, "nft"}}, Shadow: true, BootFile: filepath.Join(dir, "ruleset.nft")}
+	ctx := context.Background()
+	st := model.NewState()
+	st.Settings.OutputPolicy = model.PolicyDrop
+	st.Settings.InputPolicy = model.PolicyAccept
+	st.Settings.Builtins.Loopback = false // it would accept the ping first
+	st.Rules = []*model.Rule{{ID: 3, Chain: model.ChainOutput, Position: 1, Action: model.ActionReject, Match: model.Match{Destinations: []string{"127.0.0.2"}}}}
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatalf("apply: %v", err)
+	}
+	ping := func() error {
+		return exec.Command("docker", "exec", name, "ping", "-c", "2", "-W", "1", "127.0.0.2").Run()
+	}
+	if err := ping(); err != nil {
+		t.Errorf("shadow mode blocked a ping: %v", err)
+	}
+	tb, err := fw.Table(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if tb.Counters["r3"].Packets == 0 {
+		t.Errorf("what would be rejected isn't counted: %+v", tb.Counters)
+	}
+	// A rule shadow mode leaves out, which nftables refuses when enforcing.
+	broken := st.Clone()
+	broken.Rules = append(broken.Rules, &model.Rule{ID: 4, Chain: model.ChainInput, Position: 1, Raw: "tcp dport 22 jump nowhere", Comment: "jump"})
+	if err := fw.Check(ctx, broken); err == nil || !strings.Contains(err.Error(), "shadow mode is off") || !strings.Contains(err.Error(), "input #4 (jump)") {
+		t.Errorf("the ruleset for leaving shadow mode wasn't checked: %v", err)
+	}
+	// Enforcing, the same rule rejects.
+	enforcing := &Firewall{Runner: fw.Runner}
+	if err := enforcing.Apply(ctx, st); err != nil {
+		t.Fatal(err)
+	}
+	if ping() == nil {
+		t.Error("enforcing didn't reject")
+	}
+	// The boot files: both load, and filter nothing.
+	if err := fw.SaveBoot(st); err != nil {
+		t.Fatal(err)
+	}
+	for _, f := range []string{"ruleset.nft", "ruleset-fallback.nft"} {
+		b, _ := os.ReadFile(filepath.Join(dir, f))
+		c := exec.Command("docker", "exec", "-i", name, "nft", "-f", "/dev/stdin")
+		c.Stdin = strings.NewReader(string(b))
+		if out, err := c.CombinedOutput(); err != nil {
+			t.Fatalf("%s doesn't load: %v %s", f, err, out)
+		}
+		if err := ping(); err != nil {
+			t.Errorf("%s blocked a ping: %v", f, err)
+		}
+	}
+	if tb, err := fw.Table(ctx); err != nil || !tb.Fallback {
+		t.Errorf("the shadow fallback isn't recognised: %+v %v", tb, err)
+	}
+}
+
+// TestPruneCounters: the totals of objects gone (an import with --replace
+// gives every object a new ID) are forgotten; built-ins, policies and
+// disabled objects keep theirs.
+func TestPruneCounters(t *testing.T) {
+	base := map[string]Counter{}
+	for _, m := range []string{"r1", "r2", "r3", "f4", "f4-hairpin", "f4-6", "f5-hairpin", "s6", "d7", "d8", "b-established", "p-input", "fallback-input"} {
+		base[m] = Counter{Packets: 1}
+	}
+	st := model.NewState()
+	st.Rules = []*model.Rule{{ID: 1}, {ID: 3, Disabled: true}}
+	st.Forwards = []*model.PortForward{{ID: 4}}
+	st.Redirects = []*model.Redirect{{ID: 8}}
+	pruneCounters(base, st)
+	var got []string
+	for m := range base {
+		got = append(got, m)
+	}
+	sort.Strings(got)
+	if strings.Join(got, " ") != "b-established d8 f4 f4-6 f4-hairpin fallback-input p-input r1 r3" {
+		t.Errorf("kept %v", got)
+	}
+}

internal/render/fallback.go

@@ -0,0 +1,89 @@
+package render
+
+import (
+	"fmt"
+	"strings"
+
+	"housecarl/internal/model"
+)
+
+// FallbackMark starts the comment of every fallback rule ("hc:fallback-…"):
+// how housecarld tells the fallback is in force.
+const FallbackMark = "fallback"
+
+// Fallback builds the ruleset housecarl-ruleset.service loads at boot when
+// the saved one fails to: nothing new gets in from the external zones'
+// interfaces, everything else is let through, and IPv4 leaving an
+// external zone is masqueraded (when any source NAT is configured), so the
+// inside keeps working until housecarld puts the real ruleset back. With
+// no external zone it can't tell which interface faces the internet, and
+// lets nothing but loopback through.
+//
+// It is deliberately small: no sets, flowtable, meters or quotas, and
+// interfaces by name only, so it loads before they exist.
+func Fallback(st *model.State) string {
+	var b strings.Builder
+	line := func(format string, args ...any) {
+		fmt.Fprintf(&b, format, args...)
+		b.WriteByte('\n')
+	}
+	// rule writes a counted rule: what it matches, then its verdict.
+	rule := func(name, match, verdict string) {
+		line("\t\t%s counter %s comment \"hc:%s-%s\"", match, verdict, FallbackMark, name)
+	}
+	ext := st.ExternalInterfaces()
+	line("# Housecarl's fallback: loaded at boot only if ruleset.nft fails to load,")
+	line("# until housecarld starts and puts the real ruleset back.")
+	line("table inet %s {}", Table)
+	line("delete table inet %s", Table)
+	line("table inet %s {", Table)
+	if len(ext) == 0 {
+		// Nothing says which interface faces the internet: close it all.
+		for _, c := range []string{"input", "forward"} {
+			line("\tchain %s {", c)
+			line("\t\ttype filter hook %s priority filter; policy drop;", c)
+			line("\t\tiifname \"lo\" accept")
+			line("\t\tcounter comment \"hc:%s-%s\"", FallbackMark, c)
+			line("\t}")
+		}
+		line("}")
+		return b.String()
+	}
+	e := ifaces(ext)
+	line("\tchain input {")
+	line("\t\ttype filter hook input priority filter; policy accept;")
+	line("\t\tiifname \"lo\" accept")
+	line("\t\tct state established,related accept")
+	rule("icmpv6", fmt.Sprintf("iifname %s icmpv6 type { %s, echo-request, echo-reply }", e, icmpv6Errors), "accept")
+	rule("icmpv6-nd", fmt.Sprintf("iifname %s icmpv6 type { %s } ip6 hoplimit 255", e, icmpv6ND), "accept")
+	rule("icmpv6-mld", fmt.Sprintf("iifname %s ip6 saddr fe80::/10 icmpv6 type { %s }", e, icmpv6MLD), "accept")
+	rule("dhcpv6-client", fmt.Sprintf("iifname %s ip6 saddr fe80::/10 udp dport 546", e), "accept")
+	rule("input", "iifname "+e, "drop")
+	line("\t}")
+	line("\tchain forward {")
+	line("\t\ttype filter hook forward priority filter; policy accept;")
+	if st.Settings.Builtins.MSSClamp {
+		line("\t\toifname %s tcp flags syn tcp option maxseg size set rt mtu", e)
+	}
+	line("\t\tct state established,related accept")
+	rule("forward", "iifname "+e, "drop")
+	line("\t}")
+	if masquerades(st) {
+		line("\tchain srcnat {")
+		line("\t\ttype nat hook postrouting priority srcnat; policy accept;")
+		rule("masquerade", "meta nfproto ipv4 oifname "+e, "masquerade")
+		line("\t}")
+	}
+	line("}")
+	return b.String()
+}
+
+// masquerades reports whether any source NAT is configured.
+func masquerades(st *model.State) bool {
+	for _, x := range st.SNATs {
+		if !x.Disabled {
+			return true
+		}
+	}
+	return false
+}

internal/render/fallback_test.go

@@ -0,0 +1,75 @@
+package render
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"strings"
+	"testing"
+
+	"housecarl/internal/model"
+)
+
+func TestFallback(t *testing.T) {
+	st := router()
+	text := Fallback(st)
+	for _, want := range []string{
+		"delete table inet housecarl",
+		"type filter hook input priority filter; policy accept;",
+		`iifname "wan" counter drop comment "hc:fallback-input"`,
+		`iifname "wan" counter drop comment "hc:fallback-forward"`,
+		`iifname "wan" ip6 saddr fe80::/10 udp dport 546 counter accept`,
+		`ip6 hoplimit 255 counter accept`,
+		`oifname "wan" tcp flags syn tcp option maxseg size set rt mtu`,
+		`meta nfproto ipv4 oifname "wan" counter masquerade comment "hc:fallback-masquerade"`,
+	} {
+		if !strings.Contains(text, want) {
+			t.Errorf("no %q in\n%s", want, text)
+		}
+	}
+	// Nothing that could fail to load before the network is up.
+	for _, not := range []string{"flowtable", "@", "\tset ", "enp2s0", "wg0"} {
+		if strings.Contains(text, not) {
+			t.Errorf("%q in the fallback:\n%s", not, text)
+		}
+	}
+	// Established connections are accepted before the drops.
+	if strings.Index(text, "ct state established,related accept") > strings.Index(text, "hc:fallback-input") {
+		t.Errorf("established after the drop:\n%s", text)
+	}
+
+	// No source NAT: no masquerade.
+	st.SNATs = nil
+	if text := Fallback(st); strings.Contains(text, "masquerade") {
+		t.Errorf("masquerade without source NAT:\n%s", text)
+	}
+
+	// No external zone: everything but loopback is dropped.
+	for _, z := range st.Zones {
+		z.External = false
+	}
+	text = Fallback(st)
+	if strings.Count(text, "policy drop;") != 2 || strings.Contains(text, "policy accept") || strings.Contains(text, "established") {
+		t.Errorf("no external zone:\n%s", text)
+	}
+}
+
+func TestFallbackPassesNft(t *testing.T) {
+	cmd := strings.Fields(os.Getenv("HOUSECARL_NFT"))
+	if len(cmd) == 0 {
+		t.Skip("HOUSECARL_NFT not set")
+	}
+	closed := router()
+	for _, z := range closed.Zones {
+		z.External = false
+	}
+	wild := router()
+	wild.Zones = append(wild.Zones, &model.Zone{ID: 99, Name: "lte", Interfaces: []string{"wwan*"}, External: true})
+	for name, st := range map[string]*model.State{"router": router(), "closed": closed, "wildcard": wild} {
+		c := exec.CommandContext(context.Background(), cmd[0], append(cmd[1:], "-c", "-f", "/dev/stdin")...)
+		c.Stdin = strings.NewReader(Fallback(st))
+		if out, err := c.CombinedOutput(); err != nil {
+			t.Errorf("%s: nft -c: %v\n%s\n%s", name, err, out, Fallback(st))
+		}
+	}
+}

internal/render/render.go

@@ -8,6 +8,14 @@
 // counter and a comment "hc:<kind><id>", which is how counters are read
 // back, and Ruleset.Lines maps each line to its object, for nftables'
 // error messages.
+//
+// In shadow mode (Options.Shadow, docs/shadow.md) the same rules are
+// written so that they change nothing: drops and rejects accept instead
+// (an accept only ends Housecarl's chain; the packet goes on to other
+// tables' chains, which decide), policies accept, NAT only counts, and
+// what alters packets (MSS clamping, the flowtable) is left out. What
+// would be dropped, rejected or translated carries a marker in its
+// comment ("[would drop]"), which the tracer reads.
 package render
 
 import (
@@ -39,14 +47,30 @@ type Options struct {
 	// quota_r12_v4), so a reload keeps what was used.
 	Quotas    map[string]int64
 	QuotaSets map[string][]Element
+	// DeviceSets are the per-device traffic counters' entries
+	// (stats_up_v4 …), so a reload keeps them.
+	DeviceSets map[string][]Element
+	// Shadow renders for shadow mode: nothing is blocked, rejected,
+	// translated or changed (see the package comment).
+	Shadow bool
+	// ShadowBefore runs the filter chains just before other tables' on
+	// their hooks (priority - 10) instead of just after (+ 10).
+	ShadowBefore bool
 }
 
+// Would marks, in shadow mode, a rule's comment with what it would do:
+// "[would drop]".
+func Would(action string) string { return "[would " + action + "]" }
+
 // Element is a dynamic set's entry with its remaining lifetime, or a
 // per-address quota's entry with what it used.
 type Element struct {
 	Addr    string `json:"address"`
 	Timeout int    `json:"timeout,omitempty"` // seconds left (0: none)
 	Used    int64  `json:"used,omitempty"`    // bytes
+	// Packets and Bytes are a per-device counter's.
+	Packets uint64 `json:"packets,omitempty"`
+	Bytes   uint64 `json:"bytes,omitempty"`
 }
 
 // QuotaName is the name of a rule's quota: the named quota of a total
@@ -106,10 +130,14 @@ type renderer struct {
 func (r *renderer) render() {
 	w := r.w
 	w.emit("", "# Built by Housecarl from its database; changes made here are lost.")
+	if r.opt.Shadow {
+		w.emit("", "# Shadow mode: nothing here drops, rejects or translates; other tables decide.")
+	}
 	w.emit("", "table inet %s {}", Table)
 	w.emit("", "delete table inet %s", Table)
 	w.emit("", "table inet %s {", Table)
 	r.sets()
+	r.deviceSets()
 	devices := r.flowtableDevices()
 	if len(devices) > 0 {
 		w.emit("built-in: flowtable", "\tflowtable ft { hook ingress priority filter; devices = { %s }; }", strings.Join(quoteAll(devices), ", "))
@@ -190,6 +218,56 @@ func (r *renderer) elements() {
 	}
 }
 
+// IsDeviceSet reports whether a set is a per-device traffic counter.
+func IsDeviceSet(name string) bool {
+	return strings.HasPrefix(name, "stats_up_") || strings.HasPrefix(name, "stats_down_")
+}
+
+// deviceSets declares the per-device traffic counters, with what they
+// counted before a reload.
+func (r *renderer) deviceSets() {
+	if !r.st.Settings.StatsPerDevice || len(r.st.ExternalInterfaces()) == 0 {
+		return
+	}
+	for _, dir := range []string{"up", "down"} {
+		for _, v6 := range []bool{false, true} {
+			name := setName("stats_"+dir, v6)
+			var elems []string
+			for _, e := range r.opt.DeviceSets[name] {
+				a, err := netip.ParseAddr(e.Addr)
+				if err != nil || a.Is6() != v6 {
+					continue
+				}
+				timeout := ""
+				if e.Timeout > 0 {
+					timeout = fmt.Sprintf(" timeout %ds", e.Timeout)
+				}
+				elems = append(elems, fmt.Sprintf("%s%s counter packets %d bytes %d", a, timeout, e.Packets, e.Bytes))
+			}
+			list := ""
+			if len(elems) > 0 {
+				list = " elements = { " + strings.Join(elems, ", ") + " }"
+			}
+			r.w.emit("built-in: stats", "\tset %s { type %s; flags dynamic,timeout; timeout 1h; size 65535; counter;%s }", name, addrType(v6), list)
+		}
+	}
+}
+
+// deviceCounts counts forwarded traffic per inside address: what leaves
+// for an external zone by its source, what comes from one by its
+// destination (IPv6 by /64).
+func (r *renderer) deviceCounts() {
+	ext := r.st.ExternalInterfaces()
+	if !r.st.Settings.StatsPerDevice || len(ext) == 0 {
+		return
+	}
+	e := ifaces(ext)
+	r.builtin("stats", "iifname != %s oifname %s update @stats_up_v4 { ip saddr }", e, e)
+	r.builtin("stats", "iifname != %s oifname %s update @stats_up_v6 { ip6 saddr and ffff:ffff:ffff:ffff:: }", e, e)
+	r.builtin("stats", "iifname %s oifname != %s update @stats_down_v4 { ip daddr }", e, e)
+	r.builtin("stats", "iifname %s oifname != %s update @stats_down_v6 { ip6 daddr and ffff:ffff:ffff:ffff:: }", e, e)
+}
+
 // SetName is the nftables name of a set's family part.
 func SetName(name string, v6 bool) string { return setName(name, v6) }
 
@@ -208,6 +286,10 @@ func addrType(v6 bool) string {
 }
 
 func (r *renderer) flowtableDevices() []string {
+	if r.opt.Shadow {
+		// Offloaded connections skip every table's forward chain.
+		return nil
+	}
 	var out []string
 	for _, d := range r.st.Settings.FlowtableDevices {
 		if r.opt.Interfaces == nil || r.opt.Interfaces[d] {
@@ -220,8 +302,19 @@ func (r *renderer) flowtableDevices() []string {
 // ---- chains
 
 func (r *renderer) chain(name, hook, prio, policy string) {
+	nat := prio == "dstnat" || prio == "srcnat"
+	if r.opt.Shadow {
+		// NAT chains are asked in turn only until one translates: to
+		// count anything, Housecarl's must come first.
+		if nat || r.opt.ShadowBefore {
+			prio += " - 10"
+		} else {
+			prio += " + 10"
+		}
+		policy = model.PolicyAccept
+	}
 	r.w.emit("", "\tchain %s {", name)
-	r.w.emit("", "\t\ttype %s hook %s priority %s; policy %s;", map[bool]string{true: "nat", false: "filter"}[prio == "dstnat" || prio == "srcnat"], hook, prio, policy)
+	r.w.emit("", "\t\ttype %s hook %s priority %s; policy %s;", map[bool]string{true: "nat", false: "filter"}[nat], hook, prio, policy)
 }
 
 func (r *renderer) end() { r.w.emit("", "\t}") }
@@ -230,6 +323,16 @@ func (r *renderer) builtin(name, format string, args ...any) {
 	r.w.emit("built-in: "+name, "\t\t"+format+" comment \"hc:b-%s\"", append(args, name)...)
 }
 
+// builtinDrop writes a built-in that drops what it matches, or in shadow
+// mode counts it.
+func (r *renderer) builtinDrop(name, format string, args ...any) {
+	if r.opt.Shadow {
+		r.w.emit("built-in: "+name, "\t\t"+format+" counter accept comment \"hc:b-%s %s\"", append(args, name, Would("drop"))...)
+		return
+	}
+	r.builtin(name, format+" counter drop", args...)
+}
+
 func (r *renderer) early() {
 	st := r.st
 	r.chain("early", "prerouting", "raw", "accept")
@@ -245,10 +348,10 @@ func (r *renderer) early() {
 			}
 		}
 		if len(v4) > 0 {
-			r.builtin("antispoof", "iifname %s ip saddr { %s } counter drop", ifaces(ext), strings.Join(v4, ", "))
+			r.builtinDrop("antispoof", "iifname %s ip saddr { %s }", ifaces(ext), strings.Join(v4, ", "))
 		}
 		if len(v6) > 0 {
-			r.builtin("antispoof6", "iifname %s ip6 saddr { %s } counter drop", ifaces(ext), strings.Join(v6, ", "))
+			r.builtinDrop("antispoof6", "iifname %s ip6 saddr { %s }", ifaces(ext), strings.Join(v6, ", "))
 		}
 	}
 	r.rules(model.ChainEarly, true)
@@ -268,7 +371,7 @@ func (r *renderer) connBuiltins() {
 		r.builtin("established", "ct state established,related counter accept")
 	}
 	if b.DropInvalid {
-		r.builtin("invalid", "ct state invalid counter drop")
+		r.builtinDrop("invalid", "ct state invalid")
 	}
 }
 
@@ -309,10 +412,12 @@ func (r *renderer) forward(flowtable bool) {
 	b := st.Settings.Builtins
 	r.chain("forward", "forward", "filter", st.Settings.ForwardPolicy)
 	ext := st.ExternalInterfaces()
-	if b.MSSClamp && len(ext) > 0 {
+	// Shadow mode changes no packet.
+	if b.MSSClamp && len(ext) > 0 && !r.opt.Shadow {
 		r.builtin("mss-clamp", "oifname %s tcp flags syn tcp option maxseg size set rt mtu", ifaces(ext))
 	}
 	r.rules(model.ChainForward, true)
+	r.deviceCounts()
 	if flowtable {
 		r.builtin("flowtable", "meta l4proto { tcp, udp } flow add @ft")
 	}
@@ -346,15 +451,21 @@ func (r *renderer) output() {
 	r.end()
 }
 
-// policyTail counts (and optionally logs) what reaches the policy.
+// policyTail counts (and optionally logs) what reaches the policy. In
+// shadow mode, where the policy accepts, that is what it would drop, and
+// it is always logged: the log is how shadow mode shows it.
 func (r *renderer) policyTail(chain, policy string) {
 	if policy != model.PolicyDrop {
 		return
 	}
 	obj := "policy: " + chain
-	if r.st.Settings.LogPolicyDrops && chain != "output" {
+	if r.opt.Shadow || r.st.Settings.LogPolicyDrops && chain != "output" {
 		r.w.emit(obj, "\t\tlimit rate 10/second burst 20 packets log prefix %s group %d", quote("hc "+chain+" policy drop: "), LogGroup)
 	}
+	if r.opt.Shadow {
+		r.w.emit(obj, "\t\tcounter comment \"hc:p-%s %s\"", chain, Would("drop"))
+		return
+	}
 	r.w.emit(obj, "\t\tcounter comment \"hc:p-%s\"", chain)
 }
 
@@ -364,13 +475,21 @@ func (r *renderer) policyTail(chain, policy string) {
 // or the others.
 func (r *renderer) rules(chain string, traffic bool) {
 	for _, rule := range r.st.Rules {
-		if rule.Chain != chain || rule.Disabled || rule.CountsTraffic() != traffic {
+		if rule.Chain != chain || rule.Disabled || rule.ChecksEveryPacket() != traffic {
 			continue
 		}
 		obj := rule.Label()
 		mark := fmt.Sprintf("r%d", rule.ID)
 		if rule.Raw != "" {
-			r.w.emit(obj, "\t\t%s", rule.Raw)
+			raw := rule.Raw
+			if r.opt.Shadow {
+				var ok bool
+				if raw, ok = ShadowRaw(raw); !ok {
+					r.w.emit(obj, "\t\t# left out in shadow mode: %s", rule.Raw)
+					continue
+				}
+			}
+			r.w.emit(obj, "\t\t%s", raw)
 			continue
 		}
 		var tail []string
@@ -420,17 +539,22 @@ func (r *renderer) verdict(rule *model.Rule, mark string) string {
 		}
 		v = append(v, fmt.Sprintf("log prefix %s group %d", quote(prefix), LogGroup))
 	}
-	switch rule.Action {
-	case model.ActionAccept, model.ActionDrop:
+	text := rule.Comment
+	switch {
+	case r.opt.Shadow && (rule.Action == model.ActionDrop || rule.Action == model.ActionReject):
+		// It ends Housecarl's chain as the drop would, and no more.
+		v = append(v, model.ActionAccept)
+		text = Would(rule.Action) + " " + text
+	case rule.Action == model.ActionAccept, rule.Action == model.ActionDrop:
 		v = append(v, rule.Action)
-	case model.ActionReject:
+	case rule.Action == model.ActionReject:
 		if rule.RejectWith == "tcp-reset" {
 			v = append(v, "reject with tcp reset")
 		} else {
 			v = append(v, "reject")
 		}
 	}
-	v = append(v, "comment "+comment(mark, rule.Comment))
+	v = append(v, "comment "+comment(mark, text))
 	return strings.Join(v, " ")
 }
 
@@ -723,7 +847,7 @@ func (r *renderer) dstnat() {
 					}
 				}
 			}
-			r.w.emit(obj, "\t\t%s counter redirect to :%s comment %s", strings.Join(nonEmpty(line.parts), " "), to, comment(mark, x.Comment))
+			r.w.emit(obj, "\t\t%s counter %s", strings.Join(nonEmpty(line.parts), " "), r.nat("redirect to :"+to, mark, x.Comment))
 		}
 	}
 	ext := st.ExternalInterfaces()
@@ -753,7 +877,7 @@ func (r *renderer) dstnat() {
 		for _, wh := range where {
 			for _, sa := range srcAlts {
 				for _, p := range f.Protocols {
-					r.w.emit(obj, "\t\t%s counter dnat ip to %s comment %s", strings.Join(nonEmpty([]string{wh, sa, p + " dport " + f.Ports}), " "), to, comment(mark, f.Name+" "+f.Comment))
+					r.w.emit(obj, "\t\t%s counter %s", strings.Join(nonEmpty([]string{wh, sa, p + " dport " + f.Ports}), " "), r.nat("dnat ip to "+to, mark, f.Name+" "+f.Comment))
 				}
 			}
 		}
@@ -827,7 +951,7 @@ func (r *renderer) srcnat() {
 		}
 		for _, sa := range srcAlts {
 			for _, p := range f.Protocols {
-				r.w.emit(obj, "\t\t%s counter masquerade comment %s", strings.Join(nonEmpty([]string{sa, "ip daddr " + f.To, p + " dport " + ports, "ct status dnat"}), " "), comment(mark, f.Name))
+				r.w.emit(obj, "\t\t%s counter %s", strings.Join(nonEmpty([]string{sa, "ip daddr " + f.To, p + " dport " + ports, "ct status dnat"}), " "), r.nat("masquerade", mark, f.Name))
 			}
 		}
 	}
@@ -859,12 +983,21 @@ func (r *renderer) srcnat() {
 					action = "snat ip to " + x.To
 				}
 			}
-			r.w.emit(obj, "\t\t%s counter %s comment %s", strings.Join(nonEmpty(line.parts), " "), action, comment(mark, x.Comment))
+			r.w.emit(obj, "\t\t%s counter %s", strings.Join(nonEmpty(line.parts), " "), r.nat(action, mark, x.Comment))
 		}
 	}
 	r.end()
 }
 
+// nat writes a NAT rule's statement and comment; in shadow mode only the
+// comment, saying what it would do ("[would dnat]").
+func (r *renderer) nat(action, mark, text string) string {
+	if r.opt.Shadow {
+		return "comment " + comment(mark, Would(strings.Fields(action)[0])+" "+text)
+	}
+	return action + " comment " + comment(mark, text)
+}
+
 // ---- helpers
 
 // hours writes a time window. nftables can't take a range past midnight,

internal/render/render_test.go

@@ -2,6 +2,7 @@ package render
 
 import (
 	"context"
+	"fmt"
 	"os"
 	"os/exec"
 	"strings"
@@ -76,6 +77,7 @@ func router() *model.State {
 		Quota: &model.Quota{Bytes: 50000 << 20, Over: true, Reset: "monthly", Day: 15}, Comment: "guest quota"})
 	rule(model.ChainForward, model.Rule{Match: model.Match{InZones: []string{"lan"}, OutZones: []string{"wan"}}, Action: model.ActionDrop,
 		Quota: &model.Quota{Bytes: 5 << 30, Over: true, PerSource: true, Reset: "daily"}, Comment: "device quota"})
+	rule(model.ChainForward, model.Rule{Match: model.Match{Sources: []string{"10.0.10.66"}}, Action: model.ActionReject, EveryPacket: true, Comment: "blocked now"})
 	rule(model.ChainEarly, model.Rule{Match: model.Match{InZones: []string{"wan"}, Sources: []string{"198.51.100.0/24"}}, Action: model.ActionDrop})
 	rule(model.ChainOutput, model.Rule{Raw: "meta skuid 0 counter accept", Comment: "raw"})
 	st.Forwards = []*model.PortForward{
@@ -186,6 +188,9 @@ func TestRenderRouter(t *testing.T) {
 	if bw := strings.Index(fwd, "limit rate over 5 mbytes/second"); bw > est {
 		t.Error("the byte rate rule comes after established")
 	}
+	if now := strings.Index(fwd, "blocked now"); now < 0 || now > est {
+		t.Error("the every-packet rule comes after established")
+	}
 	// The trusted masquerade has no IPv6 part: masquerading the IPv6 LAN
 	// isn't wanted (its sources are global).
 	if strings.Contains(text, "ip6 saddr @trusted_v6 counter masquerade") {
@@ -202,6 +207,31 @@ func TestRenderRouter(t *testing.T) {
 	}
 }
 
+func TestRenderDeviceCounts(t *testing.T) {
+	st := router()
+	st.Settings.StatsPerDevice = true
+	text := Render(st, Options{DeviceSets: map[string][]Element{"stats_up_v4": {{Addr: "10.0.10.5", Timeout: 3000, Packets: 7, Bytes: 7000}}}}).Text
+	for _, want := range []string{
+		`set stats_up_v4 { type ipv4_addr; flags dynamic,timeout; timeout 1h; size 65535; counter; elements = { 10.0.10.5 timeout 3000s counter packets 7 bytes 7000 } }`,
+		`set stats_down_v6 { type ipv6_addr; flags dynamic,timeout; timeout 1h; size 65535; counter; }`,
+		`iifname != "wan" oifname "wan" update @stats_up_v4 { ip saddr } comment "hc:b-stats"`,
+		`iifname "wan" oifname != "wan" update @stats_down_v6 { ip6 daddr and ffff:ffff:ffff:ffff:: } comment "hc:b-stats"`,
+	} {
+		if !strings.Contains(text, want) {
+			t.Errorf("missing %q", want)
+		}
+	}
+	// Counted before the established built-in, so every packet is.
+	fwd := text[strings.Index(text, "chain forward {"):]
+	if i := strings.Index(fwd, "update @stats_up_v4"); i < 0 || i > strings.Index(fwd, "hc:b-established") {
+		t.Error("the counting comes after the established built-in")
+	}
+	st.Settings.StatsPerDevice = false
+	if strings.Contains(Render(st, Options{}).Text, "stats_") {
+		t.Error("counting without the setting")
+	}
+}
+
 // TestRenderedRulesetPassesNft has the real nft check the router's
 // ruleset (and a minimal one). Set HOUSECARL_NFT to how nft is run, e.g.
 //
@@ -213,16 +243,37 @@ func TestRenderedRulesetPassesNft(t *testing.T) {
 	if len(cmd) == 0 {
 		t.Skip("HOUSECARL_NFT not set")
 	}
-	for name, st := range map[string]*model.State{"router": router(), "empty": model.NewState()} {
+	stats := router()
+	stats.Settings.StatsPerDevice = true
+	// VLAN sub-interfaces: zones of one VLAN, of all of a parent's, and
+	// the flowtable on one.
+	vlan := router()
+	vlan.Zones = append(vlan.Zones, &model.Zone{ID: 900, Name: "iot", Interfaces: []string{"enp2s0.20"}},
+		&model.Zone{ID: 901, Name: "vlans", Interfaces: []string{"enp3s0.*"}})
+	vlan.Rules = append(vlan.Rules, &model.Rule{ID: 902, Chain: model.ChainForward, Position: 99, Action: model.ActionAccept,
+		Match: model.Match{InZones: []string{"iot", "vlans"}, OutZones: []string{"wan"}}})
+	vlan.Settings.FlowtableDevices = []string{"lo", "enp2s0.20"}
+	for name, st := range map[string]*model.State{"router": router(), "empty": model.NewState(), "stats": stats, "vlan": vlan} {
 		q, qs := quotaOptions(st)
-		rs := Render(st, Options{Elements: map[string][]Element{"guests": {{Addr: "10.0.30.5", Timeout: 1700}, {Addr: "2a02:8013:7f3d:30::5"}}}, Quotas: q, QuotaSets: qs})
+		// Enforcing, and in shadow mode after and before other tables.
+		for _, mode := range []Options{{}, {Shadow: true}, {Shadow: true, ShadowBefore: true}} {
+			opt := Options{Elements: map[string][]Element{"guests": {{Addr: "10.0.30.5", Timeout: 1700}, {Addr: "2a02:8013:7f3d:30::5"}}}, Quotas: q, QuotaSets: qs,
+				DeviceSets: map[string][]Element{"stats_up_v4": {{Addr: "10.0.10.5", Timeout: 3000, Packets: 7, Bytes: 7000}}, "stats_down_v6": {{Addr: "2001:db8:1:2::", Packets: 1, Bytes: 100}}},
+				Shadow:     mode.Shadow, ShadowBefore: mode.ShadowBefore}
+			nftCheck(t, cmd, fmt.Sprintf("%s (shadow %v, before %v)", name, mode.Shadow, mode.ShadowBefore), Render(st, opt).Text)
+		}
+	}
+	nftCheck(t, cmd, "shadow fallback", ShadowFallback())
+}
+
+func nftCheck(t *testing.T, cmd []string, name, text string) {
+	t.Helper()
 	c := exec.CommandContext(context.Background(), cmd[0], append(cmd[1:], "-c", "-f", "/dev/stdin")...)
-		c.Stdin = strings.NewReader(rs.Text)
+	c.Stdin = strings.NewReader(text)
 	if out, err := c.CombinedOutput(); err != nil {
 		t.Errorf("%s: nft -c: %v\n%s", name, err, out)
-			for i, l := range strings.Split(rs.Text, "\n") {
+		for i, l := range strings.Split(text, "\n") {
 			t.Logf("%3d %s", i+1, l)
 		}
 	}
 }
-}

internal/render/shadow.go

@@ -0,0 +1,127 @@
+package render
+
+import (
+	"fmt"
+	"strings"
+
+	"housecarl/internal/model"
+)
+
+// effects are the words of a raw rule's statements that act on packets
+// (or jump away). In shadow mode a raw rule with any of them is left out,
+// unless its only one is a final drop or reject, which becomes an accept.
+var effects = map[string]bool{
+	"drop": true, "reject": true, "queue": true,
+	"dnat": true, "snat": true, "masquerade": true, "redirect": true, "tproxy": true,
+	"notrack": true, "dup": true, "fwd": true, "synproxy": true, "flow": true, "offload": true,
+	"set": true, "reset": true, "jump": true, "goto": true, "return": true,
+}
+
+// ShadowRaw rewrites a raw rule for shadow mode: as written when it only
+// matches, counts, logs or accepts; with its final drop or reject made an
+// accept (marked "[would drop]" in its comment); ok is false when it does
+// anything else to packets (NAT, setting a mark or header field, a jump,
+// a verdict map…), and must be left out.
+func ShadowRaw(raw string) (out string, ok bool) {
+	toks := tokens(raw)
+	verdict := -1
+	for i := 0; i < len(toks); i++ {
+		w := word(toks[i])
+		if w == "comment" {
+			i++ // its text
+			continue
+		}
+		if !effects[w] {
+			continue
+		}
+		if verdict >= 0 || (w != "drop" && w != "reject") {
+			return "", false
+		}
+		verdict = i
+		// What follows a verdict, up to the comment, must be a reject's
+		// "with …".
+		j := i + 1
+		for j < len(toks) && word(toks[j]) != "comment" {
+			j++
+		}
+		if j > i+1 && (w == "drop" || word(toks[i+1]) != "with") {
+			return "", false
+		}
+		i = j - 1
+	}
+	if verdict < 0 {
+		return raw, true
+	}
+	mark := Would(word(toks[verdict]))
+	head := append(append([]string{}, toks[:verdict]...), "accept")
+	for i := verdict + 1; i < len(toks); i++ {
+		if word(toks[i]) != "comment" {
+			continue
+		}
+		if i+1 < len(toks) {
+			text := strings.Trim(toks[i+1], `"`)
+			return strings.Join(append(head, "comment", quote(strings.TrimSpace(mark+" "+text))), " "), true
+		}
+	}
+	return strings.Join(append(head, "comment", quote(mark)), " "), true
+}
+
+// tokens splits a rule into words, a quoted string being one.
+func tokens(s string) []string {
+	var out []string
+	var cur strings.Builder
+	quoted := false
+	for _, c := range s {
+		switch {
+		case c == '"':
+			quoted = !quoted
+			cur.WriteRune(c)
+		case !quoted && (c == ' ' || c == '\t'):
+			if cur.Len() > 0 {
+				out = append(out, cur.String())
+				cur.Reset()
+			}
+		default:
+			cur.WriteRune(c)
+		}
+	}
+	if cur.Len() > 0 {
+		out = append(out, cur.String())
+	}
+	return out
+}
+
+// word is a token as a keyword: without the punctuation of sets and maps
+// around it; a quoted string is no keyword.
+func word(t string) string {
+	if strings.HasPrefix(t, `"`) {
+		return ""
+	}
+	return strings.ToLower(strings.Trim(t, ",{}:()"))
+}
+
+// ShadowLeftOut lists the raw rules shadow mode leaves out (ShadowRaw).
+func ShadowLeftOut(st *model.State) []*model.Rule {
+	var out []*model.Rule
+	for _, r := range st.Rules {
+		if r.Raw == "" || r.Disabled {
+			continue
+		}
+		if _, ok := ShadowRaw(r.Raw); !ok {
+			out = append(out, r)
+		}
+	}
+	return out
+}
+
+// ShadowFallback is the boot fallback in shadow mode: the other firewall
+// protects the machine, so it is an empty table that filters nothing,
+// with a rule outside every hook that tells housecarld it is in force.
+func ShadowFallback() string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "# Housecarl's fallback in shadow mode: loaded at boot only if ruleset.nft\n")
+	fmt.Fprintf(&b, "# fails to load. It filters nothing; the other firewall decides.\n")
+	fmt.Fprintf(&b, "table inet %s {}\ndelete table inet %s\n", Table, Table)
+	fmt.Fprintf(&b, "table inet %s {\n\tchain fallback {\n\t\tcounter comment \"hc:%s-shadow\"\n\t}\n}\n", Table, FallbackMark)
+	return b.String()
+}

internal/render/shadow_test.go

@@ -0,0 +1,91 @@
+package render
+
+import (
+	"regexp"
+	"strings"
+	"testing"
+
+	"housecarl/internal/model"
+)
+
+// TestShadow: in shadow mode nothing in the ruleset drops, rejects,
+// translates or changes a packet, and what would is counted and marked.
+func TestShadow(t *testing.T) {
+	st := router()
+	st.Rules = append(st.Rules,
+		&model.Rule{ID: 950, Chain: model.ChainInput, Position: 99, Raw: `tcp dport 9999 counter reject with tcp reset comment "old"`},
+		&model.Rule{ID: 951, Chain: model.ChainForward, Position: 99, Raw: "meta mark set 0x1 accept"})
+	st.Settings.OutputPolicy = model.PolicyDrop
+	text := Render(st, Options{Shadow: true}).Text
+	// What acts on packets: verdicts, NAT, changes, the flowtable.
+	acts := regexp.MustCompile(`(?m)(\b(drop|reject|dnat|snat|masquerade|redirect)\b|maxseg size set|flow add|flowtable|policy drop)[^"\n]*$`)
+	for _, l := range strings.Split(text, "\n") {
+		code, _, _ := strings.Cut(l, "comment ")
+		code = strings.ReplaceAll(code, "ct status dnat", "") // a match
+		if strings.HasPrefix(strings.TrimSpace(l), "#") {
+			continue
+		}
+		if m := acts.FindString(code); m != "" && !strings.Contains(code, "log prefix") {
+			t.Errorf("acts on packets in shadow mode: %s", l)
+		}
+	}
+	for _, want := range []string{
+		"type filter hook input priority filter + 10; policy accept;",
+		"type filter hook prerouting priority raw + 10; policy accept;",
+		"type nat hook prerouting priority dstnat - 10; policy accept;",
+		"type nat hook postrouting priority srcnat - 10; policy accept;",
+		`ct state invalid counter accept comment "hc:b-invalid [would drop]"`,
+		`counter comment "hc:p-input [would drop]"`,
+		`counter comment "hc:p-output [would drop]"`,
+		`log prefix "hc output policy drop: "`,
+		`[would reject]`,
+		`[would dnat] jellyfin`,
+		`[would masquerade]`,
+		`[would redirect]`,
+		`tcp dport 9999 counter accept comment "[would reject] old"`,
+		"# left out in shadow mode: meta mark set 0x1 accept",
+		"# Shadow mode:",
+	} {
+		if !strings.Contains(text, want) {
+			t.Errorf("no %q in\n%s", want, text)
+		}
+	}
+	before := Render(st, Options{Shadow: true, ShadowBefore: true}).Text
+	for _, want := range []string{"priority filter - 10;", "priority raw - 10;", "priority dstnat - 10;"} {
+		if !strings.Contains(before, want) {
+			t.Errorf("before: no %q", want)
+		}
+	}
+	// Enforcing is unchanged by it.
+	if strings.Contains(Render(st, Options{}).Text, "would") {
+		t.Error("shadow markers when enforcing")
+	}
+	if got := ShadowLeftOut(st); len(got) != 1 || got[0].ID != 951 {
+		t.Errorf("left out: %v", got)
+	}
+}
+
+func TestShadowRaw(t *testing.T) {
+	cases := []struct{ in, out string }{
+		{"meta skuid 0 counter accept", "meta skuid 0 counter accept"},
+		{"tcp dport 22 drop", `tcp dport 22 accept comment "[would drop]"`},
+		{`tcp dport 22 counter drop comment "no ssh"`, `tcp dport 22 counter accept comment "[would drop] no ssh"`},
+		{"ip saddr 10.0.0.1 reject with icmpx type admin-prohibited", `ip saddr 10.0.0.1 accept comment "[would reject]"`},
+		{"tcp dport 22 reject with tcp reset", `tcp dport 22 accept comment "[would reject]"`},
+		{`log prefix "drop me: " group 5 accept`, `log prefix "drop me: " group 5 accept`},
+		{"ip saddr { 10.0.0.1, 10.0.0.2 } counter", "ip saddr { 10.0.0.1, 10.0.0.2 } counter"},
+		{"tcp dport vmap { 22 : drop, 80 : accept }", ""},
+		{"meta mark set 1", ""},
+		{"ip daddr 1.2.3.4 dnat ip to 10.0.0.1", ""},
+		{"tcp dport 22 jump other", ""},
+		{"drop accept", ""},
+		{"meta nftrace set 1", ""},
+		{"ct state new notrack", ""},
+	}
+	for _, c := range cases {
+		out, ok := ShadowRaw(c.in)
+		if ok != (c.out != "") || out != c.out {
+			t.Errorf("%q: got %q %v, want %q", c.in, out, ok, c.out)
+		}
+	}
+}

internal/server/authn.go

@@ -275,7 +275,7 @@ func (s *Server) logout(w http.ResponseWriter, r *http.Request, p *auth.Principa
 }
 
 func (s *Server) whoamiOf(p *auth.Principal, se *model.Session) api.Whoami {
-	out := api.Whoami{User: p.User, Admin: p.IsAdmin(), Method: p.Method, Permissions: p.Permissions(), Build: version.Info()}
+	out := api.Whoami{User: p.User, Admin: p.IsAdmin(), Method: p.Method, Permissions: p.Permissions(), Build: version.Info(), Mode: s.mode()}
 	if se != nil {
 		out.CSRFToken = se.CSRFToken
 	}

internal/server/h_changesets.go

@@ -36,8 +36,8 @@ func changesetParam(r *http.Request) (int64, error) {
 	return id, nil
 }
 
-// notStaged refuses a changeset on a change that can't be staged (undo,
-// import, takeover): it would otherwise be applied at once, unexpectedly.
+// notStaged refuses a changeset on a change that can't be staged (an
+// import, a takeover): it would otherwise be applied at once, unexpectedly.
 func notStaged(r *http.Request, what string) error {
 	if id, err := changesetParam(r); err != nil || id == 0 {
 		return err
@@ -102,6 +102,11 @@ func (s *Server) stage(w http.ResponseWriter, r *http.Request, p *auth.Principal
 		return nil
 	}
 	op.Summary = engine.Describe(changes)
+	if op.Op == model.OpRevert {
+		var h model.HistoryEntry
+		json.Unmarshal(op.Data, &h)
+		op.Summary = engine.UndoDescription(&h)
+	}
 	op.AddedAt = s.nowFn()
 	for _, o := range cs.Ops {
 		if o.Seq >= op.Seq {
@@ -353,7 +358,17 @@ func (s *Server) applyChangeset(w http.ResponseWriter, r *http.Request, p *auth.
 	if len(cs.Ops) == 0 {
 		return badRequest("changeset %d has no changes to apply", id)
 	}
-	res, err := s.eng.Apply(r.Context(), p, cs.Description, engine.Options{Via: via(r, p)}, func(st *model.State, _ func() int64) error {
+	opts := engine.Options{Via: via(r, p)}
+	desc := cs.Description
+	for _, o := range cs.Ops {
+		if o.Op == model.OpRevert {
+			opts.RevertOf = o.ID // History links the undo
+			if desc == "" && len(cs.Ops) == 1 {
+				desc = o.Summary
+			}
+		}
+	}
+	res, err := s.eng.Apply(r.Context(), p, desc, opts, func(st *model.State, _ func() int64) error {
 		return replay(st, cs.Ops)
 	})
 	if err != nil {
@@ -368,6 +383,7 @@ func (s *Server) applyChangeset(w http.ResponseWriter, r *http.Request, p *auth.
 	s.eng.Audit(r.Context(), p, "changeset.apply", fmt.Sprintf("changeset %d", id), "success", fmt.Sprintf("history %d", res.HistoryID))
 	s.afterChange(r.Context(), p, res)
 	out := s.mutation(res, nil)
+	out.ConfirmToken = s.issueCheck(w, r, p, res)
 	out.Changeset = cs
 	writeJSON(w, http.StatusOK, out)
 	return nil

internal/server/h_config.go

@@ -4,7 +4,9 @@ import (
 	"encoding/json"
 	"errors"
 	"net/http"
+	"net/netip"
 	"sort"
+	"strconv"
 	"strings"
 
 	udiff "github.com/aymanbagabas/go-udiff"
@@ -12,7 +14,9 @@ import (
 	"housecarl/internal/api"
 	"housecarl/internal/auth"
 	"housecarl/internal/engine"
+	"housecarl/internal/matrix"
 	"housecarl/internal/model"
+	"housecarl/internal/nft"
 	"housecarl/internal/store"
 )
 
@@ -48,7 +52,9 @@ func (s *Server) change(w http.ResponseWriter, r *http.Request, p *auth.Principa
 		return nil
 	}
 	s.afterChange(r.Context(), p, res)
-	writeJSON(w, s.mutationStatus(res, op.Op == model.OpCreate), s.mutation(res, objectOf(&op, id)))
+	out := s.mutation(res, objectOf(&op, id))
+	out.ConfirmToken = s.issueCheck(w, r, p, res)
+	writeJSON(w, s.mutationStatus(res, op.Op == model.OpCreate), out)
 	return nil
 }
 
@@ -83,8 +89,10 @@ func (s *Server) mutationStatus(res *engine.Result, creates bool) int {
 }
 
 // respondChange answers a change made without an operation (undo).
-func (s *Server) respondChange(w http.ResponseWriter, r *http.Request, res *engine.Result, object func(st *model.State) any, creates bool) {
-	writeJSON(w, s.mutationStatus(res, creates), s.mutation(res, object))
+func (s *Server) respondChange(w http.ResponseWriter, r *http.Request, p *auth.Principal, res *engine.Result, object func(st *model.State) any, creates bool) {
+	out := s.mutation(res, object)
+	out.ConfirmToken = s.issueCheck(w, r, p, res)
+	writeJSON(w, s.mutationStatus(res, creates), out)
 }
 
 // rulesetDiff is the unified diff of the rulesets two configurations
@@ -303,7 +311,19 @@ func (s *Server) createObject(w http.ResponseWriter, r *http.Request, p *auth.Pr
 	if _, err := decodeObject(kind, raw); err != nil {
 		return err
 	}
-	return s.change(w, r, p, model.Op{Op: model.OpCreate, Kind: kind, Data: raw})
+	op := model.Op{Op: model.OpCreate, Kind: kind, Data: raw}
+	// ?before=ID, ?after=ID or ?first=1: where it goes in its order.
+	q := r.URL.Query()
+	var place api.MoveInput
+	place.Before, _ = strconv.ParseInt(q.Get("before"), 10, 64)
+	place.After, _ = strconv.ParseInt(q.Get("after"), 10, 64)
+	if queryBool(r, "first") {
+		place.To = "start"
+	}
+	if place.Before != 0 || place.After != 0 || place.To != "" {
+		op.Place, _ = json.Marshal(place)
+	}
+	return s.change(w, r, p, op)
 }
 
 func (s *Server) updateObject(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
@@ -375,6 +395,47 @@ func usedBy(st *model.State, kind string, o model.Object) string {
 	return ""
 }
 
+// renameRefs makes every object that refers to a zone, set or service by
+// its old name use the new one: renaming it changes them in the same
+// change, so the history shows (and undoes) them together.
+func renameRefs(st *model.State, kind, old, now string) {
+	fix := func(list []string) {
+		for i, x := range list {
+			if strings.EqualFold(x, old) {
+				list[i] = now
+			}
+		}
+	}
+	match := func(m *model.Match) {
+		switch kind {
+		case model.KindZone:
+			fix(m.InZones)
+			fix(m.OutZones)
+		case model.KindSet:
+			fix(m.Sources)
+			fix(m.Destinations)
+		case model.KindService:
+			fix(m.Services)
+		}
+	}
+	for _, x := range st.Rules {
+		match(&x.Match)
+	}
+	for _, x := range st.SNATs {
+		match(&x.Match)
+	}
+	for _, x := range st.Redirects {
+		match(&x.Match)
+	}
+	if kind == model.KindSet {
+		for _, f := range st.Forwards {
+			fix(f.Sources)
+			fix(f.PublicAddresses)
+			fix(f.HairpinSources)
+		}
+	}
+}
+
 // moveObject reorders a rule or NAT rule (moveOp).
 func (s *Server) moveObject(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
 	kind, err := pathKind(r)
@@ -412,6 +473,70 @@ func (s *Server) updateSettings(w http.ResponseWriter, r *http.Request, p *auth.
 	return s.change(w, r, p, model.Op{Op: model.OpSettings, Data: data})
 }
 
+// ---- the zone matrix
+
+// zoneMatrix answers what the rules do to new connections between zones
+// (internal/matrix); with a changeset named, as it will make them.
+func (s *Server) zoneMatrix(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	if err := require(p, auth.ResConfig, false); err != nil {
+		return err
+	}
+	_, st, _, err := s.viewState(w, r)
+	if err != nil {
+		return err
+	}
+	m := matrix.Build(st, matrix.Options{Connected: s.connected(st)})
+	if t, err := s.fw.Table(r.Context()); err == nil {
+		matrixHits(&m, t.Counters)
+	}
+	writeJSON(w, http.StatusOK, m)
+	return nil
+}
+
+// matrixHits fills in the hits of the matrix's rules, built-ins and
+// policies from the counters.
+func matrixHits(m *api.Matrix, counters map[string]nft.Counter) {
+	hits := func(marks []string) *uint64 {
+		var n uint64
+		found := false
+		for _, mk := range marks {
+			if c, ok := counters[mk]; ok {
+				n, found = n+c.Packets, true
+			}
+		}
+		if !found {
+			return nil
+		}
+		return &n
+	}
+	for i := range m.Cells {
+		c := &m.Cells[i]
+		c.Hits = hits(c.Marks)
+		for j := range c.Exceptions {
+			c.Exceptions[j].Hits = hits(c.Exceptions[j].Marks)
+		}
+	}
+}
+
+// connected maps the networks on the firewall's interfaces (link-local
+// and loopback aside) to their zones.
+func (s *Server) connected(st *model.State) map[netip.Prefix]string {
+	out := map[netip.Prefix]string{}
+	for _, i := range s.hostInterfaces(st) {
+		if i.Zone == "" {
+			continue
+		}
+		for _, a := range i.Addresses {
+			p, err := netip.ParsePrefix(a)
+			if err != nil || p.Addr().IsLinkLocalUnicast() || p.Addr().IsLoopback() {
+				continue
+			}
+			out[p.Masked()] = i.Zone
+		}
+	}
+	return out
+}
+
 // ---- the ruleset
 
 func (s *Server) ruleset(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
@@ -423,6 +548,10 @@ func (s *Server) ruleset(w http.ResponseWriter, r *http.Request, p *auth.Princip
 		return err
 	}
 	rs := s.fw.Render(st)
+	// In shadow mode: what would be loaded once it is off.
+	if queryBool(r, "enforcing") {
+		rs = s.fw.RenderEnforcing(st)
+	}
 	w.Header().Set("Content-Type", "text/plain; charset=utf-8")
 	w.Write([]byte(rs.Text))
 	return nil
@@ -465,20 +594,34 @@ func (s *Server) revertHistory(w http.ResponseWriter, r *http.Request, p *auth.P
 	if err := require(p, auth.ResConfig, true); err != nil {
 		return err
 	}
-	if err := notStaged(r, "an undo"); err != nil {
-		return err
-	}
 	id, err := pathID(r, "id")
 	if err != nil {
 		return err
 	}
+	// Staged, an undo is an operation of its own, reviewed like any.
+	if csID, err := changesetParam(r); err != nil {
+		return err
+	} else if csID != 0 {
+		h, err := s.store.GetHistory(r.Context(), id)
+		if errors.Is(err, store.ErrNotFound) {
+			return notFound("history entry %d not found", id)
+		}
+		if err != nil {
+			return err
+		}
+		if h.RevertedBy != 0 {
+			return conflict("history entry %d was already undone by entry %d", id, h.RevertedBy)
+		}
+		data, _ := json.Marshal(h)
+		return s.stage(w, r, p, csID, model.Op{Op: model.OpRevert, ID: id, Data: data})
+	}
 	res, err := s.eng.Revert(r.Context(), p, id, engine.Options{DryRun: queryBool(r, "dry_run"), Via: via(r, p)})
 	if err != nil {
 		s.writeErr(w, r, err, res)
 		return nil
 	}
 	s.afterChange(r.Context(), p, res)
-	s.respondChange(w, r, res, nil, false)
+	s.respondChange(w, r, p, res, nil, false)
 	return nil
 }
 
@@ -496,7 +639,17 @@ func (s *Server) confirmChanges(w http.ResponseWriter, r *http.Request, p *auth.
 	if err := require(p, auth.ResConfig, true); err != nil {
 		return err
 	}
-	n, err := s.confirm(r.Context(), p)
+	var in api.ConfirmInput
+	if r.ContentLength != 0 {
+		if err := decodeJSON(r, &in); err != nil {
+			return err
+		}
+	}
+	confirm := func() (int, error) { return s.confirm(r.Context(), p) }
+	if in.Token != "" {
+		confirm = func() (int, error) { return s.confirmToken(r, p, in.Token) }
+	}
+	n, err := confirm()
 	if err != nil {
 		return err
 	}

internal/server/h_health.go

@@ -0,0 +1,77 @@
+package server
+
+import (
+	"net/http"
+	"sync"
+	"time"
+
+	"housecarl/internal/version"
+)
+
+// fallbackInForce is what status and /healthz say while the boot fallback
+// (render.Fallback) is in force.
+const fallbackInForce = "the boot fallback is in force, not Housecarl's ruleset: the saved ruleset failed to load at boot (journalctl -u housecarl-ruleset), and housecarld couldn't apply it since; it lets nothing new in from external zones (with none, nothing but loopback)"
+
+// health is GET /healthz, for monitoring (Checkmk, a load balancer). It
+// needs no authentication, so it says only what a monitor needs, and reads
+// the ruleset at most every few seconds however often it is asked.
+type health struct {
+	OK       bool   `json:"ok"`
+	Version  string `json:"version"`
+	Firewall string `json:"firewall"` // applied, unconfigured, fallback, missing
+	// Mode is enforcing, or shadow: the ruleset blocks nothing.
+	Mode string `json:"mode"`
+	// Probation: changes waiting to be confirmed, rolled back otherwise.
+	Probation int `json:"probation"`
+	// Warnings are the codes of the machine's problems (nftables_service).
+	Warnings []string `json:"warnings"`
+}
+
+const healthTTL = 5 * time.Second
+
+type healthCache struct {
+	mu  sync.Mutex
+	at  time.Time
+	out health
+}
+
+func (s *Server) healthz(w http.ResponseWriter, r *http.Request) {
+	c := &s.health
+	c.mu.Lock()
+	if time.Since(c.at) >= healthTTL {
+		c.out, c.at = s.checkHealth(r), time.Now()
+	}
+	out := c.out
+	c.mu.Unlock()
+	code := http.StatusOK
+	if !out.OK {
+		code = http.StatusServiceUnavailable
+	}
+	writeJSON(w, code, out)
+}
+
+func (s *Server) checkHealth(r *http.Request) health {
+	out := health{Version: version.String(), Warnings: []string{}, Mode: s.mode()}
+	t, err := s.fw.Table(r.Context())
+	switch {
+	case err != nil && s.eng.State().Empty():
+		out.Firewall, out.OK = "unconfigured", true
+	case err != nil:
+		out.Firewall = "missing"
+	case t.Fallback:
+		out.Firewall = "fallback"
+	default:
+		out.Firewall, out.OK = "applied", true
+	}
+	pr := s.probationStatus()
+	out.Probation = len(pr.Pending) + pr.Takeovers
+	// The other tables matter in shadow mode (shadow_unprotected).
+	var ruleset []byte
+	if s.cfg.Shadow {
+		ruleset, _ = s.fw.ListRuleset(r.Context())
+	}
+	for _, p := range s.hostProblems(r.Context(), ruleset) {
+		out.Warnings = append(out.Warnings, p.Code)
+	}
+	return out
+}

internal/server/h_health_test.go

@@ -0,0 +1,65 @@
+package server
+
+import (
+	"encoding/json"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"housecarl/internal/api"
+)
+
+func TestHealthz(t *testing.T) {
+	e := newEnv(t)
+	link := filepath.Join(t.TempDir(), "nftables.service")
+	e.srv.unitLinks = []string{link}
+	get := func() (int, health) {
+		t.Helper()
+		e.srv.health.at = time.Time{} // no caching between the steps
+		resp, err := http.Get(e.ts.URL + "/healthz")
+		if err != nil {
+			t.Fatal(err)
+		}
+		defer resp.Body.Close()
+		var h health
+		json.NewDecoder(resp.Body).Decode(&h)
+		return resp.StatusCode, h
+	}
+	// Nothing configured yet: nothing to be in force.
+	if code, h := get(); code != 200 || !h.OK || h.Firewall != "unconfigured" || h.Version == "" {
+		t.Errorf("unconfigured: %d %+v", code, h)
+	}
+	if code := e.call(e.token, "POST", "/zones", map[string]any{"name": "wan", "interfaces": []string{"wan"}, "external": true}, nil); code != 201 {
+		t.Fatalf("creating a zone: %d", code)
+	}
+	if code, h := get(); code != 200 || h.Firewall != "applied" || h.Probation != 1 || len(h.Warnings) != 0 {
+		t.Errorf("applied, on probation: %d %+v", code, h)
+	}
+	os.WriteFile(link, nil, 0o644)
+	if _, h := get(); strings.Join(h.Warnings, ",") != "nftables_service" {
+		t.Errorf("warnings %v", h.Warnings)
+	}
+	e.fw.mu.Lock()
+	e.fw.fallback = true
+	e.fw.mu.Unlock()
+	if code, h := get(); code != 503 || h.OK || h.Firewall != "fallback" {
+		t.Errorf("fallback: %d %+v", code, h)
+	}
+	var st api.Status
+	e.call(e.token, "GET", "/status", nil, &st)
+	if st.Applied || !strings.Contains(st.ApplyError, "boot fallback") {
+		t.Errorf("status with the fallback: applied %v, %q", st.Applied, st.ApplyError)
+	}
+	// Asked often, the ruleset is read once per few seconds.
+	e.fw.mu.Lock()
+	e.fw.fallback = false
+	e.fw.mu.Unlock()
+	resp, _ := http.Get(e.ts.URL + "/healthz")
+	resp.Body.Close()
+	if resp.StatusCode != 503 {
+		t.Errorf("not cached: %d", resp.StatusCode)
+	}
+}

internal/server/h_import.go

@@ -25,7 +25,7 @@ func (s *Server) importNft(w http.ResponseWriter, r *http.Request, p *auth.Princ
 	if err := decodeJSON(r, &in); err != nil {
 		return err
 	}
-	res, err := importer.Import(importer.Input{JSON: in.JSON, Text: in.Text, Defines: in.Defines, Table: in.Table})
+	res, err := importer.Import(importer.Input{JSON: in.JSON, Text: in.Text, Defines: in.Defines, Table: in.Table, Zones: in.Zones})
 	if err != nil {
 		return badRequest("%v", err)
 	}
@@ -51,6 +51,12 @@ func (s *Server) importNft(w http.ResponseWriter, r *http.Request, p *auth.Princ
 		return nil
 	}
 	s.afterChange(r.Context(), p, er)
+	if er.Applied {
+		// Until it is taken over, the old table still applies too: the
+		// status warns (takeover_pending; not in shadow mode, where that
+		// is the point).
+		s.store.SetMeta(r.Context(), importedTableKey, res.Table)
+	}
 	out.MutationResponse = api.MutationResponse{Applied: er.Applied, DryRun: er.DryRun, HistoryID: er.HistoryID, Changes: er.Changes, Warnings: er.Warnings}
 	if out.Changes == nil {
 		out.Changes = []model.Change{}
@@ -58,6 +64,7 @@ func (s *Server) importNft(w http.ResponseWriter, r *http.Request, p *auth.Princ
 	if pr := s.probationStatus(); er.Applied && len(pr.Pending) > 0 {
 		out.ConfirmBy = &pr.Deadline
 	}
+	out.ConfirmToken = s.issueCheck(w, r, p, er)
 	writeJSON(w, http.StatusOK, out)
 	return nil
 }
@@ -111,6 +118,9 @@ func (s *Server) takeover(w http.ResponseWriter, r *http.Request, p *auth.Princi
 	if err := notStaged(r, "a takeover"); err != nil {
 		return err
 	}
+	if err := s.refuseInShadow("delete other tables (the firewall in force)"); err != nil {
+		return err
+	}
 	var in api.TakeoverInput
 	if err := decodeJSON(r, &in); err != nil {
 		return err
@@ -148,6 +158,12 @@ func (s *Server) takeover(w http.ResponseWriter, r *http.Request, p *auth.Princi
 	} else if pr := s.probationStatus(); pr.Takeovers > 0 {
 		msg += fmt.Sprintf("; confirm with 'hc confirm' within %ds, or they come back", pr.SecondsLeft)
 	}
-	writeJSON(w, http.StatusOK, map[string]any{"deleted": done, "message": msg})
+	answer := map[string]any{"deleted": done, "message": msg}
+	if !queryBool(r, "dry_run") && len(done) > 0 {
+		if tok := s.issueCheck(w, r, p, nil); tok != "" {
+			answer["confirm_token"] = tok
+		}
+	}
+	writeJSON(w, http.StatusOK, answer)
 	return nil
 }

internal/server/h_stats.go

@@ -0,0 +1,266 @@
+package server
+
+import (
+	"net/http"
+	"strconv"
+	"strings"
+
+	"housecarl/internal/api"
+	"housecarl/internal/auth"
+	"housecarl/internal/model"
+	"housecarl/internal/stats"
+	"housecarl/internal/validate"
+)
+
+// Statistics: the firewall's figures over time (internal/stats).
+
+// StatsSources fills in what the statistics collector needs from the
+// server: the interfaces' zones, and which counters count drops.
+func (s *Server) StatsSources(src *stats.Sources) {
+	src.Zones = s.interfaceZones
+	src.Classify = s.classifyMark
+}
+
+// SetStats gives the server the statistics collector (nil: statistics
+// are off).
+func (s *Server) SetStats(c *stats.Collector) { s.stats = c }
+
+// interfaceZones maps the machine's interfaces to their zones.
+func (s *Server) interfaceZones() map[string]string {
+	out := map[string]string{}
+	for _, i := range s.hostInterfaces(s.eng.State()) {
+		if i.Zone != "" {
+			out[i.Name] = i.Zone
+		}
+	}
+	return out
+}
+
+// classifyMark says which chain a counter is in and whether what it
+// counts is dropped: rules that drop or reject, the drop policies, and
+// the built-ins that drop (antispoofing, invalid packets).
+func (s *Server) classifyMark(mark string) (string, bool) {
+	st := s.eng.State()
+	switch {
+	case strings.HasPrefix(mark, "p-"):
+		chain := strings.TrimPrefix(mark, "p-")
+		policy := map[string]string{"input": st.Settings.InputPolicy, "forward": st.Settings.ForwardPolicy, "output": st.Settings.OutputPolicy}[chain]
+		return chain, policy == model.PolicyDrop
+	case strings.HasPrefix(mark, "b-antispoof"):
+		return model.ChainEarly, true
+	case mark == "b-invalid":
+		return "invalid", true
+	case strings.HasPrefix(mark, "r"):
+		num := strings.TrimLeft(mark[1:], "0123456789")
+		id, _ := strconv.ParseInt(strings.TrimSuffix(mark[1:], num), 10, 64)
+		if o, ok := st.ByID(model.KindRule, id).(*model.Rule); ok && o != nil {
+			return o.Chain, o.Action == model.ActionDrop || o.Action == model.ActionReject
+		}
+	}
+	return "", false
+}
+
+// markName names a counter's object: "rule input #12 (ssh)", "built-in:
+// established", "input policy".
+func markName(st *model.State, mark string) string {
+	if c, ok := strings.CutPrefix(mark, "p-"); ok {
+		return c + " policy"
+	}
+	if l := markLabel(st, api.TraceStep{Kind: "rule", Mark: mark}); l != "" {
+		return l
+	}
+	if len(mark) < 2 {
+		return mark
+	}
+	// An object deleted since it counted.
+	kind := map[byte]string{'r': "rule", 'f': "port forward", 's': "source NAT", 'd': "redirect"}[mark[0]]
+	if num := strings.TrimLeft(mark[1:], "0123456789"); kind != "" && len(num) < len(mark)-1 {
+		return kind + " #" + strings.TrimSuffix(mark[1:], num) + " (deleted)"
+	}
+	return mark
+}
+
+func (s *Server) statsCollector() (*stats.Collector, error) {
+	if s.stats == nil {
+		return nil, &httpError{http.StatusServiceUnavailable, "stats_off", "statistics are off: set stats.enabled in housecarl.yaml and restart housecarld"}
+	}
+	return s.stats, nil
+}
+
+func (s *Server) statsQuerier(p *auth.Principal) (*stats.Querier, error) {
+	if err := require(p, auth.ResLog, false); err != nil {
+		return nil, err
+	}
+	c, err := s.statsCollector()
+	if err != nil {
+		return nil, err
+	}
+	q := stats.NewQuerier(c)
+	q.Now = s.nowFn
+	return q, nil
+}
+
+func since(r *http.Request, def string) (string, error) {
+	v := r.URL.Query().Get("since")
+	if v == "" {
+		v = def
+	}
+	if _, err := stats.ParseSince(v); err != nil {
+		return "", badRequest("%v", err)
+	}
+	return v, nil
+}
+
+// StatsStatus is the collector's health, with the setting that counts
+// traffic per device.
+type StatsStatus struct {
+	stats.Status
+	Latest *stats.Latest `json:"latest,omitempty"`
+}
+
+func (s *Server) statsStatus(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	if err := require(p, auth.ResLog, false); err != nil {
+		return err
+	}
+	out := StatsStatus{}
+	if s.stats != nil {
+		out.Status = s.stats.Status()
+		out.Latest = s.stats.Latest()
+		if !p.IsAdmin() {
+			out.Database = ""
+		}
+	}
+	out.PerDevice = s.eng.State().Settings.StatsPerDevice
+	writeJSON(w, http.StatusOK, out)
+	return nil
+}
+
+func (s *Server) statsSummary(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	q, err := s.statsQuerier(p)
+	if err != nil {
+		return err
+	}
+	v, err := since(r, "24h")
+	if err != nil {
+		return err
+	}
+	d, _ := stats.ParseSince(v)
+	sum, err := q.Summary(r.Context(), d, queryInt(r, "limit", 10))
+	if err != nil {
+		return err
+	}
+	st := s.eng.State()
+	for i := range sum.Rules {
+		sum.Rules[i].Name = markName(st, sum.Rules[i].Key)
+	}
+	writeJSON(w, http.StatusOK, sum)
+	return nil
+}
+
+func (s *Server) statsTimeline(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	q, err := s.statsQuerier(p)
+	if err != nil {
+		return err
+	}
+	v, err := since(r, "24h")
+	if err != nil {
+		return err
+	}
+	d, _ := stats.ParseSince(v)
+	var keys []string
+	for _, k := range strings.Split(r.URL.Query().Get("keys"), ",") {
+		if k = strings.TrimSpace(k); k != "" {
+			keys = append(keys, k)
+		}
+	}
+	if len(keys) == 0 {
+		return badRequest("say which figures: keys=zrx:*,ztx:* (docs/stats.md)")
+	}
+	tl, err := q.Timeline(r.Context(), d, r.URL.Query().Get("step"), keys)
+	if err != nil {
+		return badRequest("%v", err)
+	}
+	writeJSON(w, http.StatusOK, tl)
+	return nil
+}
+
+func (s *Server) statsTop(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	q, err := s.statsQuerier(p)
+	if err != nil {
+		return err
+	}
+	v, err := since(r, "24h")
+	if err != nil {
+		return err
+	}
+	d, _ := stats.ParseSince(v)
+	es, err := q.Top(r.Context(), r.URL.Query().Get("kind"), d, queryInt(r, "limit", 20))
+	if err != nil {
+		return badRequest("%v", err)
+	}
+	writeJSON(w, http.StatusOK, es)
+	return nil
+}
+
+// statsHits are the packets each rule, built-in and policy counted over
+// the range, named; rules that counted none are listed too, with 0.
+func (s *Server) statsHits(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	q, err := s.statsQuerier(p)
+	if err != nil {
+		return err
+	}
+	v, err := since(r, "30d")
+	if err != nil {
+		return err
+	}
+	d, _ := stats.ParseSince(v)
+	es, err := q.Sums(r.Context(), "hits:", d)
+	if err != nil {
+		return err
+	}
+	st := s.eng.State()
+	seen := map[string]bool{}
+	for i := range es {
+		es[i].Name = markName(st, es[i].Key)
+		seen[es[i].Key] = true
+	}
+	for _, rule := range st.Rules {
+		if mark := "r" + strconv.FormatInt(rule.ID, 10); !seen[mark] && !rule.Disabled {
+			es = append(es, stats.Entry{Key: mark, Name: "rule " + rule.Label()})
+		}
+	}
+	writeJSON(w, http.StatusOK, es)
+	return nil
+}
+
+func (s *Server) statsHeatmap(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	q, err := s.statsQuerier(p)
+	if err != nil {
+		return err
+	}
+	v, err := since(r, "28d")
+	if err != nil {
+		return err
+	}
+	d, _ := stats.ParseSince(v)
+	hm, err := q.Heatmap(r.Context(), r.URL.Query().Get("key"), d)
+	if err != nil {
+		return err
+	}
+	writeJSON(w, http.StatusOK, hm)
+	return nil
+}
+
+// statsProblems warns when the connection tracking table is nearly full:
+// new connections are dropped once it is.
+func (s *Server) statsProblems() []validate.Problem {
+	if s.stats == nil {
+		return nil
+	}
+	l := s.stats.Latest()
+	if l == nil || l.ConntrackMax == 0 || l.Conntrack*100 < l.ConntrackMax*80 {
+		return nil
+	}
+	return []validate.Problem{{Severity: validate.Warning, Code: "conntrack_full", Object: "connections",
+		Message: "the connection tracking table is " + strconv.FormatInt(l.Conntrack*100/l.ConntrackMax, 10) + "% full (" + strconv.FormatInt(l.Conntrack, 10) + " of " + strconv.FormatInt(l.ConntrackMax, 10) + "); once it is, new connections are dropped: raise net.netfilter.nf_conntrack_max, or find what opens so many (Connections)"}}
+}

internal/server/h_status.go

@@ -1,7 +1,10 @@
 package server
 
 import (
+	"context"
+	"fmt"
 	"net/http"
+	"os"
 	"sort"
 	"strings"
 	"sync"
@@ -24,7 +27,10 @@ func (s *Server) status(w http.ResponseWriter, r *http.Request, p *auth.Principa
 	}
 	st := s.eng.State()
 	out := api.Status{Build: version.Info(), Started: s.started, Problems: s.eng.Problems(), Probation: s.probationStatus(),
-		Interfaces: s.hostInterfaces(st), Counts: map[string]int{}, OtherTables: []string{}}
+		Interfaces: s.hostInterfaces(st), Counts: map[string]int{}, OtherTables: []string{}, Mode: s.mode()}
+	if s.cfg.Shadow {
+		out.ShadowPosition = s.cfg.ShadowPosition
+	}
 	if out.Problems == nil {
 		out.Problems = []validate.Problem{}
 	}
@@ -34,20 +40,78 @@ func (s *Server) status(w http.ResponseWriter, r *http.Request, p *auth.Principa
 	for _, k := range model.Kinds {
 		out.Counts[k] = len(st.Objects(k))
 	}
-	if _, err := s.fw.Table(r.Context()); err != nil && st.Empty() {
+	if t, err := s.fw.Table(r.Context()); err != nil && st.Empty() {
 		out.Unconfigured = true
 	} else if err != nil {
 		out.ApplyError = err.Error()
+	} else if t.Fallback {
+		out.ApplyError = fallbackInForce
 	} else {
 		out.Applied = true
 	}
-	if data, err := s.fw.ListRuleset(r.Context()); err == nil {
+	data, err := s.fw.ListRuleset(r.Context())
+	if err == nil {
 		out.OtherTables = otherTables(data)
 	}
+	out.Problems = append(out.Problems, s.hostProblems(r.Context(), data)...)
+	out.Problems = append(out.Problems, s.statsProblems()...)
 	writeJSON(w, http.StatusOK, out)
 	return nil
 }
 
+const importedTableKey = "imported_table"
+
+// shadowProblems are shadow mode's warnings about the machine.
+func (s *Server) shadowProblems(ruleset []byte) []validate.Problem {
+	var out []validate.Problem
+	if ruleset != nil && len(inputFilters(ruleset)) == 0 {
+		out = append(out, validate.Problem{Severity: validate.Warning, Code: "shadow_unprotected", Object: "shadow mode",
+			Message: "housecarld is in shadow mode, where its ruleset blocks nothing, and no other table filters what reaches this machine (none has a filter chain on the input hook): nothing protects it; load your firewall, or leave shadow mode"})
+	}
+	for _, r := range render.ShadowLeftOut(s.eng.State()) {
+		out = append(out, validate.Problem{Severity: validate.Warning, Code: "shadow_raw_left_out", Object: r.Label(),
+			Message: "shadow mode leaves this raw rule out: it does more to packets than accept, drop or reject (NAT, setting a mark or field, a jump, a verdict map…), so its hits aren't counted and later rules see what it would have taken"})
+	}
+	return out
+}
+
+// nftablesUnitLinks are where an enabled nftables.service shows (Debian
+// enables it for sysinit.target).
+var nftablesUnitLinks = []string{
+	"/etc/systemd/system/sysinit.target.wants/nftables.service",
+	"/etc/systemd/system/multi-user.target.wants/nftables.service",
+}
+
+// hostProblems are warnings about the machine rather than the
+// configuration: another ruleset that still loads at boot, or the table
+// an import came from, still in force beside Housecarl's. ruleset is the
+// machine's (nft -j list ruleset; nil: not read). In shadow mode those
+// are expected; the warnings are that nothing else filters, and the raw
+// rules left out.
+func (s *Server) hostProblems(ctx context.Context, ruleset []byte) []validate.Problem {
+	if s.cfg.Shadow {
+		return s.shadowProblems(ruleset)
+	}
+	others := otherTables(ruleset)
+	var out []validate.Problem
+	for _, link := range s.unitLinks {
+		if _, err := os.Lstat(link); err == nil {
+			out = append(out, validate.Problem{Severity: validate.Warning, Code: "nftables_service", Object: "nftables.service",
+				Message: "nftables.service is enabled: at boot it loads /etc/nftables.conf, and stopping or restarting it flushes every table, Housecarl's included; disable it (systemctl disable nftables, without --now)"})
+			break
+		}
+	}
+	if t, err := s.store.GetMeta(ctx, importedTableKey); err == nil && t != "" {
+		for _, o := range others {
+			if o == t {
+				out = append(out, validate.Problem{Severity: validate.Warning, Code: "takeover_pending", Object: "table " + t,
+					Message: fmt.Sprintf("%s, the table you imported, still applies beside Housecarl's: a packet must get past both; once Housecarl's rules are right, delete it with 'hc takeover %q'", t, t)})
+			}
+		}
+	}
+	return out
+}
+
 // hostInterfaces lists the machine's interfaces with their zones.
 func (s *Server) hostInterfaces(st *model.State) []api.HostInterface {
 	out := []api.HostInterface{}
@@ -260,7 +324,13 @@ func (l *logRing) Latest(n int, after time.Time, keep func(api.LogEntry) bool) [
 
 // LogPacket records a packet the firewall logged (internal/nflog calls
 // it).
-func (s *Server) LogPacket(e api.LogEntry) { s.logs.Add(e) }
+func (s *Server) LogPacket(e api.LogEntry) {
+	e.Shadow = s.cfg.Shadow
+	s.logs.Add(e)
+	if s.stats != nil {
+		s.stats.Packet(e)
+	}
+}
 
 func (s *Server) listLog(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
 	if err := require(p, auth.ResLog, false); err != nil {

internal/server/h_traffic.go

@@ -0,0 +1,318 @@
+package server
+
+import (
+	"context"
+	"fmt"
+	"net/http"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"housecarl/internal/api"
+	"housecarl/internal/auth"
+	"housecarl/internal/model"
+	"housecarl/internal/trace"
+)
+
+// Traffic as it happens: the connections the kernel tracks (and closing
+// them, so the rules check them again), and tracing packets through the
+// whole ruleset.
+
+// Conntrack lists and closes tracked connections (internal/conntrack).
+type Conntrack interface {
+	List(ctx context.Context) ([]api.Connection, error)
+	Close(ctx context.Context, c api.Connection) error
+}
+
+// SetConntrack gives the server the kernel's connection tracking.
+func (s *Server) SetConntrack(c Conntrack) { s.conns = c }
+
+func (s *Server) conntrack() (Conntrack, error) {
+	if s.conns == nil {
+		return nil, &httpError{http.StatusServiceUnavailable, "unavailable", "connection tracking isn't available here"}
+	}
+	return s.conns, nil
+}
+
+// connMatches: a search in a connection's addresses, ports (":443"),
+// protocol, state and NAT addresses.
+func connMatches(c api.Connection, q string) bool {
+	if q == "" {
+		return true
+	}
+	hay := strings.ToLower(strings.Join([]string{c.Protocol, c.Source, ":" + strconv.Itoa(c.SourcePort), c.Destination, ":" + strconv.Itoa(c.DestinationPort),
+		c.DNATTo, c.SNATTo, c.State}, " ") + " ")
+	for _, w := range strings.Fields(strings.ToLower(q)) {
+		if strings.HasPrefix(w, ":") {
+			w += " " // a whole port
+		}
+		if !strings.Contains(hay, w) {
+			return false
+		}
+	}
+	return true
+}
+
+func (s *Server) listConnections(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	if err := require(p, auth.ResLog, false); err != nil {
+		return err
+	}
+	ct, err := s.conntrack()
+	if err != nil {
+		return err
+	}
+	all, err := ct.List(r.Context())
+	if err != nil {
+		return fmt.Errorf("listing connections: %w", err)
+	}
+	q := r.URL.Query().Get("q")
+	out := []api.Connection{}
+	for _, c := range all {
+		if connMatches(c, q) {
+			out = append(out, c)
+		}
+	}
+	sort.Slice(out, func(i, j int) bool {
+		if out[i].Source != out[j].Source {
+			return out[i].Source < out[j].Source
+		}
+		return out[i].Destination < out[j].Destination
+	})
+	if n := queryInt(r, "limit", 1000); n > 0 && len(out) > n {
+		out = out[:n]
+	}
+	w.Header().Set("X-Total-Count", strconv.Itoa(len(all)))
+	writeJSON(w, http.StatusOK, out)
+	return nil
+}
+
+// closeConnections deletes tracking entries: the rules check those
+// connections again with their next packet.
+func (s *Server) closeConnections(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	if err := require(p, auth.ResConfig, true); err != nil {
+		return err
+	}
+	if err := s.refuseInShadow("close connections"); err != nil {
+		return err
+	}
+	ct, err := s.conntrack()
+	if err != nil {
+		return err
+	}
+	var in api.CloseConnections
+	if err := decodeJSON(r, &in); err != nil {
+		return err
+	}
+	list := in.Connections
+	if strings.TrimSpace(in.Search) != "" {
+		all, err := ct.List(r.Context())
+		if err != nil {
+			return fmt.Errorf("listing connections: %w", err)
+		}
+		for _, c := range all {
+			if connMatches(c, in.Search) {
+				list = append(list, c)
+			}
+		}
+	} else if len(list) == 0 {
+		return badRequest("say which connections: a list, or a search")
+	}
+	closed := 0
+	for _, c := range list {
+		if err := ct.Close(r.Context(), c); err != nil {
+			return fmt.Errorf("closing %s %s -> %s: %w", c.Protocol, c.Source, c.Destination, err)
+		}
+		closed++
+	}
+	what := in.Search
+	if what == "" && len(list) == 1 {
+		what = fmt.Sprintf("%s %s:%d -> %s:%d", list[0].Protocol, list[0].Source, list[0].SourcePort, list[0].Destination, list[0].DestinationPort)
+	}
+	s.eng.Audit(r.Context(), p, "connections.close", what, "success", fmt.Sprintf("%d closed", closed))
+	writeJSON(w, http.StatusOK, map[string]int{"closed": closed})
+	return nil
+}
+
+// ---- tracing
+
+type tracing struct {
+	mu     sync.Mutex
+	status api.TraceStatus
+	tracer *trace.Tracer
+	cancel context.CancelFunc
+}
+
+func (s *Server) startTrace(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	if err := require(p, auth.ResConfig, true); err != nil {
+		return err
+	}
+	var in api.TraceInput
+	if err := decodeJSON(r, &in); err != nil {
+		return err
+	}
+	if in.Seconds == 0 {
+		in.Seconds = 60
+	}
+	if in.Seconds < 5 || in.Seconds > 600 {
+		return badRequest("trace for 5 to 600 seconds")
+	}
+	script, err := trace.Script(in)
+	if err != nil {
+		return badRequest("%v", err)
+	}
+	t := &s.tracing
+	t.mu.Lock()
+	if t.status.Running {
+		t.mu.Unlock()
+		return conflict("a trace is running already (started %s); stop it first", t.status.Started.Format("15:04:05"))
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(in.Seconds)*time.Second)
+	now := s.nowFn()
+	t.status = api.TraceStatus{Running: true, Input: in, Started: now, Ends: now.Add(time.Duration(in.Seconds) * time.Second), Packets: []api.TracePacket{}}
+	t.tracer = &trace.Tracer{Now: s.nowFn}
+	t.cancel = cancel
+	t.mu.Unlock()
+	s.eng.Audit(r.Context(), p, "trace.start", traceWhat(in), "success", fmt.Sprintf("%d seconds", in.Seconds))
+	go func() {
+		defer cancel()
+		err := s.fw.Trace(ctx, script, func(line string) {
+			t.mu.Lock()
+			defer t.mu.Unlock()
+			t.tracer.Line(line)
+			if t.tracer.Full() {
+				cancel()
+			}
+		})
+		t.mu.Lock()
+		defer t.mu.Unlock()
+		t.status.Running = false
+		if err != nil {
+			t.status.Error = err.Error()
+			s.log.Warn("tracing failed", "err", err)
+		}
+	}()
+	writeJSON(w, http.StatusOK, s.traceStatus())
+	return nil
+}
+
+func traceWhat(in api.TraceInput) string {
+	var parts []string
+	if in.Address != "" {
+		parts = append(parts, in.Address)
+	}
+	if in.Protocol != "" {
+		parts = append(parts, in.Protocol)
+	}
+	if in.Port != 0 {
+		parts = append(parts, "port "+strconv.Itoa(in.Port))
+	}
+	return strings.Join(parts, " ")
+}
+
+// traceStatus is the trace with each step named after Housecarl's object.
+func (s *Server) traceStatus() api.TraceStatus {
+	t := &s.tracing
+	t.mu.Lock()
+	out := t.status
+	if t.tracer != nil {
+		out.Packets = t.tracer.Packets()
+	}
+	t.mu.Unlock()
+	if out.Packets == nil {
+		out.Packets = []api.TracePacket{}
+	}
+	st := s.eng.State()
+	for i := range out.Packets {
+		p := &out.Packets[i]
+		for j := range p.Steps {
+			p.Steps[j].Label = markLabel(st, p.Steps[j])
+		}
+		p.DecidedBy.Label = markLabel(st, p.DecidedBy)
+		if p.WouldDrop != nil {
+			// A copy: the tracer still holds the step.
+			wd := *p.WouldDrop
+			wd.Label = markLabel(st, wd)
+			p.WouldDrop = &wd
+		}
+	}
+	return out
+}
+
+// markLabel names the Housecarl object behind a step: "rule input #12
+// (ssh)", "built-in: established", "port forward web", "input policy".
+func markLabel(st *model.State, step api.TraceStep) string {
+	if step.Kind == "policy" {
+		if step.Table == "inet housecarl" {
+			return step.Chain + " policy"
+		}
+		return ""
+	}
+	m := step.Mark
+	kind, id := "", int64(0)
+	switch {
+	case m == "":
+		return ""
+	case strings.HasPrefix(m, "b-"):
+		return "built-in: " + strings.TrimPrefix(m, "b-")
+	case strings.HasPrefix(m, "r"):
+		kind = model.KindRule
+	case strings.HasPrefix(m, "f"):
+		kind = model.KindForward
+	case strings.HasPrefix(m, "s"):
+		kind = model.KindSNAT
+	case strings.HasPrefix(m, "d"):
+		kind = model.KindRedirect
+	default:
+		return ""
+	}
+	num := strings.TrimLeft(m[1:], "0123456789")
+	id, _ = strconv.ParseInt(strings.TrimSuffix(m[1:], num), 10, 64)
+	if o := st.ByID(kind, id); o != nil {
+		label := o.Label() // "source NAT #3 (…)", "redirect #4", a forward's name…
+		switch {
+		case kind == model.KindRule:
+			label = "rule " + label
+		case kind == model.KindForward && !strings.HasPrefix(label, "port forward"):
+			label = "port forward " + label
+		}
+		if num != "" {
+			label += " (" + strings.TrimPrefix(num, "-") + ")"
+		}
+		return label
+	}
+	return ""
+}
+
+func (s *Server) getTrace(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	if err := require(p, auth.ResLog, false); err != nil {
+		return err
+	}
+	writeJSON(w, http.StatusOK, s.traceStatus())
+	return nil
+}
+
+func (s *Server) stopTrace(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	if err := require(p, auth.ResConfig, true); err != nil {
+		return err
+	}
+	t := &s.tracing
+	t.mu.Lock()
+	if t.cancel != nil && t.status.Running {
+		t.cancel()
+	}
+	t.mu.Unlock()
+	// Let the trace end, so the answer shows it stopped.
+	for i := 0; i < 50; i++ {
+		t.mu.Lock()
+		running := t.status.Running
+		t.mu.Unlock()
+		if !running {
+			break
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	writeJSON(w, http.StatusOK, s.traceStatus())
+	return nil
+}

internal/server/ops.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"net/http"
 	"sort"
+	"strconv"
 	"strings"
 
 	"housecarl/internal/api"
@@ -31,6 +32,12 @@ func applyOp(st *model.State, op *model.Op) (int64, error) {
 		return moveOp(st, op)
 	case model.OpSettings:
 		return 0, settingsOp(st, op)
+	case model.OpRevert:
+		var h model.HistoryEntry
+		if err := json.Unmarshal(op.Data, &h); err != nil {
+			return 0, badRequest("revert: %v", err)
+		}
+		return 0, engine.Undo(st, &h)
 	}
 	return 0, badRequest("unknown operation %q", op.Op)
 }
@@ -57,7 +64,16 @@ func createOp(st *model.State, op *model.Op) (int64, error) {
 		*pos = lastPosition(st, op.Kind, chainOf(o))
 	}
 	data, _ := json.Marshal(o)
-	return op.ID, st.Put(op.Kind, op.ID, data)
+	if err := st.Put(op.Kind, op.ID, data); err != nil {
+		return 0, err
+	}
+	// Placed in the same change, so it is never in force anywhere else.
+	if len(op.Place) > 0 {
+		if _, err := moveOp(st, &model.Op{Op: model.OpMove, Kind: op.Kind, Ref: strconv.FormatInt(op.ID, 10), Data: op.Place}); err != nil {
+			return 0, err
+		}
+	}
+	return op.ID, nil
 }
 
 func updateOp(st *model.State, op *model.Op) (int64, error) {
@@ -88,10 +104,21 @@ func updateOp(st *model.State, op *model.Op) (int64, error) {
 		return 0, err
 	}
 	setID(next, id)
+	if old, now := o.Label(), next.Label(); old != now && isNamed(op.Kind) {
+		if set, ok := o.(*model.AddressSet); ok && set.Dynamic {
+			return 0, conflict("set %s is dynamic: its entries, and the API tokens and programs that fill it, go by its name; make a new set instead", old)
+		}
+		renameRefs(st, op.Kind, old, now)
+	}
 	data, _ := json.Marshal(next)
 	return id, st.Put(op.Kind, id, data)
 }
 
+// isNamed reports whether other objects refer to a kind by name.
+func isNamed(kind string) bool {
+	return kind == model.KindZone || kind == model.KindSet || kind == model.KindService
+}
+
 func deleteOp(st *model.State, op *model.Op) error {
 	o, err := findObject(st, op.Kind, op.Ref)
 	if err != nil {

internal/server/probation.go

@@ -2,8 +2,10 @@ package server
 
 import (
 	"context"
+	"crypto/subtle"
 	"encoding/json"
 	"fmt"
+	"net/http"
 	"sync"
 	"time"
 
@@ -31,6 +33,21 @@ type probation struct {
 	timer    *time.Timer
 	// restores are nft scripts that bring back tables a takeover deleted.
 	restores []string
+	// check is the latest change's confirmation token (reachCheck): only
+	// it can confirm, since the ruleset in force is the one it tests.
+	check *reachCheck
+}
+
+// reachCheck is what a client must show to confirm its change by itself:
+// the token, from the same user (and API token), the same address, and a
+// new connection (another remote address and port than the change's).
+type reachCheck struct {
+	token    string
+	userID   int64
+	username string
+	tokenID  int64
+	ip       string
+	remote   string
 }
 
 // system is the principal of rollbacks.
@@ -64,7 +81,8 @@ func (s *Server) startProbation(ctx context.Context, p *auth.Principal, historyI
 	pr := &s.probation
 	pr.mu.Lock()
 	defer pr.mu.Unlock()
-	if p.Method == "socket" && len(pr.pending) == 0 && len(pr.restores) == 0 {
+	// In shadow mode a change can't cut anyone off: it blocks nothing.
+	if s.cfg.Shadow || p.Method == "socket" && len(pr.pending) == 0 && len(pr.restores) == 0 {
 		s.saveBoot()
 		return
 	}
@@ -88,6 +106,60 @@ func (s *Server) startProbation(ctx context.Context, p *auth.Principal, historyI
 	s.saveProbation(ctx)
 }
 
+// issueCheck gives the client that made a network change on probation a
+// token to confirm it by itself, unless the settings want a person to
+// confirm. The answer then closes its connection (HTTP/2: a GOAWAY), so
+// the confirmation has to open a new one through the new ruleset; an
+// established connection would get through whatever the change did.
+func (s *Server) issueCheck(w http.ResponseWriter, r *http.Request, p *auth.Principal, res *engine.Result) string {
+	if p == nil || p.Method == "socket" || (res != nil && !res.Applied) || s.eng.State().Settings.ManualConfirm {
+		return ""
+	}
+	pr := &s.probation
+	pr.mu.Lock()
+	defer pr.mu.Unlock()
+	if len(pr.pending) == 0 && len(pr.restores) == 0 {
+		return ""
+	}
+	c := &reachCheck{token: auth.RandomString(24), username: p.Name(), ip: s.clientIP(r), remote: r.RemoteAddr}
+	if p.User != nil {
+		c.userID = p.User.ID
+	}
+	if p.Token != nil {
+		c.tokenID = p.Token.ID
+	}
+	pr.check = c
+	w.Header().Set("Connection", "close")
+	return c.token
+}
+
+// confirmToken confirms with a change's token: the proof that the client
+// that made it still reaches the API, from where it made it.
+func (s *Server) confirmToken(r *http.Request, p *auth.Principal, token string) (int, error) {
+	pr := &s.probation
+	pr.mu.Lock()
+	c := pr.check
+	pending := len(pr.pending) + len(pr.restores)
+	pr.mu.Unlock()
+	var tokenID int64
+	if p.Token != nil {
+		tokenID = p.Token.ID
+	}
+	switch {
+	case pending == 0:
+		return 0, nil
+	case c == nil || subtle.ConstantTimeCompare([]byte(c.token), []byte(token)) != 1:
+		return 0, conflict("this confirmation is for an earlier change: a newer one is on probation, and only its own client (or a person) can confirm it")
+	case p.User == nil || p.User.ID != c.userID || p.Name() != c.username || tokenID != c.tokenID:
+		return 0, forbidden("only the client that made the change can confirm it this way")
+	case s.clientIP(r) != c.ip:
+		return 0, forbidden("the confirmation comes from %s, the change from %s: it must come from the same place", s.clientIP(r), c.ip)
+	case r.RemoteAddr == c.remote:
+		return 0, conflict("the confirmation came over the change's own connection, which proves nothing; it must open a new one")
+	}
+	return s.confirmWith(r.Context(), p, fmt.Sprintf("confirmed automatically: %s reached the API again over a new connection", c.ip))
+}
+
 func (s *Server) saveBoot() {
 	if err := s.fw.SaveBoot(s.eng.State()); err != nil {
 		s.log.Error("saving the ruleset for the next boot failed", "err", err)
@@ -110,8 +182,12 @@ func (s *Server) probationStatus() api.Probation {
 	return out
 }
 
-// confirm makes the pending changes permanent.
+// confirm makes the pending changes permanent (a person's confirmation).
 func (s *Server) confirm(ctx context.Context, p *auth.Principal) (int, error) {
+	return s.confirmWith(ctx, p, "")
+}
+
+func (s *Server) confirmWith(ctx context.Context, p *auth.Principal, how string) (int, error) {
 	pr := &s.probation
 	pr.mu.Lock()
 	defer pr.mu.Unlock()
@@ -122,10 +198,14 @@ func (s *Server) confirm(ctx context.Context, p *auth.Principal) (int, error) {
 	if pr.timer != nil {
 		pr.timer.Stop()
 	}
-	pr.pending, pr.restores, pr.deadline, pr.timer = nil, nil, time.Time{}, nil
+	pr.pending, pr.restores, pr.deadline, pr.timer, pr.check = nil, nil, time.Time{}, nil, nil
 	s.saveProbation(ctx)
 	s.saveBoot()
-	s.eng.Audit(ctx, p, "config.confirm", "", "success", fmt.Sprintf("%d changes confirmed", n))
+	detail := fmt.Sprintf("%d changes confirmed", n)
+	if how != "" {
+		detail += "; " + how
+	}
+	s.eng.Audit(ctx, p, "config.confirm", "", "success", detail)
 	return n, nil
 }
 
@@ -136,7 +216,7 @@ func (s *Server) rollback(why string) {
 	pr.mu.Lock()
 	defer pr.mu.Unlock()
 	pending, restores := pr.pending, pr.restores
-	pr.pending, pr.restores, pr.deadline, pr.timer = nil, nil, time.Time{}, nil
+	pr.pending, pr.restores, pr.deadline, pr.timer, pr.check = nil, nil, time.Time{}, nil, nil
 	for i := len(restores) - 1; i >= 0; i-- {
 		if err := s.fw.ApplyScript(ctx, restores[i]); err != nil {
 			s.log.Error("bringing back a table a takeover deleted failed", "err", err)

internal/server/rename_test.go

@@ -0,0 +1,84 @@
+package server
+
+import (
+	"strings"
+	"testing"
+
+	"housecarl/internal/api"
+)
+
+// TestRename: renaming a zone, set or service changes what refers to it,
+// in the same change, which undoes as one.
+func TestRename(t *testing.T) {
+	e := newEnv(t)
+	e.srv.probationFor = 0
+	post := func(path string, body map[string]any) {
+		t.Helper()
+		if code := e.call(e.token, "POST", path, body, nil); code != 201 {
+			t.Fatalf("POST %s: %d", path, code)
+		}
+	}
+	post("/zones", map[string]any{"name": "wan", "interfaces": []string{"wan"}, "external": true})
+	post("/zones", map[string]any{"name": "dhcp", "interfaces": []string{"eth1"}})
+	post("/sets", map[string]any{"name": "trusted_ips", "entries": []string{"10.0.10.0/24"}})
+	post("/sets", map[string]any{"name": "guests", "dynamic": true})
+	post("/services", map[string]any{"name": "jarl", "entries": []string{"tcp/8053"}})
+	post("/rules", map[string]any{"chain": "forward", "in_zones": []string{"dhcp"}, "out_zones": []string{"wan"}, "sources": []string{"trusted_ips"}})
+	post("/rules", map[string]any{"chain": "input", "in_zones": []string{"dhcp"}, "services": []string{"jarl"}})
+	post("/snats", map[string]any{"out_zones": []string{"wan"}, "sources": []string{"trusted_ips"}})
+	post("/forwards", map[string]any{"name": "jf", "ports": "8096", "to": "10.0.10.57", "public_addresses": []string{"217.155.8.194"}, "sources": []string{"trusted_ips"}})
+
+	var res api.MutationResponse
+	if code := e.call(e.token, "PATCH", "/zones/dhcp", map[string]any{"name": "lan"}, &res); code != 200 {
+		t.Fatalf("renaming the zone: %d %+v", code, res)
+	}
+	zoneRename := res.HistoryID
+	e.call(e.token, "PATCH", "/sets/trusted_ips", map[string]any{"name": "trusted"}, &res)
+	setRename := res.HistoryID
+	e.call(e.token, "PATCH", "/services/jarl", map[string]any{"name": "jarl-api"}, &res)
+	serviceRename := res.HistoryID
+	st := e.eng.State()
+	if st.Zone("dhcp") != nil || st.Zone("lan") == nil {
+		t.Fatalf("zones %+v", st.Zones)
+	}
+	for _, r := range st.Rules {
+		if strings.Join(r.InZones, ",") != "lan" {
+			t.Errorf("rule %d in zones %v", r.ID, r.InZones)
+		}
+		if r.Chain == "forward" && strings.Join(r.Sources, ",") != "trusted" {
+			t.Errorf("rule %d sources %v", r.ID, r.Sources)
+		}
+		if r.Chain == "input" && strings.Join(r.Services, ",") != "jarl-api" {
+			t.Errorf("rule %d services %v", r.ID, r.Services)
+		}
+	}
+	if s := st.SNATs[0].Sources; strings.Join(s, ",") != "trusted" {
+		t.Errorf("snat sources %v", s)
+	}
+	if s := st.Forwards[0].Sources; strings.Join(s, ",") != "trusted" {
+		t.Errorf("forward sources %v", s)
+	}
+	if ps := e.eng.Problems(); len(ps) != 0 {
+		for _, p := range ps {
+			if p.Severity == "error" {
+				t.Errorf("problem after renaming: %+v", p)
+			}
+		}
+	}
+	// Undoing the renames (newest first) puts the references back too.
+	for _, id := range []int64{serviceRename, setRename, zoneRename} {
+		if code := e.call(e.token, "POST", "/history/"+itoa(id)+"/revert", nil, nil); code != 200 {
+			t.Fatalf("undo %d: %d", id, code)
+		}
+	}
+	st = e.eng.State()
+	for _, r := range st.Rules {
+		if strings.Join(r.InZones, ",") != "dhcp" || r.Chain == "forward" && r.Sources[0] != "trusted_ips" || r.Chain == "input" && r.Services[0] != "jarl" {
+			t.Errorf("after undo: rule %+v", r.Match)
+		}
+	}
+	// A dynamic set keeps its name: tokens and programs go by it.
+	if code := e.call(e.token, "PATCH", "/sets/guests", map[string]any{"name": "visitors"}, nil); code != 409 {
+		t.Errorf("renaming a dynamic set: %d", code)
+	}
+}

internal/server/routes.go

@@ -10,15 +10,12 @@ import (
 	"time"
 
 	"housecarl/internal/api"
-	"housecarl/internal/version"
 )
 
 func (s *Server) routes() {
 	m := s.mux
 	p := api.Prefix
-	m.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
-		writeJSON(w, http.StatusOK, map[string]any{"ok": true, "version": version.String()})
-	})
+	m.HandleFunc("GET /healthz", s.healthz)
 	// Authentication, users and tokens
 	m.HandleFunc("POST "+p+"/auth/login", s.public(s.login))
 	m.HandleFunc("POST "+p+"/auth/logout", s.authed(s.logout))
@@ -42,6 +39,11 @@ func (s *Server) routes() {
 	m.HandleFunc("POST "+p+"/sets/{ref}/elements", s.authed(s.addElement))
 	m.HandleFunc("DELETE "+p+"/sets/{ref}/elements/{addr}", s.authed(s.deleteElement))
 	m.HandleFunc("GET "+p+"/quotas", s.authed(s.listQuotas))
+	m.HandleFunc("GET "+p+"/connections", s.authed(s.listConnections))
+	m.HandleFunc("POST "+p+"/connections/close", s.authed(s.closeConnections))
+	m.HandleFunc("GET "+p+"/trace", s.authed(s.getTrace))
+	m.HandleFunc("POST "+p+"/trace", s.authed(s.startTrace))
+	m.HandleFunc("DELETE "+p+"/trace", s.authed(s.stopTrace))
 	m.HandleFunc("POST "+p+"/rules/{ref}/reset-quota", s.authed(s.resetQuota))
 	m.HandleFunc("GET "+p+"/changesets", s.authed(s.listChangesets))
 	m.HandleFunc("POST "+p+"/changesets", s.authed(s.createChangeset))
@@ -58,6 +60,13 @@ func (s *Server) routes() {
 	m.HandleFunc("DELETE "+p+"/{kind}/{ref}", s.authed(s.deleteObject))
 	m.HandleFunc("POST "+p+"/{kind}/{ref}/move", s.authed(s.moveObject))
 	m.HandleFunc("GET "+p+"/ruleset", s.authed(s.ruleset))
+	m.HandleFunc("GET "+p+"/matrix", s.authed(s.zoneMatrix))
+	m.HandleFunc("GET "+p+"/stats/status", s.authed(s.statsStatus))
+	m.HandleFunc("GET "+p+"/stats/summary", s.authed(s.statsSummary))
+	m.HandleFunc("GET "+p+"/stats/timeline", s.authed(s.statsTimeline))
+	m.HandleFunc("GET "+p+"/stats/top", s.authed(s.statsTop))
+	m.HandleFunc("GET "+p+"/stats/hits", s.authed(s.statsHits))
+	m.HandleFunc("GET "+p+"/stats/heatmap", s.authed(s.statsHeatmap))
 	m.HandleFunc("GET "+p+"/history", s.authed(s.listHistory))
 	m.HandleFunc("GET "+p+"/history/{id}", s.authed(s.getHistory))
 	m.HandleFunc("POST "+p+"/history/{id}/revert", s.authed(s.revertHistory))

internal/server/routes_cfg.go

@@ -1,9 +1,36 @@
 package server
 
-import "encoding/json"
+import (
+	"encoding/json"
+
+	"housecarl/internal/trace"
+)
 
 // otherTables lists the tables in an `nft -j list ruleset` other than
 // Housecarl's ("inet main").
+// inputFilters lists the tables other than Housecarl's with a filter
+// chain on the input hook: what filters the machine in shadow mode.
+func inputFilters(data []byte) []string {
+	var doc struct {
+		Nftables []map[string]json.RawMessage `json:"nftables"`
+	}
+	out := []string{}
+	if json.Unmarshal(data, &doc) != nil {
+		return out
+	}
+	for _, item := range doc.Nftables {
+		raw, ok := item["chain"]
+		if !ok {
+			continue
+		}
+		var c struct{ Family, Table, Type, Hook string }
+		if json.Unmarshal(raw, &c) == nil && c.Type == "filter" && c.Hook == "input" && !(c.Family == "inet" && (c.Table == "housecarl" || c.Table == trace.Table)) {
+			out = append(out, c.Family+" "+c.Table)
+		}
+	}
+	return out
+}
+
 func otherTables(data []byte) []string {
 	var doc struct {
 		Nftables []map[string]json.RawMessage `json:"nftables"`
@@ -18,7 +45,7 @@ func otherTables(data []byte) []string {
 			continue
 		}
 		var t struct{ Family, Name string }
-		if json.Unmarshal(raw, &t) == nil && !(t.Family == "inet" && t.Name == "housecarl") {
+		if json.Unmarshal(raw, &t) == nil && !(t.Family == "inet" && (t.Name == "housecarl" || t.Name == trace.Table)) {
 			out = append(out, t.Family+" "+t.Name)
 		}
 	}

internal/server/server.go

@@ -18,12 +18,14 @@ import (
 	"syscall"
 	"time"
 
+	"housecarl/internal/api"
 	"housecarl/internal/auth"
 	"housecarl/internal/certs"
 	"housecarl/internal/engine"
 	"housecarl/internal/model"
 	"housecarl/internal/nft"
 	"housecarl/internal/render"
+	"housecarl/internal/stats"
 	"housecarl/internal/store"
 	"housecarl/internal/version"
 )
@@ -44,6 +46,11 @@ type Config struct {
 	// ACME keeps the certificate housecarld obtains itself (tls.acme; nil
 	// when off).
 	ACME *certs.Manager
+	// Shadow is shadow mode (docs/shadow.md): nothing goes on probation,
+	// takeovers and closing connections are refused. ShadowPosition is
+	// after or before, for the status.
+	Shadow         bool
+	ShadowPosition string
 }
 
 // Interfaces lists the host's interfaces with their addresses, and
@@ -57,6 +64,8 @@ type Interfaces interface {
 // engine (internal/nft.Firewall).
 type Firewall interface {
 	Render(st *model.State) *render.Ruleset
+	// RenderEnforcing is the ruleset outside shadow mode.
+	RenderEnforcing(st *model.State) *render.Ruleset
 	Table(ctx context.Context) (*nft.Table, error)
 	AddElement(ctx context.Context, set, addr string, timeout int) error
 	DeleteElement(ctx context.Context, set, addr string) error
@@ -67,6 +76,8 @@ type Firewall interface {
 	SaveBoot(st *model.State) error
 	// ResetQuota starts a rule's quota again (perAddress: its sets).
 	ResetQuota(ctx context.Context, ruleID int64, perAddress bool) error
+	// Trace follows packets (internal/trace) until ctx ends.
+	Trace(ctx context.Context, script string, line func(string)) error
 }
 
 // Server is the HTTP server.
@@ -87,7 +98,17 @@ type Server struct {
 	srvs []*http.Server
 
 	probation probation
+	health    healthCache
 	logs      *logRing
+	// unitLinks are where an enabled nftables.service shows (tests change
+	// them).
+	unitLinks []string
+	// conns is the kernel's connection tracking (nil: unavailable).
+	conns Conntrack
+	// tracing is the packet trace running (or the last one).
+	tracing tracing
+	// stats collects statistics (nil: off).
+	stats *stats.Collector
 	// csMu orders changes to changesets (staging, applying, discarding).
 	csMu  sync.Mutex
 	nowFn func() time.Time
@@ -113,11 +134,27 @@ func New(cfg Config, eng *engine.Engine, fw Firewall, ifaces Interfaces, log *sl
 		log = slog.Default()
 	}
 	s := &Server{cfg: cfg, eng: eng, fw: fw, ifaces: ifaces, store: eng.Store(), log: log, mux: http.NewServeMux(), limiter: newRateLimiter(),
-		web: web, started: time.Now(), logs: newLogRing(2000), nowFn: time.Now}
+		web: web, started: time.Now(), logs: newLogRing(2000), nowFn: time.Now, unitLinks: nftablesUnitLinks}
 	s.routes()
 	return s
 }
 
+// mode is housecarld's mode: enforcing, or shadow.
+func (s *Server) mode() string {
+	if s.cfg.Shadow {
+		return api.ModeShadow
+	}
+	return api.ModeEnforcing
+}
+
+// refuseInShadow refuses what would change traffic in shadow mode.
+func (s *Server) refuseInShadow(what string) error {
+	if !s.cfg.Shadow {
+		return nil
+	}
+	return conflict("housecarld is in shadow mode, where it changes nothing about traffic, so it doesn't %s; leave shadow mode first (shadow.enabled in housecarl.yaml, docs/shadow.md)", what)
+}
+
 // Handler returns the HTTP handler with all middleware.
 func (s *Server) Handler() http.Handler {
 	return s.recoverer(s.securityHeaders(s.logRequests(s.mux)))

internal/server/server_test.go

@@ -23,6 +23,7 @@ import (
 	"housecarl/internal/model"
 	"housecarl/internal/nft"
 	"housecarl/internal/render"
+	"housecarl/internal/stats"
 	"housecarl/internal/store"
 )
 
@@ -37,6 +38,11 @@ type fakeFW struct {
 	deleted    []string
 	restored   []string
 	resets     []string
+	traced     string
+	traceLines []string
+	fallback   bool // the boot fallback is in force
+	shadow     bool
+	ruleset    string // nft -j list ruleset ("": a default)
 }
 
 func (f *fakeFW) Check(_ context.Context, st *model.State) error { return nil }
@@ -54,14 +60,19 @@ func (f *fakeFW) last() *model.State {
 	defer f.mu.Unlock()
 	return f.applied[len(f.applied)-1]
 }
-func (f *fakeFW) Render(st *model.State) *render.Ruleset { return render.Render(st, render.Options{}) }
+func (f *fakeFW) Render(st *model.State) *render.Ruleset {
+	return render.Render(st, render.Options{Shadow: f.shadow})
+}
+func (f *fakeFW) RenderEnforcing(st *model.State) *render.Ruleset {
+	return render.Render(st, render.Options{})
+}
 func (f *fakeFW) Table(context.Context) (*nft.Table, error) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
 	if len(f.applied) == 0 {
 		return nil, errors.New("No such file or directory: list table inet housecarl")
 	}
-	return &nft.Table{Counters: map[string]nft.Counter{"r1": {Packets: 3, Bytes: 180}}, Elements: f.elements}, nil
+	return &nft.Table{Counters: map[string]nft.Counter{"r1": {Packets: 3, Bytes: 180}}, Elements: f.elements, Fallback: f.fallback}, nil
 }
 func (f *fakeFW) AddElement(_ context.Context, set, addr string, timeout int) error {
 	f.mu.Lock()
@@ -82,6 +93,9 @@ func (f *fakeFW) DeleteElement(_ context.Context, set, addr string) error {
 	return nil
 }
 func (f *fakeFW) ListRuleset(context.Context) ([]byte, error) {
+	if f.ruleset != "" {
+		return []byte(f.ruleset), nil
+	}
 	return []byte(`{"nftables":[{"table":{"family":"inet","name":"main"}},{"table":{"family":"inet","name":"housecarl"}}]}`), nil
 }
 func (f *fakeFW) DeleteTable(_ context.Context, fam, name string) error {
@@ -96,6 +110,17 @@ func (f *fakeFW) ResetQuota(_ context.Context, id int64, perAddress bool) error
 	f.resets = append(f.resets, fmt.Sprintf("r%d:%v", id, perAddress))
 	return nil
 }
+func (f *fakeFW) Trace(ctx context.Context, script string, line func(string)) error {
+	f.mu.Lock()
+	f.traced = script
+	lines := f.traceLines
+	f.mu.Unlock()
+	for _, l := range lines {
+		line(l)
+	}
+	<-ctx.Done()
+	return nil
+}
 func (f *fakeFW) TableText(_ context.Context, fam, name string) (string, error) {
 	return "table " + fam + " " + name + " {}", nil
 }
@@ -127,6 +152,11 @@ type env struct {
 }
 
 func newEnv(t *testing.T) *env {
+	t.Helper()
+	return newEnvWith(t, Config{})
+}
+
+func newEnvWith(t *testing.T, cfg Config) *env {
 	t.Helper()
 	ctx := context.Background()
 	st, err := store.Open(":memory:")
@@ -144,10 +174,10 @@ func newEnv(t *testing.T) *env {
 	if err != nil {
 		t.Fatal(err)
 	}
-	fw := &fakeFW{elements: map[string][]render.Element{}}
+	fw := &fakeFW{elements: map[string][]render.Element{}, shadow: cfg.Shadow}
 	eng.Firewall = fw
 	ifs := hostIfaces{"wan": {netip.MustParsePrefix("217.155.8.194/32")}, "eth1": {netip.MustParsePrefix("10.0.10.1/24")}}
-	srv := New(Config{}, eng, fw, ifs, nil, nil)
+	srv := New(cfg, eng, fw, ifs, nil, nil)
 	ts := httptest.NewServer(srv.Handler())
 	t.Cleanup(ts.Close)
 	return &env{t: t, ts: ts, srv: srv, eng: eng, fw: fw, token: tok}
@@ -722,3 +752,481 @@ func TestQuotaResets(t *testing.T) {
 		t.Errorf("reset by hand: %d %v", code, e.fw.resets)
 	}
 }
+
+// TestAutoConfirm: the client that made a change confirms it by itself:
+// with its token, from the same address, over a new connection. Nobody
+// else can, an older token can't, and with manual_confirm there is none.
+func TestAutoConfirm(t *testing.T) {
+	e := newEnv(t)
+	e.srv.probationFor = 5 * time.Second
+	post := func(token, path string, body any) (*http.Response, []byte) {
+		b, _ := json.Marshal(body)
+		req, _ := http.NewRequest("POST", e.ts.URL+api.Prefix+path, bytes.NewReader(b))
+		req.Header.Set("Authorization", "Bearer "+token)
+		req.Header.Set("Content-Type", "application/json")
+		resp, err := http.DefaultClient.Do(req)
+		if err != nil {
+			t.Fatal(err)
+		}
+		defer resp.Body.Close()
+		out, _ := io.ReadAll(resp.Body)
+		return resp, out
+	}
+	resp, body := post(e.token, "/zones", model.Zone{Name: "lan", Interfaces: []string{"eth1"}})
+	var mr api.MutationResponse
+	json.Unmarshal(body, &mr)
+	if mr.ConfirmToken == "" || !resp.Close {
+		t.Fatalf("no token, or the connection stays open: %+v close=%v", mr, resp.Close)
+	}
+	first := mr.ConfirmToken
+	// A newer change: only its token confirms now.
+	_, body = post(e.token, "/zones", model.Zone{Name: "dmz", Interfaces: []string{"eth2"}})
+	json.Unmarshal(body, &mr)
+	if resp, _ := post(e.token, "/confirm", api.ConfirmInput{Token: first}); resp.StatusCode != 409 {
+		t.Errorf("an older token: %d", resp.StatusCode)
+	}
+	// Another user can't use it.
+	ctx := context.Background()
+	other := &model.User{Username: "sam", Grants: auth.Roles["operator"]}
+	e.srv.store.CreateUser(ctx, other)
+	otok, pid, sh := auth.NewToken()
+	e.srv.store.CreateToken(ctx, &model.Token{UserID: other.ID, Name: "t", PublicID: pid, SecretHash: sh})
+	if resp, _ := post(otok, "/confirm", api.ConfirmInput{Token: mr.ConfirmToken}); resp.StatusCode != 403 {
+		t.Errorf("another user's confirmation: %d", resp.StatusCode)
+	}
+	// Not over the change's own connection, nor from elsewhere.
+	c := e.srv.probation.check
+	p := &auth.Principal{User: &model.User{ID: c.userID, Username: c.username}, Method: "token", Token: &model.Token{ID: c.tokenID}}
+	same := httptest.NewRequest("POST", "/", nil)
+	same.RemoteAddr = c.remote
+	if _, err := e.srv.confirmToken(same, p, c.token); err == nil || !strings.Contains(err.Error(), "new one") {
+		t.Errorf("over the same connection: %v", err)
+	}
+	elsewhere := httptest.NewRequest("POST", "/", nil)
+	elsewhere.RemoteAddr = "192.0.2.9:4444"
+	if _, err := e.srv.confirmToken(elsewhere, p, c.token); err == nil || !strings.Contains(err.Error(), "same place") {
+		t.Errorf("from elsewhere: %v", err)
+	}
+	// The client itself, over a new connection: confirmed.
+	resp, body = post(e.token, "/confirm", api.ConfirmInput{Token: mr.ConfirmToken})
+	if resp.StatusCode != 200 || !strings.Contains(string(body), `"confirmed":2`) {
+		t.Fatalf("auto-confirm: %d %s", resp.StatusCode, body)
+	}
+	if pr := e.srv.probationStatus(); len(pr.Pending) != 0 {
+		t.Errorf("still pending: %+v", pr)
+	}
+	// manual_confirm: no token; a person confirms.
+	e.call(e.token, "PATCH", "/settings", map[string]any{"manual_confirm": true}, nil)
+	e.call(e.token, "POST", "/confirm", nil, nil)
+	_, body = post(e.token, "/zones", model.Zone{Name: "guest", Interfaces: []string{"eth3"}})
+	mr = api.MutationResponse{}
+	json.Unmarshal(body, &mr)
+	if mr.ConfirmToken != "" || mr.ConfirmBy == nil {
+		t.Errorf("manual confirmation: %+v", mr)
+	}
+}
+
+// TestStagedUndo: an undo staged in a changeset changes nothing until the
+// changeset is applied; then History links the two entries.
+func TestStagedUndo(t *testing.T) {
+	e := newEnv(t)
+	e.srv.probationFor = 5 * time.Second
+	var mr api.MutationResponse
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "lan", Interfaces: []string{"eth1"}}, &mr)
+	e.call(e.token, "POST", "/confirm", nil, nil)
+	made := mr.HistoryID
+	var cs model.Changeset
+	e.call(e.token, "POST", "/changesets", nil, &cs)
+	q := "?changeset=" + itoa(cs.ID)
+	if code := e.call(e.token, "POST", "/history/"+itoa(made)+"/revert"+q, nil, &mr); code != 200 || !mr.Staged {
+		t.Fatalf("staging the undo: %d %+v", code, mr)
+	}
+	if op := mr.Changeset.Ops[0]; op.Op != model.OpRevert || !strings.HasPrefix(op.Summary, "undo "+itoa(made)+":") {
+		t.Errorf("the operation: %+v", op)
+	}
+	if len(e.eng.State().Zones) != 1 {
+		t.Fatal("a staged undo was applied")
+	}
+	// Staging it again can't work: the changeset already undoes it.
+	var eb api.ErrorBody
+	if code := e.call(e.token, "POST", "/history/"+itoa(made)+"/revert"+q, nil, &eb); code != 409 {
+		t.Errorf("a second undo: %d %+v", code, eb)
+	}
+	if code := e.call(e.token, "POST", "/changesets/"+itoa(cs.ID)+"/apply", nil, &mr); code != 200 || !mr.Applied {
+		t.Fatalf("apply: %d %+v", code, mr)
+	}
+	if len(e.eng.State().Zones) != 0 {
+		t.Error("the undo wasn't applied")
+	}
+	h, _ := e.srv.store.GetHistory(context.Background(), mr.HistoryID)
+	orig, _ := e.srv.store.GetHistory(context.Background(), made)
+	if h.RevertOf != made || orig.RevertedBy != h.ID || !strings.HasPrefix(h.Description, "undo ") {
+		t.Errorf("history: %+v, original %+v", h, orig)
+	}
+}
+
+// TestMatrix: the zone matrix, now and as a changeset will make it.
+func TestMatrixEndpoint(t *testing.T) {
+	e := newEnv(t)
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "lan", Interfaces: []string{"eth1"}}, nil)
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "wan", Interfaces: []string{"wan"}, External: true}, nil)
+	var m api.Matrix
+	e.call(e.token, "GET", "/matrix", nil, &m)
+	cell := func(from, to string) api.MatrixCell {
+		for _, c := range m.Cells {
+			if c.From == from && c.To == to {
+				return c
+			}
+		}
+		return api.MatrixCell{}
+	}
+	if c := cell("lan", "wan"); c.Verdict != "blocked" || c.DecidedBy != "forward policy" {
+		t.Errorf("lan → wan: %+v", c)
+	}
+	var cs model.Changeset
+	e.call(e.token, "POST", "/changesets", nil, &cs)
+	q := "?changeset=" + itoa(cs.ID)
+	e.call(e.token, "POST", "/rules"+q, model.Rule{Chain: "forward", Match: model.Match{InZones: []string{"lan"}, OutZones: []string{"wan"}}, Action: "accept"}, nil)
+	e.call(e.token, "GET", "/matrix"+q, nil, &m)
+	if c := cell("lan", "wan"); c.Verdict != "allowed" {
+		t.Errorf("lan → wan with the changeset: %+v", c)
+	}
+	// Hits: summed over an object's counters; none without a counter.
+	hm := api.Matrix{Cells: []api.MatrixCell{{Marks: []string{"p-forward"}, Exceptions: []api.MatrixException{
+		{Marks: []string{"b-antispoof", "b-antispoof6"}}, {Marks: []string{"r9"}}}}}}
+	matrixHits(&hm, map[string]nft.Counter{"p-forward": {Packets: 4}, "b-antispoof": {Packets: 2}, "b-antispoof6": {Packets: 3}})
+	if c := hm.Cells[0]; c.Hits == nil || *c.Hits != 4 || c.Exceptions[0].Hits == nil || *c.Exceptions[0].Hits != 5 || c.Exceptions[1].Hits != nil {
+		t.Errorf("hits: %+v", c)
+	}
+}
+
+// TestHostProblems: an enabled nftables.service, and an imported table
+// still in force, are warned about.
+func TestHostProblems(t *testing.T) {
+	e := newEnv(t)
+	dir := t.TempDir()
+	link := dir + "/nftables.service"
+	e.srv.unitLinks = []string{link}
+	codes := func() string {
+		var st api.Status
+		e.call(e.token, "GET", "/status", nil, &st)
+		var out []string
+		for _, p := range st.Problems {
+			out = append(out, p.Code)
+		}
+		return strings.Join(out, " ")
+	}
+	if c := codes(); strings.Contains(c, "nftables_service") || strings.Contains(c, "takeover_pending") {
+		t.Errorf("warned without cause: %s", c)
+	}
+	os.Symlink("/usr/lib/systemd/system/nftables.service", link) // dangling is fine: Lstat
+	// The fake ruleset lists inet main: imported from it, it still applies.
+	e.srv.store.SetMeta(context.Background(), importedTableKey, "inet main")
+	if c := codes(); !strings.Contains(c, "nftables_service") || !strings.Contains(c, "takeover_pending") {
+		t.Errorf("not warned: %s", c)
+	}
+	e.srv.store.SetMeta(context.Background(), importedTableKey, "inet gone")
+	if c := codes(); strings.Contains(c, "takeover_pending") {
+		t.Errorf("warned about a table that is gone: %s", c)
+	}
+}
+
+type fakeConns struct {
+	list   []api.Connection
+	closed []api.Connection
+}
+
+func (f *fakeConns) List(context.Context) ([]api.Connection, error) { return f.list, nil }
+func (f *fakeConns) Close(_ context.Context, c api.Connection) error {
+	f.closed = append(f.closed, c)
+	return nil
+}
+
+// TestConnections: connections listed and searched; closed one by one, or
+// all a search finds; closing needs config:write, and is audited.
+func TestConnections(t *testing.T) {
+	e := newEnv(t)
+	var code int
+	if code = e.call(e.token, "GET", "/connections", nil, nil); code != 503 {
+		t.Errorf("without conntrack: %d", code)
+	}
+	fc := &fakeConns{list: []api.Connection{
+		{Family: "ipv4", Protocol: "tcp", Source: "192.168.1.50", SourcePort: 51432, Destination: "198.51.100.9", DestinationPort: 443, SNATTo: "203.0.113.10", State: "established"},
+		{Family: "ipv4", Protocol: "udp", Source: "192.168.1.51", SourcePort: 5353, Destination: "192.168.1.1", DestinationPort: 53},
+		{Family: "ipv6", Protocol: "tcp", Source: "2001:db8:1::50", SourcePort: 40000, Destination: "2001:db8:2::9", DestinationPort: 4430},
+	}}
+	e.srv.SetConntrack(fc)
+	var cs []api.Connection
+	e.call(e.token, "GET", "/connections?q=192.168.1.50", nil, &cs)
+	if len(cs) != 1 || cs[0].DestinationPort != 443 {
+		t.Errorf("search by address: %+v", cs)
+	}
+	e.call(e.token, "GET", "/connections?q=:443", nil, &cs)
+	if len(cs) != 1 || cs[0].Source != "192.168.1.50" {
+		t.Errorf("search by port (not :4430): %+v", cs)
+	}
+	var out map[string]int
+	if code := e.call(e.token, "POST", "/connections/close", api.CloseConnections{Search: "tcp"}, &out); code != 200 || out["closed"] != 2 || len(fc.closed) != 2 {
+		t.Errorf("close a search: %d %+v %+v", code, out, fc.closed)
+	}
+	if code := e.call(e.token, "POST", "/connections/close", api.CloseConnections{}, nil); code != 400 {
+		t.Errorf("close nothing said: %d", code)
+	}
+	var tok api.Token
+	e.call(e.token, "POST", "/me/tokens", api.TokenInput{Name: "monitor", Role: "log"}, &tok)
+	if code := e.call(tok.Secret, "GET", "/connections", nil, nil); code != 200 {
+		t.Errorf("listing with log:read: %d", code)
+	}
+	if code := e.call(tok.Secret, "POST", "/connections/close", api.CloseConnections{Search: "udp"}, nil); code != 403 {
+		t.Errorf("closing with log:read: %d", code)
+	}
+}
+
+// TestTrace: a trace runs with the filter's table, collects packets named
+// after Housecarl's objects, and stops when asked; one at a time.
+func TestTrace(t *testing.T) {
+	e := newEnv(t)
+	var mr api.MutationResponse
+	e.call(e.token, "POST", "/rules", model.Rule{Chain: "input", Match: model.Match{Services: []string{"tcp/9999"}}, Action: "drop", Comment: "block 9999"}, &mr)
+	var rule model.Rule
+	json.Unmarshal(mr.Object, &rule)
+	e.fw.traceLines = []string{
+		`trace id 1 inet housecarl input packet: iif "wan" ip saddr 198.51.100.7 ip daddr 203.0.113.10 tcp sport 4000 tcp dport 9999 tcp flags == syn`,
+		`trace id 1 inet housecarl input rule ct state established,related accept comment "hc:b-established" (verdict continue)`,
+		fmt.Sprintf(`trace id 1 inet housecarl input rule tcp dport 9999 counter drop comment "hc:r%d block 9999" (verdict drop)`, rule.ID),
+	}
+	var st api.TraceStatus
+	if code := e.call(e.token, "POST", "/trace", api.TraceInput{Port: 9999, Seconds: 30}, &st); code != 200 || !st.Running {
+		t.Fatalf("start: %d %+v", code, st)
+	}
+	if code := e.call(e.token, "POST", "/trace", api.TraceInput{Port: 22}, nil); code != 409 {
+		t.Errorf("a second trace: %d", code)
+	}
+	time.Sleep(100 * time.Millisecond)
+	e.call(e.token, "GET", "/trace", nil, &st)
+	if len(st.Packets) != 1 || st.Packets[0].Verdict != "dropped" || st.Packets[0].DecidedBy.Label != "rule input #"+itoa(rule.ID)+" (block 9999)" ||
+		st.Packets[0].Steps[0].Label != "built-in: established" {
+		t.Errorf("packets: %+v", st.Packets)
+	}
+	if !strings.Contains(e.fw.traced, "th dport 9999") {
+		t.Errorf("the trace table: %s", e.fw.traced)
+	}
+	if code := e.call(e.token, "DELETE", "/trace", nil, &st); code != 200 || st.Running || len(st.Packets) != 1 {
+		t.Errorf("stop: %d %+v", code, st)
+	}
+	if code := e.call(e.token, "POST", "/trace", api.TraceInput{}, nil); code != 400 {
+		t.Errorf("tracing everything: %d", code)
+	}
+}
+
+// TestStatsEndpoints: statistics off, then on: the summary's top lists,
+// hits per rule (named, with those that counted none), and which
+// counters count drops.
+func TestStatsEndpoints(t *testing.T) {
+	e := newEnv(t)
+	if code := e.call(e.token, "GET", "/stats/summary", nil, nil); code != http.StatusServiceUnavailable {
+		t.Errorf("statistics off: %d", code)
+	}
+	var st StatsStatus
+	e.call(e.token, "GET", "/stats/status", nil, &st)
+	if st.Enabled {
+		t.Error("enabled without a collector")
+	}
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "wan", Interfaces: []string{"wan"}, External: true}, nil)
+	e.call(e.token, "POST", "/rules", model.Rule{Chain: "input", Match: model.Match{Sources: []string{"198.51.100.0/24"}}, Action: "drop", Comment: "bad net"}, nil)
+	rule := e.eng.State().Rules[0]
+
+	ss, err := stats.OpenStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer ss.Close()
+	src := stats.Sources{Table: e.fw.Table, SysNet: t.TempDir(), Proc: t.TempDir()}
+	e.srv.StatsSources(&src)
+	c := stats.NewCollector(ss, stats.Options{RetainMinute: 7 * 24 * time.Hour, RetainHour: 90 * 24 * time.Hour, RetainDay: 730 * 24 * time.Hour}, src, nil)
+	e.srv.SetStats(c)
+	c.Start(context.Background())
+	e.srv.LogPacket(api.LogEntry{Time: time.Now(), Source: "198.51.100.7", Protocol: "tcp", DestPort: 22, Verdict: "drop"})
+	c.Sample(context.Background())
+
+	var sum stats.Summary
+	if code := e.call(e.token, "GET", "/stats/summary?since=1h", nil, &sum); code != http.StatusOK {
+		t.Fatalf("summary: %d", code)
+	}
+	if len(sum.BlockedSources) != 1 || sum.BlockedSources[0].Key != "198.51.100.7" || len(sum.BlockedPorts) != 1 || sum.BlockedPorts[0].Key != "tcp/22" {
+		t.Errorf("summary: %+v", sum)
+	}
+	var hits []stats.Entry
+	e.call(e.token, "GET", "/stats/hits", nil, &hits)
+	if len(hits) != 1 || hits[0].Key != "r"+itoa(rule.ID) || hits[0].Count != 0 || !strings.Contains(hits[0].Name, "bad net") {
+		t.Errorf("hits: %+v", hits)
+	}
+	var top []stats.Entry
+	if code := e.call(e.token, "GET", "/stats/top?kind=blocked_port&since=7d", nil, &top); code != http.StatusOK || len(top) != 1 {
+		t.Errorf("top: %d %+v", code, top)
+	}
+	if code := e.call(e.token, "GET", "/stats/top?kind=nonsense", nil, nil); code != http.StatusBadRequest {
+		t.Errorf("an unknown list: %d", code)
+	}
+	if code := e.call(e.token, "GET", "/stats/timeline?keys=zrx:*,drops&since=2h", nil, nil); code != http.StatusOK {
+		t.Errorf("timeline: %d", code)
+	}
+	e.call(e.token, "GET", "/stats/status", nil, &st)
+	if !st.Enabled || st.LastSample == nil {
+		t.Errorf("status: %+v", st)
+	}
+	for mark, want := range map[string]string{"r" + itoa(rule.ID): "input true", "p-input": "input true", "p-output": "output false", "b-antispoof6": "early true", "b-established": " false"} {
+		chain, drop := e.srv.classifyMark(mark)
+		if got := fmt.Sprint(chain, " ", drop); got != want {
+			t.Errorf("%s: %q, want %q", mark, got, want)
+		}
+	}
+}
+
+// TestShadowMode: in shadow mode changes aren't on probation, what would
+// change traffic is refused, and the status, health and log say so.
+func TestShadowMode(t *testing.T) {
+	e := newEnvWith(t, Config{Shadow: true, ShadowPosition: "after"})
+	e.srv.SetConntrack(&fakeConns{})
+	link := t.TempDir() + "/nftables.service"
+	os.Symlink("/usr/lib/systemd/system/nftables.service", link)
+	e.srv.unitLinks = []string{link}
+	e.srv.store.SetMeta(context.Background(), importedTableKey, "inet main")
+	var mr api.MutationResponse
+	if code := e.call(e.token, "POST", "/zones", model.Zone{Name: "wan", Interfaces: []string{"wan"}, External: true}, &mr); code != 201 || mr.ConfirmToken != "" || mr.ConfirmBy != nil {
+		t.Errorf("a change: %d %+v", code, mr)
+	}
+	var pr api.Probation
+	e.call(e.token, "GET", "/probation", nil, &pr)
+	if len(pr.Pending) != 0 || e.fw.boot == nil {
+		t.Errorf("on probation in shadow mode: %+v (saved for the boot: %v)", pr, e.fw.boot != nil)
+	}
+	var er api.ErrorBody
+	if code := e.call(e.token, "POST", "/takeover", api.TakeoverInput{Tables: []string{"inet main"}}, &er); code != 409 || !strings.Contains(er.Error.Message, "shadow mode") || len(e.fw.deleted) != 0 {
+		t.Errorf("takeover: %d %+v %v", code, er, e.fw.deleted)
+	}
+	if code := e.call(e.token, "POST", "/connections/close", api.CloseConnections{Search: "tcp"}, &er); code != 409 {
+		t.Errorf("closing connections: %d", code)
+	}
+	if code := e.call(e.token, "POST", "/rules", model.Rule{Chain: model.ChainInput, Raw: "meta mark set 1"}, &er); code != 201 {
+		t.Fatalf("a raw rule: %d %+v", code, er)
+	}
+	var st api.Status
+	e.call(e.token, "GET", "/status", nil, &st)
+	var codes []string
+	for _, p := range st.Problems {
+		codes = append(codes, p.Code)
+	}
+	c := strings.Join(codes, " ")
+	if st.Mode != api.ModeShadow || st.ShadowPosition != "after" {
+		t.Errorf("status mode: %q %q", st.Mode, st.ShadowPosition)
+	}
+	// Expected in shadow mode: the other firewall, and its service.
+	if strings.Contains(c, "nftables_service") || strings.Contains(c, "takeover_pending") || !strings.Contains(c, "shadow_unprotected") || !strings.Contains(c, "shadow_raw_left_out") {
+		t.Errorf("warnings: %s", c)
+	}
+	e.fw.ruleset = `{"nftables":[{"chain":{"family":"inet","table":"main","name":"input","type":"filter","hook":"input","prio":0,"policy":"drop"}}]}`
+	e.call(e.token, "GET", "/status", nil, &st)
+	for _, p := range st.Problems {
+		if p.Code == "shadow_unprotected" {
+			t.Error("unprotected while inet main filters")
+		}
+	}
+	var h health
+	resp, err := http.Get(e.ts.URL + "/healthz")
+	if err != nil {
+		t.Fatal(err)
+	}
+	json.NewDecoder(resp.Body).Decode(&h)
+	resp.Body.Close()
+	if h.Mode != api.ModeShadow || !h.OK {
+		t.Errorf("healthz: %+v", h)
+	}
+	var who api.Whoami
+	e.call(e.token, "GET", "/auth/whoami", nil, &who)
+	if who.Mode != api.ModeShadow {
+		t.Errorf("whoami: %+v", who)
+	}
+	e.srv.LogPacket(api.LogEntry{Time: time.Now(), Prefix: "hc input policy drop: ", Verdict: "drop"})
+	var logs []api.LogEntry
+	e.call(e.token, "GET", "/log", nil, &logs)
+	if len(logs) != 1 || !logs[0].Shadow {
+		t.Errorf("log: %+v", logs)
+	}
+	shadow := e.text("/ruleset")
+	enforcing := e.text("/ruleset?enforcing=1")
+	if !strings.Contains(shadow, "Shadow mode") || strings.Contains(enforcing, "Shadow mode") {
+		t.Errorf("rulesets:\n%s\n---\n%s", shadow, enforcing)
+	}
+	// Enforcing, the same change goes on probation.
+	n := newEnv(t)
+	n.call(n.token, "POST", "/zones", model.Zone{Name: "wan", Interfaces: []string{"wan"}, External: true}, nil)
+	n.call(n.token, "GET", "/probation", nil, &pr)
+	if len(pr.Pending) != 1 {
+		t.Errorf("not on probation enforcing: %+v", pr)
+	}
+	n.call(n.token, "GET", "/status", nil, &st)
+	if st.Mode != api.ModeEnforcing {
+		t.Errorf("enforcing status mode: %q", st.Mode)
+	}
+}
+
+// text GETs a plain-text answer.
+func (e *env) text(path string) string {
+	e.t.Helper()
+	req, _ := http.NewRequest("GET", e.ts.URL+api.Prefix+path, nil)
+	req.Header.Set("Authorization", "Bearer "+e.token)
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		e.t.Fatal(err)
+	}
+	defer resp.Body.Close()
+	b, _ := io.ReadAll(resp.Body)
+	return string(b)
+}
+
+// TestCreatePlaced: a rule created ?before= another is placed in the same
+// change (one history entry), also when staged in a changeset.
+func TestCreatePlaced(t *testing.T) {
+	e := newEnv(t)
+	var a, b, c api.MutationResponse
+	e.call(e.token, "POST", "/rules", model.Rule{Chain: "input", Match: model.Match{Services: []string{"ssh"}}}, &a)
+	e.call(e.token, "POST", "/rules", model.Rule{Chain: "input", Match: model.Match{Services: []string{"dns"}}}, &b)
+	var first model.Rule
+	json.Unmarshal(a.Object, &first)
+	if code := e.call(e.token, "POST", "/rules?before="+itoa(first.ID), model.Rule{Chain: "input", Match: model.Match{Services: []string{"http"}}, Comment: "placed"}, &c); code != 201 {
+		t.Fatalf("create placed: %d %+v", code, c)
+	}
+	order := func() string {
+		var out []string
+		for _, r := range e.eng.State().Rules {
+			out = append(out, strings.Join(r.Services, ""))
+		}
+		return strings.Join(out, ",")
+	}
+	if o := order(); o != "http,ssh,dns" {
+		t.Errorf("order: %s", o)
+	}
+	var hs []model.HistoryEntry
+	e.call(e.token, "GET", "/history", nil, &hs)
+	if len(hs) != 3 || len(hs[0].Changes) != 1 || hs[0].Changes[0].Action() != "create" {
+		t.Errorf("history: %d entries, newest %+v", len(hs), hs[0])
+	}
+	// Staged, the place goes with it.
+	var cs model.Changeset
+	e.call(e.token, "POST", "/changesets", nil, &cs)
+	e.call(e.token, "POST", "/rules?first=1&changeset="+itoa(cs.ID), model.Rule{Chain: "input", Match: model.Match{Services: []string{"ntp"}}}, nil)
+	if code := e.call(e.token, "POST", "/changesets/"+itoa(cs.ID)+"/apply", nil, nil); code != 200 {
+		t.Fatalf("apply: %d", code)
+	}
+	if o := order(); o != "ntp,http,ssh,dns" {
+		t.Errorf("order after the changeset: %s", o)
+	}
+	// A bad place: refused, nothing created.
+	if code := e.call(e.token, "POST", "/rules?before=99999", model.Rule{Chain: "input"}, nil); code != 400 {
+		t.Errorf("before a rule that isn't there: %d", code)
+	}
+	if n := len(e.eng.State().Rules); n != 4 {
+		t.Errorf("%d rules", n)
+	}
+}

internal/stats/collector.go

@@ -0,0 +1,546 @@
+package stats
+
+import (
+	"cmp"
+	"context"
+	"errors"
+	"fmt"
+	"log/slog"
+	"slices"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"housecarl/internal/api"
+	"housecarl/internal/nft"
+)
+
+// Resolutions and retention, and the sources the figures come from.
+type Options struct {
+	TopN         int
+	RetainMinute time.Duration
+	RetainHour   time.Duration
+	RetainDay    time.Duration
+	Location     *time.Location // for days (default: local time)
+	// Interval is how often the collector samples (default: a minute).
+	Interval time.Duration
+}
+
+// Sources are what the collector reads. Nil functions are left out.
+type Sources struct {
+	// Table reads Housecarl's table: the counters by mark (totals across
+	// reloads), the quotas, the per-device sets.
+	Table func(context.Context) (*nft.Table, error)
+	// Zones maps the interfaces to their zones.
+	Zones func() map[string]string
+	// Classify says which chain a mark's counter is in and whether what
+	// it counts is dropped ("r12" → forward, true).
+	Classify func(mark string) (chain string, drop bool)
+	// SysNet and Proc are /sys/class/net and /proc (tests point them
+	// elsewhere).
+	SysNet, Proc string
+}
+
+// Counter keys. Traffic: rx:IF, tx:IF (bytes), rxp:IF, txp:IF (packets),
+// zrx:ZONE, ztx:ZONE (bytes received from and sent to a zone's
+// interfaces). Hits: hits:MARK (packets), hitb:MARK (bytes). Drops:
+// drop:CHAIN and drops (packets). Quotas: quota:r12 (bytes). Gauges
+// (g:KEY, see gauge) are below.
+const (
+	GaugeConntrack    = "conntrack"
+	GaugeConntrackMax = "conntrack_max"
+	GaugeRSS          = "rss"  // housecarld's resident memory, bytes
+	GaugeCPU          = "cpu"  // housecarld's CPU, tenths of a percent of one CPU
+	GaugeMem          = "mem"  // the host's memory in use, bytes
+	GaugeLoad         = "load" // the host's 1-minute load average, times 100
+)
+
+// Top-list kinds: per hour and per day, the TopN largest.
+const (
+	KindBlockedSource = "blocked_src"  // logged drops by source address
+	KindBlockedPort   = "blocked_port" // logged drops by what they tried: tcp/22, icmp
+	KindDeviceUp      = "device_up"    // bytes sent to the internet, by inside address
+	KindDeviceDown    = "device_down"  // bytes received from it
+	// A per-address quota's use by address is "quota_r12".
+)
+
+// Kinds are the top lists Top accepts (and quota_rN).
+var Kinds = []string{KindBlockedSource, KindBlockedPort, KindDeviceUp, KindDeviceDown}
+
+// Status describes the collector's health.
+type Status struct {
+	Enabled     bool       `json:"enabled"`
+	Database    string     `json:"database,omitempty"`
+	Size        int64      `json:"size"`
+	LastSample  *time.Time `json:"last_sample,omitempty"`
+	SampleError string     `json:"sample_error,omitempty"`
+	FirstData   *time.Time `json:"first_data,omitempty"`
+	// PerDevice: the firewall counts traffic per device (a setting).
+	PerDevice bool `json:"per_device"`
+	Retention struct {
+		Minute string `json:"minute"`
+		Hour   string `json:"hour"`
+		Day    string `json:"day"`
+	} `json:"retention"`
+}
+
+// Latest are the newest gauge values.
+type Latest struct {
+	At           time.Time `json:"at"`
+	Conntrack    int64     `json:"conntrack"`
+	ConntrackMax int64     `json:"conntrack_max"`
+	RSS          int64     `json:"rss"`
+	CPUPermille  int64     `json:"cpu_permille"`
+	MemUsed      int64     `json:"mem_used"`
+	Load         float64   `json:"load"`
+}
+
+// Collector gathers statistics into a Store.
+type Collector struct {
+	opts  Options
+	src   Sources
+	store *Store
+	log   *slog.Logger
+	now   func() time.Time
+
+	// Each source's last totals: the next sample stores the difference.
+	ifs, table deltas
+	cpuTicks   int64
+	cpuAt      time.Time
+	lastPrune  time.Time
+
+	mu     sync.Mutex
+	hours  map[int64]map[string]map[string]int64 // live hours: hour -> kind -> key -> count
+	status Status
+	latest *Latest
+}
+
+// maxKeys bounds each kind of an hour's top list in memory.
+const maxKeys = 100_000
+
+// NewCollector creates a collector. Run starts it.
+func NewCollector(store *Store, opts Options, src Sources, log *slog.Logger) *Collector {
+	if opts.Location == nil {
+		opts.Location = time.Local
+	}
+	if log == nil {
+		log = slog.Default()
+	}
+	if opts.TopN <= 0 {
+		opts.TopN = 100
+	}
+	if opts.Interval <= 0 {
+		opts.Interval = time.Minute
+	}
+	if src.SysNet == "" {
+		src.SysNet = "/sys/class/net"
+	}
+	if src.Proc == "" {
+		src.Proc = "/proc"
+	}
+	c := &Collector{opts: opts, src: src, store: store, log: log, now: time.Now, hours: map[int64]map[string]map[string]int64{}}
+	c.status.Enabled = true
+	c.status.Database = store.Path()
+	c.status.Retention.Minute = humanDuration(opts.RetainMinute)
+	c.status.Retention.Hour = humanDuration(opts.RetainHour)
+	c.status.Retention.Day = humanDuration(opts.RetainDay)
+	return c
+}
+
+// SetClock replaces the clock (tests).
+func (c *Collector) SetClock(now func() time.Time) { c.now = now }
+
+// Store returns the database.
+func (c *Collector) Store() *Store { return c.store }
+
+// Options returns the collector's options.
+func (c *Collector) Options() Options { return c.opts }
+
+// Status returns the collector's health.
+func (c *Collector) Status() Status {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	s := c.status
+	s.Size = c.store.Size()
+	return s
+}
+
+// Latest returns the newest gauges (nil before the first sample).
+func (c *Collector) Latest() *Latest {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if c.latest == nil {
+		return nil
+	}
+	l := *c.latest
+	return &l
+}
+
+func (c *Collector) minuteOf(t time.Time) int64 { return t.Unix() - t.Unix()%60 }
+func (c *Collector) hourOf(t time.Time) int64   { return t.Unix() - t.Unix()%3600 }
+
+// dayOf returns the start of the local day containing t.
+func (c *Collector) dayOf(t time.Time) int64 {
+	t = t.In(c.opts.Location)
+	return time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, c.opts.Location).Unix()
+}
+
+// Run collects until ctx is done.
+func (c *Collector) Run(ctx context.Context) {
+	c.Start(ctx)
+	for {
+		t := time.NewTimer(time.Until(c.nextSample()))
+		select {
+		case <-ctx.Done():
+			t.Stop()
+			c.write(context.Background(), &Batch{})
+			return
+		case <-t.C:
+			c.Sample(ctx)
+		}
+	}
+}
+
+// nextSample: two seconds into the next minute, so the minute that just
+// ended is complete (or after the interval, when it isn't a minute).
+func (c *Collector) nextSample() time.Time {
+	if c.opts.Interval != time.Minute {
+		return c.now().Add(c.opts.Interval)
+	}
+	return time.Unix(c.minuteOf(c.now())+62, 0)
+}
+
+// Start takes the first sample (the baseline the next ones are counted
+// from), continues the current hour's top lists, and adds hours that
+// ended while housecarld wasn't running to their days.
+func (c *Collector) Start(ctx context.Context) {
+	if v, _ := c.store.Meta(ctx, "first_data"); v != "" {
+		if n, err := strconv.ParseInt(v, 10, 64); err == nil {
+			t := time.Unix(n, 0)
+			c.status.FirstData = &t
+		}
+	}
+	cur := c.hourOf(c.now())
+	b := &Batch{}
+	merged, _ := c.store.Meta(ctx, "merged_through")
+	if last, err := strconv.ParseInt(merged, 10, 64); err == nil {
+		for h := max(last+3600, cur-48*3600); h < cur; h += 3600 {
+			kinds, _ := c.store.KindsAt(ctx, ResHour, h)
+			for _, k := range kinds {
+				if es, err := c.store.TopsAt(ctx, ResHour, h, k); err == nil && len(es) > 0 {
+					b.AddTops(ResDay, c.dayOf(time.Unix(h, 0)), k, es, 3*c.opts.TopN)
+				}
+			}
+		}
+	}
+	b.SetMeta("merged_through", strconv.FormatInt(cur-3600, 10))
+	kinds, _ := c.store.KindsAt(ctx, ResHour, cur)
+	c.mu.Lock()
+	for _, k := range kinds {
+		if es, err := c.store.TopsAt(ctx, ResHour, cur, k); err == nil {
+			m := c.topsMap(cur, k)
+			for _, e := range es {
+				m[e.Key] = e.Count
+			}
+		}
+	}
+	c.mu.Unlock()
+	if err := c.store.Commit(ctx, b); err != nil {
+		c.log.Warn("statistics: catching up failed", "err", err)
+	}
+	c.sample(ctx, -1)
+	c.lastPrune = c.now()
+	c.prune(ctx)
+	c.log.Info("statistics collector started", "database", c.store.Path())
+}
+
+// Sample reads every source and writes what changed since the last
+// sample.
+func (c *Collector) Sample(ctx context.Context) {
+	now := c.now()
+	// The interval that just ended: with a minute's interval, sampling two
+	// seconds into a minute, the previous minute.
+	minute := c.minuteOf(now.Add(-c.opts.Interval / 2))
+	vals := c.sample(ctx, minute)
+	b := &Batch{}
+	if len(vals) > 0 {
+		t := time.Unix(minute, 0)
+		b.AddCounters(ResMinute, minute, vals)
+		b.AddCounters(ResHour, c.hourOf(t), vals)
+		b.AddCounters(ResDay, c.dayOf(t), vals)
+		c.mu.Lock()
+		if c.status.FirstData == nil {
+			c.status.FirstData = &t
+			b.SetMeta("first_data", strconv.FormatInt(minute, 10))
+		}
+		c.mu.Unlock()
+	}
+	c.write(ctx, b)
+	if now.Sub(c.lastPrune) >= time.Hour {
+		c.lastPrune = now
+		c.prune(ctx)
+	}
+}
+
+// write adds the live hours' top lists to b, ends the hours that are
+// over, and commits.
+func (c *Collector) write(ctx context.Context, b *Batch) {
+	c.mu.Lock()
+	cur := c.hourOf(c.now())
+	for h, kinds := range c.hours {
+		for k, m := range kinds {
+			b.SetTops(ResHour, h, k, top(m, c.opts.TopN))
+		}
+		if h < cur {
+			// The hour is over: add it to its day.
+			for k, m := range kinds {
+				b.AddTops(ResDay, c.dayOf(time.Unix(h, 0)), k, top(m, c.opts.TopN), 3*c.opts.TopN)
+			}
+			delete(c.hours, h)
+			b.SetMeta("merged_through", strconv.FormatInt(h, 10))
+		}
+	}
+	c.mu.Unlock()
+	if err := c.store.Commit(ctx, b); err != nil {
+		c.log.Error("statistics: writing failed", "err", err)
+	}
+}
+
+// deltas turn a source's totals into what changed since its last read.
+type deltas struct {
+	prev, next map[string]int64
+	based      bool
+}
+
+// of returns how much key grew. The first read is the baseline (nothing
+// counts); after it, a new key counts whole (a new interface, rule or
+// device starts from zero), and a total that went down started again (an
+// interface came back, a quota was reset, a device's entry expired).
+func (d *deltas) of(key string, cur int64) int64 {
+	if d.next == nil {
+		d.next = map[string]int64{}
+	}
+	d.next[key] = cur
+	if !d.based {
+		return 0
+	}
+	p, ok := d.prev[key]
+	if !ok || cur < p {
+		return cur
+	}
+	return cur - p
+}
+
+// done keeps this read's totals for the next (a failed read skips it, so
+// its keys aren't new next time).
+func (d *deltas) done() { d.prev, d.next, d.based = d.next, nil, true }
+
+// sample reads the sources; minute is the bucket the changes go to (-1:
+// only the baseline). It returns the counters.
+func (c *Collector) sample(ctx context.Context, minute int64) map[string]int64 {
+	now := c.now()
+	vals := map[string]int64{}
+	add := func(k string, v int64) {
+		if v != 0 {
+			vals[k] += v
+		}
+	}
+	gauge := func(k string, v int64) {
+		vals["g:"+k] += v
+		vals["gn:"+k]++
+	}
+	tops := map[string]map[string]int64{}
+	addTop := func(kind, key string, v int64) {
+		if v <= 0 {
+			return
+		}
+		if tops[kind] == nil {
+			tops[kind] = map[string]int64{}
+		}
+		tops[kind][key] += v
+	}
+	var errs []string
+
+	// Interfaces, and their zones.
+	var zones map[string]string
+	if c.src.Zones != nil {
+		zones = c.src.Zones()
+	}
+	for name, ic := range readInterfaces(c.src.SysNet) {
+		rx, tx := c.ifs.of("rx:"+name, ic.RxBytes), c.ifs.of("tx:"+name, ic.TxBytes)
+		add("rx:"+name, rx)
+		add("tx:"+name, tx)
+		add("rxp:"+name, c.ifs.of("rxp:"+name, ic.RxPackets))
+		add("txp:"+name, c.ifs.of("txp:"+name, ic.TxPackets))
+		if z := zones[name]; z != "" {
+			add("zrx:"+z, rx)
+			add("ztx:"+z, tx)
+		}
+	}
+	c.ifs.done()
+
+	// Housecarl's table: hits, drops, quotas, devices.
+	if c.src.Table != nil {
+		t, err := c.src.Table(ctx)
+		switch {
+		case err == nil:
+			for mark, ctr := range t.Counters {
+				p := c.table.of("hits:"+mark, int64(ctr.Packets))
+				add("hits:"+mark, p)
+				add("hitb:"+mark, c.table.of("hitb:"+mark, int64(ctr.Bytes)))
+				if c.src.Classify != nil && p > 0 {
+					if chain, drop := c.src.Classify(mark); drop {
+						add("drop:"+chain, p)
+						add("drops", p)
+					}
+				}
+			}
+			for name, used := range t.Quotas {
+				add("quota:"+strings.TrimPrefix(name, "quota_"), c.table.of("quota:"+name, used))
+			}
+			for set, es := range t.QuotaSets {
+				kind := strings.TrimSuffix(strings.TrimSuffix(set, "_v4"), "_v6")
+				for _, e := range es {
+					addTop(kind, e.Addr, c.table.of(set+"/"+e.Addr, e.Used))
+				}
+			}
+			for set, es := range t.DeviceSets {
+				kind := map[bool]string{true: KindDeviceUp, false: KindDeviceDown}[strings.HasPrefix(set, "stats_up")]
+				for _, e := range es {
+					key := e.Addr
+					if strings.HasSuffix(set, "_v6") {
+						key += "/64"
+					}
+					addTop(kind, key, c.table.of(set+"/"+e.Addr, int64(e.Bytes)))
+				}
+			}
+			c.table.done()
+		case errors.Is(err, nft.ErrNoTable):
+		default:
+			errs = append(errs, "reading the table: "+err.Error())
+		}
+	}
+
+	// Gauges.
+	l := &Latest{At: now}
+	if n, m, ok := readConntrack(c.src.Proc); ok {
+		l.Conntrack, l.ConntrackMax = n, m
+		gauge(GaugeConntrack, n)
+		gauge(GaugeConntrackMax, m)
+	}
+	if rss, ticks, ok := readSelf(c.src.Proc); ok {
+		l.RSS = rss
+		gauge(GaugeRSS, rss)
+		if !c.cpuAt.IsZero() && ticks >= c.cpuTicks {
+			secs := max(int64(now.Sub(c.cpuAt)/time.Second), 1)
+			l.CPUPermille = (ticks - c.cpuTicks) * 1000 / clockTicks / secs
+			gauge(GaugeCPU, l.CPUPermille)
+		}
+		c.cpuTicks, c.cpuAt = ticks, now
+	}
+	if mem, load, ok := readHost(c.src.Proc); ok {
+		l.MemUsed, l.Load = mem, load
+		gauge(GaugeMem, mem)
+		gauge(GaugeLoad, int64(load*100))
+	}
+
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.latest = l
+	c.status.LastSample = &now
+	c.status.SampleError = strings.Join(errs, "; ")
+	if minute < 0 {
+		return nil
+	}
+	h := c.hourOf(time.Unix(minute, 0))
+	for kind, m := range tops {
+		for k, v := range m {
+			c.addTop(h, kind, k, v)
+		}
+	}
+	return vals
+}
+
+// Packet counts a packet the firewall logged: drops by source and by
+// what they tried.
+func (c *Collector) Packet(e api.LogEntry) {
+	if e.Verdict != "drop" && e.Verdict != "reject" {
+		return
+	}
+	what := e.Protocol
+	switch {
+	case e.DestPort > 0:
+		what = fmt.Sprintf("%s/%d", e.Protocol, e.DestPort)
+	case e.ICMPType != "":
+		what = e.Protocol + "/" + e.ICMPType
+	}
+	t := e.Time
+	if t.IsZero() {
+		t = c.now()
+	}
+	h := c.hourOf(t)
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if e.Source != "" {
+		c.addTop(h, KindBlockedSource, e.Source, 1)
+	}
+	if what != "" {
+		c.addTop(h, KindBlockedPort, what, 1)
+	}
+}
+
+// addTop counts into an hour's top list. Called with c.mu held.
+func (c *Collector) addTop(hour int64, kind, key string, n int64) {
+	m := c.topsMap(hour, kind)
+	if _, ok := m[key]; !ok && len(m) >= maxKeys {
+		return
+	}
+	m[key] += n
+}
+
+func (c *Collector) topsMap(hour int64, kind string) map[string]int64 {
+	h := c.hours[hour]
+	if h == nil {
+		h = map[string]map[string]int64{}
+		c.hours[hour] = h
+	}
+	m := h[kind]
+	if m == nil {
+		m = map[string]int64{}
+		h[kind] = m
+	}
+	return m
+}
+
+// top returns the n largest entries of m.
+func top(m map[string]int64, n int) []Entry {
+	out := make([]Entry, 0, len(m))
+	for k, v := range m {
+		out = append(out, Entry{Key: k, Count: v})
+	}
+	slices.SortFunc(out, func(a, b Entry) int {
+		if a.Count != b.Count {
+			return cmp.Compare(b.Count, a.Count)
+		}
+		return cmp.Compare(a.Key, b.Key)
+	})
+	if len(out) > n {
+		out = out[:n]
+	}
+	return out
+}
+
+func (c *Collector) prune(ctx context.Context) {
+	now := c.now().Unix()
+	for res, keep := range map[string]time.Duration{ResMinute: c.opts.RetainMinute, ResHour: c.opts.RetainHour, ResDay: c.opts.RetainDay} {
+		if keep <= 0 {
+			continue
+		}
+		if n, err := c.store.Prune(ctx, res, now-int64(keep.Seconds())); err != nil {
+			c.log.Warn("statistics: pruning failed", "err", err)
+		} else if n > 0 {
+			c.log.Debug("statistics: pruned", "resolution", res, "rows", n)
+		}
+	}
+}

internal/stats/collector_test.go

@@ -0,0 +1,238 @@
+package stats
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"log/slog"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"housecarl/internal/api"
+	"housecarl/internal/nft"
+	"housecarl/internal/render"
+)
+
+// fakeHost is a /sys/class/net and a /proc to read from.
+type fakeHost struct {
+	t          *testing.T
+	sys, proc  string
+	table      *nft.Table
+	tableError error
+}
+
+func newFakeHost(t *testing.T) *fakeHost {
+	dir := t.TempDir()
+	h := &fakeHost{t: t, sys: filepath.Join(dir, "sys"), proc: filepath.Join(dir, "proc"),
+		table: &nft.Table{Counters: map[string]nft.Counter{}, QuotaUsage: nft.QuotaUsage{Quotas: map[string]int64{}, QuotaSets: map[string][]render.Element{}},
+			DeviceSets: map[string][]render.Element{}}}
+	h.write("proc/sys/net/netfilter/nf_conntrack_count", "120")
+	h.write("proc/sys/net/netfilter/nf_conntrack_max", "65536")
+	h.write("proc/self/stat", "4242 (housecarld) S 1 4242 4242 0 -1 4194560 3000 0 0 0 150 50 0 0 20 0 9 0 100 1000000 3000")
+	h.write("proc/self/status", "Name:\thousecarld\nVmRSS:\t   20480 kB\nThreads:\t9\n")
+	h.write("proc/meminfo", "MemTotal:        8000000 kB\nMemFree:         1000000 kB\nMemAvailable:    6000000 kB\n")
+	h.write("proc/loadavg", "0.50 0.40 0.30 1/200 4242\n")
+	return h
+}
+
+func (h *fakeHost) write(rel, content string) {
+	p := filepath.Join(filepath.Dir(h.sys), rel)
+	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+		h.t.Fatal(err)
+	}
+	if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
+		h.t.Fatal(err)
+	}
+}
+
+func (h *fakeHost) iface(name string, rx, tx int64) {
+	h.write("sys/"+name+"/statistics/rx_bytes", fmt.Sprint(rx))
+	h.write("sys/"+name+"/statistics/tx_bytes", fmt.Sprint(tx))
+	h.write("sys/"+name+"/statistics/rx_packets", fmt.Sprint(rx/100))
+	h.write("sys/"+name+"/statistics/tx_packets", fmt.Sprint(tx/100))
+}
+
+func (h *fakeHost) sources() Sources {
+	return Sources{
+		SysNet: h.sys, Proc: h.proc,
+		Table: func(context.Context) (*nft.Table, error) {
+			if h.tableError != nil {
+				return nil, h.tableError
+			}
+			return h.table, nil
+		},
+		Zones: func() map[string]string { return map[string]string{"eth0": "wan", "eth1": "lan"} },
+		Classify: func(mark string) (string, bool) {
+			switch mark {
+			case "r1":
+				return "forward", true
+			case "p-input":
+				return "input", true
+			}
+			return "input", false
+		},
+	}
+}
+
+func TestCollector(t *testing.T) {
+	ctx := context.Background()
+	h := newFakeHost(t)
+	h.iface("eth0", 1000, 500)
+	h.iface("eth1", 2000, 2000)
+	h.iface("lo", 9999, 9999)
+	h.table.Counters["r1"] = nft.Counter{Packets: 10, Bytes: 1000}
+	h.table.Counters["r2"] = nft.Counter{Packets: 100, Bytes: 9000}
+	h.table.Quotas["quota_r5"] = 1 << 20
+	h.table.DeviceSets["stats_up_v4"] = []render.Element{{Addr: "10.0.10.5", Bytes: 5000}}
+
+	store, err := OpenStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	loc := time.FixedZone("test", 0)
+	now := time.Date(2026, 10, 19, 10, 0, 30, 0, loc)
+	c := NewCollector(store, Options{TopN: 10, RetainMinute: 7 * 24 * time.Hour, RetainHour: 90 * 24 * time.Hour, RetainDay: 730 * 24 * time.Hour, Location: loc},
+		h.sources(), slog.New(slog.NewTextHandler(io.Discard, nil)))
+	c.SetClock(func() time.Time { return now })
+	c.Start(ctx)
+
+	// A minute of traffic.
+	h.iface("eth0", 5000, 1500)
+	h.table.Counters["r1"] = nft.Counter{Packets: 25, Bytes: 2500}
+	h.table.Counters["p-input"] = nft.Counter{Packets: 3, Bytes: 180} // a new counter counts whole
+	h.table.Quotas["quota_r5"] = 3 << 20
+	h.table.DeviceSets["stats_up_v4"] = []render.Element{{Addr: "10.0.10.5", Bytes: 8000}}
+	h.table.DeviceSets["stats_down_v6"] = []render.Element{{Addr: "2001:db8:1:2::", Bytes: 70000}}
+	for range 2 {
+		c.Packet(api.LogEntry{Time: now, Source: "198.51.100.7", Protocol: "tcp", DestPort: 22, Verdict: "drop"})
+	}
+	c.Packet(api.LogEntry{Time: now, Source: "198.51.100.8", Protocol: "icmp", ICMPType: "echo-request", Verdict: "drop"})
+	c.Packet(api.LogEntry{Time: now, Source: "10.0.10.5", Protocol: "udp", DestPort: 53, Verdict: "accept"}) // not a drop
+	now = time.Date(2026, 10, 19, 10, 1, 2, 0, loc)
+	c.Sample(ctx)
+
+	q := NewQuerier(c)
+	sum := func(prefix string) map[string]int64 {
+		es, err := q.Sums(ctx, prefix, time.Hour)
+		if err != nil {
+			t.Fatal(err)
+		}
+		m := map[string]int64{}
+		for _, e := range es {
+			m[e.Key] = e.Count
+		}
+		return m
+	}
+	check := func(what string, got, want int64) {
+		t.Helper()
+		if got != want {
+			t.Errorf("%s = %d, want %d", what, got, want)
+		}
+	}
+	check("rx:eth0", sum("rx:")["eth0"], 4000)
+	check("rx:eth1 (no change)", sum("rx:")["eth1"], 0)
+	check("rx:lo", sum("rx:")["lo"], 0)
+	check("zrx:wan", sum("zrx:")["wan"], 4000)
+	check("ztx:wan", sum("ztx:")["wan"], 1000)
+	check("hits:r1", sum("hits:")["r1"], 15)
+	check("hits:p-input", sum("hits:")["p-input"], 3)
+	check("drop:forward", sum("drop:")["forward"], 15)
+	check("drop:input", sum("drop:")["input"], 3)
+	check("quota:r5", sum("quota:")["r5"], 2<<20)
+
+	s, err := q.Summary(ctx, time.Hour, 10)
+	if err != nil {
+		t.Fatal(err)
+	}
+	check("drops", s.Drops, 18)
+	if len(s.Zones) == 0 || s.Zones[0].Name != "wan" || s.Zones[0].Received != 4000 {
+		t.Errorf("zones %+v", s.Zones)
+	}
+	if len(s.BlockedSources) != 2 || s.BlockedSources[0] != (Entry{Key: "198.51.100.7", Count: 2}) {
+		t.Errorf("blocked sources %+v", s.BlockedSources)
+	}
+	if len(s.BlockedPorts) != 2 || s.BlockedPorts[0].Key != "tcp/22" || s.BlockedPorts[1].Key != "icmp/echo-request" {
+		t.Errorf("blocked ports %+v", s.BlockedPorts)
+	}
+	if len(s.DevicesUp) != 1 || s.DevicesUp[0] != (Entry{Key: "10.0.10.5", Count: 3000}) {
+		t.Errorf("devices up %+v", s.DevicesUp)
+	}
+	if len(s.DevicesDown) != 1 || s.DevicesDown[0] != (Entry{Key: "2001:db8:1:2::/64", Count: 70000}) {
+		t.Errorf("devices down %+v", s.DevicesDown)
+	}
+	check("peak connections", s.PeakConnections, 120)
+
+	// Gauges average; the CPU needs two samples.
+	tl, err := q.Timeline(ctx, time.Hour, "1m", []string{"g:" + GaugeRSS, "g:" + GaugeMem, "g:" + GaugeLoad, "rx:*"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	last := func(i int) int64 {
+		for j := len(tl.Times) - 1; j >= 0; j-- {
+			if tl.Times[j] == time.Date(2026, 10, 19, 10, 0, 0, 0, loc).Unix() {
+				return tl.Series[i].Values[j]
+			}
+		}
+		return -2
+	}
+	check("rss", last(0), 20480*1024)
+	check("mem", last(1), 2000000*1024)
+	check("load", last(2), 50)
+	if len(tl.Series) != 4 || tl.Series[3].Key != "rx:eth0" {
+		t.Errorf("rx:* expanded to %+v", tl.Series[3:])
+	}
+
+	// Totals that go down started again: an interface came back, a quota
+	// was reset. A table that can't be read is skipped, and its keys
+	// aren't new the next time.
+	h.iface("eth0", 100, 1500)
+	h.table.Quotas["quota_r5"] = 1000
+	h.tableError = fmt.Errorf("nft failed")
+	now = time.Date(2026, 10, 19, 10, 2, 2, 0, loc)
+	c.Sample(ctx)
+	if st := c.Status(); st.SampleError == "" {
+		t.Error("no sample error")
+	}
+	h.tableError = nil
+	now = time.Date(2026, 10, 19, 10, 3, 2, 0, loc)
+	c.Sample(ctx)
+	check("rx:eth0 after a reset", sum("rx:")["eth0"], 4100)
+	check("quota:r5 after a reset", sum("quota:")["r5"], 2<<20+1000)
+	check("hits:r1 after a failed read", sum("hits:")["r1"], 15)
+
+	// The hour ends: its top lists go to its day.
+	now = time.Date(2026, 10, 19, 11, 0, 2, 0, loc)
+	c.Sample(ctx)
+	day, err := store.TopsAt(ctx, ResDay, time.Date(2026, 10, 19, 0, 0, 0, 0, loc).Unix(), KindBlockedSource)
+	if err != nil || len(day) != 2 || day[0].Count != 2 {
+		t.Errorf("the day's blocked sources: %+v %v", day, err)
+	}
+	if es, _ := q.Top(ctx, KindBlockedSource, 30*24*time.Hour, 5); len(es) != 2 {
+		t.Errorf("30 days of blocked sources: %+v", es)
+	}
+
+	// A restart continues the hour's top lists.
+	c.Packet(api.LogEntry{Time: now, Source: "203.0.113.9", Protocol: "tcp", DestPort: 23, Verdict: "drop"})
+	c.write(ctx, &Batch{})
+	c2 := NewCollector(store, c.opts, h.sources(), c.log)
+	c2.SetClock(func() time.Time { return now })
+	c2.Start(ctx)
+	c2.Packet(api.LogEntry{Time: now, Source: "203.0.113.9", Protocol: "tcp", DestPort: 23, Verdict: "drop"})
+	c2.write(ctx, &Batch{})
+	hour, _ := store.TopsAt(ctx, ResHour, time.Date(2026, 10, 19, 11, 0, 0, 0, loc).Unix(), KindBlockedSource)
+	if len(hour) != 1 || hour[0].Count != 2 {
+		t.Errorf("after a restart, the hour's blocked sources: %+v", hour)
+	}
+
+	// The heatmap of drops.
+	hm, err := q.Heatmap(ctx, "", 7*24*time.Hour)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if hm.Avg[0][10] != 18 { // a Monday, 10:00
+		t.Errorf("heatmap Monday 10:00 = %v", hm.Avg[0][10])
+	}
+}

internal/stats/query.go

@@ -0,0 +1,480 @@
+package stats
+
+import (
+	"cmp"
+	"context"
+	"fmt"
+	"slices"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// Querier answers questions about stored statistics.
+type Querier struct {
+	Store *Store
+	Opts  Options
+	Now   func() time.Time
+}
+
+// NewQuerier returns a querier for a collector's data.
+func NewQuerier(c *Collector) *Querier {
+	return &Querier{Store: c.store, Opts: c.opts, Now: c.now}
+}
+
+func (q *Querier) loc() *time.Location {
+	if q.Opts.Location != nil {
+		return q.Opts.Location
+	}
+	return time.Local
+}
+
+// Resolution picks the finest stored resolution that covers a time range
+// starting since ago.
+func (q *Querier) Resolution(since time.Duration) string {
+	switch {
+	case since <= 48*time.Hour && since <= q.Opts.RetainMinute:
+		return ResMinute
+	case since <= q.Opts.RetainHour:
+		return ResHour
+	}
+	return ResDay
+}
+
+// topsResolution picks the resolution for top lists, which exist per hour
+// and per day.
+func (q *Querier) topsResolution(since time.Duration) string {
+	if since <= 7*24*time.Hour && since <= q.Opts.RetainHour {
+		return ResHour
+	}
+	return ResDay
+}
+
+// bucketStart returns the start of the bucket containing t at a resolution.
+func (q *Querier) bucketStart(res string, t time.Time) int64 {
+	switch res {
+	case ResMinute:
+		return t.Unix() - t.Unix()%60
+	case ResHour:
+		return t.Unix() - t.Unix()%3600
+	}
+	l := t.In(q.loc())
+	return time.Date(l.Year(), l.Month(), l.Day(), 0, 0, 0, 0, q.loc()).Unix()
+}
+
+// sums returns the totals of counters (optionally with a key prefix) in
+// [from, to).
+func (s *Store) sums(ctx context.Context, res string, from, to int64, prefix string) (map[string]int64, error) {
+	rows, err := s.db.QueryContext(ctx, `SELECT key, SUM(value) FROM counters WHERE res = ? AND ts >= ? AND ts < ? AND key LIKE ? ESCAPE '\' GROUP BY key`,
+		res, from, to, likePrefix(prefix))
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	out := map[string]int64{}
+	for rows.Next() {
+		var k string
+		var v int64
+		if err := rows.Scan(&k, &v); err != nil {
+			return nil, err
+		}
+		out[k] = v
+	}
+	return out, rows.Err()
+}
+
+func likePrefix(p string) string {
+	r := strings.NewReplacer(`\`, `\\`, `%`, `\%`, `_`, `\_`)
+	return r.Replace(p) + "%"
+}
+
+// series returns the values of a counter per bucket in [from, to).
+func (s *Store) series(ctx context.Context, res string, from, to int64, key string) (map[int64]int64, error) {
+	rows, err := s.db.QueryContext(ctx, `SELECT ts, value FROM counters WHERE res = ? AND key = ? AND ts >= ? AND ts < ?`, res, key, from, to)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	out := map[int64]int64{}
+	for rows.Next() {
+		var ts, v int64
+		if err := rows.Scan(&ts, &v); err != nil {
+			return nil, err
+		}
+		out[ts] = v
+	}
+	return out, rows.Err()
+}
+
+// tops sums top lists (of one or more kinds) in [from, to).
+func (s *Store) tops(ctx context.Context, res string, kinds []string, from, to int64, limit int) ([]Entry, error) {
+	args := []any{res}
+	for _, k := range kinds {
+		args = append(args, k)
+	}
+	args = append(args, from, to, limit)
+	rows, err := s.db.QueryContext(ctx, `SELECT key, SUM(count) AS n FROM tops WHERE res = ? AND kind IN (`+strings.TrimSuffix(strings.Repeat("?,", len(kinds)), ",")+`) AND ts >= ? AND ts < ?
+		GROUP BY key ORDER BY n DESC, key LIMIT ?`, args...)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	out := []Entry{}
+	for rows.Next() {
+		var e Entry
+		if err := rows.Scan(&e.Key, &e.Count); err != nil {
+			return nil, err
+		}
+		out = append(out, e)
+	}
+	return out, rows.Err()
+}
+
+func sorted(m map[string]int64, prefix string) []Entry {
+	out := []Entry{}
+	for k, v := range m {
+		if strings.HasPrefix(k, prefix) && v != 0 {
+			out = append(out, Entry{Key: strings.TrimPrefix(k, prefix), Count: v})
+		}
+	}
+	slices.SortFunc(out, func(a, b Entry) int {
+		if a.Count != b.Count {
+			return cmp.Compare(b.Count, a.Count)
+		}
+		return cmp.Compare(a.Key, b.Key)
+	})
+	return out
+}
+
+// Top returns the largest entries of a top list over the last since.
+func (q *Querier) Top(ctx context.Context, kind string, since time.Duration, limit int) ([]Entry, error) {
+	if !slices.Contains(Kinds, kind) && !quotaKind(kind) {
+		return nil, fmt.Errorf("unknown list %q (use %s, or quota_rN)", kind, strings.Join(Kinds, ", "))
+	}
+	now := q.Now()
+	res := q.topsResolution(since)
+	return q.Store.tops(ctx, res, []string{kind}, q.bucketStart(res, now.Add(-since)), now.Unix()+1, limit)
+}
+
+func quotaKind(k string) bool {
+	n, ok := strings.CutPrefix(k, "quota_r")
+	_, err := strconv.ParseInt(n, 10, 64)
+	return ok && err == nil
+}
+
+// Sums returns the totals of the counters with a prefix over the last
+// since, largest first, the prefix taken off ("hits:" → "r12").
+func (q *Querier) Sums(ctx context.Context, prefix string, since time.Duration) ([]Entry, error) {
+	now := q.Now()
+	res := q.Resolution(since)
+	m, err := q.Store.sums(ctx, res, q.bucketStart(res, now.Add(-since)), now.Unix()+1, prefix)
+	if err != nil {
+		return nil, err
+	}
+	return sorted(m, prefix), nil
+}
+
+// Traffic is what an interface or zone received and sent, in bytes.
+type Traffic struct {
+	Name     string `json:"name"`
+	Received int64  `json:"received"`
+	Sent     int64  `json:"sent"`
+}
+
+// Summary is an overview of the last since.
+type Summary struct {
+	Since      string    `json:"since"`
+	From       time.Time `json:"from"`
+	Zones      []Traffic `json:"zones"`
+	Interfaces []Traffic `json:"interfaces"`
+	// Drops are the packets dropped (rules, built-ins, policies), and by
+	// chain.
+	Drops        int64   `json:"drops"`
+	DropsByChain []Entry `json:"drops_by_chain"`
+	// Rules are the busiest rules, built-ins and policies, by packets.
+	Rules []Entry `json:"rules"`
+	// Connections: the most tracked at once (per-bucket averages), and
+	// the table's size.
+	PeakConnections int64   `json:"peak_connections"`
+	ConntrackMax    int64   `json:"conntrack_max"`
+	BlockedSources  []Entry `json:"blocked_sources"`
+	BlockedPorts    []Entry `json:"blocked_ports"`
+	DevicesUp       []Entry `json:"devices_up"`
+	DevicesDown     []Entry `json:"devices_down"`
+}
+
+// Summary sums up the last since, with top lists of up to topLimit.
+func (q *Querier) Summary(ctx context.Context, since time.Duration, topLimit int) (*Summary, error) {
+	now := q.Now()
+	res := q.Resolution(since)
+	from := q.bucketStart(res, now.Add(-since))
+	all, err := q.Store.sums(ctx, res, from, now.Unix()+1, "")
+	if err != nil {
+		return nil, err
+	}
+	s := &Summary{Since: humanDuration(since), From: time.Unix(from, 0), Zones: []Traffic{}, Interfaces: []Traffic{}}
+	traffic := func(rx, tx string) []Traffic {
+		by := map[string]*Traffic{}
+		for k, v := range all {
+			var name string
+			var in bool
+			if n, ok := strings.CutPrefix(k, rx); ok {
+				name, in = n, true
+			} else if n, ok := strings.CutPrefix(k, tx); ok {
+				name = n
+			} else {
+				continue
+			}
+			t := by[name]
+			if t == nil {
+				t = &Traffic{Name: name}
+				by[name] = t
+			}
+			if in {
+				t.Received += v
+			} else {
+				t.Sent += v
+			}
+		}
+		out := []Traffic{}
+		for _, t := range by {
+			out = append(out, *t)
+		}
+		slices.SortFunc(out, func(a, b Traffic) int {
+			if c := cmp.Compare(b.Received+b.Sent, a.Received+a.Sent); c != 0 {
+				return c
+			}
+			return cmp.Compare(a.Name, b.Name)
+		})
+		return out
+	}
+	s.Zones, s.Interfaces = traffic("zrx:", "ztx:"), traffic("rx:", "tx:")
+	s.Drops = all["drops"]
+	s.DropsByChain = sorted(all, "drop:")
+	s.Rules = sorted(all, "hits:")
+	if len(s.Rules) > topLimit {
+		s.Rules = s.Rules[:topLimit]
+	}
+	if tl, err := q.Timeline(ctx, since, "auto", []string{"g:" + GaugeConntrack, "g:" + GaugeConntrackMax}); err == nil {
+		for _, v := range tl.Series[0].Values {
+			s.PeakConnections = max(s.PeakConnections, v)
+		}
+		for _, v := range tl.Series[1].Values {
+			s.ConntrackMax = max(s.ConntrackMax, v)
+		}
+	}
+	for _, l := range []struct {
+		kind string
+		dst  *[]Entry
+	}{{KindBlockedSource, &s.BlockedSources}, {KindBlockedPort, &s.BlockedPorts}, {KindDeviceUp, &s.DevicesUp}, {KindDeviceDown, &s.DevicesDown}} {
+		es, err := q.Top(ctx, l.kind, since, topLimit)
+		if err != nil {
+			return nil, err
+		}
+		*l.dst = es
+	}
+	return s, nil
+}
+
+// Timeline is a set of series over time.
+type Timeline struct {
+	Step   string    `json:"step"`
+	Times  []int64   `json:"times"` // bucket starts (Unix seconds)
+	Series []*Series `json:"series"`
+}
+
+// Series is one counter over time. For a gauge (g:KEY) the values are
+// averages, and -1 where there is no sample.
+type Series struct {
+	Key    string  `json:"key"`
+	Values []int64 `json:"values"`
+}
+
+var steps = []struct {
+	name string
+	d    time.Duration
+	res  string
+}{
+	{"1m", time.Minute, ResMinute}, {"5m", 5 * time.Minute, ResMinute}, {"15m", 15 * time.Minute, ResMinute},
+	{"1h", time.Hour, ResHour}, {"6h", 6 * time.Hour, ResHour}, {"1d", 24 * time.Hour, ResDay},
+}
+
+// Timeline returns counters over the last since. keys are counter names;
+// a key ending in ":*" means every counter of that group. step is "auto"
+// or one of 1m, 5m, 15m, 1h, 6h, 1d.
+func (q *Querier) Timeline(ctx context.Context, since time.Duration, step string, keys []string) (*Timeline, error) {
+	idx := -1
+	for i, st := range steps {
+		ok := st.res == ResMinute && since <= q.Opts.RetainMinute || st.res == ResHour && since <= q.Opts.RetainHour || st.res == ResDay
+		if step == st.name {
+			if !ok {
+				return nil, fmt.Errorf("step %s needs %s data, which is only kept for %s", st.name, map[string]string{ResMinute: "per-minute", ResHour: "hourly"}[st.res], humanDuration(map[string]time.Duration{ResMinute: q.Opts.RetainMinute, ResHour: q.Opts.RetainHour}[st.res]))
+			}
+			idx = i
+			break
+		}
+		if step == "auto" || step == "" {
+			if ok && since/st.d <= 300 {
+				idx = i
+				break
+			}
+		}
+	}
+	if idx < 0 {
+		if step != "auto" && step != "" {
+			return nil, fmt.Errorf("unknown step %q (use auto, 1m, 5m, 15m, 1h, 6h or 1d)", step)
+		}
+		idx = len(steps) - 1
+	}
+	st := steps[idx]
+	now := q.Now()
+	// Buckets aligned to the step (days to local midnight).
+	var starts []int64
+	if st.res == ResDay {
+		for t := time.Unix(q.bucketStart(ResDay, now.Add(-since)), 0).In(q.loc()); !t.After(now); t = t.AddDate(0, 0, 1) {
+			starts = append(starts, t.Unix())
+		}
+	} else {
+		sd := int64(st.d.Seconds())
+		first := now.Add(-since).Unix()
+		first -= first % sd
+		for t := first; t <= now.Unix(); t += sd {
+			starts = append(starts, t)
+		}
+	}
+	from, to := starts[0], now.Unix()+1
+	// Expand group keys.
+	var expanded []string
+	for _, k := range keys {
+		if g, ok := strings.CutSuffix(k, ":*"); ok {
+			m, err := q.Store.sums(ctx, st.res, from, to, g+":")
+			if err != nil {
+				return nil, err
+			}
+			for _, e := range sorted(m, "") {
+				expanded = append(expanded, e.Key)
+			}
+			continue
+		}
+		expanded = append(expanded, k)
+	}
+	tl := &Timeline{Step: st.name, Times: starts, Series: []*Series{}}
+	bucketed := func(key string) ([]int64, error) {
+		raw, err := q.Store.series(ctx, st.res, from, to, key)
+		if err != nil {
+			return nil, err
+		}
+		vals := make([]int64, len(starts))
+		for ts, v := range raw {
+			// The bucket is the last start <= ts.
+			i, found := slices.BinarySearch(starts, ts)
+			if !found {
+				i--
+			}
+			if i >= 0 && i < len(vals) {
+				vals[i] += v
+			}
+		}
+		return vals, nil
+	}
+	for _, k := range expanded {
+		vals, err := bucketed(k)
+		if err != nil {
+			return nil, err
+		}
+		// A gauge (g:KEY) is the average of its samples: its sum divided
+		// by gn:KEY, their number. A bucket without samples is -1.
+		if g, ok := strings.CutPrefix(k, "g:"); ok {
+			n, err := bucketed("gn:" + g)
+			if err != nil {
+				return nil, err
+			}
+			for i := range vals {
+				if n[i] > 0 {
+					vals[i] /= n[i]
+				} else {
+					vals[i] = -1
+				}
+			}
+		}
+		tl.Series = append(tl.Series, &Series{Key: k, Values: vals})
+	}
+	return tl, nil
+}
+
+// Heatmap is average activity per weekday and hour of day (local time).
+type Heatmap struct {
+	Key string `json:"key"`
+	// Avg[d][h] is the average per hour on weekday d (0 = Monday) at
+	// hour h; -1 where there is no data.
+	Avg [7][24]float64 `json:"avg"`
+	// Samples[d][h] is how many hours the average is based on.
+	Samples [7][24]int `json:"samples"`
+	From    time.Time  `json:"from"`
+}
+
+// Heatmap averages a counter (default "drops") by weekday and hour over
+// the last since (at most the hourly retention).
+func (q *Querier) Heatmap(ctx context.Context, key string, since time.Duration) (*Heatmap, error) {
+	if key == "" {
+		key = "drops"
+	}
+	since = min(since, q.Opts.RetainHour)
+	now := q.Now()
+	from := q.bucketStart(ResHour, now.Add(-since))
+	raw, err := q.Store.series(ctx, ResHour, from, now.Unix()+1, key)
+	if err != nil {
+		return nil, err
+	}
+	hm := &Heatmap{Key: key, From: time.Unix(from, 0)}
+	var sum [7][24]int64
+	for ts, v := range raw {
+		t := time.Unix(ts, 0).In(q.loc())
+		d := (int(t.Weekday()) + 6) % 7
+		sum[d][t.Hour()] += v
+		hm.Samples[d][t.Hour()]++
+	}
+	for d := range 7 {
+		for h := range 24 {
+			if n := hm.Samples[d][h]; n > 0 {
+				hm.Avg[d][h] = float64(sum[d][h]) / float64(n)
+			} else {
+				hm.Avg[d][h] = -1
+			}
+		}
+	}
+	return hm, nil
+}
+
+// ParseSince parses a range such as 90m, 24h, 7d or 2w.
+func ParseSince(s string) (time.Duration, error) {
+	if s == "" {
+		return 24 * time.Hour, nil
+	}
+	mult := time.Duration(0)
+	switch {
+	case strings.HasSuffix(s, "d"):
+		mult = 24 * time.Hour
+	case strings.HasSuffix(s, "w"):
+		mult = 7 * 24 * time.Hour
+	}
+	if mult > 0 {
+		var n int
+		if _, err := fmt.Sscanf(s[:len(s)-1], "%d", &n); err != nil || n <= 0 || fmt.Sprint(n) != s[:len(s)-1] {
+			return 0, fmt.Errorf("invalid range %q (use e.g. 90m, 24h, 7d, 2w)", s)
+		}
+		return time.Duration(n) * mult, nil
+	}
+	d, err := time.ParseDuration(s)
+	if err != nil || d <= 0 {
+		return 0, fmt.Errorf("invalid range %q (use e.g. 90m, 24h, 7d, 2w)", s)
+	}
+	return d, nil
+}
+
+func humanDuration(d time.Duration) string {
+	if d%(24*time.Hour) == 0 && d > 0 {
+		return strconv.Itoa(int(d/(24*time.Hour))) + "d"
+	}
+	return d.String()
+}

internal/stats/sources.go

@@ -0,0 +1,136 @@
+package stats
+
+import (
+	"bytes"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+)
+
+// What the collector reads from the kernel without privileges: the
+// interfaces' counters (/sys/class/net), the connection tracking table's
+// size, and housecarld's and the host's memory and load (/proc).
+
+// clockTicks is USER_HZ, the unit of the CPU times in /proc/PID/stat. It
+// is 100 on every Linux architecture Housecarl runs on.
+const clockTicks = 100
+
+// ifCounters are an interface's totals since it came up.
+type ifCounters struct {
+	RxBytes, TxBytes, RxPackets, TxPackets int64
+}
+
+// readInterfaces reads every interface's counters but the loopback's.
+func readInterfaces(sysNet string) map[string]ifCounters {
+	out := map[string]ifCounters{}
+	entries, err := os.ReadDir(sysNet)
+	if err != nil {
+		return out
+	}
+	for _, e := range entries {
+		name := e.Name()
+		if name == "lo" {
+			continue
+		}
+		dir := filepath.Join(sysNet, name, "statistics")
+		read := func(f string) (int64, bool) {
+			b, err := os.ReadFile(filepath.Join(dir, f))
+			if err != nil {
+				return 0, false
+			}
+			n, err := strconv.ParseInt(strings.TrimSpace(string(b)), 10, 64)
+			return n, err == nil
+		}
+		var c ifCounters
+		var ok bool
+		if c.RxBytes, ok = read("rx_bytes"); !ok {
+			continue
+		}
+		c.TxBytes, _ = read("tx_bytes")
+		c.RxPackets, _ = read("rx_packets")
+		c.TxPackets, _ = read("tx_packets")
+		out[name] = c
+	}
+	return out
+}
+
+// readConntrack returns how many connections the kernel tracks, and how
+// many it can (ok false without connection tracking).
+func readConntrack(proc string) (count, max int64, ok bool) {
+	read := func(f string) (int64, bool) {
+		b, err := os.ReadFile(filepath.Join(proc, "sys/net/netfilter", f))
+		if err != nil {
+			return 0, false
+		}
+		n, err := strconv.ParseInt(strings.TrimSpace(string(b)), 10, 64)
+		return n, err == nil
+	}
+	count, ok1 := read("nf_conntrack_count")
+	max, ok2 := read("nf_conntrack_max")
+	return count, max, ok1 && ok2
+}
+
+// readSelf returns housecarld's resident memory (bytes) and CPU time
+// (clock ticks).
+func readSelf(proc string) (rss, cpu int64, ok bool) {
+	dir := filepath.Join(proc, "self")
+	stat, err := os.ReadFile(filepath.Join(dir, "stat"))
+	if err != nil {
+		return 0, 0, false
+	}
+	// The command name is in parentheses and may contain spaces: the
+	// fields that follow start after the last ')'.
+	i := bytes.LastIndexByte(stat, ')')
+	if i < 0 {
+		return 0, 0, false
+	}
+	f := strings.Fields(string(stat[i+1:]))
+	// f[0] is field 3 (state); utime and stime are fields 14 and 15.
+	if len(f) < 13 {
+		return 0, 0, false
+	}
+	utime, _ := strconv.ParseInt(f[11], 10, 64)
+	stime, _ := strconv.ParseInt(f[12], 10, 64)
+	status, err := os.ReadFile(filepath.Join(dir, "status"))
+	if err != nil {
+		return 0, 0, false
+	}
+	for _, line := range strings.Split(string(status), "\n") {
+		if k, v, ok := strings.Cut(line, ":"); ok && k == "VmRSS" {
+			rss = kib(v)
+		}
+	}
+	return rss, utime + stime, true
+}
+
+// readHost returns the memory in use (bytes) and the 1-minute load
+// average.
+func readHost(proc string) (memUsed int64, load float64, ok bool) {
+	b, err := os.ReadFile(filepath.Join(proc, "meminfo"))
+	if err != nil {
+		return 0, 0, false
+	}
+	var total, avail int64
+	for _, line := range strings.Split(string(b), "\n") {
+		k, v, _ := strings.Cut(line, ":")
+		switch k {
+		case "MemTotal":
+			total = kib(v)
+		case "MemAvailable":
+			avail = kib(v)
+		}
+	}
+	if b, err := os.ReadFile(filepath.Join(proc, "loadavg")); err == nil {
+		if f := strings.Fields(string(b)); len(f) > 0 {
+			load, _ = strconv.ParseFloat(f[0], 64)
+		}
+	}
+	return total - avail, load, total > 0
+}
+
+// kib parses a "1234 kB" value from /proc into bytes.
+func kib(v string) int64 {
+	n, _ := strconv.ParseInt(strings.TrimSpace(strings.TrimSuffix(strings.TrimSpace(v), "kB")), 10, 64)
+	return n * 1024
+}

internal/stats/store.go

@@ -0,0 +1,304 @@
+// Package stats keeps the firewall's figures over time: traffic per
+// interface and zone, hits per rule, drops, blocked sources, connections,
+// quotas and the machine's health, sampled every minute. It stores
+// aggregates in a SQLite database of its own, separate from Housecarl's
+// configuration, so it can be deleted at any time without losing anything
+// important.
+package stats
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"fmt"
+	"os"
+
+	_ "modernc.org/sqlite" // registers the "sqlite" driver
+)
+
+// Resolutions of stored data.
+const (
+	ResMinute = "m"
+	ResHour   = "h"
+	ResDay    = "d"
+)
+
+// Store is the statistics database.
+type Store struct {
+	db   *sql.DB
+	path string
+}
+
+var schema = []string{
+	`CREATE TABLE counters (
+		res   TEXT NOT NULL,
+		ts    INTEGER NOT NULL,
+		key   TEXT NOT NULL,
+		value INTEGER NOT NULL,
+		PRIMARY KEY (res, ts, key)
+	) WITHOUT ROWID;
+	CREATE INDEX counters_by_key ON counters (res, key, ts);
+	CREATE TABLE tops (
+		res   TEXT NOT NULL,
+		ts    INTEGER NOT NULL,
+		kind  TEXT NOT NULL,
+		key   TEXT NOT NULL,
+		count INTEGER NOT NULL,
+		PRIMARY KEY (res, ts, kind, key)
+	) WITHOUT ROWID;
+	CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);`,
+}
+
+// Size is the database's size on disk, with its write-ahead log.
+func (s *Store) Size() int64 {
+	var n int64
+	for _, f := range []string{s.path, s.path + "-wal"} {
+		if fi, err := os.Stat(f); err == nil {
+			n += fi.Size()
+		}
+	}
+	return n
+}
+
+// Oldest is the start of the oldest stored bucket (0 when empty).
+func (s *Store) Oldest(ctx context.Context) (int64, error) {
+	var ts sql.NullInt64
+	err := s.db.QueryRowContext(ctx, `SELECT MIN(ts) FROM counters`).Scan(&ts)
+	return ts.Int64, err
+}
+
+// OpenStore opens (creating if needed) the statistics database. Use
+// ":memory:" for tests.
+func OpenStore(path string) (*Store, error) {
+	dsn := "file:" + path + "?_pragma=busy_timeout(10000)&_pragma=synchronous(NORMAL)&_txlock=immediate"
+	if path == ":memory:" {
+		dsn = "file::memory:?_txlock=immediate"
+	} else {
+		dsn += "&_pragma=journal_mode(WAL)"
+	}
+	db, err := sql.Open("sqlite", dsn)
+	if err != nil {
+		return nil, err
+	}
+	if path == ":memory:" {
+		db.SetMaxOpenConns(1)
+	}
+	s := &Store{db: db, path: path}
+	if err := s.migrate(); err != nil {
+		db.Close()
+		return nil, fmt.Errorf("statistics database %s: %w", path, err)
+	}
+	return s, nil
+}
+
+// Path returns the database file ("" in memory).
+func (s *Store) Path() string {
+	if s.path == ":memory:" {
+		return ""
+	}
+	return s.path
+}
+
+// Close closes the database.
+func (s *Store) Close() error { return s.db.Close() }
+
+func (s *Store) migrate() error {
+	if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS schema_version (version INTEGER NOT NULL)`); err != nil {
+		return err
+	}
+	var v int
+	err := s.db.QueryRow(`SELECT version FROM schema_version`).Scan(&v)
+	if errors.Is(err, sql.ErrNoRows) {
+		if _, err := s.db.Exec(`INSERT INTO schema_version (version) VALUES (0)`); err != nil {
+			return err
+		}
+	} else if err != nil {
+		return err
+	}
+	if v > len(schema) {
+		return fmt.Errorf("schema version %d is newer than this Housecarl; upgrade Housecarl or delete the statistics database", v)
+	}
+	for i := v; i < len(schema); i++ {
+		tx, err := s.db.Begin()
+		if err != nil {
+			return err
+		}
+		if _, err := tx.Exec(schema[i]); err != nil {
+			tx.Rollback()
+			return err
+		}
+		if _, err := tx.Exec(`UPDATE schema_version SET version = ?`, i+1); err != nil {
+			tx.Rollback()
+			return err
+		}
+		if err := tx.Commit(); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// Batch collects writes that are committed together.
+type Batch struct {
+	counters []counterRow
+	tops     []topsWrite
+	meta     map[string]string
+}
+
+type counterRow struct {
+	res   string
+	ts    int64
+	key   string
+	value int64
+}
+
+type topsWrite struct {
+	res, kind string
+	ts        int64
+	entries   []Entry
+	replace   bool // replace the (res, ts, kind) rows instead of adding
+	keep      int  // after adding, keep only the largest entries (0: all)
+}
+
+// Entry is a key and its count.
+type Entry struct {
+	Key   string `json:"key"`
+	Count int64  `json:"count"`
+	// Name is what the key stands for (a rule's label, a device's host
+	// name), set by the API.
+	Name string `json:"name,omitempty"`
+}
+
+// AddCounters adds values to the counters of a bucket.
+func (b *Batch) AddCounters(res string, ts int64, values map[string]int64) {
+	for k, v := range values {
+		b.counters = append(b.counters, counterRow{res, ts, k, v})
+	}
+}
+
+// SetTops replaces a bucket's top list of a kind.
+func (b *Batch) SetTops(res string, ts int64, kind string, entries []Entry) {
+	b.tops = append(b.tops, topsWrite{res: res, ts: ts, kind: kind, entries: entries, replace: true})
+}
+
+// AddTops adds counts to a bucket's top list and keeps the keep largest.
+func (b *Batch) AddTops(res string, ts int64, kind string, entries []Entry, keep int) {
+	b.tops = append(b.tops, topsWrite{res: res, ts: ts, kind: kind, entries: entries, keep: keep})
+}
+
+// SetMeta stores a value that survives restarts.
+func (b *Batch) SetMeta(key, value string) {
+	if b.meta == nil {
+		b.meta = map[string]string{}
+	}
+	b.meta[key] = value
+}
+
+// Commit writes the batch in one transaction.
+func (s *Store) Commit(ctx context.Context, b *Batch) error {
+	tx, err := s.db.BeginTx(ctx, nil)
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	if len(b.counters) > 0 {
+		st, err := tx.PrepareContext(ctx, `INSERT INTO counters (res, ts, key, value) VALUES (?, ?, ?, ?)
+			ON CONFLICT (res, ts, key) DO UPDATE SET value = value + excluded.value`)
+		if err != nil {
+			return err
+		}
+		for _, c := range b.counters {
+			if _, err := st.ExecContext(ctx, c.res, c.ts, c.key, c.value); err != nil {
+				st.Close()
+				return err
+			}
+		}
+		st.Close()
+	}
+	for _, t := range b.tops {
+		if t.replace {
+			if _, err := tx.ExecContext(ctx, `DELETE FROM tops WHERE res = ? AND ts = ? AND kind = ?`, t.res, t.ts, t.kind); err != nil {
+				return err
+			}
+		}
+		for _, e := range t.entries {
+			if _, err := tx.ExecContext(ctx, `INSERT INTO tops (res, ts, kind, key, count) VALUES (?, ?, ?, ?, ?)
+				ON CONFLICT (res, ts, kind, key) DO UPDATE SET count = count + excluded.count`, t.res, t.ts, t.kind, e.Key, e.Count); err != nil {
+				return err
+			}
+		}
+		if t.keep > 0 {
+			if _, err := tx.ExecContext(ctx, `DELETE FROM tops WHERE res = ? AND ts = ? AND kind = ? AND key NOT IN (
+				SELECT key FROM tops WHERE res = ? AND ts = ? AND kind = ? ORDER BY count DESC, key LIMIT ?)`,
+				t.res, t.ts, t.kind, t.res, t.ts, t.kind, t.keep); err != nil {
+				return err
+			}
+		}
+	}
+	for k, v := range b.meta {
+		if _, err := tx.ExecContext(ctx, `INSERT INTO meta (key, value) VALUES (?, ?) ON CONFLICT (key) DO UPDATE SET value = excluded.value`, k, v); err != nil {
+			return err
+		}
+	}
+	return tx.Commit()
+}
+
+// Meta returns a stored value ("" if unset).
+func (s *Store) Meta(ctx context.Context, key string) (string, error) {
+	var v string
+	err := s.db.QueryRowContext(ctx, `SELECT value FROM meta WHERE key = ?`, key).Scan(&v)
+	if errors.Is(err, sql.ErrNoRows) {
+		return "", nil
+	}
+	return v, err
+}
+
+// Prune deletes data of a resolution older than before (a Unix time).
+func (s *Store) Prune(ctx context.Context, res string, before int64) (int64, error) {
+	var n int64
+	for _, table := range []string{"counters", "tops"} {
+		r, err := s.db.ExecContext(ctx, `DELETE FROM `+table+` WHERE res = ? AND ts < ?`, res, before)
+		if err != nil {
+			return n, err
+		}
+		c, _ := r.RowsAffected()
+		n += c
+	}
+	return n, nil
+}
+
+// TopsAt returns a bucket's top list of a kind.
+func (s *Store) TopsAt(ctx context.Context, res string, ts int64, kind string) ([]Entry, error) {
+	rows, err := s.db.QueryContext(ctx, `SELECT key, count FROM tops WHERE res = ? AND ts = ? AND kind = ? ORDER BY count DESC, key`, res, ts, kind)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []Entry
+	for rows.Next() {
+		var e Entry
+		if err := rows.Scan(&e.Key, &e.Count); err != nil {
+			return nil, err
+		}
+		out = append(out, e)
+	}
+	return out, rows.Err()
+}
+
+// KindsAt returns the top-list kinds stored for one bucket.
+func (s *Store) KindsAt(ctx context.Context, res string, ts int64) ([]string, error) {
+	rows, err := s.db.QueryContext(ctx, `SELECT DISTINCT kind FROM tops WHERE res = ? AND ts = ?`, res, ts)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []string
+	for rows.Next() {
+		var k string
+		if err := rows.Scan(&k); err != nil {
+			return nil, err
+		}
+		out = append(out, k)
+	}
+	return out, rows.Err()
+}

internal/tablewatch/tablewatch.go

@@ -0,0 +1,100 @@
+// Package tablewatch tells when an nftables table is deleted: by
+// `nft flush ruleset` (which a ruleset file such as /etc/nftables.conf
+// usually starts with, and which nftables.service runs when it stops), or
+// by hand. It listens to nftables' netlink notifications, as `nft monitor`
+// does, and calls back once they have been quiet for a moment, since a
+// ruleset reload brings many. In shadow mode housecarld then puts its
+// table back (docs/shadow.md).
+package tablewatch
+
+import (
+	"context"
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"os"
+	"time"
+
+	"golang.org/x/sys/unix"
+)
+
+// Settle is how long the notifications must be quiet before the callback.
+const Settle = 2 * time.Second
+
+// Watch calls deleted after a table was deleted, until ctx ends. It
+// returns an error only if it can't listen at all.
+func Watch(ctx context.Context, deleted func()) error {
+	fd, err := unix.Socket(unix.AF_NETLINK, unix.SOCK_RAW|unix.SOCK_CLOEXEC|unix.SOCK_NONBLOCK, unix.NETLINK_NETFILTER)
+	if err != nil {
+		return fmt.Errorf("opening a netlink socket: %w", err)
+	}
+	if err := unix.Bind(fd, &unix.SockaddrNetlink{Family: unix.AF_NETLINK, Groups: 1 << (unix.NFNLGRP_NFTABLES - 1)}); err != nil {
+		unix.Close(fd)
+		return fmt.Errorf("listening to nftables' changes: %w", err)
+	}
+	// A non-blocking descriptor goes through Go's poller, so closing the
+	// file ends a Read in progress.
+	f := os.NewFile(uintptr(fd), "nfnetlink")
+	go func() {
+		<-ctx.Done()
+		f.Close()
+	}()
+	events := make(chan struct{}, 1)
+	go func() {
+		buf := make([]byte, 256<<10)
+		for {
+			n, err := f.Read(buf)
+			switch {
+			case err != nil && ctx.Err() != nil:
+				close(events)
+				return
+			case errors.Is(err, unix.ENOBUFS):
+				// Notifications were lost: one may have been a deletion.
+			case err != nil:
+				time.Sleep(time.Second)
+				continue
+			case !DeletesTable(buf[:n]):
+				continue
+			}
+			select {
+			case events <- struct{}{}:
+			default:
+			}
+		}
+	}()
+	var timer <-chan time.Time
+	for {
+		select {
+		case _, ok := <-events:
+			if !ok {
+				return nil
+			}
+			timer = time.After(Settle)
+		case <-timer:
+			timer = nil
+			deleted()
+		}
+	}
+}
+
+// DeletesTable reports whether a netlink datagram holds a table being
+// deleted.
+func DeletesTable(b []byte) bool {
+	want := uint16(unix.NFNL_SUBSYS_NFTABLES<<8 | unix.NFT_MSG_DELTABLE)
+	for len(b) >= unix.NLMSG_HDRLEN {
+		size := int(binary.NativeEndian.Uint32(b[0:4]))
+		typ := binary.NativeEndian.Uint16(b[4:6])
+		if size < unix.NLMSG_HDRLEN || size > len(b) {
+			return false
+		}
+		if typ == want {
+			return true
+		}
+		size = (size + unix.NLMSG_ALIGNTO - 1) &^ (unix.NLMSG_ALIGNTO - 1)
+		if size >= len(b) {
+			return false
+		}
+		b = b[size:]
+	}
+	return false
+}

internal/tablewatch/tablewatch_test.go

@@ -0,0 +1,37 @@
+package tablewatch
+
+import (
+	"encoding/binary"
+	"testing"
+
+	"golang.org/x/sys/unix"
+)
+
+// msg is a netlink message of a type, with a few bytes of payload.
+func msg(typ uint16) []byte {
+	b := make([]byte, unix.NLMSG_HDRLEN+4)
+	binary.NativeEndian.PutUint32(b[0:4], uint32(len(b)))
+	binary.NativeEndian.PutUint16(b[4:6], typ)
+	return b
+}
+
+func TestDeletesTable(t *testing.T) {
+	nft := func(m uint16) uint16 { return unix.NFNL_SUBSYS_NFTABLES<<8 | m }
+	newRule, delTable, delChain := msg(nft(unix.NFT_MSG_NEWRULE)), msg(nft(unix.NFT_MSG_DELTABLE)), msg(nft(unix.NFT_MSG_DELCHAIN))
+	cases := []struct {
+		b    []byte
+		want bool
+	}{
+		{delTable, true},
+		{append(append([]byte{}, newRule...), delTable...), true},
+		{append(append([]byte{}, newRule...), delChain...), false},
+		{newRule, false},
+		{delTable[:10], false},
+		{nil, false},
+	}
+	for i, c := range cases {
+		if got := DeletesTable(c.b); got != c.want {
+			t.Errorf("%d: %v", i, got)
+		}
+	}
+}

internal/trace/trace.go

@@ -0,0 +1,271 @@
+// Package trace follows packets through the whole ruleset, every table's
+// chains, with nftables' own tracing: a table of its own (housecarl-trace,
+// hooked before everything else) marks the packets a filter picks
+// (meta nftrace set 1), and `nft monitor trace` reports each chain and
+// rule they pass, and what decided their fate. Packets are counted per
+// trace id: one packet through the hooks it crosses.
+package trace
+
+import (
+	"fmt"
+	"net/netip"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+
+	"housecarl/internal/api"
+)
+
+// Table is the tracing table's name (family inet).
+const Table = "housecarl-trace"
+
+// Script builds the tracing table for a filter: packets from or to the
+// address, on the port (either way), of the protocol. A limit keeps a
+// flood from swamping the trace.
+func Script(in api.TraceInput) (string, error) {
+	var addr netip.Prefix
+	if in.Address != "" {
+		p, err := netip.ParsePrefix(in.Address)
+		if err != nil {
+			a, err2 := netip.ParseAddr(in.Address)
+			if err2 != nil {
+				return "", fmt.Errorf("address %q: give an address or a prefix (192.168.1.50, 2001:db8::/64)", in.Address)
+			}
+			p = netip.PrefixFrom(a, a.BitLen())
+		}
+		addr = p.Masked()
+	}
+	if in.Port < 0 || in.Port > 65535 {
+		return "", fmt.Errorf("port %d: 1 to 65535", in.Port)
+	}
+	proto := strings.ToLower(in.Protocol)
+	switch proto {
+	case "", "tcp", "udp", "sctp", "icmp", "icmpv6":
+	default:
+		return "", fmt.Errorf("protocol %q: tcp, udp, sctp, icmp, icmpv6, or nothing for any", in.Protocol)
+	}
+	if !addr.IsValid() && in.Port == 0 {
+		return "", fmt.Errorf("say what to trace: an address, a port, or both")
+	}
+	if in.Port != 0 && (proto == "icmp" || proto == "icmpv6") {
+		return "", fmt.Errorf("ICMP has no ports")
+	}
+	var l4 string
+	switch {
+	case proto != "":
+		l4 = "meta l4proto " + proto
+	case in.Port != 0:
+		l4 = "meta l4proto { tcp, udp, sctp }"
+	}
+	addrs := []string{""}
+	if addr.IsValid() {
+		ip := "ip"
+		if addr.Addr().Is6() {
+			ip = "ip6"
+		}
+		addrs = []string{ip + " saddr " + addr.String(), ip + " daddr " + addr.String()}
+	}
+	ports := []string{""}
+	if in.Port != 0 {
+		ports = []string{fmt.Sprintf("th dport %d", in.Port), fmt.Sprintf("th sport %d", in.Port)}
+	}
+	var rules []string
+	for _, a := range addrs {
+		for _, p := range ports {
+			parts := []string{}
+			for _, x := range []string{a, l4, p, "limit rate 20/second", "meta nftrace set 1"} {
+				if x != "" {
+					parts = append(parts, x)
+				}
+			}
+			rules = append(rules, "\t\t"+strings.Join(parts, " "))
+		}
+	}
+	body := strings.Join(rules, "\n")
+	return fmt.Sprintf(`table inet %[1]s {}
+delete table inet %[1]s
+table inet %[1]s {
+	chain pre {
+		type filter hook prerouting priority -301; policy accept;
+%[2]s
+	}
+	chain out {
+		type filter hook output priority -301; policy accept;
+%[2]s
+	}
+}
+`, Table, body), nil
+}
+
+// ---- reading `nft monitor trace`
+
+// "trace id 38ff3662 inet housecarl input rule … (verdict accept)"
+var lineRe = regexp.MustCompile(`^trace id (\S+) (\S+) (\S+) (\S+) (packet: (.*)|rule (.*) \(verdict ([^)]*)\)|policy (\S+)|(.*))$`)
+
+var (
+	fieldRe = regexp.MustCompile(`\b(iif|oif) "([^"]*)"`)
+	addrRe  = regexp.MustCompile(`\b(ip6?) (saddr|daddr) (\S+)`)
+	portRe  = regexp.MustCompile(`\b(tcp|udp|sctp) (sport|dport) (\d+)`)
+	flagsRe = regexp.MustCompile(`\btcp flags == (\S+)`)
+	icmpRe  = regexp.MustCompile(`\b(icmp|icmpv6) type (\S+)`)
+	protoRe = regexp.MustCompile(`\bmeta l4proto (\S+)|\bip protocol (\S+)|\bip6 nexthdr (\S+)`)
+	markRe  = regexp.MustCompile(`comment "hc:(\S+)`)
+)
+
+// Tracer collects packets from trace lines.
+type Tracer struct {
+	// Max is how many packets it keeps (0: 200).
+	Max     int
+	Now     func() time.Time
+	packets []*api.TracePacket
+	byID    map[string]*api.TracePacket
+	// seen are the chains a packet entered: the kernel reuses trace ids
+	// (they come from the packet's buffer), so entering one again means
+	// another packet.
+	seen map[*api.TracePacket]map[string]bool
+}
+
+// Packets returns what was traced so far, oldest first.
+func (t *Tracer) Packets() []api.TracePacket {
+	out := make([]api.TracePacket, 0, len(t.packets))
+	for _, p := range t.packets {
+		out = append(out, *p)
+	}
+	return out
+}
+
+// Full reports whether it keeps no more packets.
+func (t *Tracer) Full() bool {
+	max := t.Max
+	if max == 0 {
+		max = 200
+	}
+	return len(t.packets) >= max
+}
+
+// Line reads one line of `nft monitor trace`. The tracing table's own
+// steps are left out.
+func (t *Tracer) Line(line string) {
+	m := lineRe.FindStringSubmatch(strings.TrimSpace(line))
+	if m == nil {
+		return
+	}
+	id, table, chain := m[1], m[2]+" "+m[3], m[4]
+	if m[3] == Table {
+		return
+	}
+	if t.byID == nil {
+		t.byID, t.seen = map[string]*api.TracePacket{}, map[*api.TracePacket]map[string]bool{}
+	}
+	p := t.byID[id]
+	isPacket := m[6] != "" || strings.HasPrefix(m[5], "packet:")
+	if p != nil && isPacket && t.seen[p][table+" "+chain] {
+		p = nil // the same id, another packet
+	}
+	if p == nil {
+		if t.Full() {
+			return
+		}
+		now := time.Now
+		if t.Now != nil {
+			now = t.Now
+		}
+		p = &api.TracePacket{ID: id, Time: now(), Verdict: "accepted"}
+		t.byID[id] = p
+		t.seen[p] = map[string]bool{}
+		t.packets = append(t.packets, p)
+	}
+	switch {
+	case isPacket:
+		t.seen[p][table+" "+chain] = true
+		readPacket(p, m[6])
+	case m[7] != "":
+		step := api.TraceStep{Table: table, Chain: chain, Kind: "rule", Rule: m[7], Verdict: m[8]}
+		if mk := markRe.FindStringSubmatch(m[7]); mk != nil {
+			step.Mark = strings.TrimSuffix(mk[1], `"`)
+		}
+		p.Steps = append(p.Steps, step)
+		decide(p, step)
+	case m[9] != "":
+		step := api.TraceStep{Table: table, Chain: chain, Kind: "policy", Verdict: m[9]}
+		p.Steps = append(p.Steps, step)
+		decide(p, step)
+	}
+}
+
+// wouldRe finds shadow mode's marker in a rule's comment (render.Would):
+// "[would drop]", "[would reject]".
+var wouldRe = regexp.MustCompile(`"hc:\S+ \[would (drop|reject)\]|"\[would (drop|reject)\]`)
+
+// decide notes a packet's fate: the first drop (or reject) ends it. In
+// shadow mode Housecarl's rules accept what they would drop: that is
+// noted apart, and decides nothing.
+func decide(p *api.TracePacket, s api.TraceStep) {
+	if s.Kind == "rule" && wouldRe.MatchString(s.Rule) {
+		if p.WouldDrop == nil && p.Verdict != "dropped" {
+			step := s
+			p.WouldDrop = &step
+		}
+		return
+	}
+	if p.Verdict == "dropped" {
+		return
+	}
+	switch strings.Fields(s.Verdict + " ")[0] {
+	case "drop", "reject":
+		p.Verdict, p.DecidedBy = "dropped", s
+	case "accept":
+		// What let it through is a filter chain's accept: Housecarl's
+		// input, forward or output (NAT chains accept everything they
+		// see). Another table's accept counts when Housecarl's says
+		// nothing.
+		filter := s.Table == "inet housecarl" && (s.Chain == "input" || s.Chain == "forward" || s.Chain == "output")
+		if filter || p.DecidedBy.Verdict == "" && s.Table != "inet housecarl" {
+			p.DecidedBy = s
+		}
+	}
+}
+
+// readPacket fills in a packet's headers from its "packet:" line (once:
+// the first hook's view, before any NAT).
+func readPacket(p *api.TracePacket, f string) {
+	for _, m := range fieldRe.FindAllStringSubmatch(f, -1) {
+		if m[1] == "iif" && p.In == "" {
+			p.In = m[2]
+		}
+		if m[1] == "oif" && p.Out == "" {
+			p.Out = m[2]
+		}
+	}
+	if p.Source != "" {
+		return
+	}
+	for _, m := range addrRe.FindAllStringSubmatch(f, -1) {
+		if m[2] == "saddr" {
+			p.Source = m[3]
+		} else {
+			p.Destination = m[3]
+		}
+	}
+	for _, m := range portRe.FindAllStringSubmatch(f, -1) {
+		p.Protocol = m[1]
+		n, _ := strconv.Atoi(m[3])
+		if m[2] == "sport" {
+			p.SourcePort = n
+		} else {
+			p.DestinationPort = n
+		}
+	}
+	if m := flagsRe.FindStringSubmatch(f); m != nil {
+		p.TCPFlags = m[1]
+	}
+	if m := icmpRe.FindStringSubmatch(f); m != nil {
+		p.Protocol, p.ICMPType = m[1], m[2]
+	}
+	if p.Protocol == "" {
+		if m := protoRe.FindStringSubmatch(f); m != nil {
+			p.Protocol = strings.TrimSpace(m[1] + m[2] + m[3])
+		}
+	}
+}

internal/trace/trace_test.go

@@ -0,0 +1,120 @@
+package trace
+
+import (
+	"strings"
+	"testing"
+
+	"housecarl/internal/api"
+)
+
+// Lines nft 1.1.3's `nft monitor trace` printed for a connection to a
+// blocked port, and for one that loopback accepted.
+const sample = `trace id 38ff3662 inet housecarl-trace pre packet: iif "wan" ip saddr 198.51.100.7 ip daddr 203.0.113.10 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 44978 ip protocol 6 ip length 60 tcp sport 43168 tcp dport 9999 tcp flags == syn tcp window 65495
+trace id 38ff3662 inet housecarl-trace pre rule ip saddr 198.51.100.7 meta nftrace set 1 (verdict continue)
+trace id 38ff3662 inet housecarl-trace pre policy accept
+trace id 38ff3662 inet housecarl input packet: iif "wan" ip saddr 198.51.100.7 ip daddr 203.0.113.10 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 44978 ip protocol 6 ip length 60 tcp sport 43168 tcp dport 9999 tcp flags == syn tcp window 65495
+trace id 38ff3662 inet housecarl input rule ct state established,related counter packets 0 bytes 0 accept comment "hc:b-established" (verdict continue)
+trace id 38ff3662 inet housecarl input rule tcp dport 9999 counter packets 1 bytes 60 drop comment "hc:r12 block 9999" (verdict drop)
+trace id a232d9a0 inet housecarl output packet: oif "lo" ip saddr 127.0.0.1 ip daddr 127.0.0.1 ip protocol 6 tcp sport 43168 tcp dport 22 tcp flags == syn
+trace id a232d9a0 inet housecarl output policy accept
+trace id a232d9a0 ip other input rule ip saddr 127.0.0.1 accept (verdict accept)
+trace id 77 inet housecarl forward packet: iif "lan" oif "wan" ip saddr 192.168.1.20 ip daddr 198.51.100.10 tcp sport 3500 tcp dport 7000 tcp flags == syn
+trace id 77 inet housecarl forward rule ip saddr @trusted_v4 counter accept comment "hc:r40 trusted" (verdict accept)
+trace id 77 inet housecarl srcnat rule oifname "wan" counter masquerade comment "hc:s91" (verdict accept)`
+
+func TestTracer(t *testing.T) {
+	var tr Tracer
+	for _, l := range strings.Split(sample, "\n") {
+		tr.Line(l)
+	}
+	ps := tr.Packets()
+	if len(ps) != 3 {
+		t.Fatalf("packets: %+v", ps)
+	}
+	p := ps[0]
+	if p.In != "wan" || p.Source != "198.51.100.7" || p.DestinationPort != 9999 || p.Protocol != "tcp" || p.TCPFlags != "syn" {
+		t.Errorf("headers: %+v", p)
+	}
+	if len(p.Steps) != 2 || p.Verdict != "dropped" || p.DecidedBy.Mark != "r12" || p.DecidedBy.Chain != "input" {
+		t.Errorf("steps: %+v", p)
+	}
+	if q := ps[1]; q.Verdict != "accepted" || q.Out != "lo" || len(q.Steps) != 2 || q.DecidedBy.Kind != "policy" {
+		t.Errorf("second: %+v", q)
+	}
+	// Accepted by the forward rule, not by source NAT, which came after.
+	if q := ps[2]; q.Verdict != "accepted" || q.DecidedBy.Mark != "r40" {
+		t.Errorf("third: decided by %+v", q.DecidedBy)
+	}
+}
+
+// The kernel reuses trace ids: a connection's next packet can come with
+// the same one. Entering a chain again starts another packet.
+func TestReusedID(t *testing.T) {
+	var tr Tracer
+	for _, l := range strings.Split(`trace id 5 inet housecarl output packet: oif "lo" ip saddr 127.0.0.1 ip daddr 127.0.0.1 tcp sport 49078 tcp dport 8070 tcp flags == syn
+trace id 5 inet housecarl output policy accept
+trace id 5 inet housecarl srcnat packet: oif "lo" ip saddr 127.0.0.1 ip daddr 127.0.0.1 tcp sport 49078 tcp dport 8070 tcp flags == syn
+trace id 5 inet housecarl srcnat policy accept
+trace id 5 inet housecarl output packet: oif "lo" ip saddr 127.0.0.1 ip daddr 127.0.0.1 tcp sport 49078 tcp dport 8070 tcp flags == ack
+trace id 5 inet housecarl output policy accept`, "\n") {
+		tr.Line(l)
+	}
+	ps := tr.Packets()
+	if len(ps) != 2 || len(ps[0].Steps) != 2 || len(ps[1].Steps) != 1 || ps[1].TCPFlags != "ack" {
+		t.Errorf("packets: %+v", ps)
+	}
+}
+
+func TestScript(t *testing.T) {
+	s, err := Script(api.TraceInput{Address: "192.168.1.50", Port: 443})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, want := range []string{
+		"table inet housecarl-trace {}\ndelete table inet housecarl-trace\n",
+		"type filter hook prerouting priority -301; policy accept;",
+		"ip saddr 192.168.1.50/32 meta l4proto { tcp, udp, sctp } th dport 443 limit rate 20/second meta nftrace set 1",
+		"ip daddr 192.168.1.50/32 meta l4proto { tcp, udp, sctp } th sport 443 limit rate 20/second meta nftrace set 1",
+	} {
+		if !strings.Contains(s, want) {
+			t.Errorf("missing %q in\n%s", want, s)
+		}
+	}
+	if _, err := Script(api.TraceInput{}); err == nil {
+		t.Error("tracing everything")
+	}
+	if _, err := Script(api.TraceInput{Port: 22, Protocol: "icmp"}); err == nil {
+		t.Error("ICMP with a port")
+	}
+}
+
+// In shadow mode Housecarl's rules accept what they would drop: the
+// packet's fate is the other table's, and the would-be drop is noted.
+func TestShadowTrace(t *testing.T) {
+	var tr Tracer
+	for _, l := range strings.Split(`trace id 9 ip main input packet: iif "wan" ip saddr 198.51.100.7 ip daddr 203.0.113.10 tcp sport 4000 tcp dport 9999 tcp flags == syn
+trace id 9 ip main input rule tcp dport 9999 accept (verdict accept)
+trace id 9 inet housecarl input rule tcp dport 9999 counter packets 1 bytes 60 accept comment "hc:r12 [would drop] block 9999" (verdict accept)
+trace id 10 ip main input packet: iif "wan" ip saddr 198.51.100.7 ip daddr 203.0.113.10 tcp sport 4001 tcp dport 7000 tcp flags == syn
+trace id 10 ip main input rule tcp dport 7000 accept (verdict accept)
+trace id 10 inet housecarl input rule counter packets 1 bytes 60 comment "hc:p-input [would drop]" (verdict continue)
+trace id 10 inet housecarl input policy accept
+trace id 11 ip main input packet: iif "wan" ip saddr 198.51.100.7 ip daddr 203.0.113.10 tcp sport 4002 tcp dport 22 tcp flags == syn
+trace id 11 ip main input rule tcp dport 22 accept (verdict accept)
+trace id 11 inet housecarl input rule tcp dport 22 counter packets 1 bytes 60 accept comment "hc:r3 ssh" (verdict accept)`, "\n") {
+		tr.Line(l)
+	}
+	ps := tr.Packets()
+	if len(ps) != 3 {
+		t.Fatalf("packets: %+v", ps)
+	}
+	if p := ps[0]; p.Verdict != "accepted" || p.WouldDrop == nil || p.WouldDrop.Mark != "r12" || p.DecidedBy.Table != "ip main" {
+		t.Errorf("first: %+v", p)
+	}
+	if p := ps[1]; p.Verdict != "accepted" || p.WouldDrop == nil || p.WouldDrop.Mark != "p-input" {
+		t.Errorf("second: %+v", p)
+	}
+	if p := ps[2]; p.WouldDrop != nil || p.DecidedBy.Mark != "r3" {
+		t.Errorf("third: %+v", p)
+	}
+}

internal/validate/validate.go

@@ -85,6 +85,36 @@ func Check(st *model.State, env *Env) []Problem {
 	return c.out
 }
 
+// rawIgnores names what a raw rule has besides its text, which the
+// ruleset wouldn't use (the action too is the text's; it isn't checked).
+func rawIgnores(r *model.Rule) []string {
+	m := r.Match
+	var out []string
+	for _, x := range []struct {
+		name string
+		set  bool
+	}{
+		{"zones", len(m.InZones) > 0 || len(m.OutZones) > 0},
+		{"addresses", len(m.Sources) > 0 || len(m.Destinations) > 0},
+		{"services", len(m.Services) > 0},
+		{"family", m.Family != ""},
+		{"connection state", len(r.CtState) > 0 || len(r.CtStatus) > 0},
+		{"time window", len(r.Days) > 0 || r.HourFrom != "" || r.HourTo != ""},
+		{"rate limit", r.Limit != nil},
+		{"connection limit", r.ConnLimit != 0},
+		{"quota", r.Quota != nil},
+		{"address grouping", r.GroupV4 != 0 || r.GroupV6 != 0},
+		{"log", r.Log || r.LogPrefix != ""},
+		{"reject with", r.RejectWith != ""},
+		{"every packet", r.EveryPacket},
+	} {
+		if x.set {
+			out = append(out, x.name)
+		}
+	}
+	return out
+}
+
 // maxByteRate is the largest byte rate per period nftables (1.1.3) can
 // hold: the kernel's token bucket multiplies the rate by the period.
 var maxByteRate = map[string]int64{"second": 4294967294, "minute": 307445734, "hour": 5124095, "day": 213503}
@@ -155,6 +185,12 @@ func (c *checker) settings() {
 			}
 		}
 	}
+	if s.StatsPerDevice && len(c.st.ExternalInterfaces()) == 0 {
+		c.add(Warning, "stats_no_external", "settings", "traffic per device counts what goes to and comes from external zones, and no zone is external (with interfaces)")
+	}
+	if s.StatsPerDevice && len(s.FlowtableDevices) > 0 {
+		c.add(Warning, "stats_flowtable", "settings", "the flowtable's fast path skips the forward chain once a connection is established, so traffic per device only counts each connection's first packets; turn the flowtable off for full figures")
+	}
 	if p := s.ProbationSeconds; p != 0 && (p < 10 || p > 3600) {
 		c.add(Error, "probation_range", "settings", "the probation is %d seconds; use 10 to 3600", p)
 	}
@@ -187,8 +223,8 @@ func (c *checker) zones() {
 			}
 		}
 	}
-	if len(c.st.Zones) > 0 && !external {
-		c.add(Warning, "no_external_zone", "zones", "no zone is external: antispoofing, MSS clamping and the DHCPv6 client rule need one, and port forwards without public addresses")
+	if !c.st.Empty() && !external {
+		c.add(Warning, "no_external_zone", "zones", "no zone is external: antispoofing, MSS clamping and the DHCPv6 client rule need one, and port forwards without public addresses; and if the saved ruleset fails to load at boot, the fallback can't tell which interface faces the internet, and lets nothing but loopback through")
 	}
 }
 
@@ -197,11 +233,19 @@ func (c *checker) sets() {
 	for _, s := range c.st.Sets {
 		obj := "set " + s.Name
 		c.name("set", obj, s.Name, seen)
+		for _, pre := range []string{"meter_", "conn_", "quota_", "stats_"} {
+			if strings.HasPrefix(strings.ToLower(s.Name), pre) {
+				c.add(Error, "set_name_reserved", obj, "set names starting with %s are Housecarl's own (rate limits, quotas, statistics); choose another name", pre)
+			}
+		}
 		for _, e := range s.Entries {
 			if _, err := model.ParseAddress(e); err != nil {
 				c.add(Error, "set_entry", obj, "%v", err)
 			}
 		}
+		if len(s.Entries) > 0 && s.Dynamic {
+			c.add(Error, "set_dynamic_entries", obj, "a dynamic set gets its entries at run time (hc set entries add, the Entries button), each with its timeout; leave out the configured ones, or make it a set that isn't dynamic")
+		}
 		if len(s.Entries) == 0 && !s.Dynamic {
 			c.add(Warning, "set_empty", obj, "the set is empty: rules using it match nothing")
 		}
@@ -331,6 +375,9 @@ func (c *checker) rule(r *model.Rule) {
 		if strings.ContainsAny(r.Raw, ";\n\r") {
 			c.add(Error, "raw_rule", obj, "a raw rule is one nftables rule on one line, without ;")
 		}
+		if ignored := rawIgnores(r); len(ignored) > 0 {
+			c.add(Error, "raw_conflict", obj, "a raw rule is used as written, so its %s would be ignored: write them in the nftables rule, or leave them out", strings.Join(ignored, ", "))
+		}
 		return
 	}
 	in := r.Chain != model.ChainOutput
@@ -399,11 +446,14 @@ func (c *checker) rule(r *model.Rule) {
 			c.add(Error, "rule_quota_reset", obj, "the day of the month is 1 to 31 (a later day than a month has means its last day)")
 		}
 	}
-	if r.CountsTraffic() && r.Action == model.ActionAccept {
+	switch {
+	case r.CountsTraffic() && r.Action == model.ActionAccept:
 		c.add(Error, "rule_traffic_accept", obj, "a rule with a quota or a byte rate counts every packet, before established connections are accepted, so it can only drop, reject or log; let the traffic in with a rule of its own")
+	case r.ChecksEveryPacket() && r.Action == model.ActionAccept:
+		c.add(Error, "rule_every_packet_accept", obj, "a rule checked against every packet comes before established connections are accepted, so accepting there would skip every later rule; it can drop, reject or log")
 	}
-	if r.CountsTraffic() && r.Chain == model.ChainForward && len(c.st.Settings.FlowtableDevices) > 0 && !r.Disabled {
-		c.add(Warning, "flowtable_traffic", obj, "the flowtable's fast path skips the forward chain once a connection is established, so this quota or byte rate only counts each connection's first packets; turn the flowtable off (settings) to count all of it")
+	if r.ChecksEveryPacket() && r.Chain == model.ChainForward && len(c.st.Settings.FlowtableDevices) > 0 && !r.Disabled {
+		c.add(Warning, "flowtable_traffic", obj, "the flowtable's fast path skips the forward chain once a connection is established, so this rule only sees each connection's first packets; turn the flowtable off (settings) for it to see all of them")
 	}
 	if r.GroupV4 < 0 || r.GroupV4 > 32 || r.GroupV6 < 0 || r.GroupV6 > 128 {
 		c.add(Error, "rule_group", obj, "group addresses by a prefix length: 1 to 32 for IPv4, 1 to 128 for IPv6")

internal/validate/validate_test.go

@@ -65,7 +65,17 @@ func TestCodes(t *testing.T) {
 		}},
 		{"set_entry", func(st *model.State) { st.Sets[0].Entries = []string{"10.0.10.5/24"} }},
 		{"set_empty", func(st *model.State) { st.Sets[0].Entries = nil }},
+		{"set_name_reserved", func(st *model.State) { st.Sets[0].Name = "stats_up" }},
+		{"stats_flowtable", func(st *model.State) {
+			st.Settings.StatsPerDevice = true
+			st.Settings.FlowtableDevices = []string{"wan", "eth1"}
+		}},
+		{"stats_no_external", func(st *model.State) {
+			st.Settings.StatsPerDevice = true
+			st.Zones[0].External = false
+		}},
 		{"set_timeout", func(st *model.State) { st.Sets[0].Timeout = 60 }},
+		{"set_dynamic_entries", func(st *model.State) { st.Sets[0].Dynamic = true }},
 		{"service_empty", func(st *model.State) { st.Services[0].Entries = nil }},
 		{"service_entry", func(st *model.State) { st.Services[0].Entries = []string{"tcp/99999"} }},
 		{"unknown_zone", func(st *model.State) { st.Rules[1].InZones = []string{"dmz"} }},
@@ -83,6 +93,7 @@ func TestCodes(t *testing.T) {
 		{"rule_chain", func(st *model.State) { rule(st).Chain = "prerouting" }},
 		{"rule_action", func(st *model.State) { rule(st).Action = "allow" }},
 		{"raw_rule", func(st *model.State) { rule(st).Raw = "tcp dport 22 accept; udp dport 53 accept" }},
+		{"raw_conflict", func(st *model.State) { rule(st).Raw = "tcp dport 22 accept" }}, // it still has sources and services
 		{"early_conntrack", func(st *model.State) { rule(st).Chain = model.ChainEarly; rule(st).CtState = []string{"new"} }},
 		{"early_reject", func(st *model.State) { rule(st).Chain = model.ChainEarly; rule(st).Action = model.ActionReject }},
 		{"ct_state", func(st *model.State) { rule(st).CtState = []string{"fresh"} }},
@@ -102,6 +113,7 @@ func TestCodes(t *testing.T) {
 		{"rule_quota_reset", func(st *model.State) { rule(st).Quota = &model.Quota{Bytes: 1, Reset: "daily", Day: 3} }},
 		{"rule_group", func(st *model.State) { rule(st).GroupV6 = 129 }},
 		{"rule_traffic_accept", func(st *model.State) { rule(st).Quota = &model.Quota{Bytes: 1 << 20} }},
+		{"rule_every_packet_accept", func(st *model.State) { rule(st).EveryPacket = true }},
 		{"flowtable_traffic", func(st *model.State) {
 			st.Settings.FlowtableDevices = []string{"wan"}
 			st.Rules[1].Action, st.Rules[1].Quota = model.ActionDrop, &model.Quota{Bytes: 1 << 20, Over: true}
@@ -153,3 +165,29 @@ func TestHostWarnings(t *testing.T) {
 		}
 	}
 }
+
+func TestNoExternalZone(t *testing.T) {
+	st := base()
+	if got := codes(Check(st, nil)); strings.Contains(got, "no_external_zone") {
+		t.Errorf("an external zone, yet: [%s]", got)
+	}
+	st.Zones[0].External = false
+	if got := codes(Check(st, nil)); !strings.Contains(got, "no_external_zone") {
+		t.Errorf("no external zone, and no warning: [%s]", got)
+	}
+	// An empty configuration applies nothing, so has no fallback either.
+	if got := codes(Check(&model.State{Settings: model.DefaultSettings()}, nil)); strings.Contains(got, "no_external_zone") {
+		t.Errorf("empty configuration: [%s]", got)
+	}
+}
+
+func TestVLANInterfaces(t *testing.T) {
+	st := base()
+	st.Zones[1].Interfaces = []string{"enp2s0.10", "enp2s0.4094", "br0.20", "enp3s0.*"}
+	st.Settings.FlowtableDevices = []string{"enp2s0.10"}
+	for _, p := range Check(st, nil) {
+		if p.Severity == Error {
+			t.Errorf("VLAN interfaces refused: %+v", p)
+		}
+	}
+}

internal/web/static/assets/api.js

@@ -22,11 +22,13 @@ export class APIError extends Error {
 let onUnauthorized = () => {};
 export function setUnauthorizedHandler(fn) { onUnauthorized = fn; }
 
-// onChange is told about every applied change (the probation banner).
+// onChange is told about every applied change (the probation banner), with
+// the answer (its confirm_token, to confirm it by itself).
 let onChange = () => {};
 export function setChangeHandler(fn) { onChange = fn; }
 
-// api calls the REST API. opts: {body, query, stage (a changeset ID)}.
+// api calls the REST API. opts: {body, query, stage (a changeset ID),
+// timeout (ms)}.
 export async function api(method, path, opts = {}) {
   const headers = { "Accept": "application/json" };
   if (opts.stage) headers["X-Housecarl-Changeset"] = String(opts.stage);
@@ -43,7 +45,13 @@ export async function api(method, path, opts = {}) {
     const s = q.toString();
     if (s) url += (url.includes("?") ? "&" : "?") + s;
   }
-  const resp = await fetch(url, { method, headers, body, credentials: "same-origin" });
+  let signal;
+  if (opts.timeout) {
+    const ac = new AbortController();
+    setTimeout(() => ac.abort(), opts.timeout);
+    signal = ac.signal;
+  }
+  const resp = await fetch(url, { method, headers, body, credentials: "same-origin", signal });
   if (resp.status === 204) return null;
   const ct = resp.headers.get("Content-Type") || "";
   const data = ct.includes("application/json") ? await resp.json() : await resp.text();
@@ -173,11 +181,12 @@ export function report(r) {
     const n = ((r.changeset || {}).ops || []).length;
     toast("Staged: " + ((r.changeset.ops || [])[n - 1] || {}).summary + " — " + (n === 1 ? "1 pending change" : n + " pending changes") + " to review and apply");
   } else if (r.dry_run) toast("Checked by Housecarl and nftables: nothing was changed");
-  else if (r.applied && r.confirm_by) toast("Applied on probation (history #" + r.history_id + "): confirm it in the banner", "warn", 6000);
+  else if (r.applied && r.confirm_by && !r.confirm_token) toast("Applied on probation (history #" + r.history_id + "): confirm it in the banner", "warn", 6000);
+  else if (r.applied && r.confirm_by) toast("Applied (history #" + r.history_id + ")");
   else if (r.applied) toast("Applied and saved (history #" + r.history_id + ")");
   else if (!(r.changes || []).length) toast("Nothing to change");
   for (const w of r.warnings || []) toast(w.message + (w.object ? " (" + w.object + ")" : ""), "warn", 10000);
-  if (r.applied) onChange();
+  if (r.applied) onChange(r);
 }
 
 // can reports whether the user may read (or write) a resource.

internal/web/static/assets/app.js

@@ -3,6 +3,10 @@ import { h, mount, toast, errorBox, modal } from "./ui.js";
 import { wordmark, logo } from "./brand.js";
 import { api, get, state, can, setUnauthorizedHandler, setChangeHandler, loadDraft, onDraftChange, setImmediate, useDraft } from "./api.js";
 import * as changes from "./pages/changes.js";
+import * as connections from "./pages/connections.js";
+import * as tracePage from "./pages/trace.js";
+import * as matrixPage from "./pages/matrix.js";
+import * as statsPage from "./pages/stats.js";
 import * as dashboard from "./pages/dashboard.js";
 import * as rules from "./pages/rules.js";
 import * as nat from "./pages/nat.js";
@@ -22,10 +26,14 @@ const routes = [
   { pattern: /^$/, page: dashboard, nav: "dashboard" },
   { pattern: /^rules$/, page: rules, nav: "rules" },
   { pattern: /^nat$/, page: nat, nav: "nat" },
+  { pattern: /^matrix$/, page: matrixPage, nav: "matrix" },
   { pattern: /^zones$/, page: zones, nav: "zones" },
   { pattern: /^sets$/, page: sets, nav: "sets" },
   { pattern: /^services$/, page: services, nav: "services" },
   { pattern: /^log$/, page: log, nav: "log" },
+  { pattern: /^connections$/, page: connections, nav: "connections" },
+  { pattern: /^trace$/, page: tracePage, nav: "trace" },
+  { pattern: /^stats$/, page: statsPage, nav: "stats" },
   { pattern: /^settings$/, page: settings, nav: "settings" },
   { pattern: /^import$/, page: importPage, nav: "import" },
   { pattern: /^changes$/, page: changes, nav: "changes" },
@@ -41,13 +49,19 @@ const routes = [
 const navItems = [
   ["dashboard", "#/", "Dashboard"],
   ["rules", "#/rules", "Rules", "config"],
+  ["matrix", "#/matrix", "Matrix", "config"],
   ["nat", "#/nat", "NAT", "config"],
   { menu: "Objects", items: [
     ["zones", "#/zones", "Zones", "config"],
     ["sets", "#/sets", "Address sets", "config"],
     ["services", "#/services", "Services", "config"],
   ] },
+  { menu: "Traffic", items: [
+    ["stats", "#/stats", "Statistics", "log"],
     ["log", "#/log", "Log", "log"],
+    ["connections", "#/connections", "Connections", "log"],
+    ["trace", "#/trace", "Trace", "log"],
+  ] },
   ["changes", "#/changes", "Changes", "config"],
   ["history", "#/history", "History", "config"],
   { menu: "Server", items: [
@@ -120,7 +134,11 @@ function layout() {
         me.admin ? h("a", { href: "#/users" }, "Users") : null,
         h("button", { type: "button", class: "menu-item", onclick: logout }, "Log out"),
       ], { right: true, active: ["account", "users"].includes(current.nav), label: "Account menu" })));
-  mount(app, h("div", { class: "shell" }, header, barEl, bannerEl, mainEl, footer(me.build)));
+  // Shadow mode (housecarl.yaml): say so on every page.
+  const shadow = me.mode === "shadow" ? h("div", { class: "probation-banner shadow-banner", role: "status" },
+    h("strong", null, "Shadow mode:"), " Housecarl's ruleset is loaded but blocks, rejects and translates nothing; the other firewall decides. ",
+    "What its counters, log, tracer and statistics call drops are what Housecarl would drop. Changes aren't on probation.") : null;
+  mount(app, h("div", { class: "shell" }, header, shadow, barEl, bannerEl, mainEl, footer(me.build)));
   renderNav();
   refreshProbation();
 }
@@ -158,10 +176,11 @@ export async function applyDraft(btn, d, description) {
   try {
     const r = await api("POST", "/changesets/" + d.id + "/apply", { body: { description: description || "" } });
     useDraft(null);
-    if (r.confirm_by) toast("Applied changeset #" + d.id + " on probation (history #" + r.history_id + "): confirm it in the banner", "warn", 8000);
+    if (r.confirm_by && !r.confirm_token) toast("Applied changeset #" + d.id + " on probation (history #" + r.history_id + "): confirm it in the banner", "warn", 8000);
     else toast("Applied changeset #" + d.id + " (history #" + r.history_id + ")");
     for (const w of r.warnings || []) toast(w.message + (w.object ? " (" + w.object + ")" : ""), "warn", 10000);
-    await refreshProbation();
+    if (r.confirm_token) await autoConfirm(r.confirm_token);
+    else await refreshProbation();
     return r;
   } catch (e) {
     btn.disabled = false;
@@ -176,7 +195,25 @@ export async function applyDraft(btn, d, description) {
 
 // ---- probation: changes made here roll back unless confirmed
 
-let probationTimer = null, deadline = 0, pending = 0;
+let probationTimer = null, deadline = 0, pending = 0, checking = false;
+
+// autoConfirm confirms a change with its token. The server closed the
+// change's connection, so this request opens a new one through the new
+// ruleset: it only gets through if this browser can still reach
+// Housecarl. If it can't, the change rolls back by itself.
+export async function autoConfirm(token) {
+  checking = true;
+  await refreshProbation();
+  try {
+    await api("POST", "/confirm", { body: { token }, timeout: 10000 });
+    checking = false;
+    toast("Confirmed automatically: this browser reached Housecarl again over a new connection");
+  } catch (e) {
+    checking = false;
+    toast("Couldn't confirm the change (" + (e.name === "AbortError" ? "no answer over a new connection" : e.message) + "): unless it is confirmed, it rolls back by itself", "warn", 10000);
+  }
+  await refreshProbation();
+}
 
 // refreshProbation asks whether changes wait for confirmation, and shows
 // the banner with a countdown while they do.
@@ -204,6 +241,10 @@ function drawBanner() {
   }
   bannerEl.classList.remove("hidden");
   const write = can("config", true);
+  if (checking) {
+    mount(bannerEl, h("strong", null, "Checking you can still reach Housecarl…"), " The change is confirmed once this browser gets through over a new connection.");
+    return;
+  }
   mount(bannerEl,
     h("strong", null, pending === 1 ? "1 change is on probation" : pending + " changes are on probation"),
     " — rolled back in ", h("span", { class: "mono" }, left + "s"), " unless you confirm them. If this page stops answering, wait: the firewall puts itself back.",
@@ -303,7 +344,7 @@ export function rerender() {
   if (current.page) route();
 }
 
-setChangeHandler(() => refreshProbation());
+setChangeHandler((r) => r && r.confirm_token ? autoConfirm(r.confirm_token) : refreshProbation());
 
 setUnauthorizedHandler(() => {
   if (state.me) {

internal/web/static/assets/charts.js

@@ -0,0 +1,351 @@
+// Small SVG charts for the statistics page: a line/area chart with a
+// crosshair tooltip, a weekday x hour heatmap, and bar lists. No library;
+// colors come from CSS custom properties (--series-N, --heat-N) so light
+// and dark themes are handled in style.css. All text is inserted as text,
+// never as markup.
+import { h } from "./ui.js";
+
+const NS = "http://www.w3.org/2000/svg";
+
+function svg(tag, attrs, ...children) {
+  const el = document.createElementNS(NS, tag);
+  for (const [k, v] of Object.entries(attrs || {})) {
+    if (v !== undefined && v !== null && v !== false) el.setAttribute(k, v);
+  }
+  for (const c of children.flat(Infinity)) {
+    if (c === null || c === undefined || c === false) continue;
+    el.append(c instanceof Node ? c : document.createTextNode(String(c)));
+  }
+  return el;
+}
+
+// compact formats a count: 1,234 / 12.9K / 4.2M.
+export function compact(n) {
+  const a = Math.abs(n);
+  if (a >= 1e9) return (n / 1e9).toFixed(a >= 1e10 ? 0 : 1).replace(/\.0$/, "") + "B";
+  if (a >= 1e6) return (n / 1e6).toFixed(a >= 1e7 ? 0 : 1).replace(/\.0$/, "") + "M";
+  if (a >= 1e4) return (n / 1e3).toFixed(a >= 1e5 ? 0 : 1).replace(/\.0$/, "") + "K";
+  return Math.round(n).toLocaleString();
+}
+
+// niceTicks returns 0..max in clean steps (1, 2 or 5 x 10^k), at most n+1 ticks.
+export function niceTicks(max, n = 4) {
+  if (max <= 0) return [0, 1];
+  const raw = max / n;
+  const mag = Math.pow(10, Math.floor(Math.log10(raw)));
+  const step = [1, 2, 5, 10].map((m) => m * mag).find((s) => s >= raw);
+  const ticks = [];
+  for (let v = 0; v < max + step * 0.999; v += step) ticks.push(Math.round(v * 1e6) / 1e6);
+  return ticks;
+}
+
+const MIN = 60, HOUR = 3600, DAY = 86400;
+const tickSteps = [5 * MIN, 10 * MIN, 15 * MIN, 30 * MIN, HOUR, 2 * HOUR, 3 * HOUR, 6 * HOUR, 12 * HOUR, DAY, 2 * DAY, 7 * DAY, 14 * DAY, 30 * DAY];
+
+// timeTicks picks local-time-aligned ticks between t0 and t1 (Unix seconds).
+function timeTicks(t0, t1, maxTicks) {
+  const span = t1 - t0;
+  const step = tickSteps.find((s) => span / s <= maxTicks) || 30 * DAY;
+  const out = [];
+  const d = new Date(t0 * 1000);
+  if (step >= DAY) {
+    d.setHours(0, 0, 0, 0);
+  } else {
+    d.setSeconds(0, 0);
+    const mins = d.getHours() * 60 + d.getMinutes();
+    const stepMin = step / 60;
+    d.setHours(0, Math.floor(mins / stepMin) * stepMin, 0, 0);
+  }
+  for (let t = d.getTime() / 1000; t <= t1; ) {
+    if (t >= t0) out.push(t);
+    if (step >= DAY) {
+      const nd = new Date(t * 1000);
+      nd.setDate(nd.getDate() + step / DAY);
+      t = nd.getTime() / 1000;
+    } else t += step;
+  }
+  return { ticks: out, step };
+}
+
+const weekdays = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"];
+const months = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
+const pad = (n) => String(n).padStart(2, "0");
+
+function tickLabel(t, step) {
+  const d = new Date(t * 1000);
+  if (step >= DAY) return step >= 7 * DAY ? months[d.getMonth()] + " " + d.getDate() : weekdays[d.getDay()] + " " + d.getDate();
+  const hm = pad(d.getHours()) + ":" + pad(d.getMinutes());
+  return d.getHours() === 0 && d.getMinutes() === 0 ? weekdays[d.getDay()] + " " + d.getDate() : hm;
+}
+
+// longTime describes a bucket for tooltips and tables.
+export function bucketLabel(t, stepSecs) {
+  const d = new Date(t * 1000);
+  const day = weekdays[d.getDay()] + " " + d.getDate() + " " + months[d.getMonth()];
+  if (stepSecs >= DAY) return day;
+  const end = new Date((t + stepSecs) * 1000);
+  const hm = (x) => pad(x.getHours()) + ":" + pad(x.getMinutes());
+  return day + ", " + hm(d) + (stepSecs > MIN ? "–" + hm(end) : "");
+}
+
+export const stepSeconds = { "1m": MIN, "5m": 5 * MIN, "15m": 15 * MIN, "1h": HOUR, "6h": 6 * HOUR, "1d": DAY };
+
+// tooltip is one floating readout per chart.
+function tooltip() {
+  const el = h("div", { class: "chart-tip", role: "status" });
+  el.hidden = true;
+  return el;
+}
+
+// chartCard wraps a chart with its title and a table-view toggle (the
+// accessible twin of every chart).
+export function chartCard(title, subtitle, chart, tableView, opts = {}) {
+  const body = h("div", { class: "chart-body" }, chart);
+  const tableBox = h("div", { class: "chart-table" }, tableView);
+  tableBox.hidden = true;
+  const toggle = h("button", { class: "small link", type: "button", "aria-pressed": "false" }, "Table");
+  toggle.addEventListener("click", () => {
+    const showTable = tableBox.hidden;
+    tableBox.hidden = !showTable;
+    body.hidden = showTable;
+    toggle.textContent = showTable ? "Chart" : "Table";
+    toggle.setAttribute("aria-pressed", String(showTable));
+  });
+  return h("figure", { class: "panel chart-card" + (opts.wide ? " wide" : "") },
+    h("figcaption", { class: "chart-head" },
+      h("div", { class: "grow" }, h("h2", null, title), subtitle ? h("div", { class: "muted small" }, subtitle) : null),
+      opts.legend || null,
+      tableView ? toggle : null),
+    body, tableBox);
+}
+
+// legend lists series with a short line key (the mark the series uses).
+export function legend(series) {
+  return h("ul", { class: "chart-legend" }, series.map((s) =>
+    h("li", null, h("span", { class: "key line s" + s.slot }), s.label)));
+}
+
+// endLabelWidth is the room lineChart needs for the end labels of series.
+export function endLabelWidth(series) {
+  return series.length > 1 && series.length <= 4 ? Math.max(...series.map((s) => s.label.length)) * 7 + 12 : 8;
+}
+
+// lineChart draws series over time. series: [{label, values, slot}] (slot
+// = categorical color 1..3); times: bucket starts (Unix seconds); step in
+// seconds. With area, the first series gets a 10% wash under its line.
+// gutter {left, right} sets the least room for the axis labels and the end
+// labels: charts shown one above the other pass the same gutter, so their
+// time axes line up. format formats values (axis and readout); a null
+// value is a gap in the line (no data for that time).
+export function lineChart({ series, times, step, area = false, height = 220, endLabels = true, gutter = {}, format = compact }) {
+  const root = h("div", { class: "chart", tabindex: "0", "aria-label": series.map((s) => s.label).join(", ") + " over time" });
+  const tip = tooltip();
+  root.append(tip);
+  let width = 0, focusIdx = -1, geom = null;
+
+  function draw() {
+    const w = root.clientWidth;
+    if (!w || w === width) return;
+    width = w;
+    root.querySelector("svg")?.remove();
+    const max = Math.max(1, ...series.flatMap((s) => s.values).filter((v) => v != null));
+    const yTicks = niceTicks(max, 4);
+    const yMax = yTicks[yTicks.length - 1];
+    const labelW = Math.max(...yTicks.map((t) => format(t).length)) * 7 + 10;
+    const endW = endLabels ? endLabelWidth(series) : 8;
+    const m = { top: 10, right: Math.max(endW, gutter.right || 0), bottom: 24, left: Math.max(labelW, gutter.left || 0) };
+    const pw = Math.max(40, w - m.left - m.right), ph = height - m.top - m.bottom;
+    const t0 = times[0], t1 = times[times.length - 1] + step;
+    const x = (t) => m.left + ((t - t0) / Math.max(1, t1 - t0)) * pw;
+    const cx = (i) => x(times[i] + step / 2); // the middle of a bucket
+    const y = (v) => m.top + ph - (v / yMax) * ph;
+    const s = svg("svg", { width: w, height, viewBox: `0 0 ${w} ${height}`, role: "img", "aria-hidden": "true" });
+    // Grid and axes.
+    for (const t of yTicks) {
+      s.append(svg("line", { class: t === 0 ? "axis" : "grid", x1: m.left, x2: m.left + pw, y1: y(t), y2: y(t) }));
+      s.append(svg("text", { class: "tick", x: m.left - 6, y: y(t) + 4, "text-anchor": "end" }, format(t)));
+    }
+    const { ticks, step: tstep } = timeTicks(t0, t1, Math.max(2, Math.floor(pw / 90)));
+    for (const t of ticks) {
+      s.append(svg("text", { class: "tick", x: x(t), y: height - 6, "text-anchor": "middle" }, tickLabel(t, tstep)));
+    }
+    // Series.
+    series.forEach((sr, k) => {
+      // A gap (null) starts a new piece of line.
+      const pts = sr.values.map((v, i) => (v == null ? null : [cx(i), y(v)]));
+      const d = pts.map((p, i) => (p ? (i && pts[i - 1] ? "L" : "M") + p[0].toFixed(1) + " " + p[1].toFixed(1) : "")).join("");
+      if (area && k === 0 && pts.length && pts.every(Boolean)) {
+        s.append(svg("path", { class: "area s" + sr.slot, d: d + `L${pts[pts.length - 1][0].toFixed(1)} ${y(0)}L${pts[0][0].toFixed(1)} ${y(0)}Z` }));
+      }
+      s.append(svg("path", { class: "line s" + sr.slot, d }));
+      // A point without neighbours has no line: show it as a dot.
+      pts.forEach((p, i) => {
+        if (p && !pts[i - 1] && !pts[i + 1]) s.append(svg("circle", { class: "dot s" + sr.slot, cx: p[0], cy: p[1], r: 3 }));
+      });
+    });
+    // Direct end labels when there is room and they don't collide.
+    if (m.right > 8) {
+      const ends = series.map((sr) => ({ label: sr.label, y: y(sr.values.findLast((v) => v != null) || 0) })).sort((a, b) => a.y - b.y);
+      if (ends.every((e, i) => i === 0 || e.y - ends[i - 1].y >= 13)) {
+        for (const e of ends) s.append(svg("text", { class: "end-label", x: m.left + pw + 6, y: e.y + 4 }, e.label));
+      }
+    }
+    // Crosshair (drawn on hover/focus).
+    const cross = svg("line", { class: "crosshair", y1: m.top, y2: m.top + ph, visibility: "hidden" });
+    const dots = series.map((sr) => svg("circle", { class: "dot s" + sr.slot, r: 4, visibility: "hidden" }));
+    s.append(cross, ...dots);
+    // The hit area is the whole plot: readers aim at a time, not a line.
+    const hit = svg("rect", { class: "hit", x: m.left, y: m.top, width: pw, height: ph });
+    s.append(hit);
+    root.prepend(s);
+    geom = { cx, y, m, pw, ph, cross, dots, hit };
+    hit.addEventListener("pointermove", (e) => {
+      const r = s.getBoundingClientRect();
+      const px = e.clientX - r.left;
+      const i = Math.round(((px - m.left) / pw) * (times.length) - 0.5);
+      show(Math.max(0, Math.min(times.length - 1, i)));
+    });
+    hit.addEventListener("pointerleave", hide);
+    if (focusIdx >= 0) show(focusIdx);
+  }
+
+  function show(i) {
+    if (!geom) return;
+    focusIdx = i;
+    const xpos = geom.cx(i);
+    geom.cross.setAttribute("x1", xpos);
+    geom.cross.setAttribute("x2", xpos);
+    geom.cross.setAttribute("visibility", "visible");
+    series.forEach((sr, k) => {
+      geom.dots[k].setAttribute("cx", xpos);
+      geom.dots[k].setAttribute("cy", geom.y(sr.values[i] ?? 0));
+      geom.dots[k].setAttribute("visibility", sr.values[i] == null ? "hidden" : "visible");
+    });
+    tip.replaceChildren(
+      h("div", { class: "tip-head" }, bucketLabel(times[i], step)),
+      ...series.map((sr) => h("div", { class: "tip-row" },
+        h("span", { class: "key line s" + sr.slot }), h("strong", null, sr.values[i] == null ? "–" : format === compact ? sr.values[i].toLocaleString() : format(sr.values[i])), " ", h("span", { class: "muted" }, sr.label))));
+    tip.hidden = false;
+    // Keep the readout inside the chart, on the far side of the pointer.
+    const left = xpos > root.clientWidth / 2 ? xpos - tip.offsetWidth - 12 : xpos + 12;
+    tip.style.left = Math.max(0, left) + "px";
+    tip.style.top = geom.m.top + "px";
+  }
+
+  function hide() {
+    if (!geom) return;
+    geom.cross.setAttribute("visibility", "hidden");
+    geom.dots.forEach((d) => d.setAttribute("visibility", "hidden"));
+    tip.hidden = true;
+    focusIdx = -1;
+  }
+
+  // Keyboard: arrows move the crosshair, as hover does.
+  root.addEventListener("focus", () => show(focusIdx >= 0 ? focusIdx : times.length - 1));
+  root.addEventListener("blur", hide);
+  root.addEventListener("keydown", (e) => {
+    if (e.key === "ArrowLeft" || e.key === "ArrowRight") {
+      e.preventDefault();
+      show(Math.max(0, Math.min(times.length - 1, (focusIdx < 0 ? times.length - 1 : focusIdx) + (e.key === "ArrowLeft" ? -1 : 1))));
+    }
+  });
+  new ResizeObserver(draw).observe(root);
+  return root;
+}
+
+// timeTable is the table view of a line chart (busiest buckets first is
+// not useful here: keep time order, newest first).
+export function timeTable(series, times, step, format) {
+  const cell = (v) => (v == null ? "–" : format ? format(v) : v.toLocaleString());
+  const rows = times.map((t, i) => ({ t, i })).reverse();
+  return h("div", { class: "table-wrap scroll" }, h("table", null,
+    h("thead", null, h("tr", null, h("th", null, "Time"), series.map((s) => h("th", { class: "num" }, s.label)))),
+    h("tbody", null, rows.map((r) => h("tr", null, h("td", null, bucketLabel(r.t, step)),
+      series.map((s) => h("td", { class: "num" }, cell(s.values[r.i]))))))));
+}
+
+// heatmap draws average activity by weekday (rows, Monday first) and hour
+// (columns) with a sequential one-hue ramp; cells without data stay empty.
+export function heatmap(hm, unit) {
+  const days = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"];
+  let peak = 0;
+  hm.avg.forEach((row) => row.forEach((v) => { peak = Math.max(peak, v); }));
+  const steps = 7;
+  const level = (v) => (peak <= 0 ? 0 : Math.min(steps - 1, Math.floor((v / peak) * steps)));
+  const tip = tooltip();
+  const grid = h("div", { class: "heatmap", role: "grid", "aria-label": "Average " + unit + " per hour by weekday and hour of day" });
+  grid.append(h("div", { class: "hm-corner" }));
+  for (let hr = 0; hr < 24; hr++) grid.append(h("div", { class: "hm-col" }, hr % 3 === 0 ? pad(hr) : ""));
+  days.forEach((day, d) => {
+    grid.append(h("div", { class: "hm-row" }, day));
+    for (let hr = 0; hr < 24; hr++) {
+      const v = hm.avg[d][hr];
+      const n = hm.samples[d][hr];
+      const text = v < 0 ? day + " " + pad(hr) + ":00 – no data" :
+        day + " " + pad(hr) + ":00 – " + Math.round(v).toLocaleString() + " " + unit + " per hour (average of " + n + (n === 1 ? " hour)" : " hours)");
+      const cell = h("div", { class: "hm-cell " + (v < 0 ? "hm-empty" : "heat-" + level(v)), tabindex: "-1", "aria-label": text });
+      const showTip = () => {
+        tip.textContent = text;
+        tip.hidden = false;
+        const r = cell.getBoundingClientRect(), pr = grid.getBoundingClientRect();
+        tip.style.left = Math.max(0, Math.min(pr.width - tip.offsetWidth, r.left - pr.left - tip.offsetWidth / 2 + r.width / 2)) + "px";
+        tip.style.top = (r.bottom - pr.top + 6) + "px";
+      };
+      cell.addEventListener("pointerenter", showTip);
+      cell.addEventListener("focus", showTip);
+      cell.addEventListener("pointerleave", () => { tip.hidden = true; });
+      cell.addEventListener("blur", () => { tip.hidden = true; });
+      grid.append(cell);
+    }
+  });
+  // Arrow keys move between cells.
+  grid.tabIndex = 0;
+  let cur = -1;
+  const cells = () => [...grid.querySelectorAll(".hm-cell")];
+  grid.addEventListener("keydown", (e) => {
+    const delta = { ArrowLeft: -1, ArrowRight: 1, ArrowUp: -24, ArrowDown: 24 }[e.key];
+    if (delta === undefined) return;
+    e.preventDefault();
+    cur = Math.max(0, Math.min(7 * 24 - 1, (cur < 0 ? 0 : cur) + delta));
+    cells()[cur].focus();
+  });
+  const scale = h("div", { class: "hm-scale" },
+    h("span", { class: "muted small" }, "0"),
+    Array.from({ length: steps }, (_, i) => h("span", { class: "hm-swatch heat-" + i })),
+    h("span", { class: "muted small" }, compact(peak) + " " + unit + "/hour"),
+    h("span", { class: "hm-swatch hm-empty" }), h("span", { class: "muted small" }, "no data"));
+  return h("div", { class: "heatmap-wrap" }, grid, tip, scale);
+}
+
+export function heatmapTable(hm, unit) {
+  const days = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"];
+  return h("div", { class: "table-wrap scroll" }, h("table", null,
+    h("thead", null, h("tr", null, h("th", null, "Hour"), days.map((d) => h("th", { class: "num" }, d)))),
+    h("tbody", null, Array.from({ length: 24 }, (_, hr) => h("tr", null, h("td", null, pad(hr) + ":00"),
+      days.map((_, d) => h("td", { class: "num" }, hm.avg[d][hr] < 0 ? "–" : Math.round(hm.avg[d][hr]).toLocaleString())))))),
+    h("div", { class: "muted small" }, "Average " + unit + " per hour."));
+}
+
+// barList shows ranked entries as labelled horizontal bars in one hue
+// (one series: no legend, the title names it). Values are always printed,
+// so the list is its own table view.
+// extra(key), when given, adds something after an entry's label (e.g. a
+// link that excludes it).
+export function barList(entries, { total, label = (k) => k, empty = "No data yet.", href, extra, format = compact } = {}) {
+  if (!entries.length) return h("div", { class: "empty" }, empty);
+  // Entries may be in a natural order (e.g. latency buckets), not by size.
+  const peak = Math.max(1, ...entries.map((e) => e.count));
+  const sum = total || entries.reduce((a, e) => a + e.count, 0);
+  return h("ol", { class: "bar-list" }, entries.map((e) => {
+    const share = sum ? (e.count / sum) * 100 : 0;
+    const name = label(e.key);
+    const link = href ? href(e.key) : null;
+    // Entries with a name (a rule's label): the name, then the key.
+    const shown = e.name ? [e.name, h("span", { class: "muted" }, " " + name)] : name;
+    return h("li", { title: (e.name ? e.name + " (" + name + ")" : name) + ": " + format(e.count) + " (" + share.toFixed(1) + "%)" },
+      h("div", { class: "bar-text" },
+        link ? h("a", { href: link, class: "bar-label" }, shown) : h("span", { class: "bar-label" }, shown),
+        h("span", { class: "bar-value" }, extra ? [extra(e.key), " "] : null, format(e.count), h("span", { class: "muted" }, " · " + share.toFixed(share < 10 ? 1 : 0) + "%"))),
+      h("div", { class: "bar-track" }, h("div", { class: "bar-fill", style: { width: Math.max(0.5, (e.count / peak) * 100) + "%" } })));
+  }));
+}

internal/web/static/assets/pages/connections.js

@@ -0,0 +1,49 @@
+// Connections: what the kernel tracks, and closing them so the rules
+// check them again.
+import { h, mount, input, debounce, toast, confirmDialog, badge } from "../ui.js";
+import { get, api, can } from "../api.js";
+import { onLeave } from "../app.js";
+import { fmtSeconds } from "../widgets.js";
+
+export async function render(el, params, query) {
+  const search = input({ type: "search", placeholder: "address, :port, protocol, state…", value: query.get("q") || "" });
+  const out = h("div");
+  const count = h("span", { class: "muted small" });
+  const write = can("config", true);
+  let shown = [];
+  const hostPort = (a, p) => !p ? a : a.includes(":") ? "[" + a + "]:" + p : a + ":" + p;
+  const close = async (list, what) => {
+    if (!await confirmDialog("Close connections", "Close " + what + "? Their next packet is checked by the rules as they are now: what they block is stopped, what they allow carries on.", "Close", "danger")) return;
+    try {
+      const r = await api("POST", "/connections/close", { body: list ? { connections: list } : { search: search.value.trim() } });
+      toast("Closed " + r.closed + (r.closed === 1 ? " connection" : " connections"));
+      draw();
+    } catch (e) { toast(e.message, "err"); }
+  };
+  const draw = async () => {
+    shown = await get("/connections", { q: search.value.trim(), limit: 500 });
+    count.textContent = shown.length === 1 ? "1 connection" : shown.length + " connections";
+    mount(out, shown.length ? h("div", { class: "table-wrap" }, h("table", null,
+      h("thead", null, h("tr", null, ["Protocol", "From", "To", "NAT", "State", "Expires", ""].map((t) => h("th", null, t)))),
+      h("tbody", null, shown.map((c) => h("tr", null,
+        h("td", null, c.protocol),
+        h("td", { class: "mono" }, hostPort(c.source, c.source_port)),
+        h("td", { class: "mono" }, hostPort(c.destination, c.destination_port)),
+        h("td", { class: "mono small" }, c.dnat_to ? "to " + hostPort(c.dnat_to, c.dnat_port || c.destination_port) : "", c.snat_to ? (c.dnat_to ? ", " : "") + "as " + c.snat_to : ""),
+        h("td", null, c.state || "", c.assured ? null : [" ", badge("one way", "")]),
+        h("td", { class: "small" }, fmtSeconds(c.timeout)),
+        h("td", { class: "actions-cell" }, write ? h("button", { class: "small reveal", onclick: () => close([c], "this connection") }, "Close") : null)))))) :
+      h("div", { class: "empty" }, search.value.trim() ? "No connection matches." : "No connections tracked."));
+  };
+  search.addEventListener("input", debounce(draw, 300));
+  mount(el,
+    h("div", { class: "page-head" },
+      h("div", { class: "grow" }, h("h1", null, "Connections"),
+        h("div", { class: "muted" }, "The connections the firewall tracks. Their packets are let through by the built-in for established connections, without the rules being checked again, so a rule added later doesn't stop them. Close them, and their next packet is checked by the rules as they are now.")),
+      write ? h("div", { class: "actions" }, h("button", { class: "danger", onclick: () => close(null, search.value.trim() ? "every connection matching “" + search.value.trim() + "”" : "every connection") }, "Close all shown")) : null),
+    h("div", { class: "filter-row" }, search, count),
+    h("div", { class: "panel" }, out));
+  await draw();
+  const t = setInterval(() => { if (!document.hidden && document.activeElement !== search) draw().catch(() => {}); }, 5000);
+  onLeave(() => clearInterval(t));
+}

internal/web/static/assets/pages/dashboard.js

@@ -1,18 +1,23 @@
-// Dashboard: the ruleset's state, problems, interfaces, the busiest rules
-// and the latest drops.
-import { h, mount, table, badge, problemsList, time } from "../ui.js";
+// Dashboard: the ruleset's state, problems, interfaces, the busiest rules,
+// the quotas and the latest drops.
+import { h, mount, table, badge, problemsList, time, plural } from "../ui.js";
 import { get, can } from "../api.js";
 import { onLeave } from "../app.js";
-import { describe, hits, humanCount } from "../widgets.js";
+import { describe, hits, humanCount, fmtBytes, usageBar } from "../widgets.js";
+import { lineChart, chartCard, legend, timeTable, stepSeconds } from "../charts.js";
+import { bytes } from "./stats.js";
 
 export async function render(el) {
   const draw = async () => {
     const st = await get("/status");
     const config = can("config", false), logs = can("log", false);
-    const [rules, cs, log] = await Promise.all([
+    const [rules, cs, log, tl, zones, quotas] = await Promise.all([
       config ? get("/rules") : [],
       logs ? get("/counters").then((c) => c.counters).catch(() => ({})) : {},
       logs ? get("/log", { limit: 8 }).catch(() => []) : [],
+      logs ? get("/stats/timeline", { since: "24h", step: "1h", keys: "zrx:*,ztx:*,drops" }).catch(() => null) : null,
+      config ? get("/zones").catch(() => []) : [],
+      logs ? get("/quotas").catch(() => []) : [],
     ]);
     const busiest = rules.filter((r) => cs["r" + r.id] && cs["r" + r.id].packets > 0)
       .sort((a, b) => cs["r" + b.id].packets - cs["r" + a.id].packets).slice(0, 8);
@@ -35,6 +40,7 @@ export async function render(el) {
         card("Dropped by policy", policyDrops("input") + " / " + policyDrops("forward"), "input / forward", logs ? "#/log" : null),
         card("Problems", st.problems.length, st.problems.length ? "see below" : "none", null)),
       st.problems.length ? h("div", { class: "panel" }, h("h2", null, "Problems"), problemsList(st.problems)) : null,
+      tl ? statsCharts(tl, zones) : null,
       h("div", { class: "panel" }, h("h2", null, "Interfaces"), table([
         { label: "Interface", render: (i) => [h("span", { class: "dot " + (i.up ? "ok" : "err") }), h("span", { class: "mono" }, h("strong", null, i.name))] },
         { label: "Zone", render: (i) => i.zone || h("span", { class: "muted" }, "none") },
@@ -44,11 +50,12 @@ export async function render(el) {
         h("div", { class: "panel-head" }, h("h2", null, "Busiest rules"), h("a", { class: "button small", href: "#/rules" }, "All rules")),
         table([
           { label: "Chain", render: (r) => r.chain },
-          { label: "Rule", render: (r) => r.raw || describe(r) },
+          { label: "Rule", render: (r) => h("a", { href: "#/rules?chain=" + r.chain + "&rule=" + r.id }, "#" + r.id + " ", r.raw || describe(r)) },
           { label: "Action", render: (r) => badge(r.action, { accept: "ok", drop: "err", reject: "warn" }[r.action]) },
           { label: "Hits", render: (r) => hits(cs, "r" + r.id) },
           { label: "Comment", render: (r) => r.comment },
         ], busiest)) : null,
+      quotas.length ? quotaPanel(quotas) : null,
       log.length ? h("div", { class: "panel" },
         h("div", { class: "panel-head" }, h("h2", null, "Latest logged packets"), h("a", { class: "button small", href: "#/log" }, "The log")),
         table([
@@ -63,3 +70,38 @@ export async function render(el) {
   const t = setInterval(() => { if (!document.hidden) draw().catch(() => {}); }, 15000);
   onLeave(() => clearInterval(t));
 }
+
+// quotaPanel lists every rule's quota with what it used: the total, or
+// for a per-address quota the busiest address (the rest are on the rule).
+function quotaPanel(quotas) {
+  const perAddress = (q) => q.quota.per_source || q.quota.per_destination;
+  const used = (q) => perAddress(q) ? ((q.addresses || [])[0] || { used: 0 }).used : q.used;
+  return h("div", { class: "panel" },
+    h("div", { class: "panel-head" }, h("h2", null, "Quotas"), h("a", { class: "button small", href: "#/stats" }, "Statistics")),
+    table([
+      { label: "Rule", render: (q) => h("a", { href: "#/rules?rule=" + q.rule_id }, "#" + q.rule_id + " ", q.rule) },
+      { label: "Allowance", render: (q) => fmtBytes(q.quota.bytes) + (q.quota.per_source ? " each source" : q.quota.per_destination ? " each destination" : "") },
+      { label: "Used", render: (q) => perAddress(q)
+        ? [plural((q.addresses || []).length, "address", "addresses"), q.addresses && q.addresses.length ? ", busiest " + fmtBytes(used(q)) : ""]
+        : fmtBytes(q.used) },
+      { label: "", render: (q) => usageBar(used(q), q.quota.bytes) },
+      { label: "Next reset", render: (q) => q.next_reset ? time(q.next_reset) : h("span", { class: "muted" }, "by hand") },
+    ], quotas));
+}
+
+// statsCharts: the last 24 hours' traffic with the internet and drops,
+// per hour (the Statistics page has the rest).
+function statsCharts(tl, zones) {
+  const ext = new Set(zones.filter((z) => z.external).map((z) => z.name));
+  const step = stepSeconds[tl.step];
+  const total = (prefix) => tl.times.map((_, i) => tl.series.filter((x) => x.key.startsWith(prefix) && ext.has(x.key.slice(prefix.length))).reduce((a, x) => a + x.values[i], 0));
+  const traffic = [{ label: "Downloaded", values: total("zrx:"), slot: 1 }, { label: "Uploaded", values: total("ztx:"), slot: 2 }];
+  const drops = [{ label: "Dropped", values: (tl.series.find((x) => x.key === "drops") || { values: tl.times.map(() => 0) }).values, slot: 2 }];
+  const gutter = { left: 64, right: 8 };
+  return h("div", { class: "chart-grid" },
+    ext.size ? chartCard("Internet traffic", "bytes per hour from and to external zones, last 24 hours",
+      lineChart({ series: traffic, times: tl.times, step, area: true, height: 150, endLabels: false, gutter, format: bytes }),
+      timeTable(traffic, tl.times, step, bytes), { legend: legend(traffic) }) : null,
+    chartCard("Dropped packets", h("span", null, "per hour, last 24 hours · ", h("a", { href: "#/stats" }, "Statistics")),
+      lineChart({ series: drops, times: tl.times, step, area: true, height: 150, endLabels: false, gutter }), timeTable(drops, tl.times, step)));
+}

internal/web/static/assets/pages/history.js

@@ -1,6 +1,6 @@
 // History: every configuration change, with undo.
 import { h, mount, table, time, confirmDialog, modal, errorBox, badge } from "../ui.js";
-import { get, change, can } from "../api.js";
+import { get, change, can, state } from "../api.js";
 import { rerender } from "../app.js";
 
 export async function render(el, params) {
@@ -20,10 +20,10 @@ export async function render(el, params) {
 }
 
 async function undo(e) {
-  if (!await confirmDialog("Undo change #" + e.id, "Undo “" + e.description + "”? This is recorded as a new change, which can be undone too.", "Undo")) return;
+  if (!await confirmDialog("Undo change #" + e.id, "Undo “" + e.description + "”? This is recorded as a new change, which can be undone too." + (!state.immediate ? " It is added to your pending changes, to review and apply." : ""), "Undo")) return;
   try {
-    // An undo can't be staged: it is applied at once.
-    await change("POST", "/history/" + e.id + "/revert", undefined, { immediate: true });
+    // Like any edit: staged for review, or applied at once in live mode.
+    await change("POST", "/history/" + e.id + "/revert");
     rerender();
   } catch (err) { modal("Can't undo", errorBox(err)); }
 }

internal/web/static/assets/pages/import.js

@@ -1,39 +1,48 @@
 // Import: an existing nftables ruleset as Housecarl's configuration.
 import { h, mount, field, textarea, input, checkbox, errorBox, copyBox } from "../ui.js";
 import { api } from "../api.js";
-import { refreshProbation } from "../app.js";
+import { refreshProbation, autoConfirm } from "../app.js";
 
 export async function render(el) {
   const json = textarea("", { rows: 6, class: "mono", placeholder: "the output of: sudo nft -j list ruleset" });
   const text = textarea("", { rows: 4, class: "mono", placeholder: "optional, the output of: sudo nft -a list ruleset (keeps what Housecarl can't express as raw rules)" });
   const defines = textarea("", { rows: 4, class: "mono", placeholder: "optional, your /etc/nftables.conf: its defines name the sets" });
   const table = input({ placeholder: "inet main (when there are several)", class: "mono" });
+  const zones = input({ placeholder: "enp2s0=lan, wg0=vpn (optional)", class: "mono" });
   const replace = checkbox("replace Housecarl's configuration", false);
   const out = h("div");
   const run = async (dry) => {
     mount(out);
+    const zoneNames = {};
+    for (const part of zones.value.split(",").map((s) => s.trim()).filter(Boolean)) {
+      const [iface, name] = part.split("=").map((s) => (s || "").trim());
+      if (!iface || !name) { mount(out, h("div", { class: "msg err" }, "Zone names: write each as interface=name (enp2s0=lan).")); return; }
+      zoneNames[iface] = name;
+    }
     let parsed;
     try { parsed = JSON.parse(json.value); } catch { mount(out, h("div", { class: "msg err" }, "The ruleset isn't JSON: paste the output of nft -j list ruleset.")); return; }
     try {
-      const r = await api("POST", "/import/nft", { body: { json: parsed, text: text.value, defines: defines.value, table: table.value.trim(), replace: replace.input.checked },
+      const r = await api("POST", "/import/nft", { body: { json: parsed, text: text.value, defines: defines.value, table: table.value.trim(), zones: zoneNames, replace: replace.input.checked },
         query: dry ? { dry_run: 1 } : undefined });
       const st = r.state;
       mount(out, h("div", { class: "msg " + (dry ? "info" : "ok") },
         dry ? "Would import: " : "Imported: ",
         `${st.zones.length} zones, ${st.sets.length} sets, ${st.services.length} services, ${st.rules.length} rules, ${st.forwards.length} port forwards, ${st.snats.length} source NAT rules, ${st.redirects.length} redirects.`,
-        r.confirm_by ? " It is on probation: confirm it in the banner." : ""),
+        r.confirm_by && !r.confirm_token ? " It is on probation: confirm it in the banner." : ""),
       r.notes.length ? h("div", { class: "panel" }, h("h2", null, "Notes"), h("ul", null, r.notes.map((n) => h("li", null, n)))) : null,
       r.raw.length ? h("div", { class: "panel" }, h("h2", null, "Kept as raw rules"), h("ul", { class: "mono small" }, r.raw.map((n) => h("li", null, n)))) : null,
       !dry ? h("p", null, "Check the ", h("a", { href: "#/rules" }, "rules"), " and ", h("a", { href: "#/nat" }, "NAT"),
         "; both rulesets apply until the old table goes: ", copyBox("sudo hc takeover \"inet main\"")) : null);
-      if (!dry) refreshProbation();
+      if (!dry && r.confirm_token) autoConfirm(r.confirm_token);
+      else if (!dry) refreshProbation();
     } catch (e) { mount(out, errorBox(e)); }
   };
   mount(el,
     h("div", { class: "page-head" }, h("div", { class: "grow" }, h("h1", null, "Import a ruleset"),
       h("div", { class: "muted" }, "Turns an nftables ruleset into zones, sets, services, rules, port forwards and NAT. On the firewall, 'sudo hc import --defines /etc/nftables.conf -n' does the same from the command line."))),
     h("div", { class: "panel" }, h("div", { class: "form-grid" },
-      field("Ruleset (JSON)", json), field("Ruleset (text)", text), field("Ruleset file", defines), field("Table", table), h("label"), replace),
+      field("Ruleset (JSON)", json), field("Ruleset (text)", text), field("Ruleset file", defines), field("Table", table),
+      field("Zone names", zones, "Name zones by what the network is for; interfaces given one name share a zone. Otherwise a define holding the interface names it, or the interface's own name."), h("label"), replace),
       h("div", { class: "actions form-actions" },
         h("button", { onclick: () => run(true) }, "Preview"),
         h("button", { class: "primary", onclick: () => run(false) }, "Import"))),

internal/web/static/assets/pages/log.js

@@ -1,7 +1,7 @@
 // The log: packets rules (with logging on) and the policies logged,
 // refreshed every few seconds.
-import { h, mount, input, debounce } from "../ui.js";
-import { get } from "../api.js";
+import { h, mount, input, debounce, badge } from "../ui.js";
+import { get, state } from "../api.js";
 import { onLeave } from "../app.js";
 
 export async function render(el, params, query) {
@@ -16,11 +16,14 @@ export async function render(el, params, query) {
       h("thead", null, h("tr", null, ["Time", "Why", "In", "Out", "From", "To", "Protocol"].map((t) => h("th", null, t)))),
       h("tbody", null, es.map((e) => h("tr", null,
         h("td", { class: "mono small" }, new Date(e.time).toLocaleTimeString(undefined, { hourCycle: "h23" })),
-        h("td", null, e.mark && e.mark.startsWith("r") ? h("a", { href: "#/rules", title: "rule " + e.mark.slice(1) }, e.prefix) : e.prefix),
+        h("td", null, e.mark && e.mark.startsWith("r") ? h("a", { href: "#/rules?rule=" + e.mark.slice(1), title: "rule #" + e.mark.slice(1) }, e.prefix) : e.prefix,
+          e.shadow && /^(drop|reject)$/.test(e.verdict) ? [" ", badge("would " + e.verdict, "warn")] : null),
         h("td", { class: "mono" }, e.in || ""), h("td", { class: "mono" }, e.out || ""),
         h("td", { class: "mono" }, addr(e.source, e.source_port)), h("td", { class: "mono" }, addr(e.destination, e.destination_port)),
         h("td", null, e.protocol, e.icmp_type ? " " + e.icmp_type : "", e.tcp_flags ? h("span", { class: "muted small" }, " " + e.tcp_flags) : null)))))) :
-      h("div", { class: "empty" }, "Nothing logged yet. Turn logging on for a rule (Edit → Log), or for what the policies drop (Settings)."));
+      h("div", { class: "empty" }, state.me.mode === "shadow"
+        ? "Nothing logged yet. In shadow mode, what the policies would drop is logged; turn logging on for a rule with Edit → Log."
+        : "Nothing logged yet. Turn logging on for a rule (Edit → Log), or for what the policies drop (Settings)."));
   };
   search.addEventListener("input", debounce(draw, 300));
   mount(el,

internal/web/static/assets/pages/matrix.js

@@ -0,0 +1,50 @@
+// Matrix: what the rules do to new connections between zones, and to and
+// from the firewall itself.
+import { h, mount, modal, badge } from "../ui.js";
+import { get, state } from "../api.js";
+
+export async function render(el) {
+  const m = await get("/matrix");
+  const cols = [...m.zones, "firewall"];
+  const cells = new Map(m.cells.map((c) => [c.from + "\u0000" + c.to, c]));
+  const pending = state.draft && !state.immediate && (state.draft.ops || []).length;
+  const name = (z) => z === "firewall" ? h("em", null, "firewall") : z;
+  mount(el,
+    h("div", { class: "page-head" }, h("div", { class: "grow" }, h("h1", null, "Zone matrix"),
+      h("div", { class: "muted" }, "What happens to a new connection from a zone (rows) to a zone (columns), or to and from the firewall itself, worked out from the rules in order: all allowed, all blocked, or partly (some services, addresses or times let through, the rest not). Click a cell for the rules."))),
+    pending ? h("div", { class: "msg info" }, "Shown with your pending changes.") : null,
+    !m.zones.length ? h("div", { class: "panel" }, h("div", { class: "empty" }, "No zones yet: add them under Objects → Zones.")) :
+    h("div", { class: "panel" }, h("div", { class: "table-wrap" }, h("table", { class: "matrix" },
+      h("thead", null, h("tr", null, h("th", { class: "corner" }, "from ↓ to →"), cols.map((c) => h("th", null, name(c))))),
+      h("tbody", null, cols.map((from) => h("tr", null, h("th", null, name(from)), cols.map((to) => {
+        const c = cells.get(from + "\u0000" + to);
+        if (!c) return h("td", { class: "matrix-none" }, "–");
+        const open = c.open || [];
+        const counts = [c.allowed ? "+" + c.allowed : "", c.blocked ? "−" + c.blocked : ""].filter(Boolean).join(" ");
+        const title = (c.same_zone ? c.note + ". " : "") + "The rest: " + c.verdict + " by " + c.decided_by +
+          (c.allowed ? "; " + c.allowed + " rule(s) let part of it through" : "") + (c.blocked ? "; " + c.blocked + " block part of it" : "");
+        return h("td", { class: "matrix-" + c.summary + (c.same_zone ? " matrix-same" : ""), title },
+          h("button", { type: "button", class: "matrix-cell", onclick: () => detail(c) },
+            c.summary === "partly" ? "partly" : c.verdict, counts ? h("span", { class: "matrix-n" }, counts) : null,
+            open.length && !c.same_zone ? h("span", { class: "matrix-open" }, open.slice(0, 3).join(" · ") + (open.length > 3 ? " · +" + (open.length - 3) + " more" : "")) : null));
+      })))))),
+      h("p", { class: "muted small" }, "Zone to zone is the forward chain; to the firewall, input; from it, output. +N: rules (and built-ins such as ping or port forwards) that let part of the traffic through before the rest falls to the deciding rule or the policy; −N: those that block part of it. Under each cell, what is let through. A zone to itself (faded): only traffic routed between its interfaces passes the firewall. Replies and established connections aren't shown: the built-in lets those through.")));
+}
+
+function detail(c) {
+  const ruleLink = (id, chain, label) => id ? h("a", { href: "#/rules?chain=" + chain + "&rule=" + id, onclick: () => document.querySelector(".modal-back") && document.querySelector(".modal-back").remove() }, label) : label;
+  const v = (verdict) => badge(verdict, { allowed: "ok", blocked: "err", logged: "", unknown: "warn" }[verdict]);
+  const hits = (n) => h("td", { class: "small num" }, n === undefined || n === null ? "–" : n.toLocaleString());
+  const summary = { allowed: "all allowed", blocked: "all blocked", partly: "partly allowed" }[c.summary];
+  modal(c.from + " → " + c.to, h("div", null,
+    h("p", null, h("strong", null, summary), h("span", { class: "muted" }, " · new connections, checked in this order (the " + c.chain + " chain):")),
+    c.note ? h("div", { class: "msg info" }, c.note[0].toUpperCase() + c.note.slice(1) + ".") : null,
+    h("div", { class: "table-wrap" }, h("table", null,
+      h("thead", null, h("tr", null, h("th", null, ""), h("th", null, "What"), h("th", null, "By"), h("th", { class: "num" }, "Hits"))),
+      h("tbody", null,
+        c.exceptions.map((e) => h("tr", null, h("td", null, v(e.verdict)), h("td", null, e.what), h("td", { class: "small" }, ruleLink(e.rule_id, e.chain, e.label)), hits(e.hits))),
+        h("tr", { class: "matrix-rest" }, h("td", null, v(c.verdict)), h("td", null, h("strong", null, c.exceptions.length ? "the rest" : "everything")),
+          h("td", { class: "small" }, ruleLink(c.rule_id, c.chain, c.decided_by)), hits(c.hits))))),
+    h("p", { class: "muted small" }, "Hits: each rule's packets since housecarld started, in all its traffic: a rule can cover several pairs.")),
+    [{ label: "Close", value: null }], { wide: true });
+}

internal/web/static/assets/pages/rules.js

@@ -2,7 +2,7 @@
 import { h, mount, table, modal, errorBox, field, input, select, checkbox, confirmDialog, badge, textarea, toast } from "../ui.js";
 import { get, api, change, can } from "../api.js";
 import { rerender } from "../app.js";
-import { names, matchEditor, describe, hits, counters, listEditor, rowClass, live, fmtBytes, quotas } from "../widgets.js";
+import { names, matchEditor, describe, hits, counters, listEditor, rowClass, live, fmtBytes, quotas, usageBar } from "../widgets.js";
 
 const chains = [
   ["input", "Input", "Traffic to the firewall itself."],
@@ -24,28 +24,33 @@ const builtins = {
 };
 
 export async function render(el, params, query) {
-  const chain = query.get("chain") || "input";
+  // ?rule=ID (from the matrix, the log, the tracer, statistics): show
+  // that rule, in its chain, highlighted.
+  const target = Number(query.get("rule")) || 0;
+  let chain = query.get("chain");
+  if (!chain && target) {
+    try { chain = (await get("/rules/" + target)).chain; } catch { /* gone: the default chain */ }
+  }
+  chain = chain || "input";
   const [all, settings, cs, qs] = await Promise.all([get("/rules", { chain }), get("/settings"), counters(), quotas()]);
   // As nftables checks them: the rules that count traffic first.
-  const rules = [...all.filter(countsTraffic), ...all.filter((r) => !countsTraffic(r))];
+  const rules = [...all.filter(everyPacket), ...all.filter((r) => !everyPacket(r))];
   const write = can("config", true);
   const info = chains.find((c) => c[0] === chain) || chains[0];
   const policy = settings[chain + "_policy"];
-  const bi = (builtins[chain] || []).filter(([flag]) => flag === "" || settings.builtins[flag]);
-  const actionBadge = (r) => badge(r.action + (r.log ? " + log" : ""), { accept: "ok", drop: "err", reject: "warn", log: "" }[r.action]);
+  const actionBadge = (r) => r.raw ? badge("as written", "") : badge(r.action + (r.log ? " + log" : ""), { accept: "ok", drop: "err", reject: "warn", log: "" }[r.action]);
   mount(el,
     h("div", { class: "page-head" },
       h("div", { class: "grow" }, h("h1", null, "Rules"), h("div", { class: "muted" }, "Checked top to bottom; the first rule that decides wins. ", info[2])),
       write ? h("div", { class: "actions" }, h("button", { class: "primary", onclick: () => ruleDialog(null, chain) }, "Add rule")) : null),
     h("div", { class: "tabs" }, chains.map(([k, label]) => h("a", { href: "#/rules?chain=" + k, class: k === chain ? "active" : "" }, label))),
     h("div", { class: "panel" },
-      bi.length ? h("div", { class: "builtins muted small" }, h("strong", null, "Added by Housecarl first: "),
-        bi.map(([, mark, text], i) => [i ? " · " : "", text, mark ? [" (", hits(cs, "b-" + mark), ")"] : ""]), " — ",
-        h("a", { href: "#/settings" }, "settings")) : null,
+      builtinsRow(chain, settings, cs),
       table([
-        { label: "#", render: (r) => r.pending === "deleted" ? "" : String(live(rules).indexOf(r) + 1) },
+        { label: "Order", render: (r) => r.pending === "deleted" ? "" : String(live(rules).indexOf(r) + 1) },
+        { label: "ID", render: (r) => r.id ? h("span", { class: "muted mono small" }, "#" + r.id) : "" },
         { label: "Match", render: (r) => r.raw ? [badge("raw", ""), " ", h("span", { class: "mono small" }, r.raw)]
-          : [countsTraffic(r) ? [badge("every packet", "accent"), " "] : null, describe(r), extras(r, qs[r.id])] },
+          : [everyPacket(r) ? [badge("every packet", "accent"), " "] : null, describe(r), extras(r, qs[r.id])] },
         { label: "Action", render: actionBadge },
         { label: "Hits", render: (r) => hits(cs, "r" + r.id) },
         { label: "Comment", render: (r) => r.comment },
@@ -53,7 +58,7 @@ export async function render(el, params, query) {
           if (r.pending === "deleted") return badge("deleted", "err");
           // Moves stay among the rules of the same kind (counting traffic,
           // or not): nftables checks those first.
-          const rs = live(rules).filter((x) => countsTraffic(x) === countsTraffic(r)), i = rs.indexOf(r);
+          const rs = live(rules).filter((x) => everyPacket(x) === everyPacket(r)), i = rs.indexOf(r);
           return [
           h("button", { class: "small reveal", title: "Move up", disabled: i === 0, onclick: () => move(r, { before: rs[i - 1].id }) }, "↑"), " ",
           h("button", { class: "small reveal", title: "Move down", disabled: i === rs.length - 1, onclick: () => move(r, { after: rs[i + 1].id }) }, "↓"), " ",
@@ -62,15 +67,78 @@ export async function render(el, params, query) {
           h("button", { class: "small reveal", onclick: () => del(r) }, "Delete"),
         ];
         } } : null,
-      ].filter(Boolean), rules, { empty: "No rules in this chain yet.", rowClass }),
-      rules.some(countsTraffic) ? h("div", { class: "muted small" }, "Rules marked “every packet” have a quota or a byte rate: they are checked before established connections are accepted, so they count all of a connection's traffic, not only its first packet. They can drop, reject or log, not accept.") : null,
+      ].filter(Boolean), rules, { empty: "No rules in this chain yet.", rowClass: (r) => rowClass(r) + (target && r.id === target ? " row-target" : "") }),
+      rules.some(everyPacket) ? h("div", { class: "muted small" }, "Rules marked “every packet” are checked before established connections are accepted: they see all of a connection's traffic, not only its first packet, so a quota counts all of it and a block stops connections already open. They can drop, reject or log, not accept.") : null,
       h("div", { class: "policy small" }, "Then the policy: ", badge(policy, policy === "drop" ? "err" : "ok"),
         policy === "drop" ? [" — ", hits(cs, "p-" + chain), " packets reached it"] : null)));
+  if (target) {
+    const row = el.querySelector("tr.row-target");
+    if (row) requestAnimationFrame(() => row.scrollIntoView({ block: "center" }));
+    else el.querySelector(".tabs").after(h("div", { class: "msg warn" }, "There is no rule #" + target + " (any more)."));
+  }
 }
 
-// countsTraffic: a quota or a byte rate (as model.Rule.CountsTraffic).
-function countsTraffic(r) {
-  return !r.raw && (!!r.quota || !!(r.limit && ["bytes", "kbytes", "mbytes"].includes(r.limit.unit)));
+// everyPacket: checked before established connections are accepted (as
+// model.Rule.ChecksEveryPacket): asked to, or a quota or a byte rate.
+function everyPacket(r) {
+  return !r.raw && (!!r.every_packet || !!r.quota || !!(r.limit && ["bytes", "kbytes", "mbytes"].includes(r.limit.unit)));
+}
+
+// What each built-in does (a switch covers all its rules).
+const builtinText = {
+  loopback: "Accept everything on lo (the firewall talking to itself)",
+  established: "Accept established and related connections: replies, and the rest of connections a rule let in",
+  drop_invalid: "Drop invalid packets: what connection tracking can't place",
+  icmpv6: "Accept the ICMPv6 that IPv6 needs: errors, echo, neighbour discovery, MLD (RFC 4890)",
+  ping_internal: "Accept ping to the firewall from internal zones",
+  dhcpv6_client: "Accept DHCPv6 replies on external zones (your ISP's prefix delegation)",
+  antispoof: "Drop sources that can't come from outside, on external zones (Settings: the lists and your prefixes)",
+  mss_clamp: "Clamp TCP's MSS to the route's MTU on external zones (needed with PPPoE)",
+};
+
+// builtinsRow sums up the chain's built-ins in one line, with a button to
+// see and switch them.
+function builtinsRow(chain, settings, cs) {
+  const list = builtins[chain] || [];
+  const on = list.filter(([flag]) => flag === "" || settings.builtins[flag]);
+  if (!list.length) return null;
+  return h("div", { class: "builtins muted small" },
+    on.length ? (on.length === 1 ? "Housecarl adds 1 rule before yours. " : "Housecarl adds " + on.length + " rules before yours. ") : "Housecarl adds no rules before yours here. ",
+    h("button", { class: "small", onclick: () => builtinsDialog(chain, settings, cs) }, "Built-ins…"));
+}
+
+// builtinsDialog lists the chain's built-ins: what each does, its hits,
+// and a switch (a setting, so it covers every chain the built-in is in).
+async function builtinsDialog(chain, settings, cs) {
+  const list = builtins[chain] || [];
+  const flags = [...new Set(list.map(([flag]) => flag))];
+  const write = can("config", true);
+  const boxes = {};
+  const rows = flags.map((flag) => {
+    const marks = list.filter(([f]) => f === flag).map(([, mark]) => mark).filter(Boolean);
+    const box = flag ? h("input", { type: "checkbox", checked: settings.builtins[flag], disabled: !write }) : null;
+    if (box) boxes[flag] = box;
+    return { flag, marks, box, text: flag ? builtinText[flag] : list.find(([f]) => f === flag)[2] };
+  });
+  const err = h("div");
+  const body = h("div", null,
+    h("p", { class: "muted" }, "Checked in this order, before your rules (rules with a quota or a byte rate come before the established built-in, so they see every packet). Each is a setting: switching one changes every chain it is in."),
+    table([
+      { label: "On", render: (x) => x.box || h("span", { class: "muted small" }, "always") },
+      { label: "Built-in", render: (x) => x.flag ? x.text : [x.text, h("div", { class: "muted small" }, "Part of each port forward.")] },
+      { label: "Hits", render: (x) => x.marks.length ? hits(cs, ...x.marks.map((m) => "b-" + m)) : h("span", { class: "muted" }, "–") },
+    ], rows),
+    err);
+  await modal("Built-ins of the " + chain + " chain", body, write ? [
+    { label: "Cancel", value: null },
+    { label: "Save", kind: "primary", onclick: async () => {
+      mount(err);
+      const b = {};
+      for (const [flag, box] of Object.entries(boxes)) if (box.checked !== settings.builtins[flag]) b[flag] = box.checked;
+      if (!Object.keys(b).length) return true;
+      try { await change("PATCH", "/settings", { builtins: b }); rerender(); return true; } catch (e) { mount(err, errorBox(e)); return false; }
+    } },
+  ] : [{ label: "Close", value: null }], { wide: true });
 }
 
 function extras(r, q) {
@@ -116,16 +184,10 @@ async function usageDialog(r, q) {
     table([
       { label: "Address", render: (a) => h("span", { class: "mono" }, a.address) },
       { label: "Used", render: (a) => fmtBytes(a.used) },
-      { label: "", render: (a) => h("div", { class: "usage" }, h("div", { class: "usage-bar" + (a.used >= r.quota.bytes ? " full" : "") }, bar(a.used / r.quota.bytes))) },
+      { label: "", render: (a) => usageBar(a.used, r.quota.bytes) },
     ], q.addresses)), [{ label: "Close", value: null }], { wide: true });
 }
 
-function bar(f) {
-  const seg = h("div", { class: "seg leased" });
-  seg.style.width = Math.min(100, f * 100) + "%";
-  return seg;
-}
-
 async function resetQuota(r) {
   if (!await confirmDialog("Reset the quota", "Start the quota of " + (r.comment || "rule #" + r.id) + " again now" +
     (r.quota.per_source || r.quota.per_destination ? ", for every address" : "") + "? This is done at once, not staged.", "Reset")) return;
@@ -149,11 +211,15 @@ async function del(r) {
 export async function ruleDialog(r, chain) {
   const n = await names();
   r = r || { chain, action: "accept" };
-  const ch = select([["input", "input (to the firewall)"], ["forward", "forward (through it)"], ["output", "output (from it)"], ["early", "early (before conntrack)"]], r.chain || chain);
+  const ch = select(["input", "forward", "output", "early"], r.chain || chain);
   const match = matchEditor(r, n);
   const action = select([["accept", "accept"], ["drop", "drop"], ["reject", "reject"], ["log", "log only (and go on)"]], r.action);
   const rejectWith = select([["", "with an ICMP error"], ["tcp-reset", "with a TCP reset (TCP rules only)"]], r.reject_with || "");
-  const showReject = () => rejectWith.classList.toggle("hidden", action.value !== "reject");
+  const every = checkbox("every packet: also stop connections already open, not just new ones", r.every_packet);
+  const showReject = () => {
+    rejectWith.classList.toggle("hidden", action.value !== "reject");
+    every.classList.toggle("hidden", action.value === "accept");
+  };
   action.addEventListener("change", showReject);
   showReject();
   const states = listEditor(r.ct_state, { suggestions: ["new", "established", "related", "invalid", "untracked"], placeholder: "new", mono: false, addLabel: "Add state" });
@@ -190,27 +256,95 @@ export async function ruleDialog(r, chain) {
   const comment = input({ value: r.comment || "" });
   const disabled = checkbox("disabled: keep the rule but don't use it", r.disabled);
   const err = h("div");
+
+  // The parts most rules don't need are behind "+" buttons: open when the
+  // rule uses them, closed (and unset) with ✕.
+  const adders = h("div", { class: "adders" });
+  const sections = {};
+  const section = (key, label, open, fields) => {
+    const nodes = fields.flat().filter(Boolean);
+    const sec = { open };
+    const add = h("button", { type: "button", class: "small", onclick: () => show(true) }, "+ " + label);
+    const close = h("button", { type: "button", class: "small close-section", title: "Remove " + label.toLowerCase(), onclick: () => show(false) }, "✕");
+    nodes[0].append(" ", close);
+    const show = (on) => {
+      sec.open = on;
+      for (const n of nodes) n.classList.toggle("hidden", !on);
+      add.classList.toggle("hidden", on);
+      if (sec.onShow) sec.onShow(on);
+      if (on) { const f = nodes.find((n) => n.querySelector && n.querySelector("input, select, textarea")); if (f) f.querySelector("input, select, textarea").focus(); }
+      updateGroup();
+    };
+    adders.append(add, " ");
+    sections[key] = sec;
+    sec.nodes = nodes;
+    sec.init = () => show(open);
+    return nodes;
+  };
+  const stateFields = section("state", "Connection state", (r.ct_state || []).length > 0 || (r.ct_status || []).length > 0, [
+    field("Connection state", states, "new: a connection's first packet; established: its later ones; related: packets that belong to another connection (ICMP errors, FTP data); invalid: what connection tracking can't place. Empty: any."),
+    field("Connection status", statuses, "dnat: the connection was port forwarded; snat: its source was translated. Empty: any."),
+  ]);
+  const timeFields = section("time", "Time window", (r.days || []).length > 0 || !!r.hour_from, [
+    field("Days", days),
+    field("Hours", h("div", null, hourFrom, " to ", hourTo), "22:00 to 06:00 passes midnight. In the firewall's local time."),
+  ]);
+  const rateFields = section("rate", "Rate limit", !!r.limit, [
+    field("Rate limit", h("div", null, rate, " ", unit, " per ", per, " ", burst, " ", limitKey, over),
+      "In bytes, it is a bandwidth (KB or MB per second or minute); traffic beyond it is dropped, not queued. The burst is in the same unit."),
+  ]);
+  const quotaFields = section("quota", "Quota", !!r.quota, [
+    field("Quota", h("div", null, quotaAmount, " ", quotaUnit, " ", quotaKey, " resets ", quotaReset, " ", quotaDay, quotaOver),
+      "An allowance of traffic; Housecarl keeps the count across changes and restarts."),
+  ]);
+  const connFields = section("conn", "Connection limit", !!r.conn_limit, [
+    field("Connection limit", conn, "Match a source that already has more than this many connections."),
+  ]);
+  const rawFields = section("raw", "Write it in nftables", !!r.raw, [
+    field("Raw nftables", raw, "One nftables rule, for what Housecarl can't say, used as written: it replaces the zones, addresses, services, action, limits and log (write them in it). Checked by nftables."),
+  ]);
+  // Grouping only means something with a per-address limit, quota or
+  // connection limit.
+  const groupFields = field("Group addresses", h("div", null, "IPv4 /", group4, " IPv6 /", group6),
+    "Per-address limits, connection limits and quotas count addresses by these prefixes. Default: each IPv4 address, IPv6 by /64 (one device has several addresses).").filter(Boolean);
+  function updateGroup() {
+    if (!sections.raw) return; // still building
+    const on = (sections.rate.open && limitKey.value !== "") || (sections.quota.open && quotaKey.value !== "") || sections.conn.open;
+    for (const n of groupFields) n.classList.toggle("hidden", !on);
+  }
+  limitKey.addEventListener("change", updateGroup);
+  quotaKey.addEventListener("change", updateGroup);
+
+  // Written in nftables, a rule is its text: the fields it replaces go
+  // (and the other parts close).
+  const matchNodes = match.flat().filter(Boolean);
+  const actionField = field("Action", h("div", null, action, " ", rejectWith, every),
+    "Every packet: checked before established connections are accepted, so blocking takes effect on what is already connected.").filter(Boolean);
+  const logField = field("Log", h("div", null, log, logPrefix)).filter(Boolean);
+  const addersLabel = h("label");
+  sections.raw.onShow = (on) => {
+    for (const n of [...matchNodes, ...actionField, ...logField]) n.classList.toggle("hidden", on);
+    for (const [k, sec] of Object.entries(sections)) {
+      if (k === "raw") continue;
+      if (on && sec.open) for (const n of sec.nodes) n.classList.add("hidden");
+      if (on) sec.open = false;
+    }
+    for (const b of adders.children) if (!b.textContent.includes("nftables")) b.classList.toggle("hidden", on);
+    updateGroup();
+  };
   const body = h("div", null,
     h("div", { class: "form-grid" },
       field("Chain", ch),
-      ...match.flat(),
-      field("Action", h("div", null, action, " ", rejectWith)),
-      field("Connection state", states),
-      field("Connection status", statuses, "dnat: the connection was port forwarded."),
-      field("Days", days),
-      field("Hours", h("div", null, hourFrom, " to ", hourTo), "22:00 to 06:00 passes midnight."),
-      field("Rate limit", h("div", null, rate, " ", unit, " per ", per, " ", burst, " ", limitKey, over),
-        "Empty: no limit. In bytes, it is a bandwidth (KB or MB per second or minute); traffic beyond it is dropped, not queued. The burst is in the same unit."),
-      field("Quota", h("div", null, quotaAmount, " ", quotaUnit, " ", quotaKey, " resets ", quotaReset, " ", quotaDay, quotaOver),
-        "Empty: none. An allowance of traffic; Housecarl keeps the count across changes and restarts."),
-      field("Group addresses", h("div", null, "IPv4 /", group4, " IPv6 /", group6),
-        "Per-address limits, connection limits and quotas count addresses by these prefixes. Default: each IPv4 address, IPv6 by /64 (one device has several addresses)."),
-      field("Connection limit", conn, "Match a source that already has more than this many connections."),
-      field("Log", h("div", null, log, logPrefix)),
-      field("Raw nftables", raw, "Instead of all the above: one nftables rule, for what Housecarl can't say. Checked by nftables."),
+      ...matchNodes,
+      ...actionField,
+      ...stateFields, ...timeFields, ...rateFields, ...quotaFields, ...connFields, ...groupFields, ...rawFields,
+      addersLabel, adders,
+      ...logField,
       field("Comment", comment),
       r.id ? [h("label"), disabled] : null),
     err);
+  for (const sec of Object.values(sections)) sec.init();
+  updateGroup();
   await modal(r.id ? "Edit rule" : "Add rule", body, [
     { label: "Cancel", value: null },
     { label: "Check", onclick: () => save(true) },
@@ -219,15 +353,21 @@ export async function ruleDialog(r, chain) {
 
   async function save(dry) {
     mount(err);
-    const v = { chain: ch.value, ...match.value(), action: action.value, reject_with: action.value === "reject" ? rejectWith.value : "",
-      ct_state: states.value(), ct_status: statuses.value(), days: days.value(), hour_from: hourFrom.value.trim(), hour_to: hourTo.value.trim(),
-      limit: rate.value ? { rate: parseInt(rate.value, 10), unit: unit.value, per: per.value, burst: parseInt(burst.value || "0", 10), over: over.input.checked,
+    const on = (k) => sections[k].open;
+    const isRaw = on("raw");
+    const m = isRaw ? { in_zones: [], not_in: false, out_zones: [], not_out: false, sources: [], not_sources: false, destinations: [], not_destinations: false, services: [], family: "" } : match.value();
+    const v = { chain: ch.value, ...m, action: isRaw ? "accept" : action.value, reject_with: !isRaw && action.value === "reject" ? rejectWith.value : "",
+      ct_state: on("state") ? states.value() : [], ct_status: on("state") ? statuses.value() : [],
+      days: on("time") ? days.value() : [], hour_from: on("time") ? hourFrom.value.trim() : "", hour_to: on("time") ? hourTo.value.trim() : "",
+      limit: on("rate") && rate.value ? { rate: parseInt(rate.value, 10), unit: unit.value, per: per.value, burst: parseInt(burst.value || "0", 10), over: over.input.checked,
         per_source: limitKey.value === "source", per_destination: limitKey.value === "destination" } : null,
-      quota: quotaAmount.value ? { bytes: parseInt(quotaAmount.value, 10) * (quotaUnit.value === "gb" ? 1 << 30 : 1 << 20), over: quotaOver.input.checked,
+      quota: on("quota") && quotaAmount.value ? { bytes: parseInt(quotaAmount.value, 10) * (quotaUnit.value === "gb" ? 1 << 30 : 1 << 20), over: quotaOver.input.checked,
         per_source: quotaKey.value === "source", per_destination: quotaKey.value === "destination",
         reset: quotaReset.value, day: quotaReset.value === "monthly" ? parseInt(quotaDay.value || "1", 10) : 0 } : null,
-      group_v4: parseInt(group4.value || "0", 10), group_v6: parseInt(group6.value || "0", 10),
-      conn_limit: parseInt(conn.value || "0", 10), log: log.input.checked, log_prefix: logPrefix.value.trim(), raw: raw.value.trim(), comment: comment.value.trim() };
+      group_v4: isRaw ? 0 : parseInt(group4.value || "0", 10), group_v6: isRaw ? 0 : parseInt(group6.value || "0", 10),
+      conn_limit: on("conn") ? parseInt(conn.value || "0", 10) : 0,
+      every_packet: !isRaw && action.value !== "accept" && every.input.checked, log: !isRaw && log.input.checked, log_prefix: isRaw ? "" : logPrefix.value.trim(),
+      raw: on("raw") ? raw.value.trim() : "", comment: comment.value.trim() };
     if (r.id) v.disabled = disabled.input.checked;
     try {
       await change(r.id ? "PATCH" : "POST", r.id ? "/rules/" + r.id : "/rules", v, { query: dry ? { dry_run: 1 } : undefined });

internal/web/static/assets/pages/sets.js

@@ -31,19 +31,29 @@ async function setDialog(s) {
   s = s || { entries: [] };
   const name = input({ value: s.name || "", placeholder: "trusted" });
   const list = listEditor(s.entries, { placeholder: "192.168.1.0/24, 2001:db8:1::/64, 192.168.1.100-192.168.1.199", addLabel: "Add entry" });
-  const dynamic = checkbox("dynamic: entries are added at run time", s.dynamic);
+  const dynamic = checkbox("dynamic: entries are added at run time (the Entries button, hc set entries, a program with a token), each for a time", s.dynamic);
+  // A dynamic set has no configured entries; only it has a timeout.
+  const entryFields = field("Entries", list).filter(Boolean);
+  const timeoutFields = field("Entry timeout", timeout, "How long an entry stays unless it is added with its own. Empty: until removed (or a reboot).").filter(Boolean);
+  const showKind = () => {
+    for (const n of entryFields) n.classList.toggle("hidden", dynamic.input.checked);
+    for (const n of timeoutFields) n.classList.toggle("hidden", !dynamic.input.checked);
+  };
+  dynamic.input.addEventListener("change", showKind);
+  showKind();
   const timeout = input({ value: s.timeout ? fmtSeconds(s.timeout) : "", placeholder: "1h, 1d (none: until removed)", class: "short" });
   const comment = input({ value: s.comment || "" });
   const err = h("div");
   await modal(s.id ? "Edit set " + s.name : "Add set", h("div", null, h("div", { class: "form-grid" },
-    field("Name", name), field("Entries", list), h("label"), dynamic, field("Entry timeout", timeout, "A dynamic set's default."), field("Comment", comment)), err), [
+    field("Name", name), ...entryFields, h("label"), dynamic, ...timeoutFields, field("Comment", comment)), err), [
     { label: "Cancel", value: null },
     { label: s.id ? "Save" : "Add set", kind: "primary", onclick: async () => {
       mount(err);
       const t = parseSeconds(timeout.value);
       if (t === null) { mount(err, h("div", { class: "msg err" }, "Write the timeout as 1h, 30m or 1d.")); return false; }
       try {
-        await change(s.id ? "PATCH" : "POST", s.id ? "/sets/" + s.id : "/sets", { name: name.value.trim(), entries: list.value(), dynamic: dynamic.input.checked, timeout: t, comment: comment.value.trim() });
+        await change(s.id ? "PATCH" : "POST", s.id ? "/sets/" + s.id : "/sets", { name: name.value.trim(), entries: dynamic.input.checked ? [] : list.value(), dynamic: dynamic.input.checked,
+          timeout: dynamic.input.checked ? t : 0, comment: comment.value.trim() });
         rerender();
         return true;
       } catch (e) { mount(err, errorBox(e)); return false; }

internal/web/static/assets/pages/settings.js

@@ -27,6 +27,8 @@ export async function render(el) {
   const v6 = listEditor(s.antispoof_v6, { addLabel: "Add" });
   const flow = listEditor(s.flowtable_devices, { suggestions: ifs.map((i) => i.name), placeholder: "wan", addLabel: "Add device" });
   const probation = input({ value: fmtSeconds(s.probation_seconds || 120), class: "short" });
+  const perDevice = checkbox("count traffic per inside device (IPv6 per /64), for the Statistics page's top devices: a counter for each address talking to an external zone", s.stats_per_device);
+  const manual = checkbox("a person confirms each change (otherwise the browser or hc that made it confirms it by itself, once it reaches Housecarl again over a new connection)", s.manual_confirm);
   const err = h("div");
   const save = async (dry) => {
     mount(err);
@@ -37,7 +39,7 @@ export async function render(el) {
     try {
       const r = await change("PATCH", "/settings", { input_policy: input_.value, forward_policy: forward.value, output_policy: output.value,
         log_policy_drops: logDrops.input.checked, builtins, own_prefixes: own.value(), antispoof_v4: v4.value(), antispoof_v6: v6.value(),
-        flowtable_devices: flow.value(), probation_seconds: p === 120 ? 0 : p }, { query: dry ? { dry_run: 1 } : undefined });
+        flowtable_devices: flow.value(), probation_seconds: p === 120 ? 0 : p, manual_confirm: manual.input.checked, stats_per_device: perDevice.input.checked }, { query: dry ? { dry_run: 1 } : undefined });
       if (!dry && (r.applied || r.staged)) rerender();
     } catch (e) { mount(err, errorBox(e)); }
   };
@@ -53,8 +55,10 @@ export async function render(el) {
       field("Your prefixes", own, "Your public prefixes (your delegated IPv6 /48): they can't come from outside."),
       field("Antispoof IPv4", v4),
       field("Antispoof IPv6", v6),
-      field("Flowtable", flow, "Forwarded TCP and UDP between these devices skip the rules once established (fast path). Exact names; none: off."),
-      field("Probation", probation, "How long a change made here has to be confirmed before it is rolled back.")),
+      field("Flowtable", flow, "Forwarded TCP and UDP between these devices skip the rules once established (fast path): for a CPU that can't keep up with the line. Forward quotas, byte rates and traffic per device then only count first packets (docs/flowtable.md). Exact names, both sides (wan and lan); none: off."),
+      field("Probation", probation, "How long a change made over the network has to be confirmed before it is rolled back."),
+      h("label"), manual,
+      field("Statistics", perDevice)),
     err,
     write ? h("div", { class: "actions form-actions" },
       h("button", { class: "primary", onclick: () => save(false) }, "Save"),

internal/web/static/assets/pages/stats.js

@@ -0,0 +1,214 @@
+// Statistics: traffic per zone, drops, blocked sources, devices,
+// connections, quotas and the machine over time (docs/stats.md).
+import { h, mount, fmtTime, errorBox } from "../ui.js";
+import { get } from "../api.js";
+import { lineChart, chartCard, legend, timeTable, heatmap, heatmapTable, barList, compact, stepSeconds, endLabelWidth } from "../charts.js";
+
+const ranges = [["1h", "1 hour"], ["6h", "6 hours"], ["24h", "24 hours"], ["7d", "7 days"], ["30d", "30 days"], ["90d", "90 days"], ["365d", "a year"]];
+const rangeSecs = { "1h": 3600, "6h": 6 * 3600, "24h": 86400, "7d": 7 * 86400, "30d": 30 * 86400, "90d": 90 * 86400, "365d": 365 * 86400 };
+
+let generation = 0;
+// The zone the traffic chart shows, kept while the page refreshes.
+let shownZone = null;
+
+export async function render(el, params, query) {
+  const gen = ++generation;
+  const want = query && query.get ? query.get("since") : null;
+  const since = rangeSecs[want] ? want : "24h";
+  const status = await get("/stats/status");
+  const head = h("div", { class: "page-head" },
+    h("div", { class: "grow" }, h("h1", null, "Statistics"),
+      h("div", { class: "muted" }, status.enabled
+        ? "Sampled every minute · data since " + (status.first_data ? fmtTime(status.first_data) : "–") +
+          " · kept: minutes " + status.retention.minute + ", hours " + status.retention.hour + ", days " + status.retention.day
+        : "Statistics are off")));
+  if (!status.enabled) {
+    mount(el, head, h("div", { class: "panel" },
+      h("p", null, "Housecarl keeps traffic per zone, drops, blocked sources, connections and quotas over time, in a database of its own. Turn it on in /etc/housecarl/housecarl.yaml and restart housecarl:"),
+      h("pre", { class: "code" }, "stats:\n  enabled: true"),
+      h("p", { class: "muted" }, "See docs/stats.md.")));
+    return;
+  }
+  const heatSince = rangeSecs[since] < 7 * 86400 ? "7d" : rangeSecs[since] > 90 * 86400 ? "90d" : since;
+  let d;
+  try {
+    d = await loadAll(since, heatSince);
+  } catch (e) {
+    mount(el, head, rangeRow(since), errorBox(e));
+    return;
+  }
+  if (gen !== generation) return; // the user moved on
+  const s = d.summary;
+  const rangeLabel = "in the last " + ranges.find((r) => r[0] === since)[1];
+  const parts = [head, rangeRow(since)];
+  if (status.sample_error) parts.push(h("div", { class: "msg warn" }, "The last sample failed: " + status.sample_error));
+
+  // Headline numbers: what came from and went to the external zones.
+  const ext = new Set(d.zones.filter((z) => z.external).map((z) => z.name));
+  const sum = (f) => s.zones.filter((z) => ext.has(z.name)).reduce((a, z) => a + f(z), 0);
+  const l = status.latest || {};
+  parts.push(h("div", { class: "cards" },
+    ext.size ? tile("Downloaded", bytes(sum((z) => z.received)), "from external zones, " + rangeLabel) : null,
+    ext.size ? tile("Uploaded", bytes(sum((z) => z.sent)), "to external zones") : null,
+    tile("Dropped", compact(s.drops), "packets: rules, built-ins and policies"),
+    s.conntrack_max ? tile("Connections", compact(s.peak_connections), "most at once (of " + compact(s.conntrack_max) + " allowed)" + (l.conntrack != null ? " · now " + compact(l.conntrack) : "")) : null));
+
+  // Traffic per zone: one zone at a time, received and sent.
+  const traffic = complete(d.traffic);
+  const step = stepSeconds[traffic.step];
+  const zoneNames = s.zones.map((z) => z.name);
+  if (!zoneNames.includes(shownZone)) shownZone = zoneNames.find((z) => ext.has(z)) || zoneNames[0] || null;
+  const trafficBox = h("div");
+  const drawTraffic = () => {
+    const pick = h("div", { class: "segmented" }, zoneNames.map((z) =>
+      h("button", { type: "button", class: z === shownZone ? "active" : "", onclick: () => { shownZone = z; drawTraffic(); } }, z)));
+    const series = [
+      { label: "Received", values: values(traffic, "zrx:" + shownZone), slot: 1 },
+      { label: "Sent", values: values(traffic, "ztx:" + shownZone), slot: 2 },
+    ];
+    mount(trafficBox, chartCard("Traffic" + (shownZone ? ": " + shownZone : ""),
+      "bytes per " + stepName(traffic.step) + " received from and sent to the zone's interfaces, " + rangeLabel,
+      shownZone ? h("div", null, zoneNames.length > 1 ? h("div", { class: "filter-row" }, pick) : null,
+        lineChart({ series, times: traffic.times, step, area: true, gutter: { left: 64, right: endLabelWidth(series) }, format: bytes }))
+        : h("div", { class: "empty" }, "No traffic recorded yet: interfaces count once they are in a zone."),
+      shownZone ? timeTable(series, traffic.times, step, bytes) : null, { wide: true, legend: shownZone ? legend(series) : null }));
+  };
+  drawTraffic();
+  parts.push(trafficBox);
+  if (s.zones.length) {
+    parts.push(h("div", { class: "chart-grid" },
+      listCard("Received per zone", rangeLabel, barList(s.zones.map((z) => ({ key: z.name, count: z.received })), { format: bytes })),
+      listCard("Sent per zone", rangeLabel, barList(s.zones.map((z) => ({ key: z.name, count: z.sent })), { format: bytes }))));
+  }
+
+  // Drops over time, by chain, and when they happen.
+  const drops = complete(d.drops);
+  const dSeries = [{ label: "Dropped", values: values(drops, "drops"), slot: 2 }];
+  const anyDrops = dSeries[0].values.some((v) => v > 0);
+  parts.push(chartCard("Dropped packets", "per " + stepName(drops.step) + ", by rules, built-ins and policies that drop, " + rangeLabel,
+    anyDrops ? lineChart({ series: dSeries, times: drops.times, step: stepSeconds[drops.step], area: true, gutter: { left: 64, right: 8 } }) : h("div", { class: "empty" }, "Nothing dropped " + rangeLabel + "."),
+    anyDrops ? timeTable(dSeries, drops.times, stepSeconds[drops.step]) : null, { wide: true }));
+  parts.push(chartCard("When drops happen", "average dropped packets per hour by weekday and hour of day (local time), last " + ranges.find((r) => r[0] === heatSince)[1],
+    heatmap(d.heat, "packets"), heatmapTable(d.heat, "packets"), { wide: true }));
+
+  // Who and what.
+  const unused = d.hits.filter((e) => e.count === 0 && e.key.startsWith("r"));
+  const devHint = h("div", { class: "empty" }, status.per_device
+    ? "No traffic per device " + rangeLabel + "."
+    : ["Traffic per device is off. Turn it on under ", h("a", { href: "#/settings" }, "Settings"), " (it adds a counter for each inside address)."]);
+  parts.push(h("div", { class: "chart-grid" },
+    listCard("Busiest rules", "packets each counted, " + rangeLabel, barList(s.rules, { label: (k) => k, href: (k) => ruleHref(k) })),
+    listCard("Dropped, by chain", "early: antispoofing · invalid: packets conntrack can't place", barList(s.drops_by_chain, { empty: "Nothing dropped " + rangeLabel + "." })),
+    listCard("Blocked most", "sources of logged drops (the policies' with 'log drops' on, and rules that log)", barList(s.blocked_sources, { empty: "No logged drops " + rangeLabel + ".", href: (k) => "#/trace?address=" + encodeURIComponent(k) })),
+    listCard("What they tried", "the logged drops' protocol and port", barList(s.blocked_ports, { empty: "No logged drops " + rangeLabel + "." })),
+    listCard("Devices, received", "bytes from external zones, by inside address (IPv6 per /64)", s.devices_down.length ? barList(s.devices_down, { format: bytes, href: connHref }) : devHint),
+    listCard("Devices, sent", "bytes to external zones", s.devices_up.length ? barList(s.devices_up, { format: bytes, href: connHref }) : devHint.cloneNode(true)),
+    d.quotas.length ? listCard("Quotas", "bytes counted against each rule's quota, " + rangeLabel, barList(d.quotas, { format: bytes, label: (k) => "rule #" + k.slice(1), href: () => "#/rules" })) : null,
+    listCard("Rules that matched nothing", "in the last 30 days (or since statistics started): candidates to remove",
+      unused.length ? h("div", null, h("ul", { class: "plain-list" }, unused.slice(0, 12).map((e) => h("li", null, h("a", { href: ruleHref(e.key) }, e.name || e.key)))),
+        unused.length > 12 ? h("div", { class: "muted small" }, "and " + (unused.length - 12) + " more: hc stats rules --unused") : null) : h("div", { class: "empty" }, "Every rule matched something."))));
+
+  // Connections and the machine.
+  const g = complete(d.gauges);
+  const gStep = stepSeconds[g.step];
+  const gauge = (k, f = (v) => v) => values(g, k).map((v) => (v == null || v < 0 ? null : f(v)));
+  const any = (ss) => ss.some((x) => x.values.some((v) => v != null));
+  const cSeries = [{ label: "Tracked", values: gauge("g:conntrack"), slot: 1 }];
+  const mSeries = [{ label: "Host in use", values: gauge("g:mem"), slot: 1 }, { label: "housecarld", values: gauge("g:rss"), slot: 2 }];
+  const lSeries = [{ label: "Load", values: gauge("g:load", (v) => v / 100), slot: 1 }, { label: "housecarld CPU %", values: gauge("g:cpu", (v) => v / 10), slot: 2 }];
+  const gutter = { left: 64, right: Math.max(endLabelWidth(cSeries), endLabelWidth(mSeries), endLabelWidth(lSeries)) };
+  const empty = h("div", { class: "empty" }, "No samples yet: they are taken once a minute.");
+  parts.push(h("h2", { class: "section" }, "Connections and the machine"));
+  parts.push(chartCard("Tracked connections", "average per " + stepName(g.step) + (l.conntrack_max ? "; the kernel allows " + compact(l.conntrack_max) + " (nf_conntrack_max), and drops new connections beyond" : ""),
+    any(cSeries) ? lineChart({ series: cSeries, times: g.times, step: gStep, area: true, gutter }) : empty, any(cSeries) ? timeTable(cSeries, g.times, gStep) : null, { wide: true }));
+  parts.push(chartCard("Memory", "average per " + stepName(g.step) + (l.rss ? " · housecarld now " + bytes(l.rss) : ""),
+    any(mSeries) ? lineChart({ series: mSeries, times: g.times, step: gStep, gutter, format: bytes }) : empty.cloneNode(true), any(mSeries) ? timeTable(mSeries, g.times, gStep, bytes) : null, { wide: true, legend: legend(mSeries) }));
+  parts.push(chartCard("Load", "the host's 1-minute load average, and housecarld's CPU (percent of one CPU)",
+    any(lSeries) ? lineChart({ series: lSeries, times: g.times, step: gStep, gutter, format: (v) => (Math.round(v * 100) / 100).toString() }) : empty.cloneNode(true),
+    any(lSeries) ? timeTable(lSeries, g.times, gStep, (v) => v.toFixed(2)) : null, { wide: true, legend: legend(lSeries) }));
+  el.classList.remove("refreshing");
+  mount(el, parts);
+  scheduleRefresh(el, params, query, gen);
+}
+
+async function loadAll(since, heatSince) {
+  const [summary, zones, traffic, drops, heat, hits, gauges, quotas] = await Promise.all([
+    get("/stats/summary", { since, limit: 10 }),
+    get("/zones"),
+    get("/stats/timeline", { since, keys: "zrx:*,ztx:*" }),
+    get("/stats/timeline", { since, keys: "drops" }),
+    get("/stats/heatmap", { since: heatSince }),
+    get("/stats/hits", { since: "30d" }),
+    get("/stats/timeline", { since, keys: "g:conntrack,g:conntrack_max,g:mem,g:rss,g:load,g:cpu" }),
+    get("/stats/timeline", { since, keys: "quota:*" }),
+  ]);
+  return { summary, zones, traffic, drops, heat, hits, gauges,
+    quotas: quotas.series.map((x) => ({ key: x.key.slice(6), count: x.values.reduce((a, v) => a + Math.max(v, 0), 0) })).filter((e) => e.count > 0).sort((a, b) => b.count - a.count) };
+}
+
+function scheduleRefresh(el, params, query, gen) {
+  setTimeout(() => {
+    const hh = location.hash;
+    if (gen !== generation || !(hh === "#/stats" || hh.startsWith("#/stats?")) || !el.isConnected) return;
+    el.classList.add("refreshing");
+    render(el, params, query).catch(() => el.classList.remove("refreshing"));
+  }, 60_000);
+}
+
+// complete drops the last bucket of a timeline if it hasn't ended yet:
+// drawn, it would look like a sudden fall.
+function complete(tl) {
+  const step = stepSeconds[tl.step];
+  const n = tl.times.length;
+  if (n > 1 && tl.times[n - 1] + step > Date.now() / 1000) {
+    return { ...tl, times: tl.times.slice(0, -1), series: tl.series.map((x) => ({ ...x, values: x.values.slice(0, -1) })) };
+  }
+  return tl;
+}
+
+function values(tl, key) {
+  return (tl.series.find((x) => x.key === key) || { values: tl.times.map(() => 0) }).values;
+}
+
+function rangeRow(since) {
+  return h("div", { class: "filter-row", role: "group", "aria-label": "Time range" },
+    h("span", { class: "muted" }, "Range"),
+    h("div", { class: "segmented" }, ranges.map(([k, label]) =>
+      h("a", { href: "#/stats?since=" + k, class: k === since ? "active" : "", "aria-current": k === since ? "true" : null, title: "last " + label }, k === "365d" ? "1y" : k))));
+}
+
+// ruleHref: a counter's mark to its page ("r12" → the rules, "b-…" → the
+// built-ins under Settings).
+function ruleHref(mark) {
+  if (/^r\d+$/.test(mark)) return "#/rules?rule=" + mark.slice(1);
+  if (mark.startsWith("b-") || mark.startsWith("p-")) return "#/settings";
+  if (mark.startsWith("f")) return "#/nat";
+  return null;
+}
+
+function connHref(addr) {
+  return "#/connections?q=" + encodeURIComponent(addr.replace(/::\/64$/, ":"));
+}
+
+// bytes formats a size in binary units.
+export function bytes(n) {
+  const units = ["B", "KiB", "MiB", "GiB", "TiB"];
+  let i = 0;
+  while (n >= 1024 && i < units.length - 1) {
+    n /= 1024;
+    i++;
+  }
+  return (i === 0 || n >= 100 ? Math.round(n) : n.toFixed(1).replace(/\.0$/, "")) + " " + units[i];
+}
+
+function tile(label, value, sub) {
+  return h("div", { class: "card stat" }, h("div", { class: "label" }, label), h("div", { class: "value big" }, value), h("div", { class: "sub" }, sub));
+}
+
+function listCard(title, subtitle, body) {
+  return h("section", { class: "panel" }, h("h2", null, title), subtitle ? h("div", { class: "muted small list-sub" }, subtitle) : null, body);
+}
+
+function stepName(step) {
+  return { "1m": "minute", "5m": "5 minutes", "15m": "15 minutes", "1h": "hour", "6h": "6 hours", "1d": "day" }[step] || step;
+}

internal/web/static/assets/pages/trace.js

@@ -0,0 +1,79 @@
+// Trace: follow packets through every table's rules, and see what
+// decided their fate.
+import { h, mount, input, select, field, errorBox, badge } from "../ui.js";
+import { get, api, can } from "../api.js";
+import { onLeave } from "../app.js";
+
+export async function render(el, params, query) {
+  const address = input({ placeholder: "192.168.1.50, or 2001:db8:1::/64", class: "mono", value: (query && query.get && query.get("address")) || "" });
+  const port = input({ placeholder: "443", class: "short", type: "number", min: 1, max: 65535 });
+  const protocol = select([["", "any"], "tcp", "udp", "sctp", "icmp", "icmpv6"], "", { class: "inline" });
+  const seconds = input({ value: "60", class: "short", type: "number", min: 5, max: 600 });
+  const err = h("div");
+  const status = h("div");
+  const packets = h("div");
+  const write = can("config", true);
+  const startBtn = h("button", { class: "primary", onclick: start }, "Start tracing");
+  const stopBtn = h("button", { class: "hidden", onclick: stop }, "Stop");
+  let timer = null;
+
+  async function start() {
+    mount(err);
+    try {
+      await api("POST", "/trace", { body: { address: address.value.trim(), port: parseInt(port.value || "0", 10), protocol: protocol.value, seconds: parseInt(seconds.value || "60", 10) } });
+      poll();
+    } catch (e) { mount(err, errorBox(e)); }
+  }
+  async function stop() {
+    try { show(await api("DELETE", "/trace")); } catch (e) { mount(err, errorBox(e)); }
+  }
+  async function poll() {
+    clearTimeout(timer);
+    let st;
+    try { st = await get("/trace"); } catch (e) { mount(err, errorBox(e)); return; }
+    show(st);
+    if (st.running) timer = setTimeout(poll, 1000);
+  }
+  function show(st) {
+    startBtn.classList.toggle("hidden", st.running || !write);
+    stopBtn.classList.toggle("hidden", !st.running || !write);
+    const what = [st.input.address, st.input.protocol, st.input.port ? "port " + st.input.port : ""].filter(Boolean).join(" ");
+    mount(status, st.started && st.started !== "0001-01-01T00:00:00Z" ? h("div", { class: "msg " + (st.error ? "err" : st.running ? "info" : "ok") },
+      st.running ? ["Tracing ", h("strong", null, what), " until " + new Date(st.ends).toLocaleTimeString() + ". Make the traffic happen now."]
+        : ["Traced ", h("strong", null, what), ": " + (st.packets.length === 1 ? "1 packet" : st.packets.length + " packets") + "."],
+      st.error ? h("div", null, st.error) : null) : null);
+    mount(packets, st.packets.length ? st.packets.slice().reverse().map(packet)
+      : st.running ? h("div", { class: "empty" }, "No packets yet.") : null);
+  }
+  const hostPort = (a, p) => !a ? "" : !p ? a : a.includes(":") ? "[" + a + "]:" + p : a + ":" + p;
+  function packet(p) {
+    const head = [p.protocol, p.tcp_flags, p.icmp_type].filter(Boolean).join(" ");
+    const decided = p.decided_by.label || [p.decided_by.table, p.decided_by.chain, p.decided_by.kind].filter(Boolean).join(" ");
+    return h("div", { class: "panel trace-packet" },
+      h("div", { class: "trace-head" },
+        badge(p.verdict, p.verdict === "dropped" ? "err" : "ok"), " ",
+        h("span", { class: "mono" }, hostPort(p.source, p.source_port), " → ", hostPort(p.destination, p.destination_port)), " ",
+        h("span", { class: "muted" }, head, p.in ? " · in " + p.in : "", p.out ? " · out " + p.out : ""),
+        h("span", { class: "muted small trace-time" }, new Date(p.time).toLocaleTimeString())),
+      decided ? h("div", { class: "small" }, p.verdict === "dropped" ? "Dropped by " : "Accepted by ", h("strong", null, decided)) : null,
+      p.would_drop ? h("div", { class: "small" }, badge("shadow mode", "warn"), " Housecarl would have dropped it: ",
+        h("strong", null, p.would_drop.label || [p.would_drop.chain, p.would_drop.mark].filter(Boolean).join(" "))) : null,
+      h("table", { class: "trace-steps" }, h("tbody", null, p.steps.map((s) => h("tr", { class: /^(drop|reject)/.test(s.verdict) || /\[would (drop|reject)\]/.test(s.rule || "") ? "step-drop" : s.verdict === "accept" ? "step-accept" : "" },
+        h("td", { class: "muted small" }, s.table + " " + s.chain),
+        h("td", null, s.label ? (s.mark && s.mark.startsWith("r") ? h("a", { href: "#/rules?chain=" + s.chain + "&rule=" + s.mark.slice(1).split("-")[0] }, s.label) : s.label)
+          : s.kind === "policy" ? s.chain + " policy" : h("span", { class: "mono small" }, s.rule)),
+        h("td", { class: "small" }, s.verdict))))));
+  }
+
+  mount(el,
+    h("div", { class: "page-head" }, h("div", { class: "grow" }, h("h1", null, "Trace"),
+      h("div", { class: "muted" }, "Follows packets through the whole ruleset, Housecarl's rules and every other table (Docker, fail2ban…): each rule and policy they meet, and what accepted or dropped them. For “why doesn't this get through?”"))),
+    write ? h("div", { class: "panel" }, h("div", { class: "form-grid" },
+      field("Address", address, "Packets from or to it (either way). An address or a prefix."),
+      field("Port", h("div", null, port, " ", protocol), "Packets from or to it. Empty: any."),
+      field("For", h("div", null, seconds, " seconds"), "At most 600; it stops by itself after 200 packets.")),
+      h("div", { class: "actions form-actions" }, startBtn, stopBtn), err) : null,
+    status, packets);
+  await poll();
+  onLeave(() => clearTimeout(timer));
+}

internal/web/static/assets/style.css

@@ -373,6 +373,7 @@ pre.wrap { white-space: pre-wrap; word-break: break-all; font-size: 12px; }
 /* The probation banner: changes roll back unless confirmed. */
 .probation-banner { background: var(--warn-soft); color: var(--text); border-bottom: 1px solid var(--warn); padding: 10px 24px; font-size: 14px; }
 .probation-banner .banner-actions { margin-left: 12px; white-space: nowrap; }
+.shadow-banner { background: var(--panel); border-bottom-color: var(--border); }
 /* List editors (zones, addresses, services). */
 .list-rows { display: flex; flex-direction: column; gap: 6px; margin-bottom: 6px; }
 .list-row { display: flex; gap: 6px; align-items: center; }
@@ -409,3 +410,165 @@ input.wide { width: 100%; }
 .diff .hunk { background: var(--hunk); color: var(--accent); }
 .diff .file { font-weight: 700; padding-top: 6px; }
 select.inline { width: auto; display: inline-block; }
+/* The rule dialog's optional parts: "+" buttons, and ✕ to close one. */
+.adders { display: flex; flex-wrap: wrap; gap: 6px; }
+button.close-section { padding: 0 6px; margin-left: 4px; font-size: 11px; line-height: 1.5; }
+/* Trace: one panel per packet, its steps below. */
+.trace-head { display: flex; flex-wrap: wrap; align-items: center; gap: 6px; margin-bottom: 4px; }
+.trace-time { margin-left: auto; }
+table.trace-steps { width: 100%; margin-top: 6px; }
+table.trace-steps td { padding: 3px 8px; border-top: 1px solid var(--border); }
+tr.step-drop td { background: var(--del); }
+tr.step-accept td { background: var(--add); }
+/* The zone matrix: a cell per pair, coloured by what happens. */
+table.matrix th, table.matrix td { text-align: center; }
+table.matrix tbody th { text-align: left; }
+table.matrix th.corner { font-weight: 400; color: var(--muted); text-transform: none; }
+table.matrix td { padding: 4px; }
+table.matrix { table-layout: fixed; width: 100%; min-width: 640px; }
+table.matrix thead th:first-child, table.matrix tbody th { width: 8em; }
+.matrix-cell { display: block; width: 100%; min-width: 7em; border: 0; background: transparent; color: inherit; font-weight: 600; padding: 6px 8px; cursor: pointer; white-space: normal; text-align: center; }
+td.matrix-allowed { background: var(--add); color: var(--add-text); }
+td.matrix-blocked { background: var(--del); color: var(--del-text); }
+.matrix-n { margin-left: 6px; font-weight: 400; font-size: 12px; }
+td.matrix-partly { background: var(--warn-soft); color: var(--text); }
+td.matrix-same { opacity: 0.55; }
+/* The rule a link pointed at (?rule=ID). */
+tr.row-target td { background: var(--accent-soft); font-weight: 600; }
+tr.row-target td:first-child { box-shadow: inset 3px 0 0 var(--accent); }
+.matrix-open { display: block; font-weight: 400; font-size: 12px; opacity: 0.85; margin-top: 2px; }
+td.matrix-none { color: var(--muted); }
+tr.matrix-rest td { border-top: 2px solid var(--border); }
+
+/* ---- Statistics charts. Colors from the validated reference palette
+   (categorical slots 1-3, sequential blue), checked against this UI's
+   panel surfaces in both themes. */
+:root {
+  --series-1: #2a78d6;
+  --series-2: #eb6834;
+  --series-3: #1baf7a;
+  --chart-grid: #eceef2;
+  --chart-axis: #c9cfd8;
+  /* Sequential ramp, low to high (blue 100 -> 700). */
+  --heat-0: #cde2fb; --heat-1: #9ec5f4; --heat-2: #6da7ec; --heat-3: #3987e5;
+  --heat-4: #256abf; --heat-5: #184f95; --heat-6: #0d366b;
+}
+@media (prefers-color-scheme: dark) {
+  :root:not([data-theme="light"]) {
+    --series-1: #3987e5;
+    --series-2: #d95926;
+    --series-3: #199e70;
+    --chart-grid: #272c34;
+    --chart-axis: #3a414d;
+    /* In dark, low values recede into the surface: the ramp runs dark -> light. */
+    --heat-0: #0d366b; --heat-1: #184f95; --heat-2: #256abf; --heat-3: #3987e5;
+    --heat-4: #6da7ec; --heat-5: #9ec5f4; --heat-6: #cde2fb;
+  }
+}
+:root[data-theme="dark"] {
+  --series-1: #3987e5;
+  --series-2: #d95926;
+  --series-3: #199e70;
+  --chart-grid: #272c34;
+  --chart-axis: #3a414d;
+  --heat-0: #0d366b; --heat-1: #184f95; --heat-2: #256abf; --heat-3: #3987e5;
+  --heat-4: #6da7ec; --heat-5: #9ec5f4; --heat-6: #cde2fb;
+}
+
+.refreshing { opacity: 0.6; transition: opacity 0.2s; }
+/* Secondary row actions show when the row is hovered or focused; touch
+   screens, which can't hover, always show them. Opacity keeps them in the
+   layout (no shifting) and reachable with Tab. */
+tr .reveal { opacity: 0; transition: opacity 0.1s; }
+tr:hover .reveal, tr:focus-within .reveal { opacity: 1; }
+@media (hover: none) { tr .reveal { opacity: 1; } }
+/* Blocking steps, in order, on the Blocking page. */
+.steps-line { display: flex; flex-wrap: wrap; align-items: center; gap: 4px; margin: 4px 0 10px; }
+.step-chip { display: inline-flex; align-items: center; gap: 4px; padding: 2px 8px; border: 1px solid var(--border); border-radius: 12px; font-family: var(--mono); font-size: 13px; background: var(--panel); }
+.step-chip.accent { border-color: var(--accent); }
+.step-tools { display: inline-flex; gap: 2px; margin-left: 2px; }
+.step-tools button { padding: 0 5px; min-width: 0; }
+h2.section { margin: 24px 0 10px; font-size: 18px; }
+.segmented button { border: none; border-right: 1px solid var(--border); border-radius: 0; background: transparent; padding: 4px 12px; color: var(--text); }
+.segmented button:last-child { border-right: none; }
+.segmented button:hover { background: var(--info-soft); }
+.segmented button.active { background: var(--accent); color: var(--accent-text); font-weight: 600; }
+.card.stat .value.big { font-size: 26px; font-weight: 600; line-height: 1.2; }
+
+.chart-card { margin: 0 0 14px; }
+.chart-head { display: flex; align-items: flex-start; gap: 12px; flex-wrap: wrap; margin-bottom: 8px; }
+.chart-head .grow { flex: 1; min-width: 200px; }
+.chart-head h2 { margin: 0; font-size: 16px; }
+/* Two or three columns; the last cards stretch to fill their row, so no
+   card is left alone beside an empty space. */
+.chart-grid { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 0 14px; }
+.chart-grid > :last-child:nth-child(odd) { grid-column: 1 / -1; }
+@media (min-width: 1100px) {
+  .chart-grid { grid-template-columns: repeat(3, minmax(0, 1fr)); }
+  .chart-grid > :last-child:nth-child(odd) { grid-column: auto; }
+  .chart-grid > :last-child:nth-child(3n+1) { grid-column: 1 / -1; }
+  .chart-grid > :last-child:nth-child(3n+2) { grid-column: span 2; }
+}
+.chart-grid > .panel h2 { margin: 0; font-size: 16px; }
+.list-sub { margin: 2px 0 8px; }
+
+.chart { position: relative; outline: none; }
+.chart:focus-visible { box-shadow: 0 0 0 2px var(--accent); border-radius: var(--radius); }
+.chart svg { display: block; overflow: visible; }
+.chart .grid { stroke: var(--chart-grid); stroke-width: 1; }
+.chart .axis { stroke: var(--chart-axis); stroke-width: 1; }
+.chart .tick, .chart .end-label { fill: var(--muted); font-size: 11px; font-variant-numeric: tabular-nums; }
+.chart .end-label { fill: var(--text); }
+.chart .line { fill: none; stroke-width: 2; stroke-linejoin: round; stroke-linecap: round; }
+.chart .area { stroke: none; fill-opacity: 0.1; }
+.chart .dot { stroke: var(--panel); stroke-width: 2; }
+.chart .crosshair { stroke: var(--muted); stroke-width: 1; }
+.chart .hit { fill: transparent; cursor: crosshair; }
+.chart .s1, .key.s1 { stroke: var(--series-1); }
+.chart .s2, .key.s2 { stroke: var(--series-2); }
+.chart .s3, .key.s3 { stroke: var(--series-3); }
+.chart .area.s1, .chart .dot.s1 { fill: var(--series-1); }
+.chart .area.s2, .chart .dot.s2 { fill: var(--series-2); }
+.chart .area.s3, .chart .dot.s3 { fill: var(--series-3); }
+.key.line { display: inline-block; width: 14px; height: 0; border-top: 2px solid; vertical-align: middle; margin-right: 6px; }
+.key.line.s1 { border-color: var(--series-1); }
+.key.line.s2 { border-color: var(--series-2); }
+.key.line.s3 { border-color: var(--series-3); }
+.chart-legend { list-style: none; display: flex; gap: 14px; margin: 2px 0 0; padding: 0; font-size: 13px; color: var(--text); flex-wrap: wrap; }
+
+.chart-tip { position: absolute; z-index: 5; pointer-events: none; background: var(--panel); border: 1px solid var(--border); border-radius: var(--radius);
+  box-shadow: 0 4px 16px rgba(0, 0, 0, 0.16); padding: 6px 10px; font-size: 13px; white-space: nowrap; }
+.chart-tip .tip-head { color: var(--muted); font-size: 12px; margin-bottom: 2px; }
+.chart-tip .tip-row strong { font-variant-numeric: tabular-nums; }
+.chart-table .scroll, .chart-table .table-wrap { max-height: 360px; overflow: auto; }
+td.num, th.num { text-align: right; font-variant-numeric: tabular-nums; }
+
+.heatmap-wrap { position: relative; }
+.heatmap { display: grid; grid-template-columns: 36px repeat(24, minmax(10px, 1fr)); gap: 2px; outline: none; }
+.heatmap:focus-visible { box-shadow: 0 0 0 2px var(--accent); }
+.hm-corner, .hm-col, .hm-row { font-size: 11px; color: var(--muted); font-variant-numeric: tabular-nums; }
+.hm-row { display: flex; align-items: center; }
+.hm-cell { height: 22px; border-radius: 3px; }
+.hm-cell:hover, .hm-cell:focus { outline: 2px solid var(--text); outline-offset: -1px; }
+.hm-empty { background: transparent; box-shadow: inset 0 0 0 1px var(--chart-grid); }
+.heat-0 { background: var(--heat-0); } .heat-1 { background: var(--heat-1); } .heat-2 { background: var(--heat-2); }
+.heat-3 { background: var(--heat-3); } .heat-4 { background: var(--heat-4); } .heat-5 { background: var(--heat-5); }
+.heat-6 { background: var(--heat-6); }
+.hm-scale { display: flex; align-items: center; gap: 2px; margin-top: 10px; flex-wrap: wrap; }
+.hm-scale .muted { margin: 0 6px; }
+.hm-swatch { display: inline-block; width: 18px; height: 10px; border-radius: 2px; }
+
+.bar-list { list-style: none; margin: 0; padding: 0; }
+.bar-list li { padding: 4px 0; }
+.bar-text { display: flex; gap: 8px; justify-content: space-between; font-size: 13px; }
+.bar-label { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-family: var(--mono); font-size: 12.5px; min-width: 0; }
+.bar-value { white-space: nowrap; font-variant-numeric: tabular-nums; }
+.bar-track { height: 6px; margin-top: 3px; }
+.bar-fill { height: 6px; background: var(--series-1); border-radius: 0 3px 3px 0; min-width: 2px; }
+@media (max-width: 700px) {
+  .heatmap { grid-template-columns: 30px repeat(24, minmax(6px, 1fr)); gap: 1px; }
+  .hm-cell { height: 16px; }
+  .chart-grid { grid-template-columns: 1fr; }
+}
+.plain-list { margin: 0; padding-left: 18px; }
+.plain-list li { padding: 2px 0; }

internal/web/static/assets/widgets.js

@@ -156,6 +156,13 @@ export function fmtBytes(n) {
   return n + " bytes";
 }
 
+// usageBar draws used out of total as a bar, red once it is all used.
+export function usageBar(used, total) {
+  const seg = h("div", { class: "seg leased" });
+  seg.style.width = Math.min(100, used / total * 100) + "%";
+  return h("div", { class: "usage" }, h("div", { class: "usage-bar" + (used >= total ? " full" : "") }, seg));
+}
+
 // quotas loads what the rules' quotas used, by rule ID ({} without
 // log:read).
 export async function quotas() {

scripts/e2e.sh

@@ -2,12 +2,12 @@
 # End-to-end test with real traffic: a Debian 13 router container (nftables
 # 1.1.3) between an "internet" network and a LAN, IPv4 and IPv6, runs
 # housecarld. The maintainer's own router ruleset (internal/importer/
-# testdata/router.nft, with lab addresses) is loaded as it is today, imported,
-# taken over, and then checked with connections: port forwards, hairpin NAT,
-# the IPv6 pinhole, masquerading, antispoofing, the drop log, a change on
-# probation that cuts the client off and rolls itself back, a restart and the
-# boot ruleset. Needs Docker; no root on the host. Run from the repository
-# root:
+# testdata/router.nft, with lab addresses) is loaded as it is today,
+# imported with housecarld in shadow mode beside it, taken over, and then
+# checked with connections: port forwards, hairpin NAT, the IPv6 pinhole,
+# masquerading, antispoofing, the drop log, a change on probation that cuts
+# the client off and rolls itself back, a restart and the boot ruleset.
+# Needs Docker; no root on the host. Run from the repository root:
 #
 #   make e2e        or        scripts/e2e.sh
 #
@@ -100,8 +100,12 @@ database: /w/data/housecarl.db
 ruleset: /w/data/ruleset.nft
 backup: {dir: /w/backups}
 log: {level: debug}
+stats: {database: /w/data/stats.db}
 EOF
-start() { R sh -c '(/w/housecarld -c /w/housecarl.yaml serve >> /w/srv.log 2>&1 &); for i in $(seq 1 30); do [ -S /run/housecarl/housecarl.sock ] && exit 0; sleep 0.3; done; exit 1'; }
+{ cat "$WORK/housecarl.yaml"; echo "shadow: {enabled: true}"; } > "$WORK/housecarl-shadow.yaml"
+CONF=housecarl.yaml
+start() { R sh -c "(HOUSECARL_STATS_INTERVAL=3s /w/housecarld -c /w/$CONF serve >> /w/srv.log 2>&1 &); for i in \$(seq 1 30); do [ -S /run/housecarl/housecarl.sock ] && exit 0; sleep 0.3; done; exit 1"; }
+stop() { docker exec ${P}-router sh -c 'kill $(pidof housecarld)'; sleep 1; }
 
 # ---- today's router: the hand-written ruleset
 
@@ -112,21 +116,75 @@ R nft add table ip other
 run "the port forward works under it" 0 connect net $WAN4 8096
 contains "jellyfin"
 
-# ---- import and take over
+# ---- import in shadow mode: beside today's ruleset, changing nothing
 
 run "init" 0 R sh -c 'echo secret-pass-123 | /w/housecarld -c /w/housecarl.yaml init --password-stdin'
-run "start housecarld" 0 start
+CONF=housecarl-shadow.yaml
+run "start housecarld in shadow mode" 0 start
 run "status" 0 H status
+contains "Mode:.*SHADOW"
 contains "Other tables:.*inet main"
 run "two tables: say which" 6 H import --defines /w/router.nft -n
 contains "several tables"
 run "dry run of the import" 0 H import --table "inet main" --defines /w/router.nft -n
 contains "5 port forwards, 2 source NAT rules"
 contains "dry run: nothing was changed"
+contains "dhcp (enp2s0: the define dhcp_int)"
+run "zone names given at import" 0 H import --table "inet main" --defines /w/router.nft --zone enp2s0=lan --zone wg0=vpn -n
+contains "lan (enp2s0: as given)"
+contains "vpn (wg0: as given)"
 run "import" 0 H import --table "inet main" --defines /w/router.nft
 contains "applied and saved"
+run "the shadow ruleset is in force" 0 R nft list table inet housecarl
+contains "would dnat\] jellyfin"
+contains "priority filter + 10; policy accept;"
+not_contains "dnat ip to"
+not_contains "maxseg size set"
+not_contains "flowtable"
+run "nothing in it drops, rejects or translates" 1 R sh -c "nft list table inet housecarl | grep -v 'log prefix' | sed 's/comment.*//; s/ct status dnat//' | grep -wE 'drop|reject|dnat|snat|masquerade|redirect'"
+run "no warning about today's ruleset beside it" 0 H status
+not_contains "takeover_pending"
+not_contains "shadow_unprotected"
+run "what it loads once shadow mode is off" 0 H ruleset --enforcing
+contains "dnat ip to 10.88.10.57:443"
+run "the port forward works, by today's ruleset" 0 connect net $WAN4 8096
+contains "jellyfin"
+run "and Housecarl's would-be DNAT counted it" 0 R nft list chain inet housecarl dstnat
+contains 'packets [1-9][0-9]* bytes [0-9]* comment "hc:f[0-9]* \[would dnat\] jellyfin'
+# A drop Housecarl has and today's ruleset doesn't: logged, let through.
+SR=$(H -o id rule add forward --from $CLIENT4 --service tcp/9000 -a drop --log --first --comment "e2e shadow")
+: > "$WORK/net.log"
+run "a shadow drop lets the connection through" 0 docker exec ${P}-client sh -c "echo hi | nc -N -w 3 $NET4 9000"
+sleep 1
+run "it reached the internet" 0 grep -q "$WAN4" "$WORK/net.log"
+run "the log says it would have been dropped" 0 H log
+contains "would drop, let through"
+run "the shadow rule goes" 0 H rule delete -y "$SR"
+run "no takeover in shadow mode" 5 H takeover "inet main" -y
+contains "shadow mode"
+run "healthz" 0 R curl -sf http://127.0.0.1:8070/healthz
+contains '"mode":"shadow"'
+run "the boot ruleset is the shadow one" 0 R grep -q "would dnat" /w/data/ruleset.nft
+run "the boot fallback filters nothing" 1 R grep -wE "drop|masquerade" /w/data/ruleset-fallback.nft
+# Today's ruleset starts with flush ruleset: reloading it deletes
+# Housecarl's table, which housecarld puts back.
+run "today's ruleset reloaded" 0 R nft -f /w/router.nft
+R nft add table ip other
+sleep 4
+run "Housecarl's table is back" 0 R nft list table inet housecarl
+contains "would dnat"
+run "and logged" 0 R grep -q "put it back" /w/srv.log
+
+# ---- enforcing, and take over
+
+stop
+CONF=housecarl.yaml
+run "start housecarld enforcing" 0 start
+run "status" 0 H status
+not_contains "SHADOW"
 run "the ruleset is in force" 0 R nft list table inet housecarl
 contains "dnat ip to 10.88.10.57:443"
+not_contains "would"
 run "zones" 0 H zone list
 contains "wan.*wan.*yes"
 contains "dhcp.*enp2s0"
@@ -172,11 +230,11 @@ run "probation of 10 seconds" 0 H settings set --probation 10s
 TOKEN=$(R /w/housecarld -c /w/housecarl.yaml user token admin e2e 2>/dev/null | tail -1)
 C() { docker exec ${P}-client /w/hc --config /w/cli.yaml --server http://$LAN4:8070 --token "$TOKEN" "$@"; }
 run "the API from the LAN" 0 C whoami
-run "a change over the network is on probation" 0 C zone add guests --interface guest0
+run "a change over the network is on probation" 0 C --no-confirm zone add guests --interface guest0
 contains "applied on probation"
 run "confirmed" 0 C confirm
 contains "confirmed 1 change;"
-run "a change that cuts the client off" 0 C rule disable "$APIRULE"
+run "a change that cuts the client off" 0 C --no-confirm rule disable "$APIRULE"
 contains "on probation"
 run "the client is cut off (a timeout)" 124 docker exec ${P}-client timeout 3 /w/hc --config /w/cli.yaml --server http://$LAN4:8070 --token "$TOKEN" whoami
 sleep 12
@@ -185,6 +243,25 @@ run "the rule is enabled again" 0 H rule show "$APIRULE"
 not_contains "disabled"
 run "the rollback is in the history" 0 H history list
 contains "rollback"
+# Confirmed by the client itself: it reached the API over a new connection.
+run "a change confirmed by its client" 0 C zone add guests2 --interface guest2
+contains "applied and confirmed"
+run "nothing waits" 0 C status
+not_contains "Probation:"
+run "the confirmation is audited" 0 H audit --action config.confirm
+contains "confirmed automatically"
+# A change that cuts its client off can't be confirmed by it: rolled back.
+run "a lockout change, not confirmed" 0 C rule disable "$APIRULE"
+contains "couldn't confirm it"
+sleep 3
+run "rolled back: the client is back" 0 C whoami
+run "the rule is enabled again" 0 H rule show "$APIRULE"
+not_contains "disabled"
+run "a person must confirm, by setting" 0 H settings set --auto-confirm off
+run "then a change waits" 0 C zone add guests3 --interface guest3
+contains "applied on probation"
+run "confirmed by a person" 0 C confirm
+run "back to automatic" 0 H settings set --auto-confirm on
 
 # ---- changesets: staged, reviewed, applied together
 
@@ -209,8 +286,8 @@ run "remove the forward operation" 0 CE changeset remove 3
 contains "has 2 operations"
 run "the diff without it" 0 CE changeset diff
 not_contains "dnat ip to 10.88.10.57"
-run "applied together" 0 CE changeset apply
-contains "applied on probation"
+run "applied together, and confirmed by this client" 0 CE changeset apply
+contains "applied and confirmed"
 contains "HOUSECARL_CHANGESET still names changeset $CS"
 run "one history entry" 0 H history list
 contains "e2e batch"
@@ -221,9 +298,14 @@ contains "^jellyfin$"
 run "a stale HOUSECARL_CHANGESET is refused" 1 CE zone add stale -i stale0
 contains "is applied"
 CS2=$(H changeset create -q)
-run "an undo can't be staged" 6 H -C "$CS2" history undo 1
+UNDO=$(H zone add undome -i undo0 | sed -n 's/.*history \([0-9]*\);.*/\1/p')
+run "an undo is staged" 0 H -C "$CS2" history undo "$UNDO"
+contains "staged in changeset $CS2"
+run "not applied yet: the zone is there" 0 H zone show undome
+run "an import can't be staged" 6 sh -c "echo '{\"nftables\":[]}' | docker exec -i ${P}-router /w/hc -C $CS2 import -"
 contains "can't be staged"
 run "discarded" 0 H changeset discard "$CS2"
+run "the zone stays" 0 H zone show undome
 run "a dry run shows the ruleset's diff" 0 H -n set add e2e_dry 10.1.2.3
 contains "^+.*set e2e_dry_v4"
 
@@ -238,7 +320,7 @@ run "used up: the device is cut off" 1 send 1000
 run "the usage" 0 H quotas
 contains "e2e quota.*1.0 MB each.*by hand"
 contains "$CLIENT4  *1\.[0-9] MB"
-docker exec ${P}-router sh -c 'kill $(pidof housecarld)'; sleep 1; start
+stop; start
 run "after a restart, still used up" 1 send 1000
 run "the usage after a restart" 0 H quotas
 contains "$CLIENT4  *1\.[0-9] MB"
@@ -248,6 +330,84 @@ run "the reset is audited" 0 H audit --action quota.
 contains "quota.reset"
 run "the quota rule goes" 0 H rule delete -y "$QR"
 
+# ---- the zone matrix, of the imported ruleset
+
+run "the matrix" 0 H matrix
+contains "^firewall  *allow"
+contains "^dhcp  *partly +[0-9]"
+contains "dhcp → wan  *.*from trusted_ips"
+run "LAN to the internet" 0 H matrix dhcp wan
+contains "new connections: partly allowed"
+contains "allowed  *from trusted_ips  *rule forward #[0-9]* (Forward from trusted IPs)  *[0-9]"
+contains "blocked  *the rest  *forward policy  *[0-9]"
+not_contains "wg0_net"
+run "the internet to the firewall" 0 H matrix wan firewall
+contains "blocked  *sources that can't come from outside"
+contains "allowed  *udp/51820"
+contains "allowed  *raw: icmpv6 type"
+
+# ---- statistics (sampled every 3 seconds here)
+
+run "traffic per device on" 0 H settings set --stats-per-device on --log-drops on
+run "the device counters are in the ruleset" 0 R nft list set inet housecarl stats_up_v4
+send 500000 >/dev/null 2>&1
+run "a connection from the internet, dropped and logged" 1 connect net $WAN4 23
+sleep 8
+run "statistics are collected" 0 H stats status
+contains "Collecting:  *yes"
+contains "Per device:  *on"
+run "the summary" 0 H stats --since 1h
+contains "^wan  *[0-9.]* [KM]B"
+contains "Dropped:  *[1-9]"
+run "traffic per zone over time" 0 H stats traffic --since 1h --step 1m
+contains "wan ↓"
+run "the devices" 0 H stats top devices --since 1h
+contains "^$CLIENT4  *[0-9.]* KB  *[0-9.]* KB"
+run "the sources blocked" 0 H stats top blocked --since 1h
+contains "^$NET4  *[1-9]"
+run "what they tried" 0 H stats top tried --since 1h
+contains "^tcp/23"
+run "hits per rule" 0 H stats rules --since 1h
+contains "input policy  *[1-9]"
+contains "rule #[0-9]* (deleted)  *[1-9]"
+run "connections over time" 0 H stats connections --since 1h --step 1m
+contains "^[0-9-]* [0-9:]*  *[0-9]*  *[0-9]*"
+
+# ---- connections and tracing
+
+# A connection that stays open, masqueraded on its way out.
+docker exec -d ${P}-client sh -c "sleep 30 | nc $NET4 9000"
+sleep 1
+run "the connection is tracked" 0 H connections --search "$CLIENT4 :9000"
+contains "$CLIENT4:[0-9]*  *$NET4:9000  *as $WAN4"
+run "closed" 0 H connections close --search "$CLIENT4 :9000" -y
+contains "closed [1-9][0-9]* connection"
+run "gone" 0 H connections --search "$CLIENT4 :9000"
+not_contains ":9000"
+# A block added while traffic flows: it stops new connections, not this
+# one, unless it checks every packet.
+docker exec -d ${P}-client sh -c "rm -f /tmp/ncdone; yes | nc $NET4 9000; touch /tmp/ncdone"
+sleep 1
+BR=$(H -o id rule add forward --from $CLIENT4 --service tcp/9000 -a reject --reject-with tcp-reset --comment "e2e block" --first)
+run "a block for new connections" 0 test -n "$BR"
+sleep 2
+run "the open upload carries on" 1 docker exec ${P}-client test -f /tmp/ncdone
+run "the block checks every packet now" 0 H rule update "$BR" --every-packet
+sleep 2
+run "the upload is cut" 0 docker exec ${P}-client test -f /tmp/ncdone
+# Tracing a new connection: dropped, by that rule.
+docker exec -d ${P}-router sh -c "/w/hc trace --address $CLIENT4 --port 9000 --seconds 6 > /w/trace.out 2>&1"
+sleep 2
+connect client $NET4 9000 >/dev/null 2>&1
+sleep 6
+run "the trace" 0 cat "$WORK/trace.out"
+contains "$CLIENT4:[0-9]* -> $NET4:9000  tcp syn"
+contains "inet housecarl forward: rule forward #$BR (e2e block) -> drop"
+contains "=> DROPPED by rule forward #$BR (e2e block)"
+run "no trace table is left" 0 R nft list tables
+not_contains "housecarl-trace"
+run "the block goes" 0 H rule delete -y "$BR"
+
 # ---- the drop log
 
 run "log drops from the internet to the API" 0 H rule add input --in wan --service tcp/8070 -a drop --log --first
@@ -256,9 +416,56 @@ sleep 1
 run "the log shows it" 0 H log
 contains "hc input #.* drop.*198.18.1.10:.*198.18.1.2:8070"
 
+# ---- the flowtable, and devices that appear later
+
+run "a flowtable device that doesn't exist yet" 0 H settings set --flowtable wan --flowtable enp2s0 --flowtable hcft0
+contains "flowtable_device_missing"
+run "the flowtable has the devices that exist" 0 R nft list flowtables
+contains "devices = { enp2s0, wan }"
+run "forwarded through the flowtable" 0 connect net $WAN4 8096
+contains "jellyfin"
+run "the device appears" 0 R ip link add hcft0 type veth peer name hcft1
+R ip link set hcft0 up
+sleep 4
+run "applied again with it" 0 R nft list flowtables
+contains "hcft0"
+run "logged" 0 R grep -c "interfaces changed: applied the ruleset again" /w/srv.log
+run "it comes back as a new interface" 0 R sh -c 'ip link del hcft0 && ip link add hcft0 type veth peer name hcft1 && ip link set hcft0 up'
+sleep 4
+run "applied again" 0 R sh -c 'grep -c "interfaces changed: applied the ruleset again" /w/srv.log | grep -qx 2'
+run "the flowtable still has it" 0 R nft list flowtables
+contains "hcft0"
+
+# ---- VLAN sub-interfaces: tagged traffic between the router and the client
+
+R sh -c 'ip link add link enp2s0 name enp2s0.30 type vlan id 30 && ip addr add 10.88.30.1/24 dev enp2s0.30 && ip link set enp2s0.30 up'
+docker exec ${P}-client sh -c 'ip link add link eth0 name eth0.30 type vlan id 30 && ip addr add 10.88.30.20/24 dev eth0.30 && ip link set eth0.30 up'
+run "a zone of a VLAN" 0 H zone add iot --interface enp2s0.30
+run "nothing from the VLAN to the API yet" 1 connect client 10.88.30.1 8070
+VR=$(H -o id rule add input --in iot --service tcp/8070 --comment "e2e vlan")
+run "a rule for the VLAN's zone" 0 test -n "$VR"
+run "the VLAN reaches the API" 0 connect client 10.88.30.1 8070
+run "the rule counted it" 0 H rule list input
+contains "e2e vlan"
+run "the flowtable on the VLAN" 0 H settings set --flowtable wan --flowtable enp2s0 --flowtable hcft0 --flowtable enp2s0.30
+run "attached" 0 R nft list flowtables
+contains "enp2s0.30"
+run "the VLAN rule goes" 0 H rule delete -y "$VR"
+run "the VLAN zone goes" 0 H zone delete -y iot
+R ip link del enp2s0.30
+
+# ---- renaming a zone others refer to
+
+run "rename the imported dhcp zone" 0 H zone update dhcp --name lan
+run "its rules follow" 0 H rule list input
+contains "lan"
+not_contains "dhcp "
+run "the ruleset is unchanged by it" 0 R nft list chain inet housecarl input
+contains 'iifname "enp2s0" udp dport 67'
+
 # ---- restart and boot
 
-docker exec ${P}-router sh -c 'kill $(pidof housecarld)'; sleep 1
+stop
 run "the ruleset stays without housecarld" 0 connect net $WAN4 8096
 contains "jellyfin"
 run "restart housecarld" 0 start
@@ -270,6 +477,39 @@ run "nothing forwarded after a flush" 1 connect net $WAN4 8096
 run "the boot unit's command restores it" 0 R nft -f /w/data/ruleset.nft
 run "forwarded again" 0 connect net $WAN4 8096
 contains "jellyfin"
+run "the boot ruleset has no flowtable (its devices may not exist yet)" 1 R grep -q flowtable /w/data/ruleset.nft
+run "healthz" 0 R curl -sf http://127.0.0.1:8070/healthz
+contains '"firewall":"applied"'
+
+# ---- the boot fallback: what the boot unit runs, with the lab's paths
+
+UNIT=$(sed -n 's/^ExecStart=//p' "$ROOT/deploy/systemd/housecarl-ruleset.service" | sed 's|/var/lib/housecarl|/w/data|g; s|/usr/sbin/nft|nft|g')
+run "the fallback was saved" 0 R test -s /w/data/ruleset-fallback.nft
+R cp /w/data/ruleset.nft /w/ruleset.good
+R sh -c 'echo "this is not nftables" >> /w/data/ruleset.nft'
+R nft flush ruleset
+run "the boot unit fails, and loads the fallback" 1 R sh -c "$UNIT"
+contains "the saved ruleset failed to load"
+run "the fallback is in force" 0 R nft list table inet housecarl
+contains "hc:fallback-input"
+run "nothing new gets in from the internet" 1 connect net $WAN4 8096
+run "not to the firewall either" 1 connect net $WAN4 8070
+: > "$WORK/net.log"
+run "the LAN still reaches the internet" 0 connect client $NET4 9000
+sleep 1
+run "masqueraded" 0 grep -q "$WAN4" "$WORK/net.log"
+run "healthz says so" 0 R sh -c 'curl -s -w " %{http_code}" http://127.0.0.1:8070/healthz'
+contains '"firewall":"fallback"'
+contains " 503"
+run "status says so" 0 H status
+contains "NOT in force: the boot fallback"
+stop
+run "housecarld starts" 0 start
+run "and puts the real ruleset back" 0 connect net $WAN4 8096
+contains "jellyfin"
+run "and saves a good boot ruleset" 0 R nft -c -f /w/data/ruleset.nft
+run "healthz is happy again" 0 R curl -sf http://127.0.0.1:8070/healthz
+contains '"firewall":"applied"'
 
 # ---- backup