Release 0.2.0-rc.1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

AGENTS.md

@@ -30,6 +30,10 @@ Go module `housecarl`, `CGO_ENABLED=0`, Linux only for housecarld.
   on probation and rolled back unless confirmed (docs/design.md). An
   empty configuration applies nothing. But never force a rule that keeps
   SSH or the web UI open: the maintainer doesn't want one.
+* **Shadow mode changes no packet** (docs/shadow.md): anything new the
+  renderer writes that drops, rejects, translates or alters packets needs
+  its shadow form (counted, marked `[would …]`) or must be left out
+  there. `TestShadow` fails on what slips through.
 * **Stay within nftables 1.1.3** (Debian 13). The dev machine's newer nft
   accepts things 1.1.3 refuses; the tests with `HOUSECARL_NFT` and the
   e2e test use 1.1.3.

CHANGELOG.md

@@ -6,6 +6,170 @@ All notable changes to Housecarl are documented here. The format follows
 
 ## [Unreleased]
 
+### Added
+
+- **Ban lists** ([docs/bans.md](docs/bans.md)). A program (Guard, which
+  reads logs, or fail2ban) bans an address, of every port or of some,
+  for a time; the ban list says what the address is kept from: the
+  firewall itself (`firewall`), some hosts behind it (`hosts`, port
+  forwards to them included), or `everything`. Bans are dropped ahead of
+  every rule, established connections too, and the banned address's
+  connections are closed. IPv6 addresses are banned by /64 (`group_v6`).
+  Never banned: inside networks, the firewall's own addresses, whoever
+  asks, and the list's `never`. Bans end by themselves, are cut to the
+  list's longest time, survive restarts and reboots, and are audited, but
+  aren't configuration (no history, probation or changesets). `hc
+  banlist`, `hc bans`, `hc ban`, `hc unban`; Traffic → Bans in the web UI;
+  `/banlists` and `/banlists/{ref}/bans` in the API; a `bans` grant that
+  can name lists (`bans:write:web`), in the `viewer` and `operator`
+  roles.
+- **A fail2ban action**, `contrib/fail2ban/action.d/housecarl.conf`
+  (installed under `/usr/share/doc/housecarl/contrib/`): fail2ban's bans
+  go into a ban list, with the jail's port and protocol.
+- **Failed logins in the journal.** A failed login and a refused API
+  token are logged (`msg="login failed" ip=… user=…`), for Guard and
+  fail2ban to act on.
+
+- **Quota usage as a share.** What a quota used shows its percentage
+  (`12.3 GB of 50.0 GB used (24%)`, each address's in the quota
+  dialog), on the Rules page, the dashboard and in `hc quotas`.
+- **Reset one address's quota, or give some back.** `hc rule
+  reset-quota ID --address ADDR` starts only that device's allowance
+  again; `--give 5gbytes` takes that much off what was used instead (one
+  address's, or everyone's): extra allowance until the next reset,
+  without changing the rule. In the web UI, a rule's quota is one
+  line (what it used, how it resets) that opens a dialog with the rest:
+  usage per address, last and next reset, and Reset and Give back for
+  everyone or one address. `POST /rules/{ref}/reset-quota` takes `address` and `give`.
+- **Names beside inside addresses.** Connections, the log, quota usage,
+  the dashboard's quotas and the statistics' devices and blocked
+  sources show each inside address's name, found by a reverse lookup:
+  `laptop 192.168.1.20` in the web UI, `192.168.1.20 (laptop)` in `hc`.
+  Only inside addresses are looked up (an inside zone's networks, and
+  private ones outside the external zones'). Names are cached for 10
+  minutes and never stored. `names.enabled`, `names.resolver` and
+  `names.cache` in housecarl.yaml; `hc --no-names`; `POST /names`.
+- **Quotas on the dashboard.** With any rule holding a quota, the
+  dashboard lists them all: the allowance, what is used (for a
+  per-address quota, how many addresses and the busiest) with a bar, and
+  the next reset. Each opens its rule.
+- **Links to a rule open it.** A rule named in the matrix, the log, the
+  tracer, statistics or the dashboard's busiest rules opens the Rules
+  page on its chain, scrolled to it and highlighted (`#/rules?rule=ID`).
+  The Rules page shows each rule's ID (`#410`, as `hc` and labels name
+  it) beside its order.
+- **A clearer zone matrix.** A cell reads allow, block, or **partly**
+  (some new connections let through, the rest not), with the exceptions
+  counted by what they do (`+3 -1` instead of `+4`), and what they let
+  through: services with the addresses they're limited to (`ssh from
+  trusted`, `5 port forwards`), under the grid in `hc matrix`, in each
+  cell on the Matrix page. A zone to itself is marked (only traffic routed
+  between its interfaces passes the firewall). The details list each
+  rule's hits. API: `summary`, `allowed`, `blocked`, `open`,
+  `same_zone`, `note`, `hits`, and `short` and `hits` per exception.
+  Raw rules that are a match and then accept, drop or reject (without
+  interfaces) are read instead of shown as unknown.
+- **Shadow mode** (`shadow.enabled` in housecarl.yaml; docs/shadow.md):
+  Housecarl's ruleset is loaded beside the firewall you have but blocks,
+  rejects and translates nothing. Drops and rejects are counted and accept
+  instead (an accept only ends Housecarl's chain), policies accept and
+  always log what they would drop, NAT only counts, MSS clamping and the
+  flowtable are left out, and raw rules that do more than accept, drop or
+  reject are left out with a warning (`shadow_raw_left_out`). Comments
+  mark what would happen (`[would drop]`). `shadow.position`: Housecarl's
+  filter chains run after the other firewall's (`filter + 10`, the
+  default: what it would drop is what switching would break) or before
+  (`- 10`); its NAT chains always first. Every change is checked by
+  nftables in both forms, so leaving shadow mode can't fail. No
+  probation; takeovers and closing connections are refused (409). The
+  boot ruleset is the shadow one, and its fallback filters nothing. When
+  a reload (`flush ruleset`) deletes Housecarl's table, housecarld puts it
+  back (it listens to nftables' netlink notifications). `mode` in
+  `/status`, `/healthz` and `/auth/whoami`; a banner in the web UI;
+  would-be drops marked in the log (`shadow`) and the tracer
+  (`would_drop`); `hc ruleset --enforcing` (`GET /ruleset?enforcing=1`)
+  shows the ruleset for after. `shadow_unprotected` warns when no other
+  table filters the input hook; `nftables_service` and `takeover_pending`
+  aren't raised in shadow mode. The e2e test imports in shadow mode
+  before taking over.
+- **Zone names at import:** `hc import --zone enp2s0=lan` (repeat;
+  interfaces given one name share a zone; Zone names on the Import
+  page), and a note listing each zone and where its name came from.
+  Defines holding a VLAN sub-interface (`enp2s0.20`) name their zone,
+  and `_ifname`, `_device` and `_nic` are taken off define names too
+  (docs/import.md, "How things are named").
+- Import: sets with `flags timeout` become dynamic sets; a DNAT of
+  several ports becomes one port forward per port.
+
+### Changed
+
+- Set names starting with `ban_` are now reserved (`set_name_reserved`):
+  ban lists' sets use them.
+- **Rules read as a direction.** A match's zones are written `lan → wan`,
+  `any but wan → any`, or with the firewall itself as the other end
+  (`lan → firewall` in input, `firewall → wan` in output), instead of
+  `in lan · out wan` and `in not wan`; exclusions read `from any but X`.
+  On the Rules and NAT pages and in `hc`; the flags and JSON stay as
+  they were.
+
+### Fixed
+
+- The dashboard's bar for a per-address quota showed only the busiest
+  address, so one address that used up its allowance filled it (100%).
+  The bar is now all the addresses together, and the dashboard and the
+  rule's quota line say how many addresses used theirs up.
+- Changing how a quota counts lost or muddled what it used: per address
+  to a total started at nothing, per source to per destination kept the
+  sources' use as if it were the destinations', and other groups left
+  the old entries behind. Now a total starts at the addresses' sum,
+  wider groups add up their addresses, and what can't be carried starts
+  again with a warning (`quota_restarts`) on the change.
+- What the quotas used is saved after every change, not only every five
+  minutes: a reboot soon after a change that counts a quota differently
+  could bring back its usage counted the old way.
+- Statistics' Quotas card linked every entry to the Rules page, not to
+  its rule; each now opens its rule, named. Per-address quotas were
+  missing from it (and from the `quota:rN` series): they count their
+  addresses' sum.
+- `hc stats top devices` listed each device twice (once for what it
+  received, once for what it sent); it is one row with both again.
+- `hc rule add --before/--after/--first` made two changes (create at the
+  end, then move), so two history entries, two probations, and the rule
+  briefly in force at the end of its chain; staged in a changeset, the
+  place was lost. The create now takes the place (`POST
+  /rules?before=ID`, `after`, `first`) and makes one change.
+- `hc matrix` in a terminal: the colours' escape codes threw the columns
+  out of line.
+- `hc counters` (`GET /counters`) kept listing the objects of a
+  configuration replaced since housecarld started (`hc import
+  --replace` gives every object a new ID), beside the current ones; the
+  totals of objects that no longer exist are now forgotten.
+- Import: two base chains on one hook had their rules imported twice,
+  and the second chain's leading built-ins were moved above the first
+  chain's rules; they now follow each other by priority (read in a
+  fixed order), with the stricter policy, and a note.
+- Import: a rule inside a jumped-to chain that stays raw kept only its
+  own text and so applied to everything; the jump's condition now goes
+  in front. A jump and a rule matching the same field are combined only
+  when one's addresses lie within the other's; else the rule stays raw.
+  A jump's ports, ICMP types, connection status and hop limit were
+  dropped; a limit or log on the jump is no longer silently lost.
+- Import: a second match on the same field (`ip saddr A ip saddr != B`)
+  replaced the first; `!=` on the hop limit and TCP flags was read as
+  `==`. Such rules now stay raw, as written.
+- Import: a forward-chain accept to a forwarded host and port without
+  `ct status dnat` was folded into the port forward, which only lets
+  forwarded connections through; it stays a rule now. An IPv6 accept
+  limited to an interface no longer becomes a pinhole open to all.
+- Import: raw rules naming a set of the old table made nftables refuse
+  the ruleset; address sets are now renamed `NAME_v4`/`NAME_v6`, other
+  sets written out with their elements, and what can't be is left out
+  with a note. Rules naming a set that isn't imported stay raw instead
+  of failing validation.
+- Import: a DNAT to an IPv6 address, or with matches a port forward
+  can't hold (an interface excluded, a limit, a log), is left out with a
+  note instead of being imported without them.
+
 ## [0.1.0] - 2026-10-19
 
 The first version of Housecarl, a firewall manager for nftables, built

Makefile

@@ -112,6 +112,7 @@ install-files:
 	install -D -m 0644 CHANGELOG.md $(DESTDIR)$(DOCDIR)/CHANGELOG.md
 	install -d $(DESTDIR)$(DOCDIR)/docs
 	install -m 0644 docs/*.md $(DESTDIR)$(DOCDIR)/docs/
+	install -D -m 0644 contrib/fail2ban/action.d/housecarl.conf $(DESTDIR)$(DOCDIR)/contrib/fail2ban/action.d/housecarl.conf
 
 uninstall:
 	rm -f $(DESTDIR)$(BINDIR)/housecarld $(DESTDIR)$(BINDIR)/hc $(DESTDIR)$(UNITDIR)/housecarl.service \

README.md

@@ -49,6 +49,14 @@ table of its own.
   ruleset into zones, sets, services, rules and port forwards, keeping the
   rules' order, and keeps what it can't translate as raw rules
   ([docs/import.md](docs/import.md)).
+* **Try it first, in shadow mode.** Housecarl's ruleset runs beside the
+  firewall you have, blocking nothing: its hits, log and tracer show what
+  it would drop, so you see what switching over would break before you do
+  ([docs/shadow.md](docs/shadow.md)).
+* **Ban lists for Guard and fail2ban.** A program on any machine bans an
+  address it caught misbehaving in its logs, and the ban list says what
+  that address is kept from: the firewall, some hosts, or everything
+  ([docs/bans.md](docs/bans.md)).
 * **Easy to watch.** A zone matrix of what is allowed between zones,
   statistics over time (traffic per zone and device, drops, the sources
   blocked most, connections; [docs/stats.md](docs/stats.md)), hits per rule, a live log of what was dropped and
@@ -103,7 +111,9 @@ The web UI is on `http://127.0.0.1:8070/` (see `listen` in
 * [install.md](docs/install.md): packages, building from source, first steps
 * [rules.md](docs/rules.md): zones, address sets, services, rules and the built-ins
 * [nat.md](docs/nat.md): port forwards, hairpin NAT, IPv6 pinholes, source NAT, redirects
+* [bans.md](docs/bans.md): ban lists, where Guard and fail2ban ban addresses
 * [import.md](docs/import.md): moving from an existing nftables ruleset
+* [shadow.md](docs/shadow.md): shadow mode, Housecarl beside your firewall, changing nothing
 * [operations.md](docs/operations.md): probation, the boot and its fallback, hits and the log, monitoring, backups, troubleshooting
 * [flowtable.md](docs/flowtable.md): the fast path for forwarded traffic, and when (not) to use it
 * [certificates.md](docs/certificates.md): HTTPS with a certificate from Let's Encrypt, through Jarl
@@ -118,13 +128,14 @@ The web UI is on `http://127.0.0.1:8070/` (see `listen` in
 
 ## Status
 
-Housecarl 0.1.0 is released: zones, address sets and
+Housecarl 0.2.0 is in testing (0.2.0-rc.1): zones, address sets and
 services with ordered rules, IPv4 and IPv6, port forwards with hairpin NAT
 and IPv6 pinholes, source NAT and redirects, built-ins, probation for
 network changes (confirmed by the client that made them), the ruleset
 at boot with a fail-closed fallback, hits and the drop log, nftables
 import, changesets reviewed with the ruleset's diff (as in Jarl),
-bandwidth limits and quotas, a web UI, REST API and CLI.
+bandwidth limits and quotas, shadow mode beside another firewall, ban
+lists for Guard and fail2ban, a web UI, REST API and CLI.
 
 ## Licence
 

VERSION

@@ -1 +1 @@
-0.1.0
+0.2.0-rc.1

cmd/hc/bans.go

@@ -0,0 +1,313 @@
+package main
+
+import (
+	"fmt"
+	"io"
+	"net/url"
+	"strconv"
+	"strings"
+
+	"github.com/spf13/cobra"
+
+	"housecarl/internal/api"
+	"housecarl/internal/model"
+)
+
+// ---- ban lists (docs/bans.md)
+
+// banTime writes a ban list's time setting: 0 is the default, -1 none.
+func banTime(n int, def string) string {
+	switch {
+	case n < 0:
+		return "no limit"
+	case n == 0:
+		return def + " (default)"
+	}
+	return fmtSeconds(n)
+}
+
+// parseBanTime reads a time flag: a duration, or "none" (-1) for no
+// limit.
+func parseBanTime(s string) (int, error) {
+	if strings.EqualFold(s, "none") || s == "-1" {
+		return -1, nil
+	}
+	return parseSeconds(s)
+}
+
+func banlistCmd() *cobra.Command {
+	cmd := &cobra.Command{Use: "banlist", Aliases: []string{"banlists"}, Short: "Ban lists: where Guard and fail2ban ban addresses"}
+	cmd.AddCommand(&cobra.Command{
+		Use: "list", Short: "List the ban lists", Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			bs, err := get[[]model.BanList]("/banlists", nil)
+			if err != nil {
+				return err
+			}
+			bans, _ := get[[]model.Ban]("/bans", nil)
+			count := map[int64]int{}
+			for _, b := range bans {
+				count[b.ListID]++
+			}
+			return output(bs, func(w io.Writer) {
+				fmt.Fprintln(w, "NAME\tSCOPE\tHOSTS\tBANS\tCOMMENT")
+				for _, b := range bs {
+					scope := b.Scope
+					if b.Disabled {
+						scope += " (disabled)"
+					}
+					fmt.Fprintf(w, "%s\t%s\t%s\t%d\t%s\n", b.Name, scope, list(b.Hosts), count[b.ID], b.Comment)
+				}
+			})
+		},
+	})
+	cmd.AddCommand(showCmd("banlists", "banlist", func(w io.Writer, b model.BanList) {
+		fmt.Fprintf(w, "Name:\t%s\nScope:\t%s\n", b.Name, b.Scope)
+		if len(b.Hosts) > 0 {
+			fmt.Fprintf(w, "Hosts:\t%s\n", list(b.Hosts))
+		}
+		fmt.Fprintf(w, "Never:\t%s\n", list(b.Never))
+		fmt.Fprintf(w, "Ban time:\t%s\nLongest:\t%s\n", banTime(b.DefaultTime, "1h"), banTime(b.MaxTime, "7d"))
+		fmt.Fprintf(w, "Most bans:\t%d\nIPv6 banned by:\t/%d\n", b.Entries(), b.V6Group())
+		if b.Disabled {
+			fmt.Fprintln(w, "Disabled:\tyes")
+		}
+		if b.Comment != "" {
+			fmt.Fprintf(w, "Comment:\t%s\n", b.Comment)
+		}
+	}))
+	var scope, comment, banT, maxT string
+	var hosts, never []string
+	var entries, group int
+	var disabled bool
+	setFields := func(fl interface{ Changed(string) bool }, p patch) error {
+		if fl.Changed("scope") {
+			p["scope"] = scope
+		}
+		if fl.Changed("host") {
+			p["hosts"] = hosts
+		}
+		if fl.Changed("never") {
+			p["never"] = never
+		}
+		for _, t := range []struct{ flag, field, v string }{{"time", "default_time", banT}, {"max-time", "max_time", maxT}} {
+			if fl.Changed(t.flag) {
+				n, err := parseBanTime(t.v)
+				if err != nil {
+					return err
+				}
+				p[t.field] = n
+			}
+		}
+		if fl.Changed("max-entries") {
+			p["max_entries"] = entries
+		}
+		if fl.Changed("group-v6") {
+			p["group_v6"] = group
+		}
+		if fl.Changed("comment") {
+			p["comment"] = comment
+		}
+		if fl.Changed("disabled") {
+			p["disabled"] = disabled
+		}
+		return nil
+	}
+	flags := func(c *cobra.Command) {
+		f := c.Flags()
+		f.StringVar(&scope, "scope", "", "what bans keep addresses from: firewall, hosts or everything")
+		f.StringArrayVar(&hosts, "host", nil, "scope hosts: a machine's address, or a set (repeat)")
+		f.StringArrayVar(&never, "never", nil, "an address, prefix or set never banned (repeat)")
+		f.StringVar(&banT, "time", "", "a ban's length when none is asked for (default 1h)")
+		f.StringVar(&maxT, "max-time", "", "the longest ban (default 7d; none for no limit)")
+		f.IntVar(&entries, "max-entries", 0, "the most bans the list holds (default 10000)")
+		f.IntVar(&group, "group-v6", 0, "IPv6 prefix length banned for an address (default 64)")
+		f.StringVar(&comment, "comment", "", "a comment")
+		f.BoolVar(&disabled, "disabled", false, "the list's bans aren't in force")
+	}
+	add := &cobra.Command{
+		Use: "add NAME", Short: "Add a ban list", Args: cobra.ExactArgs(1),
+		Long: `Adds a ban list. A program bans an address into it (Guard, fail2ban's
+housecarl action, hc ban), and the list says what that address is kept
+from:
+
+  hc banlist add router --scope firewall                  # the firewall itself
+  hc banlist add web --scope hosts --host 10.0.10.57 --host 2001:db8:10::57
+  hc banlist add edge --scope everything --never trusted
+
+Give the program a token for its list alone:
+
+  hc user add guard-web --no-password --grant bans:write:web
+  hc token create web1 --user guard-web     # as root on the firewall`,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			p := patch{"name": args[0]}
+			if err := setFields(cmd.Flags(), p); err != nil {
+				return err
+			}
+			return change("POST", "/banlists", p)
+		},
+	}
+	flags(add)
+	cmd.AddCommand(add)
+	update := &cobra.Command{
+		Use: "update BANLIST", Short: "Change a ban list", Args: cobra.ExactArgs(1),
+		Long: `Changes a ban list; --host and --never replace the lists. A ban list
+can't be renamed: tokens and programs know it by its name.`,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			p := patch{}
+			if err := setFields(cmd.Flags(), p); err != nil {
+				return err
+			}
+			return change("PATCH", objPath("banlists", args[0]), p)
+		},
+	}
+	flags(update)
+	cmd.AddCommand(update)
+	cmd.AddCommand(deleteCmd("banlists", "banlist"))
+	return cmd
+}
+
+// bansCmd lists the active bans.
+func bansCmd() *cobra.Command {
+	return &cobra.Command{
+		Use: "bans [BANLIST]", Short: "List the active bans", Args: cobra.MaximumNArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			path := "/bans"
+			if len(args) == 1 {
+				path = objPath("banlists", args[0]) + "/bans"
+			}
+			bans, err := get[[]model.Ban](path, nil)
+			if err != nil {
+				return err
+			}
+			return output(bans, func(w io.Writer) {
+				fmt.Fprintln(w, "ID\tLIST\tADDRESS\tPORT\tENDS IN\tBY\tSOURCE\tREASON")
+				for _, b := range bans {
+					port := "all"
+					if b.Protocol != "" {
+						port = fmt.Sprintf("%s/%d", b.Protocol, b.Port)
+					}
+					fmt.Fprintf(w, "%d\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", b.ID, b.List, b.Address, port, fmtSeconds(b.Timeout), b.Who, orDash(b.Source), orDash(clipText(b.Reason, 60)))
+				}
+			})
+		},
+	}
+}
+
+func clipText(s string, n int) string {
+	if len(s) > n {
+		return s[:n-1] + "…"
+	}
+	return s
+}
+
+// banCmd bans an address by hand.
+func banCmd() *cobra.Command {
+	var tcp, udp []string
+	var t, reason string
+	cmd := &cobra.Command{
+		Use: "ban BANLIST ADDRESS", Short: "Ban an address", Args: cobra.ExactArgs(2),
+		Long: `Bans an address in a ban list: every port, or with --tcp and --udp some
+(numbers or service names). IPv6 addresses are banned with the list's
+grouping (a /64 unless set otherwise).
+
+  hc ban edge 203.0.113.9 --time 1d
+  hc ban web 203.0.113.9 --tcp ssh --tcp 8443 --reason "tried passwords"`,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			in := api.BanInput{Address: args[1], Reason: reason, Source: "hc"}
+			if t != "" {
+				n, err := parseBanTime(t)
+				if err != nil {
+					return err
+				}
+				in.Time = n
+			}
+			c, err := newClient()
+			if err != nil {
+				return err
+			}
+			var out api.BanResult
+			do := func(proto string, ports []string) error {
+				in.Protocol = proto
+				in.Ports = nil
+				if len(ports) > 0 {
+					in.Ports = []byte(strconv.Quote(strings.Join(ports, ",")))
+				}
+				var res api.BanResult
+				if err := c.Do(ctx(), "POST", objPath("banlists", args[0])+"/bans", nil, in, &res); err != nil {
+					return err
+				}
+				out.Bans = append(out.Bans, res.Bans...)
+				out.Closed += res.Closed
+				return nil
+			}
+			switch {
+			case len(tcp) == 0 && len(udp) == 0:
+				err = do("", nil)
+			default:
+				if len(tcp) > 0 {
+					err = do("tcp", tcp)
+				}
+				if err == nil && len(udp) > 0 {
+					err = do("udp", udp)
+				}
+			}
+			if err != nil {
+				return err
+			}
+			return output(out, func(w io.Writer) {
+				for _, b := range out.Bans {
+					what := "banned"
+					if b.Existing {
+						what = "renewed"
+					}
+					port := "every port"
+					if b.Protocol != "" {
+						port = fmt.Sprintf("%s/%d", b.Protocol, b.Port)
+					}
+					ends := "never ends"
+					if b.Timeout > 0 {
+						ends = "ends in " + fmtSeconds(b.Timeout)
+					}
+					fmt.Fprintf(w, "%s %s, %s, in %s (%s)\n", what, b.Address, port, b.List, ends)
+				}
+				if out.Closed > 0 {
+					fmt.Fprintf(w, "%d connections closed\n", out.Closed)
+				}
+			})
+		},
+	}
+	f := cmd.Flags()
+	f.StringArrayVar(&tcp, "tcp", nil, "ban only this TCP port or service (repeat)")
+	f.StringArrayVar(&udp, "udp", nil, "ban only this UDP port or service (repeat)")
+	f.StringVar(&t, "time", "", "how long (1h, 7d; default the list's; none for ever, if the list allows)")
+	f.StringVar(&reason, "reason", "", "why, for the record")
+	return cmd
+}
+
+// unbanCmd lifts bans: by ID, or every ban of an address.
+func unbanCmd() *cobra.Command {
+	return &cobra.Command{
+		Use: "unban BANLIST ADDRESS|ID", Short: "Lift a ban, or all of an address's", Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			c, err := newClient()
+			if err != nil {
+				return err
+			}
+			path, query := objPath("banlists", args[0])+"/bans", url.Values{}
+			if _, err := strconv.ParseInt(args[1], 10, 64); err == nil {
+				path += "/" + args[1]
+			} else {
+				query.Set("address", args[1])
+			}
+			var out map[string]int
+			if err := c.Do(ctx(), "DELETE", path, query, nil, &out); err != nil {
+				return err
+			}
+			if !flagQuiet {
+				fmt.Printf("%d bans lifted\n", out["removed"])
+			}
+			return nil
+		},
+	}
+}

cmd/hc/firewall.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
+	"net/url"
 	"os"
 	"os/exec"
 	"sort"
@@ -30,6 +31,9 @@ func statusCmd() *cobra.Command {
 			}
 			return output(st, func(w io.Writer) {
 				fmt.Fprintf(w, "Housecarl:\t%s, up since %s\n", st.Build.Label, st.Started.Local().Format("2006-01-02 15:04"))
+				if st.Mode == api.ModeShadow {
+					fmt.Fprintf(w, "Mode:\tSHADOW: the ruleset blocks, rejects and translates nothing; the other firewall decides (Housecarl's chains run %s it)\n", st.ShadowPosition)
+				}
 				if st.Applied {
 					fmt.Fprintln(w, "Ruleset:\tin force (table inet housecarl)")
 				} else if st.Unconfigured {
@@ -237,21 +241,28 @@ func onOff(name, v string) (bool, error) {
 // ---- the ruleset, counters, the log
 
 func rulesetCmd() *cobra.Command {
-	return &cobra.Command{
+	var enforcing bool
+	cmd := &cobra.Command{
 		Use: "ruleset", Short: "Print the nftables ruleset Housecarl builds", Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, args []string) error {
 			c, err := newClient()
 			if err != nil {
 				return err
 			}
+			var q url.Values
+			if enforcing {
+				q = url.Values{"enforcing": {"1"}}
+			}
 			var text string
-			if err := c.Do(ctx(), "GET", "/ruleset", nil, nil, &text); err != nil {
+			if err := c.Do(ctx(), "GET", "/ruleset", q, nil, &text); err != nil {
 				return err
 			}
 			fmt.Print(text)
 			return nil
 		},
 	}
+	cmd.Flags().BoolVar(&enforcing, "enforcing", false, "in shadow mode: the ruleset loaded once it is off")
+	return cmd
 }
 
 func countersCmd() *cobra.Command {
@@ -287,6 +298,13 @@ func quotasCmd() *cobra.Command {
 			if err != nil {
 				return err
 			}
+			var addrs []string
+			for _, q := range qs {
+				for _, a := range q.Addresses {
+					addrs = append(addrs, a.Address)
+				}
+			}
+			names := lookupNames(addrs...)
 			return output(qs, func(w io.Writer) {
 				fmt.Fprintln(w, "RULE\tUSED\tOF\tRESETS\tNEXT RESET")
 				for _, q := range qs {
@@ -304,9 +322,13 @@ func quotasCmd() *cobra.Command {
 					if q.Quota.PerSource || q.Quota.PerDestination {
 						of += " each"
 					}
-					fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Rule, humanBytes(q.Used), of, resets, next)
+					used := humanBytes(q.Used)
+					if !q.Quota.PerSource && !q.Quota.PerDestination {
+						used += " (" + percent(q.Used, q.Quota.Bytes) + ")"
+					}
+					fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Rule, used, of, resets, next)
 					for _, a := range q.Addresses {
-						fmt.Fprintf(w, "  %s\t%s\t\t\t\n", a.Address, humanBytes(a.Used))
+						fmt.Fprintf(w, "  %s\t%s (%s)\t\t\t\n", named(names, a.Address, a.Address), humanBytes(a.Used), percent(a.Used, q.Quota.Bytes))
 					}
 				}
 			})
@@ -314,6 +336,17 @@ func quotasCmd() *cobra.Command {
 	}
 }
 
+// percent writes used as a share of total: "25%", "<1%" for a little.
+func percent(used, total int64) string {
+	if total <= 0 || used <= 0 {
+		return "0%"
+	}
+	if used*100 < total {
+		return "<1%"
+	}
+	return fmt.Sprintf("%d%%", used*100/total)
+}
+
 func logCmd() *cobra.Command {
 	var follow bool
 	var limit int
@@ -330,6 +363,11 @@ func logCmd() *cobra.Command {
 			var last time.Time
 			print := func(es []api.LogEntry) {
 				sort.Slice(es, func(i, j int) bool { return es[i].Time.Before(es[j].Time) })
+				var addrs []string
+				for _, e := range es {
+					addrs = append(addrs, e.Source, e.Dest)
+				}
+				names := lookupNames(addrs...)
 				for _, e := range es {
 					last = e.Time
 					src, dst := e.Source, e.Dest
@@ -339,6 +377,7 @@ func logCmd() *cobra.Command {
 					if e.DestPort != 0 {
 						dst = joinPort(dst, e.DestPort)
 					}
+					src, dst = named(names, e.Source, src), named(names, e.Dest, dst)
 					what := e.Protocol
 					if e.ICMPType != "" {
 						what += " " + e.ICMPType
@@ -346,6 +385,9 @@ func logCmd() *cobra.Command {
 					if e.TCPFlags != "" {
 						what += " " + e.TCPFlags
 					}
+					if e.Shadow && (e.Verdict == "drop" || e.Verdict == "reject") {
+						what += "  (shadow mode: would " + e.Verdict + ", let through)"
+					}
 					fmt.Printf("%s  %-28s %-6s %-6s %s -> %s  %s\n", e.Time.Local().Format("15:04:05"), strings.TrimSpace(e.Prefix), orDash(e.In), orDash(e.Out), src, dst, what)
 				}
 			}
@@ -434,6 +476,7 @@ func rollbackCmd() *cobra.Command {
 func importCmd() *cobra.Command {
 	var replace bool
 	var table, defines, textFile string
+	var zones []string
 	cmd := &cobra.Command{
 		Use:   "import [FILE|-]",
 		Short: "Import an nftables ruleset (nft -j JSON) as Housecarl's configuration",
@@ -445,7 +488,9 @@ forwards and NAT. The input is nft's JSON; on the firewall:
 Without a FILE, hc runs 'nft -j list ruleset' and 'nft -a list ruleset'
 itself (root on the firewall); from elsewhere, give the JSON as FILE and
 the text with --text. --defines reads the ruleset file's define lines, to
-name the sets as you did (admins, lan_net, …). -n shows the result without
+name the sets as you did (admins, lan_net, …), and the zones; --zone
+enp2s0=lan names one yourself (repeat; interfaces given one name share a
+zone). -n shows the result without
 saving; what can't be expressed becomes raw rules, listed with notes.
 Once it looks right, import without -n, check it with 'hc ruleset', and
 'hc takeover inet main' deletes the old table.`,
@@ -473,7 +518,14 @@ Once it looks right, import without -n, check it with 'hc ruleset', and
 					return err
 				}
 			}
-			in := api.ImportInput{JSON: data, Text: string(text), Table: table, Replace: replace}
+			in := api.ImportInput{JSON: data, Text: string(text), Table: table, Replace: replace, Zones: map[string]string{}}
+			for _, z := range zones {
+				iface, name, ok := strings.Cut(z, "=")
+				if !ok || iface == "" || name == "" {
+					return fmt.Errorf("--zone %q: write it as INTERFACE=NAME (enp2s0=lan)", z)
+				}
+				in.Zones[iface] = name
+			}
 			if defines != "" {
 				b, err := os.ReadFile(defines)
 				if err != nil {
@@ -508,6 +560,7 @@ Once it looks right, import without -n, check it with 'hc ruleset', and
 	cmd.Flags().StringVar(&table, "table", "", "import only this table (\"inet main\"; default: all but Housecarl's)")
 	cmd.Flags().StringVar(&defines, "defines", "", "the ruleset file, whose defines name the sets")
 	cmd.Flags().StringVar(&textFile, "text", "", "'nft -a list ruleset' output, to keep what can't be expressed as raw rules")
+	cmd.Flags().StringArrayVar(&zones, "zone", nil, "name an interface's zone: INTERFACE=NAME (repeat)")
 	return cmd
 }
 

cmd/hc/main.go

@@ -46,6 +46,7 @@ var (
 	flagCACert    string
 	flagQuiet     bool
 	flagNoConfirm bool
+	flagNoNames   bool
 	// lastClient is the client the command made (to confirm a change by
 	// itself after it).
 	lastClient *client.Client
@@ -182,6 +183,7 @@ Exit codes: 0 ok, 1 error, 3 not authenticated or not allowed, 4 not found,
 	pf.StringVar(&flagCACert, "ca-cert", "", "PEM file with a CA to trust for the server's certificate")
 	pf.BoolVarP(&flagQuiet, "quiet", "q", false, "print only errors (and IDs of new objects)")
 	pf.BoolVar(&flagNoConfirm, "no-confirm", false, "leave a change on probation instead of confirming it once this client reaches Housecarl again")
+	pf.BoolVar(&flagNoNames, "no-names", false, "show inside addresses without their names (reverse lookups)")
 	pf.Int64VarP(&flagChangeset, "changeset", "C", 0, "stage configuration changes in this changeset instead of applying them (env HOUSECARL_CHANGESET)")
 	root.AddGroup(&cobra.Group{ID: "fw", Title: "Firewall:"}, &cobra.Group{ID: "changes", Title: "Changes:"}, &cobra.Group{ID: "admin", Title: "Administration:"})
 	add := func(group string, cmds ...*cobra.Command) {
@@ -190,7 +192,7 @@ Exit codes: 0 ok, 1 error, 3 not authenticated or not allowed, 4 not found,
 			root.AddCommand(c)
 		}
 	}
-	add("fw", statusCmd(), zoneCmd(), setCmd(), serviceCmd(), ruleCmd(), forwardCmd(), snatCmd(), redirectCmd(), settingsCmd(),
+	add("fw", statusCmd(), zoneCmd(), setCmd(), serviceCmd(), ruleCmd(), forwardCmd(), snatCmd(), redirectCmd(), banlistCmd(), bansCmd(), banCmd(), unbanCmd(), settingsCmd(),
 		rulesetCmd(), countersCmd(), quotasCmd(), logCmd(), connectionsCmd(), traceCmd(), matrixCmd(), statsCmd(), interfaceCmd())
 	add("changes", changesetCmd(), confirmCmd(), rollbackCmd(), historyCmd(), importCmd(), takeoverCmd())
 	add("admin", whoamiCmd(), userCmd(), tokenCmd(), passwordCmd(), certCmd(), auditCmd(), configCmd())

cmd/hc/main_test.go

@@ -1,8 +1,11 @@
 package main
 
 import (
+	"regexp"
+	"strings"
 	"testing"
 
+	"housecarl/internal/matrix"
 	"housecarl/internal/model"
 )
 
@@ -32,11 +35,53 @@ func TestLimitAndDescribe(t *testing.T) {
 	if _, err := parseLimit("fast"); err == nil {
 		t.Error("a bad limit accepted")
 	}
-	m := model.Match{InZones: []string{"wan"}, NotSources: true, Sources: []string{"trusted"}, Services: []string{"ssh", "tcp/8080"}}
+	for _, c := range []struct {
-	if got := describe(m); got != "in wan  from not trusted  ssh,tcp/8080" {
+		m     model.Match
-		t.Errorf("describe: %q", got)
+		chain string
+		want  string
+	}{
+		{model.Match{InZones: []string{"wan"}, NotSources: true, Sources: []string{"trusted"}, Services: []string{"ssh", "tcp/8080"}}, "input",
+			"wan → firewall  from any but trusted  ssh,tcp/8080"},
+		{model.Match{InZones: []string{"wan"}, NotIn: true, Destinations: []string{"trusted_ips"}}, "forward", "any but wan → any  to trusted_ips"},
+		{model.Match{InZones: []string{"lan"}, OutZones: []string{"wan"}}, "forward", "lan → wan"},
+		{model.Match{OutZones: []string{"wan"}}, "output", "firewall → wan"},
+		{model.Match{OutZones: []string{"wan"}, Sources: []string{"10.0.0.0/8"}}, "", "any → wan  from 10.0.0.0/8"},
+		{model.Match{Sources: []string{"trusted"}}, "input", "from trusted"},
+		{model.Match{}, "forward", "everything"},
+	} {
+		if got := describe(c.m, c.chain); got != c.want {
+			t.Errorf("describe(%+v, %s) = %q, want %q", c.m, c.chain, got, c.want)
+		}
+	}
+	for used, want := range map[int64]string{0: "0%", 5: "<1%", 250: "25%", 1000: "100%", 1300: "130%"} {
+		if got := percent(used, 1000); got != want {
+			t.Errorf("percent(%d, 1000) = %q, want %q", used, got, want)
+		}
 	}
 	if got := editList([]string{"a", "b", "c"}, []string{"d"}, []string{"b"}); len(got) != 3 || got[1] != "c" || got[2] != "d" {
 		t.Errorf("editList: %v", got)
 	}
 }
+
+// TestMatrixAligned: the grid's columns line up in colour too (the
+// escape codes take no room on screen).
+func TestMatrixAligned(t *testing.T) {
+	st := model.NewState()
+	st.Zones = []*model.Zone{{ID: 1, Name: "lan", Interfaces: []string{"eth1"}}, {ID: 2, Name: "wan", Interfaces: []string{"wan"}, External: true}}
+	st.Rules = []*model.Rule{{ID: 3, Chain: model.ChainForward, Position: 1, Action: model.ActionAccept, Match: model.Match{InZones: []string{"lan"}, Services: []string{"https"}}}}
+	st.Settings.InputPolicy, st.Settings.ForwardPolicy = model.PolicyDrop, model.PolicyDrop
+	var b strings.Builder
+	writeMatrix(&b, matrix.Build(st), true)
+	esc := regexp.MustCompile("\x1b\\[[0-9]+m")
+	lines := strings.Split(esc.ReplaceAllString(b.String(), ""), "\n")
+	col := strings.Index(lines[0], "wan")
+	for _, l := range lines[1:4] {
+		if r := []rune(l); len(r) <= col || r[col-1] != ' ' || r[col] == ' ' {
+			t.Errorf("misaligned at column %d:\n%s", col, strings.Join(lines[:4], "\n"))
+			break
+		}
+	}
+	if !strings.Contains(b.String(), "\x1b[33mpartly +1") || !strings.Contains(b.String(), "lan → wan\thttps") {
+		t.Errorf("no partly cell, or no list of what's open:\n%s", b.String())
+	}
+}

cmd/hc/objects.go

@@ -49,6 +49,46 @@ func counters() map[string]api.Counter {
 	return cs.Counters
 }
 
+// lookupNames asks for inside addresses' names, for tables: none with
+// --no-names, another output format, or when they can't be had.
+func lookupNames(addrs ...string) map[string]string {
+	out := map[string]string{}
+	if flagNoNames || flagOutput != "table" {
+		return out
+	}
+	seen := map[string]bool{}
+	var list []string
+	for _, a := range addrs {
+		if a != "" && !seen[a] {
+			seen[a] = true
+			list = append(list, a)
+		}
+	}
+	if len(list) == 0 {
+		return out
+	}
+	c, err := newClient()
+	if err != nil {
+		return out
+	}
+	var ns api.Names
+	if c.Do(ctx(), "POST", "/names", nil, api.NamesInput{Addresses: list}, &ns) != nil {
+		return out
+	}
+	for a, n := range ns.Names {
+		out[a] = strings.SplitN(n, ".", 2)[0]
+	}
+	return out
+}
+
+// named adds an address's name to text (the address, or address:port).
+func named(names map[string]string, addr, text string) string {
+	if n := names[addr]; n != "" {
+		return text + " (" + n + ")"
+	}
+	return text
+}
+
 func hits(c map[string]api.Counter, marks ...string) string {
 	var p uint64
 	found := false

cmd/hc/rules.go

@@ -3,6 +3,7 @@ package main
 import (
 	"fmt"
 	"io"
+	"net/url"
 	"regexp"
 	"strconv"
 	"strings"
@@ -63,22 +64,40 @@ func (m *matchFlags) patch(fl *pflag.FlagSet, p patch) {
 	set("family", "family", m.family)
 }
 
-// describe sums up a match: "in lan  from trusted  ssh, dns".
+// describe sums up a match in a chain ("" for NAT): "lan → firewall
-func describe(m model.Match) string {
+// from trusted  ssh,dns". Zones are a direction; the firewall itself is
+// the other end of the input and output chains.
+func describe(m model.Match, chain string) string {
 	var parts []string
-	side := func(word string, not bool, list []string) {
+	list := func(not bool, xs []string) string {
-		if len(list) == 0 {
+		s := strings.Join(xs, ",")
-			return
-		}
 		if not {
-			word += " not"
+			return "any but " + s
 		}
-		parts = append(parts, word+" "+strings.Join(list, ","))
+		return s
+	}
+	if len(m.InZones) > 0 || len(m.OutZones) > 0 {
+		in, out := "any", "any"
+		if len(m.InZones) > 0 {
+			in = list(m.NotIn, m.InZones)
+		}
+		if len(m.OutZones) > 0 {
+			out = list(m.NotOut, m.OutZones)
+		}
+		switch chain {
+		case model.ChainInput:
+			out = "firewall"
+		case model.ChainOutput:
+			in = "firewall"
+		}
+		parts = append(parts, in+" → "+out)
+	}
+	if len(m.Sources) > 0 {
+		parts = append(parts, "from "+list(m.NotSources, m.Sources))
+	}
+	if len(m.Destinations) > 0 {
+		parts = append(parts, "to "+list(m.NotDests, m.Destinations))
 	}
-	side("in", m.NotIn, m.InZones)
-	side("out", m.NotOut, m.OutZones)
-	side("from", m.NotSources, m.Sources)
-	side("to", m.NotDests, m.Destinations)
 	if len(m.Services) > 0 {
 		parts = append(parts, strings.Join(m.Services, ","))
 	}
@@ -280,7 +299,7 @@ func ruleCmd() *cobra.Command {
 			return output(rs, func(w io.Writer) {
 				fmt.Fprintln(w, "CHAIN\tID\tMATCH\tACTION\tHITS\tCOMMENT")
 				for _, r := range rs {
-					what := describe(r.Match)
+					what := describe(r.Match, r.Chain)
 					if r.Raw != "" {
 						what = "raw: " + r.Raw
 					}
@@ -303,7 +322,7 @@ func ruleCmd() *cobra.Command {
 	cmd.AddCommand(ls)
 	cmd.AddCommand(showCmd("rules", "rule", func(w io.Writer, r model.Rule) {
 		if r.Raw == "" {
-			fmt.Fprintf(w, "ID:\t%d\nChain:\t%s\nMatch:\t%s%s\nAction:\t%s\n", r.ID, r.Chain, describe(r.Match), ruleExtras(r), r.Action)
+			fmt.Fprintf(w, "ID:\t%d\nChain:\t%s\nMatch:\t%s%s\nAction:\t%s\n", r.ID, r.Chain, describe(r.Match, r.Chain), ruleExtras(r), r.Action)
 		} else {
 			fmt.Fprintf(w, "ID:\t%d\nChain:\t%s\n", r.ID, r.Chain)
 		}
@@ -340,26 +359,21 @@ output (from it) or early (before connection tracking: cheap drops).
 			if err != nil {
 				return err
 			}
+			// Placed in the same change: one history entry, and never in
+			// force at the end of the chain.
+			q := url.Values{}
+			switch {
+			case rf.first:
+				q.Set("first", "1")
+			case rf.before != 0:
+				q.Set("before", fmt.Sprint(rf.before))
+			case rf.after != 0:
+				q.Set("after", fmt.Sprint(rf.after))
+			}
 			var resp api.MutationResponse
-			if err := c.Do(ctx(), "POST", "/rules", nil, r, &resp); err != nil {
+			if err := c.Do(ctx(), "POST", "/rules", q, r, &resp); err != nil {
 				return err
 			}
-			// Then place it.
-			if resp.Applied && (rf.first || rf.before != 0 || rf.after != 0) {
-				var created model.Rule
-				if err := jsonUnmarshal(resp.Object, &created); err == nil {
-					mv := api.MoveInput{Before: rf.before, After: rf.after}
-					if rf.first {
-						mv.To = "start"
-					}
-					var moved api.MutationResponse
-					if err := c.Do(ctx(), "POST", objPath("rules", fmt.Sprint(created.ID))+"/move", nil, mv, &moved); err != nil {
-						return err
-					}
-					resp.Changes = append(resp.Changes, moved.Changes...)
-					resp.ConfirmBy = moved.ConfirmBy
-				}
-			}
 			return report(&resp)
 		},
 	}
@@ -426,22 +440,57 @@ output (from it) or early (before connection tracking: cheap drops).
 	move.Flags().BoolVar(&last, "last", false, "last in its chain")
 	move.Flags().StringVar(&chain, "chain", "", "to the end of another chain")
 	cmd.AddCommand(move)
-	cmd.AddCommand(&cobra.Command{
+	var resetAddr, give string
-		Use: "reset-quota ID", Short: "Start a rule's quota again now (every address's, for a per-address quota)", Args: cobra.ExactArgs(1),
+	resetCmd := &cobra.Command{
+		Use: "reset-quota ID [--address ADDR] [--give AMOUNT]", Short: "Start a rule's quota again now, or give some back until its next reset", Args: cobra.ExactArgs(1),
+		Long: `Starts a rule's quota again now: all of it (every address's, for a
+per-address quota), or with --address one address's. --give takes an
+amount off what was used instead: extra allowance until the quota's next
+reset, without changing the rule. Done at once, not staged or on
+probation.
+
+  hc rule reset-quota 31
+  hc rule reset-quota 31 --address 192.168.1.23
+  hc rule reset-quota 31 --give 5gbytes --address 192.168.1.23`,
 		RunE: func(cmd *cobra.Command, args []string) error {
+			in := api.QuotaReset{Address: resetAddr}
+			if give != "" {
+				bad := fmt.Errorf("--give %q: write it as 500mbytes or 5gbytes", give)
+				m := amountRe.FindStringSubmatch(strings.TrimSpace(give))
+				if m == nil {
+					return bad
+				}
+				unit, ok := byteUnit(m[2])
+				n, _ := strconv.ParseInt(m[1], 10, 64)
+				if !ok || unit == "" || n < 1 {
+					return bad
+				}
+				in.Give = n * model.ByteUnits[unit]
+			}
 			c, err := newClient()
 			if err != nil {
 				return err
 			}
-			if err := c.Do(ctx(), "POST", objPath("rules", args[0])+"/reset-quota", nil, nil, nil); err != nil {
+			if err := c.Do(ctx(), "POST", objPath("rules", args[0])+"/reset-quota", nil, in, nil); err != nil {
 				return err
 			}
 			if !flagQuiet {
-				fmt.Printf("reset the quota of rule %s\n", args[0])
+				who := ""
+				if resetAddr != "" {
+					who = " for " + resetAddr
+				}
+				if in.Give > 0 {
+					fmt.Printf("gave back %s of the quota of rule %s%s, until its next reset\n", humanBytes(in.Give), args[0], who)
+				} else {
+					fmt.Printf("reset the quota of rule %s%s\n", args[0], who)
+				}
 			}
 			return nil
 		},
-	})
+	}
+	resetCmd.Flags().StringVar(&resetAddr, "address", "", "only this address's (a per-address quota; an IPv6 address counts for its /64)")
+	resetCmd.Flags().StringVar(&give, "give", "", "take this much off what was used instead (5gbytes, 500mbytes)")
+	cmd.AddCommand(resetCmd)
 	for _, on := range []bool{true, false} {
 		verb, short := "disable", "Keep a rule but don't use it"
 		if on {
@@ -669,7 +718,7 @@ func snatCmd() *cobra.Command {
 					if to == "" {
 						to = "masquerade"
 					}
-					fmt.Fprintf(w, "%d\t%s\t%s\t%s\t%s\n", x.ID, describe(x.Match), to, hits(cs, fmt.Sprintf("s%d", x.ID)), x.Comment)
+					fmt.Fprintf(w, "%d\t%s\t%s\t%s\t%s\n", x.ID, describe(x.Match, ""), to, hits(cs, fmt.Sprintf("s%d", x.ID)), x.Comment)
 				}
 			})
 		},
@@ -749,7 +798,7 @@ func redirectCmd() *cobra.Command {
 			return output(xs, func(w io.Writer) {
 				fmt.Fprintln(w, "ID\tMATCH\tTO PORT\tHITS\tCOMMENT")
 				for _, x := range xs {
-					fmt.Fprintf(w, "%d\t%s\t%s\t%s\t%s\n", x.ID, describe(x.Match), orDash(x.ToPort), hits(cs, fmt.Sprintf("d%d", x.ID)), x.Comment)
+					fmt.Fprintf(w, "%d\t%s\t%s\t%s\t%s\n", x.ID, describe(x.Match, ""), orDash(x.ToPort), hits(cs, fmt.Sprintf("d%d", x.ID)), x.Comment)
 				}
 			})
 		},

cmd/hc/stats.go

@@ -283,6 +283,7 @@ func statsTopCmd() *cobra.Command {
 				for _, l := range [][]statEntry{down, up} {
 					for _, e := range l {
 						if _, ok := out[e.Key]; !ok {
+							out[e.Key] = [2]int64{}
 							order = append(order, e.Key)
 						}
 					}
@@ -297,10 +298,11 @@ func statsTopCmd() *cobra.Command {
 					v[1] = e.Count
 					out[e.Key] = v
 				}
+				names := lookupNames(order...)
 				return output(map[string]any{"up": up, "down": down}, func(w io.Writer) {
 					fmt.Fprintln(w, "DEVICE\tRECEIVED\tSENT")
 					for _, k := range order {
-						fmt.Fprintf(w, "%s\t%s\t%s\n", k, humanBytes(out[k][0]), humanBytes(out[k][1]))
+						fmt.Fprintf(w, "%s\t%s\t%s\n", named(names, k, k), humanBytes(out[k][0]), humanBytes(out[k][1]))
 					}
 				})
 			case "blocked", "tried":
@@ -309,10 +311,14 @@ func statsTopCmd() *cobra.Command {
 				if err != nil {
 					return err
 				}
+				var names map[string]string
+				if args[0] == "blocked" {
+					names = lookupNames(keys(es)...)
+				}
 				return output(es, func(w io.Writer) {
 					fmt.Fprintln(w, map[string]string{"blocked": "SOURCE", "tried": "WHAT"}[args[0]]+"\tPACKETS")
 					for _, e := range es {
-						fmt.Fprintf(w, "%s\t%d\n", e.Key, e.Count)
+						fmt.Fprintf(w, "%s\t%d\n", named(names, e.Key, e.Key), e.Count)
 					}
 				})
 			case "quota":
@@ -323,10 +329,11 @@ func statsTopCmd() *cobra.Command {
 				if err != nil {
 					return err
 				}
+				names := lookupNames(keys(es)...)
 				return output(es, func(w io.Writer) {
 					fmt.Fprintln(w, "ADDRESS\tUSED")
 					for _, e := range es {
-						fmt.Fprintf(w, "%s\t%s\n", e.Key, humanBytes(e.Count))
+						fmt.Fprintf(w, "%s\t%s\n", named(names, e.Key, e.Key), humanBytes(e.Count))
 					}
 				})
 			}
@@ -338,6 +345,15 @@ func statsTopCmd() *cobra.Command {
 	return cmd
 }
 
+// keys are a top list's keys.
+func keys(es []statEntry) []string {
+	var out []string
+	for _, e := range es {
+		out = append(out, e.Key)
+	}
+	return out
+}
+
 func statsRulesCmd() *cobra.Command {
 	var since string
 	var unused bool

cmd/hc/traffic.go

@@ -9,6 +9,7 @@ import (
 	"strconv"
 	"strings"
 	"time"
+	"unicode/utf8"
 
 	"github.com/spf13/cobra"
 
@@ -37,10 +38,16 @@ rules as it stands.
 			if err != nil {
 				return err
 			}
+			var addrs []string
+			for _, c := range cs {
+				addrs = append(addrs, c.Source, c.Destination)
+			}
+			names := lookupNames(addrs...)
 			return output(cs, func(w io.Writer) {
 				fmt.Fprintln(w, "PROTO\tFROM\tTO\tNAT\tSTATE\tEXPIRES")
 				for _, c := range cs {
-					fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\n", c.Protocol, hostPort(c.Source, c.SourcePort), hostPort(c.Destination, c.DestinationPort), natText(c), c.State, fmtSeconds(c.Timeout))
+					fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\n", c.Protocol, named(names, c.Source, hostPort(c.Source, c.SourcePort)),
+						named(names, c.Destination, hostPort(c.Destination, c.DestinationPort)), natText(c), c.State, fmtSeconds(c.Timeout))
 				}
 			})
 		},
@@ -205,6 +212,13 @@ func printPacket(w io.Writer, p api.TracePacket) {
 	} else {
 		fmt.Fprintf(w, "    => %s by %s\n", strings.ToUpper(p.Verdict), decided)
 	}
+	if wd := p.WouldDrop; wd != nil {
+		name := wd.Label
+		if name == "" {
+			name = strings.TrimSpace(wd.Chain + " " + wd.Mark)
+		}
+		fmt.Fprintf(w, "    => shadow mode: Housecarl would have dropped it, by %s\n", name)
+	}
 }
 
 func matrixCmd() *cobra.Command {
@@ -242,42 +256,112 @@ such as some services or addresses). With FROM and TO (a zone, or
 }
 
 func printMatrix(w io.Writer, m api.Matrix) {
+	writeMatrix(w, m, isTerminal(os.Stdout) && flagOutput == "table")
+}
+
+// writeMatrix writes the grid, in colour on a terminal.
+func writeMatrix(w io.Writer, m api.Matrix, color bool) {
 	cols := append(append([]string{}, m.Zones...), "firewall")
 	cells := map[[2]string]api.MatrixCell{}
 	for _, c := range m.Cells {
 		cells[[2]string{c.From, c.To}] = c
 	}
-	color := isTerminal(os.Stdout) && flagOutput == "table"
+	// The grid is padded by hand: tabwriter would count the colours'
-	fmt.Fprint(w, "FROM \\ TO")
+	// escape codes as text, and misalign the columns.
-	for _, c := range cols {
+	grid := [][]string{append([]string{"FROM \\ TO"}, cols...)}
-		fmt.Fprintf(w, "\t%s", c)
+	kinds := [][]string{make([]string, len(cols)+1)}
-	}
+	same := false
-	fmt.Fprintln(w)
 	for _, from := range cols {
-		fmt.Fprint(w, from)
+		row, kind := []string{from}, []string{""}
 		for _, to := range cols {
 			c, ok := cells[[2]string{from, to}]
-			text := "-"
+			if !ok {
-			if ok {
+				row, kind = append(row, "-"), append(kind, "")
-				text = map[string]string{"allowed": "allow", "blocked": "block"}[c.Verdict]
+				continue
-				if n := len(c.Exceptions); n > 0 {
-					text += fmt.Sprintf(" +%d", n)
 			}
-				if color {
+			text := map[string]string{"allowed": "allow", "blocked": "block", "partly": "partly"}[c.Summary]
-					text = map[string]string{"allowed": "\x1b[32m", "blocked": "\x1b[31m"}[c.Verdict] + text + "\x1b[0m"
+			if c.Allowed > 0 {
+				text += fmt.Sprintf(" +%d", c.Allowed)
+			}
+			if c.Blocked > 0 {
+				text += fmt.Sprintf(" -%d", c.Blocked)
+			}
+			if c.SameZone {
+				text += " *"
+				same = true
+			}
+			row, kind = append(row, text), append(kind, c.Summary)
+		}
+		grid, kinds = append(grid, row), append(kinds, kind)
+	}
+	width := make([]int, len(cols)+1)
+	for _, row := range grid {
+		for i, t := range row {
+			width[i] = max(width[i], utf8.RuneCountInString(t))
 		}
 	}
-			fmt.Fprintf(w, "\t%s", text)
+	paint := map[string]string{"allowed": "\x1b[32m", "blocked": "\x1b[31m", "partly": "\x1b[33m"}
+	for r, row := range grid {
+		var b strings.Builder
+		for i, t := range row {
+			pad := strings.Repeat(" ", width[i]-utf8.RuneCountInString(t))
+			if i == len(row)-1 {
+				pad = ""
+			} else {
+				pad += "  "
 			}
-		fmt.Fprintln(w)
+			if p := paint[kinds[r][i]]; color && p != "" {
+				t = p + t + "\x1b[0m"
+			}
+			b.WriteString(t + pad)
+		}
+		fmt.Fprintln(w, b.String())
+	}
+	// What the partly open pairs let through.
+	var open [][2]string
+	for _, from := range cols {
+		for _, to := range cols {
+			c, ok := cells[[2]string{from, to}]
+			if !ok || len(c.Open) == 0 || c.SameZone {
+				continue
+			}
+			items := c.Open
+			more := ""
+			if len(items) > 6 {
+				items, more = items[:6], fmt.Sprintf("; %d more", len(c.Open)-6)
+			}
+			open = append(open, [2]string{from + " → " + to, strings.Join(items, "; ") + more})
+		}
+	}
+	if len(open) > 0 {
+		fmt.Fprintln(w, "\nLet through for part of the traffic:")
+		for _, o := range open {
+			fmt.Fprintf(w, "  %s\t%s\n", o[0], o[1])
+		}
+	}
+	fmt.Fprintln(w, "\nallow, block: every new connection; partly: some are let through, some not.")
+	fmt.Fprintln(w, "+N: rules that let part of the traffic through; -N: rules that block part of it. 'hc matrix FROM TO' lists them.")
+	if same {
+		fmt.Fprintln(w, "*: a zone to itself: only traffic routed between its interfaces passes the firewall.")
 	}
-	fmt.Fprintln(w, "\nallow/block: new connections no exception covers; +N: rules deciding for part of the traffic first ('hc matrix FROM TO')")
 }
 
 func printCell(w io.Writer, c api.MatrixCell) {
-	fmt.Fprintf(w, "%s → %s (the %s chain), new connections:\n", c.From, c.To, c.Chain)
+	summary := map[string]string{"allowed": "all allowed", "blocked": "all blocked", "partly": "partly allowed"}[c.Summary]
+	fmt.Fprintf(w, "%s → %s (the %s chain), new connections: %s\n", c.From, c.To, c.Chain, summary)
+	if c.Note != "" {
+		fmt.Fprintf(w, "  (%s)\n", c.Note)
+	}
+	hits := func(h *uint64) string {
+		if h == nil {
+			return "-"
+		}
+		return fmt.Sprintf("%d", *h)
+	}
+	fmt.Fprintln(w, "  VERDICT\tWHAT\tBY\tHITS")
 	for _, e := range c.Exceptions {
-		fmt.Fprintf(w, "  %s\t%s\t%s\n", e.Verdict, e.What, e.Label)
+		fmt.Fprintf(w, "  %s\t%s\t%s\t%s\n", e.Verdict, e.What, e.Label, hits(e.Hits))
 	}
-	fmt.Fprintf(w, "  %s\tthe rest\t%s\n", c.Verdict, c.DecidedBy)
+	fmt.Fprintf(w, "  %s\tthe rest\t%s\t%s\n", c.Verdict, c.DecidedBy, hits(c.Hits))
+	fmt.Fprintln(w, "\nHits: each rule's packets since housecarld started, in all its traffic (a rule can cover several pairs).")
 }

cmd/housecarld/main.go

@@ -32,12 +32,14 @@ import (
 	"housecarl/internal/jarl"
 	"housecarl/internal/linkwatch"
 	"housecarl/internal/model"
+	"housecarl/internal/names"
 	"housecarl/internal/nflog"
 	"housecarl/internal/nft"
 	"housecarl/internal/render"
 	"housecarl/internal/server"
 	"housecarl/internal/stats"
 	"housecarl/internal/store"
+	"housecarl/internal/tablewatch"
 	"housecarl/internal/trace"
 	"housecarl/internal/validate"
 	"housecarl/internal/version"
@@ -153,8 +155,13 @@ func serveCmd() *cobra.Command {
 			}
 			host := &hostif.Host{}
 			eng.Env = hostEnv(host)
-			fw := &nft.Firewall{Runner: &nft.Runner{Command: nft.Wrapper()}, Interfaces: host.Exists, BootFile: a.cfg.Ruleset}
+			fw := &nft.Firewall{Runner: &nft.Runner{Command: nft.Wrapper()}, Interfaces: host.Exists, BootFile: a.cfg.Ruleset,
+				Shadow: a.cfg.Shadow.Enabled, ShadowBefore: a.cfg.Shadow.Position == config.ShadowBefore,
+				Bans: func() []render.Ban { return server.ActiveBans(context.Background(), a.store) }}
 			eng.Firewall = fw
+			if fw.Shadow {
+				a.log.Warn("shadow mode: Housecarl's ruleset is loaded but blocks, rejects and translates nothing; the other firewall decides (docs/shadow.md)", "position", a.cfg.Shadow.Position)
+			}
 			for _, p := range eng.Problems() {
 				a.log.Warn("configuration problem: "+p.Message, "object", p.Object, "code", p.Code)
 			}
@@ -171,10 +178,15 @@ func serveCmd() *cobra.Command {
 				acme = certs.New(ctx, certs.Config{ACME: a.cfg.TLS.ACME, Version: version.Short()}, jc, a.store, a.log)
 				go acme.Run(ctx)
 			}
+			var resolver *names.Resolver
+			if a.cfg.Names.Enabled {
+				resolver = names.New(a.cfg.Names.Resolver, time.Duration(a.cfg.Names.Cache))
+			}
 			srv := server.New(server.Config{
 				Listen: a.cfg.Listen, TLSCertFile: a.cfg.TLS.CertFile, TLSKeyFile: a.cfg.TLS.KeyFile, TrustedProxies: a.cfg.Proxies(),
 				SessionLifetime: time.Duration(a.cfg.Session.Lifetime), SessionIdle: time.Duration(a.cfg.Session.IdleTimeout),
 				CookieSecure: a.cfg.Session.CookieSecure, Socket: a.cfg.Socket, ACME: acme,
+				Shadow: a.cfg.Shadow.Enabled, ShadowPosition: a.cfg.Shadow.Position, Names: resolver,
 			}, eng, fw, host, a.log, web.FS())
 			// Put the ruleset in force: what was on probation when
 			// housecarld stopped is rolled back first. An empty
@@ -182,12 +194,15 @@ func serveCmd() *cobra.Command {
 			// close the machine before any rule lets anyone in.
 			// What the quotas used before housecarld (or the machine)
 			// stopped comes back with the next apply.
+			restoreQuotas := func() {
 				if v, err := a.store.GetMeta(ctx, server.QuotaUsageKey); err == nil && v != "" {
 					var u nft.QuotaUsage
 					if json.Unmarshal([]byte(v), &u) == nil {
 						fw.RestoreQuotas(&u)
 					}
 				}
+			}
+			restoreQuotas()
 			if !srv.ResumeProbation(ctx) {
 				if eng.State().Empty() {
 					a.log.Info("the configuration is empty: no ruleset is applied until the first change (or 'hc import')")
@@ -215,6 +230,29 @@ func serveCmd() *cobra.Command {
 					a.log.Warn("not watching the interfaces: a flowtable device that appears later is only added by the next change", "err", err)
 				}
 			}()
+			// In shadow mode the other firewall's ruleset is reloaded now
+			// and then, and a ruleset file usually starts with flush
+			// ruleset, which deletes Housecarl's table too: put it back.
+			if fw.Shadow {
+				go func() {
+					err := tablewatch.Watch(ctx, func() {
+						if _, err := fw.Table(ctx); !errors.Is(err, nft.ErrNoTable) || eng.State().Empty() {
+							return
+						}
+						// The quotas' usage went with the table: from
+						// what was last saved.
+						restoreQuotas()
+						if err := eng.ApplyCurrent(ctx); err != nil {
+							a.log.Error("Housecarl's table was deleted, and putting it back failed", "err", err)
+							return
+						}
+						a.log.Info("Housecarl's table was deleted (flush ruleset?): put it back")
+					})
+					if err != nil {
+						a.log.Warn("not watching nftables: if Housecarl's table is deleted, only the next change or a restart puts it back", "err", err)
+					}
+				}()
+			}
 			go nflog.Listen(ctx, render.LogGroup, a.log, srv.LogPacket)
 			go srv.RunQuotas(ctx)
 			srv.SetConntrack(conntrack.Client{})

contrib/fail2ban/action.d/housecarl.conf

@@ -0,0 +1,52 @@
+# Housecarl's action for fail2ban: a ban goes into a Housecarl ban list,
+# and the list says what the banned address is kept from (the firewall
+# itself, some hosts behind it, or everything). See docs/bans.md.
+#
+# Copy this file to /etc/fail2ban/action.d/housecarl.conf, make a ban
+# list and a token for it alone on the firewall:
+#
+#   hc banlist add web --scope hosts --host 10.0.10.57
+#   hc user add f2b-web --no-password --grant bans:write:web
+#   hc token create web --user f2b-web        # as root on the firewall
+#
+# put the token in /etc/fail2ban/housecarl.token (mode 0600), and use it
+# in a jail:
+#
+#   [sshd]
+#   enabled = true
+#   action  = housecarl[list=web, url="https://router.lan:8070"]
+#
+# The jail's port and protocol go along: port = ssh bans only tcp/22;
+# fail2ban's default (0:65535) bans every port. IPv6 addresses are banned
+# with the list's grouping (a /64 unless set otherwise). Housecarl never
+# bans inside networks, its own addresses or the list's "never".
+#
+# On the firewall itself, the local socket works too, still with the
+# token: url = http://localhost, curlopts = --unix-socket
+# /run/housecarl/housecarl.sock.
+
+[Definition]
+
+actionstart =
+actionstop =
+actioncheck =
+
+# A ban renews one already there; a bantime of -1 is the list's longest.
+actionban = <_curl> -f -X POST -H "Content-Type: application/json" "<url>/api/v1/banlists/<list>/bans"
+            -d '{"address": "<ip>", "protocol": "<protocol>", "ports": "<port>", "time": <bantime>, "source": "fail2ban/<name>"}'
+
+# A ban Housecarl already ended (its own time ran out) is no error.
+actionunban = <_curl> -X DELETE "<url>/api/v1/banlists/<list>/bans?address=<ip>"
+
+[Init]
+
+# The ban list, and Housecarl's URL (its listen address).
+list =
+url = https://127.0.0.1:8070
+# The API token, in a file only root reads.
+tokenfile = /etc/fail2ban/housecarl.token
+# More curl options: --cacert /etc/fail2ban/housecarl-ca.pem for a
+# private CA, --unix-socket for the local socket.
+curlopts =
+
+_curl = curl -sS --max-time 10 <curlopts> -o /dev/null -H "Authorization: Bearer $(cat <tokenfile>)"

docs/api.md

@@ -31,7 +31,7 @@ user name. [permissions.md](permissions.md) has what each call needs.
 | 401 | `unauthenticated`, `session_expired`, `login_failed` | |
 | 403 | `forbidden`, `csrf` | |
 | 404 | `not_found` | |
-| 409 | `conflict` | e.g. undoing a change after the objects changed again |
+| 409 | `conflict` | e.g. undoing a change after the objects changed again; a takeover or closing connections in shadow mode ([shadow.md](shadow.md)) |
 | 422 | `validation` | the change would make the configuration invalid; `problems` lists why ([validation.md](validation.md)) |
 | 422 | `firewall` | nftables refused the ruleset (`nft -c`), or applying it failed; the message has nftables' words. Nothing was changed |
 | 429 | `rate_limited` | |
@@ -50,7 +50,7 @@ Calls that change the configuration answer:
 
 * `confirm_by`: the change is on probation, and is rolled back at that
   time unless `POST /confirm` comes first. Missing for changes made
-  through the socket.
+  through the socket, and in shadow mode ([shadow.md](shadow.md)).
 * `confirm_token`: the client that made the change can confirm it by
   itself with `POST /confirm {"token"}`, from the same address, as the
   same user and token, over a new connection (the answer closes this
@@ -74,19 +74,20 @@ Creating answers 201, other changes 200.
 
 | | |
 |---|---|
-| `GET /healthz` | `ok`, `version`, `firewall` (`applied`, `unconfigured`, `fallback`, `missing`), `probation` (changes waiting to be confirmed), `warnings` (codes); HTTP 503 unless `ok`. Without authentication; read at most every five seconds ([operations.md](operations.md#monitoring)) |
+| `GET /healthz` | `ok`, `version`, `firewall` (`applied`, `unconfigured`, `fallback`, `missing`), `mode` (`enforcing`, or `shadow`: the ruleset blocks nothing, [shadow.md](shadow.md)), `probation` (changes waiting to be confirmed), `warnings` (codes); HTTP 503 unless `ok`. Without authentication; read at most every five seconds ([operations.md](operations.md#monitoring)) |
-| `GET /status` | `build`, `started`, `applied` (the ruleset is in force), `apply_error`, `problems`, `probation`, `interfaces`, `counts` (objects by kind), `other_tables` (other nftables tables, `"inet main"`) |
+| `GET /status` | `build`, `started`, `mode` (`enforcing` or `shadow`), `shadow_position` (`after` or `before`, in shadow mode), `applied` (the ruleset is in force), `apply_error`, `problems`, `probation`, `interfaces`, `counts` (objects by kind), `other_tables` (other nftables tables, `"inet main"`) |
 | `GET /interfaces` | the machine's interfaces: `name`, `up`, `addresses`, `zone` |
-| `GET /matrix[?changeset=ID]` | the zone matrix: `zones`, `cells` (`from`, `to` (a zone, or `firewall`), `chain`, `verdict` (allowed, blocked), `decided_by`, `rule_id`, `exceptions`: `rule_id`, `chain`, `label`, `verdict` (allowed, blocked, logged, unknown), `what`) ([rules.md](rules.md#the-zone-matrix)) |
+| `GET /matrix[?changeset=ID]` | the zone matrix: `zones`, `cells` (`from`, `to` (a zone, or `firewall`), `chain`, `summary` (allowed, blocked, partly), `verdict` (what the rest gets: allowed, blocked), `decided_by`, `rule_id`, `hits`, `allowed` and `blocked` (exceptions by verdict), `open` (what the allowing exceptions let through, in short), `same_zone` and `note`, `exceptions`: `rule_id`, `chain`, `label`, `verdict` (allowed, blocked, logged, unknown), `what`, `short`, `hits`). `hits` are the rule's or policy's packets since housecarld started, all its traffic (missing: no counter) ([rules.md](rules.md#the-zone-matrix)) |
-| `GET /ruleset` | the nftables ruleset Housecarl builds, as text |
+| `GET /ruleset[?enforcing=1]` | the nftables ruleset Housecarl builds, as text. In shadow mode it is the shadow ruleset; `enforcing=1` gives the one loaded once shadow mode is off |
 | `GET /counters` | `{"since", "counters": {"r12": {"packets", "bytes"}, ...}}`, by mark ([design.md](design.md#its-own-table-only)); totals since housecarld started |
 | `GET /quotas` | each rule's quota: `rule_id`, `rule`, `quota`, `used` (bytes), `addresses` (`[{"address", "used"}]` for a per-address quota), `last_reset`, `next_reset` |
 | `GET /connections[?q=&limit=]` | tracked connections: `family`, `protocol`, `source`, `source_port`, `destination`, `destination_port` (or `icmp_id`, `icmp_type`, `icmp_code`), `dnat_to`, `dnat_port`, `snat_to`, `state`, `assured`, `timeout` (seconds left), `packets` and `bytes` when the kernel counts them. `q`: all of its words in the addresses, `:port`, protocol, state |
-| `POST /connections/close` | `{"connections": [...]}` (as listed) or `{"search": "192.168.1.50"}`: delete their tracking entries (`config:write`, audited); answers `{"closed": N}` |
+| `POST /names` | `{"addresses": [...]}` (at most 2000): `{"enabled", "names": {"192.168.1.20": "laptop.example.net"}}`, the names of the inside addresses among them, by reverse lookup ([configuration.md](configuration.md#names)). Addresses without a name, outside ones, and those still being looked up after a second are left out; ask again for those. `log:read` |
+| `POST /connections/close` | `{"connections": [...]}` (as listed) or `{"search": "192.168.1.50"}`: delete their tracking entries (`config:write`, audited); answers `{"closed": N}`. 409 in shadow mode |
 | `POST /trace` | `{"address", "port", "protocol", "seconds"}` (5-600, default 60): start tracing (`config:write`, audited); 409 while one runs |
-| `GET /trace` | the trace running, or the last: `running`, `input`, `started`, `ends`, `error`, `packets`: each with `time`, `in`, `out`, `protocol`, addresses and ports, `tcp_flags`, `icmp_type`, `steps` (`table`, `chain`, `kind` rule or policy, `rule`, `mark`, `label`, `verdict`), `verdict` (accepted, dropped) and `decided_by` |
+| `GET /trace` | the trace running, or the last: `running`, `input`, `started`, `ends`, `error`, `packets`: each with `time`, `in`, `out`, `protocol`, addresses and ports, `tcp_flags`, `icmp_type`, `steps` (`table`, `chain`, `kind` rule or policy, `rule`, `mark`, `label`, `verdict`), `verdict` (accepted, dropped) and `decided_by`; in shadow mode `would_drop`, the step where Housecarl would have dropped it |
 | `DELETE /trace` | stop it |
-| `GET /log[?limit=&after=&q=]` | logged packets, newest first: `time`, `prefix`, `in`, `out`, `family`, `protocol`, `source`, `source_port`, `destination`, `destination_port`, `icmp_type`, `tcp_flags`, `length`, `mark` (the rule's). `after` (a time) for those since; `q` searches |
+| `GET /log[?limit=&after=&q=]` | logged packets, newest first: `time`, `prefix`, `in`, `out`, `family`, `protocol`, `source`, `source_port`, `destination`, `destination_port`, `icmp_type`, `tcp_flags`, `length`, `mark` (the rule's), `verdict`, `shadow` (logged in shadow mode: a drop or reject is what would have been done). `after` (a time) for those since; `q` searches |
 | `GET /catalog` | built-in services, chains, actions, roles, the log group |
 
 ### Settings
@@ -107,18 +108,18 @@ Lists replace the old ones. `probation_seconds` 0 is the default (120).
 
 ### Objects
 
-Zones, sets, services, rules, port forwards, source NAT and redirects
+Zones, sets, services, rules, port forwards, source NAT, redirects and
-share one pattern, `/{kind}` with kind `zones`, `sets`, `services`,
+ban lists share one pattern, `/{kind}` with kind `zones`, `sets`,
-`rules`, `forwards`, `snats` or `redirects`:
+`services`, `rules`, `forwards`, `snats`, `redirects` or `banlists`:
 
 | | |
 |---|---|
 | `GET /{kind}` | all of them, in order; `GET /rules?chain=input` for one chain; `GET /services` includes the built-in ones |
-| `POST /{kind}` | create; rules, forwards and NAT rules go last unless given a `position` |
+| `POST /{kind}[?before=ID\|after=ID\|first=1]` | create; rules, forwards and NAT rules go last unless given a `position`, or placed before or after another of their chain, or first, in the same change (one history entry; staged in a changeset, the place goes with it) |
-| `GET /{kind}/{ref}` | by ID, or name for zones, sets, services and forwards |
+| `GET /{kind}/{ref}` | by ID, or name for zones, sets, services, forwards and ban lists |
 | `PATCH /{kind}/{ref}` | change the fields given (a JSON merge: `null` clears) |
 | `DELETE /{kind}/{ref}` | refused while something uses it |
-| `POST /rules/{ref}/reset-quota` | start a rule's quota again now (`config:write`; audited, not staged) |
+| `POST /rules/{ref}/reset-quota` | start a rule's quota again now (`config:write`; audited, not staged). Optional body: `{"address": "192.168.1.23"}` starts only that address's again (a per-address quota; an IPv6 address counts for its /64; 404 if it hasn't used the quota); `{"give": 5368709120}` takes that many bytes off what was used instead (the address's, or every address's), until the next reset |
 | `POST /{kind}/{ref}/move` | `{"before": ID}`, `{"after": ID}` or `{"to": "start"\|"end"}`; a rule can take `"chain"` too |
 
 The objects:
@@ -144,7 +145,18 @@ The objects:
 
 A source NAT rule is a match (as a rule's, without the rule's other
 fields) with `"to"` (empty: masquerade), a redirect a match with
-`"to_port"`. Times are seconds (`timeout`).
+`"to_port"`. A ban list ([bans.md](bans.md)):
+
+```json
+{"name": "web", "scope": "hosts", "hosts": ["10.0.10.57"], "never": ["trusted"],
+ "default_time": 3600, "max_time": 604800, "max_entries": 10000, "group_v6": 64, "comment": "", "disabled": false}
+```
+
+`scope` is `firewall`, `hosts` or `everything`; 0 is the default for the
+numbers, and `max_time` -1 is no limit. A ban list can't be renamed.
+Someone with a `bans` grant but no `config:read` may read the ban lists
+the grant covers (`GET /banlists`, `GET /banlists/{ref}`). Times are
+seconds (`timeout`).
 
 ### Statistics
 
@@ -195,6 +207,22 @@ These need `sets:read` or `sets:write`, for that set
 ([permissions.md](permissions.md)); they aren't on probation or in the
 history, but are audited.
 
+### Bans
+
+| | |
+|---|---|
+| `GET /bans` | the active bans of every ban list you may read: `id`, `list_id`, `list`, `address` (an address, or an IPv6 prefix), `protocol` and `port` (none: every port), `expires` and `timeout` (seconds left; none: no end), `created`, `who`, `source`, `reason` |
+| `GET /banlists/{ref}/bans` | one list's |
+| `POST /banlists/{ref}/bans` | ban: `{"address": "203.0.113.9", "protocol": "tcp", "ports": "ssh", "time": 3600, "source": "guard/ssh", "reason": "…"}`. `ports` is a number, a string (`"22"`, `"http,https"`, `"0:65535"` or `"all"` for every port) or a list; without a protocol a number is TCP and UDP, a service its own protocols. `time` 0 is the list's default, -1 its longest; longer is cut down. 201 with `{"bans": [ban…, "existing": true for a renewed one], "closed": N}` (connections closed) |
+| `DELETE /banlists/{ref}/bans/{id}` | lift one ban |
+| `DELETE /banlists/{ref}/bans?address=A` | lift all of an address's bans (404 if it has none) |
+
+These need `bans:read` or `bans:write` for the list
+([permissions.md](permissions.md)); bans aren't on probation or in the
+history, but are audited (`ban.add`, `ban.remove`). Refusals: `422
+never_ban` (inside, the firewall's own, the caller's own, or in the
+list's `never`), `422 list_full`, `409` for a disabled list.
+
 ### Probation
 
 | | |
@@ -210,8 +238,8 @@ history, but are audited.
 | `GET /history[?limit=&before=]` | changes, newest first |
 | `GET /history/{id}` | one, with each object before and after |
 | `POST /history/{id}/revert` | undo it; 409 if an object it touched has changed since, or it was undone already |
-| `POST /import/nft` | `{"json": NFT_JSON, "text": "nft -a list ruleset output", "defines": "ruleset file", "table": "inet main", "replace": false}` ([import.md](import.md)); answers with `state` (the imported configuration), `notes` and `raw` too |
+| `POST /import/nft` | `{"json": NFT_JSON, "text": "nft -a list ruleset output", "defines": "ruleset file", "table": "inet main", "zones": {"enp2s0": "lan"}, "replace": false}` ([import.md](import.md)); answers with `state` (the imported configuration), `notes` and `raw` too |
-| `POST /takeover` | `{"tables": ["inet main"]}`: delete these tables (administrators); on probation over the network |
+| `POST /takeover` | `{"tables": ["inet main"]}`: delete these tables (administrators); on probation over the network; 409 in shadow mode |
 
 ### Certificate
 

docs/bans.md

@@ -0,0 +1,148 @@
+# Ban lists
+
+A **ban list** is where a program bans addresses: Guard (Housecarl's
+companion, which reads logs), fail2ban with Housecarl's action, or you
+with `hc ban`. The program says only *which address*, and perhaps a
+protocol and ports. The ban list says what that address is kept from.
+So a web server that sees password guessing in its own logs can ban the
+guesser at the router without knowing anything about the router.
+
+## Ban lists
+
+```sh
+hc banlist add router --scope firewall                   # the firewall itself
+hc banlist add web --scope hosts --host 10.0.10.57 --host 2001:db8:10::57
+hc banlist add edge --scope everything --never trusted
+```
+
+| Scope | Banned addresses are kept from | Where the drops are |
+|---|---|---|
+| `firewall` | the firewall itself: its SSH, DNS, web UI | `input` |
+| `hosts` | the list's hosts (addresses or sets), behind the firewall | `forward`, to those addresses |
+| `everything` | the firewall and everything behind it | `input` and `forward` |
+
+A `hosts` list covers port forwards to its hosts as well: the forward
+chain sees a forwarded connection's translated address and port, the
+host's own. A web server's Guard bans `tcp/22`, and that is right even
+when the router forwards its port 2222 to it.
+
+Other settings, with their defaults:
+
+| Setting | Default | |
+|---|---|---|
+| `never` | none | addresses, prefixes or sets this list never bans |
+| `default_time` (`--time`) | 1h | a ban's length when the program asks for none |
+| `max_time` (`--max-time`) | 7d | the longest ban; longer ones are cut to it. `-1` (`none`): no limit, so bans may be permanent |
+| `max_entries` | 10000 | the most bans the list holds; more are refused (`list_full`) |
+| `group_v6` | 64 | the IPv6 prefix length banned for an address. One customer usually has a whole /64 and can move around in it; 128 bans the one address. IPv4 addresses are banned alone |
+| `disabled` | no | the list's bans aren't in force (they are kept) |
+
+A ban list can't be renamed: the tokens and programs that ban into it
+know it by its name. Make a new one instead.
+
+## Bans
+
+```sh
+hc ban edge 203.0.113.9 --time 1d --reason "port scan"
+hc ban web 203.0.113.9 --tcp ssh --tcp 8443
+hc bans                       # every list's
+hc bans web
+hc unban web 203.0.113.9      # all of the address's bans in web
+hc unban web 17               # one, by ID
+```
+
+A ban is of **every port**, or of **a protocol's ports** (numbers or
+service names, built-in or yours). A port without a protocol is banned
+for TCP and UDP; a service name takes the service's protocols. Banning an
+address again renews the ban: its new time counts from now.
+
+Bans are dropped **ahead of every rule** in their chains, and ahead of
+the built-in that lets established connections through, so a ban cuts a
+connection that is already open. Housecarl also closes the banned
+address's tracked connections the ban covers when it bans it, because
+connections offloaded to the flowtable no longer pass the rules at all.
+Replies to connections the inside opened to a banned address are
+dropped too.
+
+Bans are kept in the database, apart from the configuration (like a
+dynamic set's entries): they aren't in the history, can't be undone,
+aren't on probation, and don't go through changesets. They end by
+themselves; every change, restart and reboot puts the active ones back
+with the time they have left. Each ban and unban is in the audit log
+(`ban.add`, `ban.remove`), with the program's source and reason.
+
+### What is never banned
+
+Whatever a list's `never` says, Housecarl refuses (`422 never_ban`) to
+ban:
+
+* the inside: addresses in networks of inside zones' interfaces, and
+  private addresses outside the external zones' networks;
+* the firewall's own addresses, and its own prefixes (settings);
+* the address of whoever asks (a program can't lock itself out);
+* loopback, link-local, multicast and unspecified addresses.
+
+A ban grouped to a prefix (IPv6) is refused when the prefix overlaps any
+of these.
+
+## Programs and tokens
+
+Give each program a token for its list alone. A `bans` grant can name
+ban lists, as a `sets` grant names sets:
+
+```sh
+hc user add guard-web --no-password --grant bans:write:web
+hc token create web --user guard-web          # as root on the firewall
+```
+
+With it, the program may ban into `web` and lift its bans, list them,
+and read the list itself (`GET /banlists/web`: its grouping and times),
+and nothing else. `viewer` has `bans:read`, `operator` `bans:write`.
+
+## Guard
+
+Guard reads log files and the systemd journal on any machine, and bans
+the addresses that misbehave into a ban list (its own documentation).
+Run it on the web server with a token for `web`, and on the router with
+one for `router` (or the local socket).
+
+## fail2ban
+
+Housecarl ships an action for fail2ban,
+`/usr/share/doc/housecarl/contrib/fail2ban/action.d/housecarl.conf`
+(`contrib/` in the source). Copy it to `/etc/fail2ban/action.d/`, put
+the token in `/etc/fail2ban/housecarl.token` (mode 0600), and use it in a
+jail:
+
+```ini
+[sshd]
+enabled = true
+action  = housecarl[list=web, url="https://router.lan:8070"]
+```
+
+The jail's `port` and `protocol` go with each ban: `port = ssh` bans only
+`tcp/22`, fail2ban's default `0:65535` bans every port. A `bantime` of
+`-1` is the list's longest ban. When fail2ban unbans an address Housecarl
+already let go (its own time ran out first), nothing happens. Add
+`curlopts = --cacert /etc/fail2ban/housecarl-ca.pem` for a private CA.
+
+## Shadow mode
+
+In shadow mode the ban rules count what they would drop
+(`[would drop]`) and let it through, like every other rule, and banning
+closes no connections.
+
+## The ruleset
+
+Each list has four sets, `ban_<id>_v4`, `ban_<id>_v6` (every port) and
+`ban_<id>_v4_svc`, `ban_<id>_v6_svc` (address, protocol and port), all
+with timeouts:
+
+```
+ip saddr @ban_3_v4 counter drop comment "hc:ban3"
+ip saddr . meta l4proto . th dport @ban_3_v4_svc counter drop comment "hc:ban3"
+ip6 saddr and ffff:ffff:ffff:ffff:: @ban_3_v6 counter drop comment "hc:ban3"
+```
+
+Set names starting with `ban_` are therefore Housecarl's own
+(`set_name_reserved`).

docs/cli.md

@@ -32,6 +32,7 @@ command and its flags.
 | `-C, --changeset ID` | `HOUSECARL_CHANGESET` (checked, see [Changesets](#changesets)) | stage configuration changes in this changeset instead of making them |
 | `-o, --output FORMAT` | | `table` (default), `json`, `yaml` or `id` |
 | `-q, --quiet` | | print only errors (and the IDs of new objects) |
+| `--no-names` | | show inside addresses without their names: tables show `192.168.1.20 (laptop)` ([configuration.md](configuration.md#names)) |
 
 Exit codes: 0 ok, 1 error, 3 not authenticated or not allowed, 4 not
 found, 5 conflict, 6 invalid or refused by validation or nftables, 7
@@ -131,15 +132,15 @@ by name.
 
 | Command | |
 |---|---|
-| `status` | the ruleset (in force or not), changes on probation, problems, interfaces, other tables |
+| `status` | the mode (in shadow mode), the ruleset (in force or not), changes on probation, problems, interfaces, other tables |
 | `interfaces` | the machine's interfaces, their addresses and zones |
-| `ruleset` | the nftables ruleset Housecarl builds |
+| `ruleset [--enforcing]` | the nftables ruleset Housecarl builds; in shadow mode, `--enforcing` shows the one loaded once it is off ([shadow.md](shadow.md)) |
 | `counters` | the hits of every rule, NAT rule, built-in and policy, by mark (`r12`, `f3`, `b-established`, `p-input`) |
 | `quotas` | what each rule's quota used (each address's, for per-address quotas), and when it resets |
 | `log [-f] [--search TEXT] [--limit N]` | the packets logged by rules and policies ([operations.md](operations.md#the-log)) |
-| `connections [--search TEXT] [--limit N]`, `connections close --search TEXT [-y]` | tracked connections; closing makes the rules check them again ([operations.md](operations.md#connections)) |
+| `connections [--search TEXT] [--limit N]`, `connections close --search TEXT [-y]` | tracked connections; closing makes the rules check them again ([operations.md](operations.md#connections)); not in shadow mode |
 | `stats [--since 24h]` | traffic per zone, drops, the busiest rules, blocked sources, devices; `stats status`, `stats traffic`, `stats top devices\|blocked\|tried\|quota ID`, `stats rules [--unused]`, `stats connections` ([stats.md](stats.md#the-cli)) |
-| `matrix [FROM TO]` | what the rules do to new connections between zones; with FROM and TO, that pair's rules in order ([rules.md](rules.md#the-zone-matrix)) |
+| `matrix [FROM TO]` | what the rules do to new connections between zones (allow, block or partly), and what is let through; with FROM and TO, that pair's rules in order, with their hits ([rules.md](rules.md#the-zone-matrix)) |
 | `trace [--address A] [--port N] [--protocol P] [--seconds S]` | follow packets through every table's rules ([operations.md](operations.md#tracing)) |
 
 ### zone, set, service
@@ -171,7 +172,7 @@ See [rules.md](rules.md).
 | `rule update ID [flags]` | only the flags given |
 | `rule move ID --before ID\|--after ID\|--first\|--last\|--chain CHAIN` | |
 | `rule enable\|disable ID` | |
-| `rule reset-quota ID` | start a rule's quota again now |
+| `rule reset-quota ID [--address ADDR] [--give 5gbytes]` | start a rule's quota again now, or one address's; `--give` takes an amount off what was used instead, until the next reset |
 | `rule delete ID` | |
 
 The match flags (`rule add`, `snat add`, `redirect add`):
@@ -203,6 +204,19 @@ accept|drop|reject|log`, `--reject-with tcp-reset`, `--log`,
 
 See [nat.md](nat.md).
 
+### banlist, ban
+
+| Command | |
+|---|---|
+| `banlist list`, `banlist show BANLIST` | ban lists, with their bans' count |
+| `banlist add NAME --scope firewall\|hosts\|everything [--host ADDR\|SET]... [--never ADDR\|SET]... [--time 1h] [--max-time 7d\|none] [--max-entries N] [--group-v6 64] [--disabled] [--comment C]` | |
+| `banlist update BANLIST [flags]`, `banlist delete BANLIST` | `--host` and `--never` replace the lists; no renaming |
+| `bans [BANLIST]` | the active bans |
+| `ban BANLIST ADDRESS [--tcp PORT\|SERVICE]... [--udp …]... [--time 1h\|none] [--reason R]` | ban every port, or some; again renews |
+| `unban BANLIST ADDRESS\|ID` | lift one ban, or all of an address's |
+
+See [bans.md](bans.md).
+
 ### settings
 
 | Command | |
@@ -219,8 +233,8 @@ See [nat.md](nat.md).
 | `history list [--limit N]` | changes, newest first |
 | `history show ID` | each object before and after |
 | `history undo ID` | apply the change backwards (refused if something it touched has changed since) |
-| `import [FILE\|-] [--text FILE] [--defines FILE] [--table T] [--replace]` | [import.md](import.md) |
+| `import [FILE\|-] [--text FILE] [--defines FILE] [--table T] [--zone IFACE=NAME]... [--replace]` | [import.md](import.md) |
-| `takeover TABLE... [-y]` | delete tables Housecarl's ruleset replaces (administrators) |
+| `takeover TABLE... [-y]` | delete tables Housecarl's ruleset replaces (administrators); not in shadow mode |
 
 ### cert
 

docs/configuration.md

@@ -30,9 +30,36 @@ reads the file when it starts.
 | `stats.database` | `/var/lib/housecarl/stats.db` | the statistics' own database; deleting it is safe |
 | `stats.retention.minute`, `.hour`, `.day` | `7d`, `90d`, `730d` | how long per-minute, hourly and daily figures are kept |
 | `stats.top_n` | `100` | entries kept per top list, per hour and per day |
+| `shadow.enabled` | `false` | shadow mode: Housecarl's ruleset is loaded beside the firewall you have, but blocks, rejects and translates nothing; its counters, log and tracer show what it would do. No change is on probation, and takeovers and closing connections are refused ([shadow.md](shadow.md)) |
+| `shadow.position` | `after` | where Housecarl's filter chains run in shadow mode: `after` the other firewall's (`filter + 10`: it sees only what that one lets through, so what it would drop is what switching would break) or `before` them (`filter - 10`: it sees every packet) |
+| `names.enabled` | `true` | show inside addresses' names beside them, found by reverse lookups (below) |
+| `names.resolver` | `""` | the DNS server to ask, such as `10.10.0.1:53`; empty: the system's (`/etc/hosts`, then `/etc/resolv.conf`) |
+| `names.cache` | `10m` | how long a name, or the lack of one, is kept |
 
 Durations are written `15s`, `12h` or `30d`.
 
+Shadow mode is here, and not a firewall setting, so that no user or API
+token can switch enforcement off; restart housecarld to switch.
+
+### Names
+
+The web UI and `hc` show a name beside each **inside** address in
+connections, the log, quotas and traffic per device: `laptop
+192.168.1.20`. Housecarl finds it with a reverse (PTR) lookup, so the
+names are your reverse zones' (and `/etc/hosts`'). An address counts as
+inside when it is in a network connected to an inside zone's interface,
+or private (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `fc00::/7`)
+and not in an external zone's network. Outside addresses get no name:
+their reverse zones belong to whoever holds the addresses, who would
+choose the names, and looking up every scanner is slow.
+
+Names are looked up when a page or `hc` asks, cached (a failed lookup is
+tried again after a minute), and never stored: statistics stay by
+address. With a split DNS (views), the server asked must answer the
+firewall as an inside client; set `names.resolver` if the system's
+resolver doesn't. IPv6 traffic per device is counted per /64, which has
+no name. `hc --no-names` leaves them out.
+
 The probation's length is a firewall setting, not here: `hc settings set
 --probation 5m`. So is counting traffic per device for the statistics:
 `hc settings set --stats-per-device on`.

docs/design.md

@@ -100,6 +100,32 @@ Every rule carries a comment `hc:<mark>`, such as `hc:r12` for rule 12,
 `hc:f3` for port forward 3, `hc:b-established` for a built-in. Counters
 and log lines are traced back to their object through it.
 
+## Shadow mode
+
+With `shadow.enabled` in the configuration file ([shadow.md](shadow.md)),
+the same configuration is rendered so that it changes nothing (render's
+`Options.Shadow`): drops and rejects accept instead, policies accept,
+NAT only counts, and what alters packets (MSS clamping, the flowtable)
+and raw rules that do more than accept, drop or reject are left out.
+This rests on how nftables runs base chains: an `accept` only ends its
+own chain, and the packet goes on to the next one on the hook, other
+tables' included; only a drop is final. Another table, the real
+firewall, decides.
+
+What would be dropped, rejected or translated is counted under the same
+marks, and its comment says so (`"hc:r12 [would drop] …"`), which is
+how the tracer tells it apart. The filter chains run at `filter + 10`
+(after the other firewall's: what Housecarl would drop is what switching
+over would break) or `- 10`; the NAT chains always at `- 10`, since the
+kernel asks NAT chains only until one translates.
+
+Every change is checked by nftables in both forms, so leaving shadow
+mode can't fail on the ruleset. There is no probation, since nothing
+can lock anyone out; takeovers and closing connections, which would
+change traffic, are refused. A ruleset reload elsewhere (`flush
+ruleset`) deletes Housecarl's table too: housecarld listens to nftables'
+netlink notifications (internal/tablewatch) and puts it back.
+
 ## Not locking yourself out
 
 A change made over the network is applied **on probation** (a changeset

docs/development.md

@@ -11,6 +11,8 @@ internal/
   render/             State → the nftables ruleset text, with each line's object
   nft/                runs nft: check, apply (keeping dynamic set entries and counters), list, the boot file
   nflog/              reads logged packets from netlink log group 4171
+  linkwatch/          netlink: interfaces that appear or go (the flowtable's devices)
+  tablewatch/         netlink: a table deleted (shadow mode puts Housecarl's back; docs/shadow.md)
   importer/           nft -j list ruleset → State, with notes and raw rules (docs/import.md)
   hostif/             the machine's interfaces
   store/              SQLite (modernc.org/sqlite, no cgo): configuration, history, users, tokens, sessions, audit
@@ -70,8 +72,11 @@ Newer nftables accepts things 1.1.3 doesn't (a meter with a timeout for
   `wan`, `enp2s0` and `wg0` inside it, IPv4 and IPv6;
 * "the internet" on the WAN side, a LAN client, and a "jellyfin" host.
 
-It imports a copy of the maintainer's ruleset (lab addresses), takes
+It imports a copy of the maintainer's ruleset (lab addresses) with
-over the old table, and checks with real traffic: port forwards with
+housecarld in shadow mode beside it (traffic unchanged, a would-be drop
+logged and let through, the table back after the old ruleset is
+reloaded), restarts it enforcing, takes over the old table, and checks
+with real traffic: port forwards with
 hairpin NAT and the IPv6 pinhole, masquerading, antispoofing, the drop
 log, a lockout change that rolls back by itself, a restart, the boot
 ruleset after a flush, and that other tables are left alone. It makes

docs/flowtable.md

@@ -58,6 +58,12 @@ over half of the forwarding cost, which shows up as speed only when the
 CPU was the limit, and otherwise as a cooler, quieter, more responsive
 machine.
 
+In shadow mode ([shadow.md](shadow.md)) Housecarl leaves its flowtable
+out: an offloaded connection skips every table's forward chain, the
+other firewall's too. If that firewall has a flowtable of its own, the
+same goes for Housecarl's forward chain: it only sees each connection's
+first packets.
+
 ## When not to use it
 
 **When the CPU keeps up anyway.** If the speed test already reaches your

docs/import.md

@@ -18,11 +18,9 @@ shows what it would make, without saving anything: zones, sets,
 services, rules, port forwards and NAT, then **notes** on every guess it
 made, and the rules it **kept raw**.
 
-`--defines` reads your ruleset file's `define` lines, so sets get your
+`--defines` reads your ruleset file's `define` lines, so sets and zones
-names (`define admins = { ... }` becomes the set `admins`).
+get your names ([How things are named](#how-things-are-named)).
-Without it, sets are named after what they hold. Interfaces named by a
+`--zone enp2s0=lan` names a zone yourself.
-define name their zone too (`define wan_interface = "ppp0"` makes the
-zone `wan`); otherwise a zone is named after its interface.
 
 From another machine, give the JSON as a file, and the text with
 `--text`:
@@ -37,6 +35,46 @@ With several tables, `--table "inet main"` picks one; by default every
 table but Housecarl's is read. The web UI has the same under Server →
 Import a ruleset.
 
+## How things are named
+
+**Zones** are named in this order:
+
+1. **As you say**: `--zone enp2s0=lan` (repeat it; Zone names on the
+   web UI's Import page). Interfaces given the same name share one zone
+   (`--zone enp2s0=lan --zone enp3s0=lan`).
+2. **After a define holding the interface**, with a trailing `_int`,
+   `_if`, `_iface`, `_ifname`, `_interface`, `_dev`, `_device` or `_nic`
+   taken off: `define wan_int = "ppp0"` makes the zone `wan`, `define
+   iot_if = enp2s0.20` (a VLAN) the zone `iot`.
+3. **After the interface itself**: `enp2s0`.
+
+If the name is taken, the zone is called `INTERFACE_zone`. A define
+names what it was *for* in your old ruleset, which isn't always what the
+network is (`define dhcp_int = enp2s0` makes `dhcp`, for your LAN). A
+note lists each zone and where its name came from; rename one with
+`--zone` and import again, or afterwards with `hc zone update dhcp
+--name lan` (its rules follow). Name zones for what the network is for:
+`lan`, `iot`, `guest`, `servers`, `vpn`, `wan`.
+
+**Address sets**: a named set of the table keeps its name; a define
+holding a list of addresses, or one prefix, gives its name to a set
+(`define trusted = { 10.0.10.0/24, 10.1.1.0/24 }` makes `trusted`,
+wherever those addresses appear together). A single address stays an
+address. When two defines hold the same addresses, the first by name
+wins.
+
+**Services**: a define holding several ports names a service (`define
+web_ports = { 80, 443 }` makes `web_ports`); one port stays a port
+(`tcp/22`).
+
+**Port forwards** are named after their comment, without a leading
+"Port forward" (`comment "Port Forward Jellyfin"` makes `jellyfin`);
+several ports in one DNAT become one forward each (`web-80`, `web-443`),
+and a DNAT map's elements are named after the map (`forwards-443`).
+
+Names that aren't valid are made so: other characters become `_` (`-`
+in forward names), and a name must start with a letter.
+
 ## What becomes what
 
 * **Interfaces** become zones. Zones are marked external when the rules
@@ -45,7 +83,11 @@ Import a ruleset.
 * **Sets and defines** of addresses become address sets; port lists
   become services.
 * **Jumps and gotos** to your own chains are flattened: the jump's
-  condition is added to each rule of the chain, in place.
+  condition is added to each rule of the chain, in place. When the jump
+  and a rule both match sources (or destinations), the narrower is kept
+  if one lies within the other; otherwise, and for any rule there that
+  stays raw, the jump's condition is written in front of the raw rule's
+  text, so it still only applies where the jump did.
 * **Rules** keep their order. A rule that is a built-in (loopback,
   established, invalid, the ICMPv6 rules, MSS clamping, the DHCPv6 client
   rule, antispoofing) becomes that built-in only when it sits where the
@@ -56,10 +98,15 @@ Import a ruleset.
   stays a rule keeps the ICMPv6 accepts after it behind it, as they were.
   Built-ins your ruleset doesn't have are turned off.
 * **DNAT** rules become port forwards, TCP and UDP of the same port
-  merged into one. The forward-chain accepts that let them through
+  merged into one, one per port when a rule has several. The
-  become part of the port forward, and a masquerade for inside sources
+  forward-chain accepts that let them through (`ct status dnat`) become
-  to the same host becomes its hairpin. An IPv6 accept to the same port
+  part of the port forward; an accept to the same host and port without
-  becomes the forward's IPv6 pinhole when there is only one candidate.
+  `ct status dnat` lets in more than forwarded connections, and stays a
+  rule. A masquerade for inside sources to the same host becomes its
+  hairpin. An IPv6 accept to the same port, from everywhere, becomes the
+  forward's IPv6 pinhole when there is only one candidate. A DNAT to an
+  IPv6 address, or with matches a port forward has no field for, is left
+  out with a note.
 * **Maps are expanded**, one element at a time. A DNAT map
   (`dnat ip to tcp dport map @forwards`, named or written in the rule)
   becomes one port forward per element; forwards from a named map are
@@ -78,13 +125,27 @@ Import a ruleset.
   Byte rates and quotas are imported too; a quota imported resets only by
   hand until you give it a period. A rule with a quota or a byte rate that
   accepts stays a raw rule ([rules.md](rules.md#bandwidth-and-quotas)).
+* **Sets with `flags timeout`** (filled by a program or a rule) become
+  dynamic sets, without their entries of the moment: whatever fills them
+  must fill Housecarl's from then on, as a note says.
 * **Policies** become the chains' policies.
-* **Anything else** becomes a raw rule with its original text, in place.
+* **Anything else** becomes a raw rule with its original text, in place:
-  A raw rule naming a set of the old table needs the new set name:
+  a match Housecarl has no field for, two matches on the same field
-  Housecarl's are `NAME_v4` and `NAME_v6`, and a note says so.
+  (`ip saddr 10.0.0.0/8 ip saddr != 10.0.10.0/24`), `!=` where Housecarl
+  only has `==`, a set of interface names. Its set references are made
+  to work in Housecarl's table: an imported address set becomes
+  `@NAME_v4` or `@NAME_v6`, any other set (interface names, ports) is
+  written out with its elements. A raw rule naming a set that can't be
+  (a meter, an empty set) is left out with a note.
+* **Logging rules** log to Housecarl's log (`hc log`), not the kernel's.
 
-Chains on hooks Housecarl doesn't manage (ingress, a bridge table) are
+Several base chains on one hook (`input` at priority -10 and at 0) are
-left out, with a note.
+put one after the other, by priority, into Housecarl's one chain, with
+the stricter policy; a note says to check it, since a packet had to get
+past each, and an accept in the first no longer ends there. A prerouting
+filter chain after connection tracking becomes the early chain, which
+runs before it: a note says so. Chains on hooks Housecarl doesn't manage
+(ingress, a bridge table) are left out, with a note.
 
 ## 2. Import
 
@@ -98,6 +159,11 @@ Housecarl's table is now in force **alongside** the old one: a packet
 must be accepted by both. Nothing should change for your traffic. If
 Housecarl already has a configuration, `--replace` replaces it.
 
+To be sure first, import in **shadow mode** ([shadow.md](shadow.md)):
+Housecarl's ruleset is then loaded but drops nothing, and its log shows
+each packet the old ruleset lets through that Housecarl would drop. When
+there are none you mind, leave shadow mode and take over.
+
 Imported over the network (not through the local socket), the import is
 on probation like any change: confirm it.
 

docs/install.md

@@ -85,6 +85,10 @@ you are changing the firewall you are connected through.
    sudo systemctl disable nftables.service
    ```
 
+   To try Housecarl beside your firewall first, turn on **shadow mode**
+   before the import: it loads Housecarl's ruleset but blocks nothing,
+   and shows what it would do ([shadow.md](shadow.md)).
+
    **Or start from scratch**, through the local socket (no probation).
    The first change puts the ruleset in force, and the input and forward
    policies drop what no rule accepts, so let yourself in first:

docs/nat.md

@@ -7,6 +7,10 @@ public port are sent to the host, and let through the forward chain.
 One object does what takes three rules by hand (the DNAT, the forward
 accept, and the hairpin masquerade).
 
+In shadow mode ([shadow.md](shadow.md)) NAT rules translate nothing:
+they count the connections they would translate, and the other
+firewall's NAT does the work.
+
 ```sh
 hc forward add 8443 --to 192.168.1.10:443 --public 203.0.113.10 --hairpin --name web
 hc forward add 27015-27030 --proto tcp,udp --to 192.168.1.20 --public 203.0.113.10 --hairpin --name game

docs/operations.md

@@ -28,7 +28,8 @@ edits) applies each edit at once instead. With hc, see
 ## Changes on probation
 
 Changes made over the network roll back unless confirmed
-([design.md](design.md#not-locking-yourself-out)). The web UI and `hc`
+([design.md](design.md#not-locking-yourself-out)); in shadow mode, where
+a change can't cut anyone off, none do ([shadow.md](shadow.md)). The web UI and `hc`
 confirm their own changes once they reach Housecarl again over a new
 connection, so normally nothing waits:
 
@@ -79,6 +80,9 @@ sudo cat /var/lib/housecarl/ruleset.nft
 sudo nft -c -f /var/lib/housecarl/ruleset.nft     # check it
 ```
 
+In shadow mode both files are shadow mode's, and the fallback filters
+nothing ([shadow.md](shadow.md#what-works-differently)).
+
 Dynamic sets start empty after a reboot. The saved ruleset has no
 flowtable, since its devices may not exist yet: housecarld adds it when
 it starts, and whenever a flowtable device appears
@@ -150,16 +154,19 @@ table on its hook, so a drop in any of them wins.
 Every rule, NAT rule, built-in and policy has a counter. `hc rule list`,
 `hc forward list` and the web UI show packets; `hc counters` shows them
 all, with bytes. They count from housecarld's start, and carry across
-changes to the ruleset.
+changes to the ruleset. A deleted object's counter goes with it; an
+import with `--replace` gives every object a new ID, so its counters
+start again. A disabled rule keeps its count for when it is enabled.
 
 ## Quotas
 
 `hc quotas` (and the rules page) show what each quota used and when it
-resets; `hc rule reset-quota ID` starts one again. housecarld resets
+resets; `hc rule reset-quota ID` starts one again (`--address` one device's,
+`--give 5gbytes` a top-up until the next reset). housecarld resets
 quotas at midnight when their period ends (late, if it was stopped then),
-and saves their usage every five minutes and when it stops; after a
+and saves their usage every five minutes, after every change and when
-reboot it puts it back. A reboot without a clean stop loses at most the
+it stops; after a reboot it puts it back. A reboot without a clean stop
-last five minutes' count.
+loses at most the last five minutes' count.
 
 ## The log
 
@@ -177,6 +184,10 @@ hc log --search 198.51.100.7
 10:02:15  hc forward policy drop: wan  eth1  198.51.100.9:443 -> 192.168.1.20:51432  tcp
 ```
 
+In shadow mode ([shadow.md](shadow.md)) what the policies would drop is
+always logged, in all three chains, and drops are marked as what would
+have been done: the packets went on.
+
 The packets go to netlink log group 4171, where housecarld reads them;
 the last 2000 are kept in memory. They aren't written to the kernel log
 or the journal. A rule's log isn't rate-limited: on a busy rule, give it
@@ -281,13 +292,14 @@ it is asked):
 
 ```sh
 $ curl -s https://firewall.example.net:8070/healthz
-{"ok":true,"version":"0.1.0","firewall":"applied","probation":0,"warnings":[]}
+{"ok":true,"version":"0.1.0","firewall":"applied","mode":"enforcing","probation":0,"warnings":[]}
 ```
 
 | Field | |
 |---|---|
 | `ok` | `true` with HTTP 200 when the firewall is as it should be; `false` with HTTP **503** otherwise |
 | `firewall` | `applied` (Housecarl's ruleset is in force), `unconfigured` (nothing configured yet, so nothing to apply), `fallback` (the [boot fallback](#if-the-ruleset-fails-to-load-at-boot) is in force), `missing` (no Housecarl table: flushed, or never applied) |
+| `mode` | `enforcing`, or `shadow`: Housecarl's ruleset is loaded but blocks nothing, and another firewall decides ([shadow.md](shadow.md)). `ok` doesn't depend on it; to be told when a machine isn't enforcing, also expect `"mode":"enforcing"` |
 | `probation` | changes waiting to be confirmed, rolled back unless they are |
 | `warnings` | codes of problems with the machine, as `hc status` shows them (`nftables_service`) |
 
@@ -297,6 +309,22 @@ down, the fallback and a missing ruleset. Reach it from where the input
 rules let the monitor in (a rule for the Checkmk server's address and
 port 8070).
 
+## Failed logins in the journal
+
+A failed login to the web UI or the API, and a refused API token, are
+written to the journal as well as the audit log, with the address first:
+
+```
+level=WARN msg="login failed" ip=203.0.113.9 user=admin reason="wrong username or password"
+level=WARN msg="token refused" ip=203.0.113.9 reason="wrong token secret"
+```
+
+Guard's `housecarl` preset, or a fail2ban filter, can ban such addresses
+([bans.md](bans.md)). The address comes before the user name, which a
+caller chooses, so a pattern anchored on `msg="login failed" ip=` can't
+be fooled by a name. (With `log.format: json` they are JSON objects with
+the same fields, in the same order.)
+
 ## Checks
 
 `housecarld check` validates the configuration in the database against

docs/permissions.md

@@ -9,6 +9,7 @@ else gets **grants**, `resource:access`:
 |---|---|---|
 | `config` | settings, zones, sets, services, rules, NAT, the ruleset, history, interfaces, probation, changesets (the web UI shows the forms locked) | change them, stage and apply changesets (your own, or anyone's for administrators), undo, import, confirm and roll back; close connections; start and stop traces |
 | `sets` | dynamic sets' entries | add and remove entries |
+| `bans` | ban lists' bans, and the lists the grant names | ban and unban ([bans.md](bans.md)) |
 | `log` | hit counters, the log, connections, traces | – |
 | `audit` | the audit log | – |
 | `*` | all of the above | |
@@ -20,6 +21,10 @@ program, such as Steward's captive portal, which should be able to let a
 client in and nothing more. A dynamic set's entries need a `sets` grant
 even for those with `config:write`.
 
+A `bans` grant names ban lists the same way: `bans:write:web` may ban
+into the list `web` (and read it) and nothing else, the grant for Guard
+or fail2ban on the web server.
+
 Only administrators may delete other tables (`hc takeover`), and manage
 users and the certificate (`hc cert`, Server → Certificate).
 
@@ -27,8 +32,8 @@ users and the certificate (`hc cert`, Server → Certificate).
 
 | Role | Grants |
 |---|---|
-| `viewer` | `config:read sets:read log:read` |
+| `viewer` | `config:read sets:read bans:read log:read` |
-| `operator` | `config:write sets:write log:read audit:read` |
+| `operator` | `config:write sets:write bans:write log:read audit:read` |
 | `log` | `log:read`, e.g. for a monitoring script |
 
 ```sh
@@ -82,6 +87,6 @@ on probation like any other.
 
 Logins (and failed ones), users, tokens, every configuration change (with
 its history entry), changesets created, applied and discarded,
-confirmations and rollbacks, imports, takeovers and
+confirmations and rollbacks, imports, takeovers,
-dynamic set entries are recorded with who, from where and how: `session`,
+dynamic set entries and bans are recorded with who, from where and how: `session`,
 `token` or `socket`.

docs/rules.md

@@ -186,8 +186,22 @@ hc rule add forward --in lan --out wan --quota 50gbytes,over,per-source,monthly=
 hc rule add forward --in guest --out wan --quota 10gbytes,over,daily -a drop
 hc quotas                      # what each used, and the next reset
 hc rule reset-quota 31         # start again now
+hc rule reset-quota 31 --address 192.168.1.23            # one device only
+hc rule reset-quota 31 --give 5gbytes --address 192.168.1.23
 ```
 
+`--give` is a top-up: it takes that much off what was used (never below
+zero), so the device gets 5 GB more until the quota's next reset, and the
+rule stays as it is. In the web UI, the quota's line under the rule
+opens its dialog: usage (each address's), resets, and **Reset** and
+**Give back…** for everyone or one address. Like a reset, these are
+done at once (audited, not staged or on probation); Housecarl puts the
+ruleset in force again with the new figure.
+
+In the web UI each quota's use is on its rule (with **Reset**), all of
+them together on the dashboard, and their history under **Traffic →
+Statistics**.
+
 | | |
 |---|---|
 | `per-source`, `per-destination` | each address its own allowance; without, one allowance for everything the rule matches |
@@ -195,12 +209,26 @@ hc rule reset-quota 31         # start again now
 
 nftables counts a quota but never resets it, and forgets it when its
 table is replaced: Housecarl carries what each quota used across every
-change and restart, saves it every five minutes (and when housecarld
+change and restart, saves it every five minutes (and after each change,
-stops) for after a reboot, and resets quotas when their period ends.
+and when housecarld stops) for after a reboot, and resets quotas when their period ends.
 A per-address quota counts an address: a device that gets a new one
 starts again. With Steward's reservations IPv4 addresses stay; IPv6 is
 grouped by /64 (below).
 
+Changing the amount, `over` or the resets keeps what was used. Changing
+how the quota counts carries what can be carried:
+
+| Change | What was used |
+|---|---|
+| per address to one total | the addresses' sum |
+| to wider groups (IPv6 /64 to /56) | added up per new group |
+| a total to per address | starts again: a total can't be split |
+| per source to per destination (or back) | starts again |
+| to finer groups (/64 to /128) | starts again: a group can't be split |
+
+A change that starts a quota again says so (warning `quota_restarts`),
+when it is staged for review and in `hc` (also with `-n`).
+
 **Rules with a quota or a byte rate count every packet.** The built-in
 that accepts established connections comes before your rules, so a rule
 there only sees each connection's first packet. Rules that count traffic
@@ -235,6 +263,18 @@ nftables names (`@trusted_v4`). The importer makes raw rules from
 what it can't translate. It goes into the ruleset as written: add
 `counter` to it to count its packets.
 
+In shadow mode ([shadow.md](shadow.md)) a raw rule's final `drop` or
+`reject` becomes `accept`; a raw rule that does anything else to packets
+(NAT, `meta mark set`, `jump`, `notrack`, a verdict map…) is left out,
+with a warning (`shadow_raw_left_out`).
+
+## Ban lists
+
+Programs such as Guard and fail2ban ban addresses into ban lists, and
+those bans come first of all in the input and forward chains (after
+loopback in input), before the built-ins and every rule. See
+[bans.md](bans.md).
+
 ## Built-ins
 
 Housecarl adds the rules every firewall needs before yours. Each can be
@@ -274,30 +314,66 @@ chain; from it: output; zone to zone: forward).
 ```
 $ hc matrix
 FROM \ TO  guest         lan           wan         firewall
-guest      block     block     block +1  block +3
+guest      block *       block         partly +1   partly +4
-lan        block     block     allow     block +4
+lan        block         block *       allow       partly +4
-wan        block +2  block +2  block +1  block +3
+wan        partly +1 -1  partly +1 -1  block -1 *  partly +3 -1
 firewall   allow         allow         allow       -
+
+Let through for part of the traffic:
+  guest → wan       http, https
+  guest → firewall  ICMPv6; ping; ssh from admins; dns
+  lan → firewall    ICMPv6; ping; ssh from admins; dns
+  wan → guest       1 port forward
+  wan → lan         1 port forward
+  wan → firewall    ICMPv6; DHCPv6; ssh from admins
+
+allow, block: every new connection; partly: some are let through, some not.
++N: rules that let part of the traffic through; -N: rules that block part of it. 'hc matrix FROM TO' lists them.
+*: a zone to itself: only traffic routed between its interfaces passes the firewall.
 ```
 
 Each cell walks that pair's rules in order, as nftables would, with the
 built-ins that act on new connections (ping from internal zones,
 ICMPv6, the DHCPv6 client, antispoofing, port forwards):
 
-* a rule that applies to the whole pair decides it ("allow", "block");
+* a rule that applies to the whole pair decides it;
 * a rule that applies to part of its traffic (some addresses, services,
-  times, rates, a quota) is an **exception** (+N), and the walk goes on;
+  times, rates, a quota) is an **exception**, and the walk goes on;
 * what's left falls to the deciding rule or the chain's policy.
 
+A cell reads **allow** or **block** when every new connection gets the
+same, and **partly** when exceptions let some through while the rest is
+blocked (or block some while the rest is allowed). **+N** counts the
+exceptions that let traffic through, **-N** those that block it. Under
+the grid (in the web UI, in each cell) is what the exceptions let
+through: their services, with the addresses they are limited to.
+
+A zone to itself (`*`, faded in the web UI) is mostly traffic that never
+reaches the firewall: within one interface's network it goes straight
+from host to host. Only what is routed between the zone's interfaces (or
+back out of the one it came in on) passes.
+
 `hc matrix guest wan` (or clicking the cell) lists the exceptions in
-order:
+order, with each rule's hits:
 
 ```
-guest → wan (the forward chain), new connections:
+guest → wan (the forward chain), new connections: partly allowed
-  allowed  http, https   rule forward #14
+  VERDICT  WHAT         BY                HITS
-  blocked  the rest      forward policy
+  allowed  http, https  rule forward #14  1520
+  blocked  the rest     forward policy    37
 ```
 
+The hits are the rule's packets since housecarld started, in all of its
+traffic: a rule without zones counts for every pair it covers. They are
+how you tell an exception that is used from one that isn't.
+
+A raw rule is read when it is a match followed by `accept`, `drop` or
+`reject` (with `counter` and `log` if you like): it counts as an exception
+for every pair of its chain, shown as `raw: …`. One that matches
+interfaces (the matrix can't tell which zones), jumps, translates, sets
+something or uses a verdict map shows as **unknown**: check it yourself.
+Raw rules have hits only if they say `counter`.
+
 A rule whose addresses all lie in networks directly connected to other
 zones is left out of a pair: traffic between those zones can't have them
 (`from wg_net` doesn't concern traffic coming in on the LAN). Routed

docs/shadow.md

@@ -0,0 +1,174 @@
+# Shadow mode
+
+In shadow mode Housecarl runs beside the firewall you have now and
+changes nothing. Its ruleset is loaded, so its counters, log, tracer and
+statistics show what it *would* do with your real traffic, but it blocks,
+rejects and translates nothing. Your current firewall keeps deciding.
+
+Use it to:
+
+* **try Housecarl** on the real machine before you trust it with the
+  firewall: the web UI, the API, `hc`, changesets, history;
+* **check an import**: import your ruleset ([import.md](import.md)) and
+  watch whether Housecarl would drop anything your firewall lets through;
+* **develop against the API**: Steward, scripts or fail2ban can fill
+  dynamic sets and see them used, with nothing at stake.
+
+## Turning it on
+
+Shadow mode is set in `/etc/housecarl/housecarl.yaml`
+([configuration.md](configuration.md)), not through the API, so no user
+or token can switch enforcement off:
+
+```yaml
+shadow:
+  enabled: true
+  position: after     # or before; see below
+```
+
+housecarld reads the file when it starts:
+
+```sh
+sudo systemctl restart housecarl
+sudo hc status        # Mode: SHADOW …
+```
+
+The web UI shows a banner on every page, `hc status` a `Mode:` line, and
+`/healthz` `"mode": "shadow"`.
+
+## Why it can't let anything through
+
+In nftables, `accept` only ends the chain it is in. The packet then
+goes on to the next base chain on the same hook, and other tables' chains
+are among those. Only a drop is final. Housecarl's table in shadow mode
+never drops, so it can't let through anything your firewall drops, and it
+can't block anything your firewall accepts.
+
+## What changes in the ruleset
+
+The same configuration is rendered, with everything that acts on packets
+turned into counting:
+
+| | Enforcing | Shadow mode |
+|---|---|---|
+| Rules that drop or reject | `drop`, `reject` | counted, then `accept` (Housecarl's chain ends as it would have). Their comment says `[would drop]` or `[would reject]` |
+| The built-ins that drop (invalid packets, antispoofing) | `drop` | counted, `[would drop]` |
+| Policies | as configured | `accept`. What a drop policy would drop is counted and **always logged**, rate-limited, in all three chains |
+| Port forwards, source NAT, redirects | translate | counted only (`[would dnat]`, `[would masquerade]`, `[would snat]`, `[would redirect]`) |
+| MSS clamping | changes SYN packets | left out |
+| The flowtable | offloads connections | left out: offloaded connections skip *every* table's forward chain, your firewall's too |
+| Raw rules | as written | as written if they only match, count, log or accept. A final `drop` or `reject` becomes `accept`. Anything else that acts on packets (NAT, `set`, `jump`, `notrack`, a verdict map…) is left out, with a warning (`shadow_raw_left_out`) |
+| Everything else | | the same: accepts, rate limits, connection limits, quotas, dynamic sets, per-device counters, logging |
+
+`hc ruleset` prints the shadow ruleset, the one loaded now.
+`hc ruleset --enforcing` prints the ruleset that will be loaded once
+shadow mode is off.
+
+**Every change is checked by nftables twice**: the shadow ruleset, and the
+enforcing one. A change that nftables would refuse once shadow mode is off
+is refused now, so leaving shadow mode can't fail on that.
+
+## Where Housecarl's chains run
+
+Your firewall's chains are usually at priority `filter` (0), its raw
+chains at `raw`. `shadow.position` puts Housecarl's filter chains just
+next to them:
+
+* **`after`** (the default) runs them at `filter + 10` (and `raw + 10`).
+  Housecarl only sees what your firewall let through. So **every would-drop
+  is traffic that would break if you switched over**: the list to work
+  through before you do. What your firewall drops and Housecarl would
+  accept (a hole) doesn't show; compare the rules for that.
+* **`before`** runs them at `filter - 10` (and `raw - 10`). Housecarl sees
+  every packet, as it would when enforcing, and its hits and drops are the
+  numbers you would get. Its would-drops include everything your firewall
+  drops too (the internet's background noise).
+
+If your firewall uses other priorities, say `filter + 20`, "after" is
+relative to `filter`, not to it: check with `nft list ruleset`.
+
+NAT chains are different: the kernel asks NAT chains in turn only until
+one translates. Housecarl's NAT chains always run first (`dstnat - 10`,
+`srcnat - 10`), whatever the position, or they would never see a
+connection your firewall translates. They translate nothing, so your
+firewall's NAT still does the work.
+
+## What you see
+
+* **Hits** (`hc counters`, the Rules page): a drop rule's hits are what it
+  would have dropped. NAT counters count connections (only a connection's
+  first packet reaches the NAT chains).
+* **The log** (`hc log`, the Log page): every packet a drop policy would
+  have dropped, and what rules with logging would have dropped, marked
+  *would drop*. `hc log` adds "(shadow mode: would drop, let through)".
+* **The tracer** (`hc trace`, the Trace page): the verdict is your
+  firewall's, the real one. When Housecarl would have dropped the packet,
+  the trace says by which rule ("shadow mode: Housecarl would have
+  dropped it, by …").
+* **Statistics** ([stats.md](stats.md)): drops, drops per chain and the
+  blocked sources are what Housecarl would have dropped.
+
+## What works differently
+
+* **No probation.** A change can't cut anyone off, so changes from the
+  network are applied and saved at once, with no confirmation. Changesets
+  work as usual; their ruleset diff is of the shadow ruleset.
+* **No takeover, no closing connections.** `hc takeover` (deleting your
+  firewall's tables) and `hc connections close` are refused (409,
+  `hc` exits with 5): both would change your traffic.
+* **Bans** ([bans.md](bans.md)) are kept and their rules count what they
+  would drop (`[would drop]`), but nothing is dropped, and banning closes
+  no connections.
+* **The boot.** `housecarl-ruleset.service` loads the shadow ruleset at
+  boot. Its fallback ([operations.md](operations.md#if-the-ruleset-fails-to-load-at-boot))
+  is an empty table that filters nothing, since your firewall protects the
+  machine.
+* **Your firewall reloading.** A ruleset file such as `/etc/nftables.conf`
+  usually starts with `flush ruleset`, and Debian's `nftables.service`
+  runs `nft flush ruleset` when it stops or restarts. Both delete
+  Housecarl's table too. housecarld watches nftables' notifications and
+  puts its table back within a few seconds (the journal says "Housecarl's
+  table was deleted … put it back"). Dynamic sets lose their entries, and
+  quotas go back to what was last saved (every five minutes).
+* **Warnings.** `nftables_service` and `takeover_pending` aren't raised:
+  another firewall beside Housecarl is the point. Instead:
+  `shadow_unprotected` when no other table has a filter chain on the input
+  hook (nothing protects the machine), and `shadow_raw_left_out` for each
+  raw rule left out ([validation.md](validation.md#the-machine)).
+* **Quotas count** what passes in shadow mode, and keep counting when
+  you switch over.
+
+## A trial, step by step
+
+From a machine whose firewall is `/etc/nftables.conf` with `table inet main`:
+
+1. Install Housecarl and create the administrator
+   ([install.md](install.md)).
+2. Turn shadow mode on in `/etc/housecarl/housecarl.yaml`, and start
+   housecarld: `sudo systemctl enable --now housecarl`.
+3. Import your ruleset: `sudo hc import --defines /etc/nftables.conf`
+   ([import.md](import.md)). Leave `nftables.service` as it is.
+4. Use your network for a while, and watch `hc log -f`, `hc counters` and
+   the dashboard. With `position: after`, each would-drop is something
+   your firewall allows and Housecarl wouldn't: fix the rule in Housecarl,
+   or decide it doesn't matter. `hc status` lists the raw rules left out.
+5. When nothing you need would be dropped, leave shadow mode.
+
+## Leaving shadow mode
+
+1. Look at what will be loaded: `sudo hc ruleset --enforcing`.
+2. Set `shadow.enabled: false` (or remove the section), and
+   `sudo systemctl restart housecarl`. Housecarl's ruleset is now in
+   force **alongside** your firewall: a packet must be accepted by both.
+   The restart applies it at once, not on probation, but your old firewall
+   is still there and still decides too.
+3. Take over: `sudo hc takeover "inet main"` deletes the old table (on
+   probation over the network), and `sudo systemctl disable nftables`
+   (without `--now`) retires it ([import.md](import.md#3-take-over)).
+
+## Going back
+
+Set `shadow.enabled: true` and restart housecarld. If Housecarl was your
+only firewall, **the machine is now unprotected**: load your other
+firewall first. Until one filters the input hook, `hc status` and
+`/healthz` warn (`shadow_unprotected`).

docs/stats.md

@@ -100,6 +100,10 @@ The blocked lists only see what is **logged**: the input and forward
 policies' drops with `hc settings set --log-drops on`, and rules with
 logging. The drop counts above see everything.
 
+In shadow mode ([shadow.md](shadow.md)) the drops, and the blocked
+lists, are what Housecarl would have dropped: the packets went on. What
+the policies would drop is always logged there.
+
 A counter that goes down (an interface that came back, a quota reset)
 counts again from zero. What happens while housecarld is stopped isn't
 counted.

docs/validation.md

@@ -51,7 +51,7 @@ are unique per kind, ignoring case.
 
 | Code | | When |
 |---|---|---|
-| `set_name_reserved` | error | a set name starting with `meter_`, `conn_`, `quota_` or `stats_`: Housecarl's own sets (rate limits, quotas, statistics) use those |
+| `set_name_reserved` | error | a set name starting with `meter_`, `conn_`, `quota_`, `stats_` or `ban_`: Housecarl's own sets (rate limits, quotas, statistics, ban lists) use those |
 | `set_entry` | error | an entry that isn't an address, prefix (without host bits) or range |
 | `set_empty` | warning | a set (not dynamic) with no entries: rules using it match nothing |
 | `set_dynamic_entries` | error | configured entries on a dynamic set: its entries come at run time |
@@ -97,6 +97,7 @@ These apply to rules, source NAT and redirects.
 | `rule_group` | error | a grouping prefix outside 1-32 (IPv4) or 1-128 (IPv6) |
 | `rule_traffic_accept` | error | a rule with a quota or a byte rate that accepts: it is checked before established connections are accepted, so it can only drop, reject or log |
 | `rule_every_packet_accept` | error | a rule checked against every packet (`every_packet`) that accepts: it comes before established connections are accepted, so it can only drop, reject or log |
+| `quota_restarts` | warning | a change makes a rule count its quota so that what it used can't carry over (a total to per address, per source to per destination, finer groups): it starts again. Only on the change that does it |
 | `flowtable_traffic` | warning | a forward rule checked against every packet (a quota, a byte rate, `every_packet`) while the flowtable is on: it only sees each connection's first packets |
 | `reject_with` | error | `tcp-reset` on a rule that matches more than TCP, or with an action other than reject |
 | `log_prefix` | error | a log prefix over 100 characters |
@@ -132,3 +133,22 @@ configuration: the dashboard and `hc status` show them.
 | `nftables_service` | warning, *host* | Debian's `nftables.service` is enabled: at boot it loads `/etc/nftables.conf`, and stopping or restarting it runs `nft flush ruleset`, deleting Housecarl's table too. Disable it with `systemctl disable nftables` (without `--now`) |
 | `conntrack_full` | warning, *host* | the connection tracking table is over 80% full; once it is full, the kernel drops new connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens so many ([stats.md](stats.md#warnings)) |
 | `takeover_pending` | warning, *host* | the table you imported from is still in force beside Housecarl's, so a packet must get past both; `hc takeover TABLE` deletes it once Housecarl's rules are right |
+| `shadow_unprotected` | warning, *host* | shadow mode, and no other table has a filter chain on the input hook: Housecarl's ruleset blocks nothing, and nothing else protects the machine. Load your firewall, or leave shadow mode ([shadow.md](shadow.md)) |
+| `shadow_raw_left_out` | warning, *host* | shadow mode leaves this raw rule out: it does more to packets than accept, drop or reject (NAT, setting a mark or field, a jump, a verdict map…). Its hits aren't counted, and the rules after it see what it would have taken ([shadow.md](shadow.md#what-changes-in-the-ruleset)) |
+
+In shadow mode `nftables_service` and `takeover_pending` aren't raised:
+another firewall beside Housecarl is the point there.
+
+## Ban lists
+
+| Code | | When |
+|---|---|---|
+| `banlist_name` | error | not a valid name (as for zones and sets) |
+| `banlist_name_taken` | error | two ban lists with one name |
+| `banlist_scope` | error | a scope other than `firewall`, `hosts` or `everything` |
+| `banlist_hosts` | error | scope `hosts` without hosts, or hosts with another scope |
+| `banlist_address` | error | a host or never entry that isn't an address, prefix or range |
+| `unknown_set` | error | a host or never entry naming a set that doesn't exist |
+| `banlist_time` | error | negative times (but `max_time` -1), or a default longer than the longest |
+| `banlist_entries` | error | `max_entries` below 0 or above 1000000 |
+| `banlist_group` | error | `group_v6` outside 16 to 128 |

internal/api/types.go

@@ -40,8 +40,18 @@ type Whoami struct {
 	TokenID   int64         `json:"token_id,omitempty"`
 	CSRFToken string        `json:"csrf_token,omitempty"`
 	Build     version.Build `json:"build"`
+	// Mode is enforcing, or shadow (docs/shadow.md).
+	Mode string `json:"mode"`
 }
 
+// Modes: housecarld's ruleset decides, or (shadow mode, the
+// configuration file's shadow.enabled) it only shows what it would do,
+// beside another firewall.
+const (
+	ModeEnforcing = "enforcing"
+	ModeShadow    = "shadow"
+)
+
 // Login is the login request.
 type Login struct {
 	Username string `json:"username"`
@@ -155,6 +165,9 @@ type LogEntry struct {
 	TCPFlags string    `json:"tcp_flags,omitempty"`
 	RuleMark string    `json:"mark,omitempty"`
 	Verdict  string    `json:"verdict,omitempty"`
+	// Shadow: logged in shadow mode, where a drop or reject is what would
+	// have been done; the packet went on.
+	Shadow bool `json:"shadow,omitempty"`
 }
 
 // Status is the dashboard's overview.
@@ -162,6 +175,11 @@ type Status struct {
 	Build    version.Build `json:"build"`
 	Started  time.Time     `json:"started"`
 	Database string        `json:"database,omitempty"`
+	// Mode is enforcing or shadow; ShadowPosition (shadow mode) after or
+	// before: where Housecarl's filter chains run beside the other
+	// firewall's.
+	Mode           string `json:"mode"`
+	ShadowPosition string `json:"shadow_position,omitempty"`
 	Applied        bool   `json:"applied"`
 	ApplyError     string `json:"apply_error,omitempty"`
 	// Unconfigured: the configuration is empty, and no ruleset was
@@ -198,6 +216,9 @@ type ImportInput struct {
 	// Replace replaces Housecarl's configuration; without it, the import
 	// is refused when there is one.
 	Replace bool `json:"replace,omitempty"`
+	// Zones names the zones by interface ({"enp2s0": "lan"}); interfaces
+	// given the same name share a zone.
+	Zones map[string]string `json:"zones,omitempty"`
 }
 
 // ImportResult reports an import.
@@ -305,6 +326,16 @@ type CertificateResponse struct {
 	Status *CertStatus `json:"status,omitempty"`
 }
 
+// QuotaReset is POST /rules/{ref}/reset-quota's optional body. Without
+// one the whole quota starts again. Address starts one address of a
+// per-address quota again (any address in its group: 10.0.10.23, or an
+// IPv6 address for its /64); Give takes that many bytes off what was used
+// instead (the address's, or every address's), until the next reset.
+type QuotaReset struct {
+	Address string `json:"address,omitempty"`
+	Give    int64  `json:"give,omitempty"`
+}
+
 // QuotaInfo is a rule's quota and what it used.
 type QuotaInfo struct {
 	RuleID int64       `json:"rule_id"`
@@ -363,6 +394,19 @@ type Connection struct {
 	Bytes   uint64 `json:"bytes,omitempty"`
 }
 
+// NamesInput asks for the names of addresses (POST /names).
+type NamesInput struct {
+	Addresses []string `json:"addresses"`
+}
+
+// Names are the names found for the inside addresses asked about, by
+// address; those without one (or still being looked up) are left out.
+// Enabled is false when names are off (names.enabled).
+type Names struct {
+	Enabled bool              `json:"enabled"`
+	Names   map[string]string `json:"names"`
+}
+
 // CloseConnections closes connections: these, or every one Search finds
 // (as GET /connections?q= does).
 type CloseConnections struct {
@@ -411,6 +455,9 @@ type TracePacket struct {
 	// Verdict is accepted or dropped; DecidedBy the step that decided.
 	Verdict   string    `json:"verdict"`
 	DecidedBy TraceStep `json:"decided_by"`
+	// WouldDrop, in shadow mode, is the step of Housecarl's where it
+	// would have dropped (or rejected) the packet, had it been enforcing.
+	WouldDrop *TraceStep `json:"would_drop,omitempty"`
 }
 
 // TraceStatus is the trace running (or the last one).
@@ -443,6 +490,26 @@ type MatrixCell struct {
 	DecidedBy  string            `json:"decided_by"`
 	RuleID     int64             `json:"rule_id,omitempty"` // the deciding rule
 	Exceptions []MatrixException `json:"exceptions"`
+	// Summary is the pair at a glance: allowed or blocked when every new
+	// connection gets the same, partly when some are let through and
+	// some not.
+	Summary string `json:"summary"`
+	// Allowed and Blocked count the exceptions that let part of the
+	// traffic through, and that block part of it.
+	Allowed int `json:"allowed"`
+	Blocked int `json:"blocked"`
+	// Open sums up what the allowing exceptions let through, shortest
+	// first in their order: "ssh", "dns", "5 port forwards".
+	Open []string `json:"open"`
+	// SameZone: from a zone to itself. Note says what of that traffic
+	// passes the firewall at all.
+	SameZone bool   `json:"same_zone,omitempty"`
+	Note     string `json:"note,omitempty"`
+	// Hits are the deciding rule's (or policy's) packets, all of its
+	// traffic since housecarld started (nil: no counter).
+	Hits *uint64 `json:"hits,omitempty"`
+	// Marks are the counters behind Hits.
+	Marks []string `json:"-"`
 }
 
 // MatrixException is a rule (or built-in) that decides for part of a
@@ -456,4 +523,44 @@ type MatrixException struct {
 	Verdict string `json:"verdict"`
 	// What it covers: "from trusted · ssh", "Monday 08:00–17:00".
 	What string `json:"what"`
+	// Short is What in a word or two, for the overview: its services,
+	// or else its first condition.
+	Short string `json:"short"`
+	// Hits are its packets, all of its traffic since housecarld started
+	// (a rule may apply to several pairs; nil: no counter).
+	Hits *uint64 `json:"hits,omitempty"`
+	// Marks are the counters behind Hits ("r12", "b-ping", "f3").
+	Marks []string `json:"-"`
+}
+
+// BanInput asks a ban list to ban an address (POST
+// /banlists/{list}/bans).
+type BanInput struct {
+	Address string `json:"address"`
+	// Protocol (tcp, udp) and Ports limit the ban; with neither, every
+	// port. Ports are numbers or service names, in a list or a string
+	// ("22", "ssh", "http,https"); "all", "0:65535" or "1-65535" mean
+	// every port. Ports without a protocol take the service's protocols,
+	// or for a number tcp and udp.
+	Protocol string          `json:"protocol,omitempty"`
+	Ports    json.RawMessage `json:"ports,omitempty"`
+	// Time is the ban's length in seconds: 0 the list's default, -1 its
+	// longest (or for ever). Longer than the list allows is cut down.
+	Time   int    `json:"time,omitempty"`
+	Source string `json:"source,omitempty"`
+	Reason string `json:"reason,omitempty"`
+}
+
+// BanResult is the answer to a ban: the bans made or renewed (one per
+// protocol and port), Existing for each already in force.
+type BanResult struct {
+	Bans []BanMade `json:"bans"`
+	// Closed counts the banned address's connections closed.
+	Closed int `json:"closed"`
+}
+
+// BanMade is a ban made, or renewed (Existing).
+type BanMade struct {
+	model.Ban
+	Existing bool `json:"existing,omitempty"`
 }

internal/auth/auth.go

@@ -147,13 +147,14 @@ func NewSessionID() (id, csrf string) {
 const (
 	ResConfig = "config" // the firewall configuration, its history and imports
 	ResSets   = "sets"   // dynamic sets' entries (a grant may name the sets)
+	ResBans   = "bans"   // ban lists' bans (a grant may name the lists)
 	ResLog    = "log"    // counters, the drop log and the status
 	ResAudit  = "audit"  // the audit log
 	ResAll    = "*"
 )
 
 // Resources lists them in display order.
-var Resources = []string{ResConfig, ResSets, ResLog, ResAudit}
+var Resources = []string{ResConfig, ResSets, ResBans, ResLog, ResAudit}
 
 // Access levels.
 const (
@@ -163,8 +164,8 @@ const (
 
 // Roles are ready-made sets of grants.
 var Roles = map[string][]model.Grant{
-	"viewer":   {{Resource: ResConfig, Access: Read}, {Resource: ResSets, Access: Read}, {Resource: ResLog, Access: Read}},
+	"viewer":   {{Resource: ResConfig, Access: Read}, {Resource: ResSets, Access: Read}, {Resource: ResBans, Access: Read}, {Resource: ResLog, Access: Read}},
-	"operator": {{Resource: ResConfig, Access: Write}, {Resource: ResSets, Access: Write}, {Resource: ResLog, Access: Read}, {Resource: ResAudit, Access: Read}},
+	"operator": {{Resource: ResConfig, Access: Write}, {Resource: ResSets, Access: Write}, {Resource: ResBans, Access: Write}, {Resource: ResLog, Access: Read}, {Resource: ResAudit, Access: Read}},
 	"log":      {{Resource: ResLog, Access: Read}},
 }
 
@@ -179,12 +180,13 @@ func RoleNames() []string {
 }
 
 // ParseGrant reads "resource:access" ("log:read", "*:write"), and for
-// sets optionally the sets it is limited to ("sets:write:guest,portal").
+// sets and bans optionally the sets or ban lists it is limited to
+// ("sets:write:guest,portal", "bans:write:web").
 func ParseGrant(s string) (model.Grant, error) {
 	parts := strings.SplitN(strings.TrimSpace(s), ":", 3)
 	g := model.Grant{Resource: strings.ToLower(parts[0])}
 	if len(parts) < 2 {
-		return g, fmt.Errorf("grant %q: write it as resource:access, e.g. log:read, or sets:write:SET1,SET2", s)
+		return g, fmt.Errorf("grant %q: write it as resource:access, e.g. log:read, sets:write:SET1,SET2 or bans:write:LIST", s)
 	}
 	g.Access = strings.ToLower(parts[1])
 	if len(parts) == 3 {
@@ -211,8 +213,8 @@ func ValidateGrant(g *model.Grant) error {
 	if g.Access != Read && g.Access != Write {
 		return fmt.Errorf("access must be read or write, not %q", g.Access)
 	}
-	if len(g.Sets) > 0 && g.Resource != ResSets {
+	if len(g.Sets) > 0 && g.Resource != ResSets && g.Resource != ResBans {
-		return fmt.Errorf("only a sets grant can name sets (sets:write:%s)", strings.Join(g.Sets, ","))
+		return fmt.Errorf("only a sets or bans grant can name sets or ban lists (sets:write:%s)", strings.Join(g.Sets, ","))
 	}
 	return nil
 }
@@ -272,20 +274,26 @@ func covers(gs []model.Grant, resource string, write bool) bool {
 
 // CanSet reports whether the principal may read or change the entries of
 // a dynamic set: a sets grant covers it if it names no sets, or names it.
-func (p *Principal) CanSet(name string, write bool) bool {
+func (p *Principal) CanSet(name string, write bool) bool { return p.canNamed(ResSets, name, write) }
+
+// CanBan reports whether the principal may read or change the bans of a
+// ban list: a bans grant covers it if it names no lists, or names it.
+func (p *Principal) CanBan(list string, write bool) bool { return p.canNamed(ResBans, list, write) }
+
+func (p *Principal) canNamed(resource, name string, write bool) bool {
 	if p == nil || p.User == nil {
 		return false
 	}
-	allowed := p.User.Admin || coversSet(p.User.Grants, name, write)
+	allowed := p.User.Admin || coversNamed(p.User.Grants, resource, name, write)
 	if p.Token != nil && p.Token.Grants != nil {
-		allowed = allowed && coversSet(p.Token.Grants, name, write)
+		allowed = allowed && coversNamed(p.Token.Grants, resource, name, write)
 	}
 	return allowed
 }
 
-func coversSet(gs []model.Grant, name string, write bool) bool {
+func coversNamed(gs []model.Grant, resource, name string, write bool) bool {
 	for _, g := range gs {
-		if (g.Resource != ResAll && g.Resource != ResSets) || (write && g.Access != Write) {
+		if (g.Resource != ResAll && g.Resource != resource) || (write && g.Access != Write) {
 			continue
 		}
 		if len(g.Sets) == 0 {

internal/config/config.go

@@ -41,8 +41,40 @@ type Config struct {
 	Ruleset string `yaml:"ruleset"`
 	// Stats keeps traffic, hits and drops over time (docs/stats.md).
 	Stats Stats `yaml:"stats"`
+	// Shadow loads the ruleset without letting it block, reject or
+	// translate anything, beside the firewall in force (docs/shadow.md).
+	Shadow Shadow `yaml:"shadow"`
+	// Names shows inside addresses' names, by reverse lookups
+	// (docs/configuration.md).
+	Names Names `yaml:"names"`
 }
 
+// Names configures the reverse lookups that put names beside inside
+// addresses in the web UI and hc.
+type Names struct {
+	Enabled bool `yaml:"enabled"`
+	// Resolver is a DNS server ("10.10.0.1:53"; "": the system's).
+	Resolver string `yaml:"resolver"`
+	// Cache is how long a name is kept.
+	Cache Duration `yaml:"cache"`
+}
+
+// Shadow is shadow mode: Housecarl's ruleset is loaded and counts, logs
+// and traces what it would do, while another firewall decides.
+type Shadow struct {
+	Enabled bool `yaml:"enabled"`
+	// Position is where Housecarl's filter chains run: after the other
+	// firewall's (it sees what that one lets through, so what it would
+	// drop is what switching would break) or before (it sees everything).
+	Position string `yaml:"position"`
+}
+
+// Shadow positions.
+const (
+	ShadowAfter  = "after"
+	ShadowBefore = "before"
+)
+
 // Stats configures statistics: a database of their own, aggregates
 // only, pruned by age.
 type Stats struct {
@@ -204,6 +236,8 @@ func Default() *Config {
 		Backup:         Backup{Dir: "/var/backups/housecarl", Retention: Duration(30 * 24 * time.Hour), KeepAtLeast: 7},
 		Stats: Stats{Enabled: true, Database: "/var/lib/housecarl/stats.db", TopN: 100,
 			Retention: StatsRetention{Minute: Duration(7 * 24 * time.Hour), Hour: Duration(90 * 24 * time.Hour), Day: Duration(730 * 24 * time.Hour)}},
+		Shadow: Shadow{Position: ShadowAfter},
+		Names:  Names{Enabled: true, Cache: Duration(10 * time.Minute)},
 		TLS:    TLS{ACME: ACME{Directory: "letsencrypt-staging", Resolvers: []string{"1.1.1.1:53", "9.9.9.9:53", "8.8.8.8:53"}, CheckPropagation: true}},
 	}
 }
@@ -275,6 +309,17 @@ func (c *Config) check() error {
 	default:
 		return fmt.Errorf("log.level must be debug, info, warn or error")
 	}
+	if c.Shadow.Position != ShadowAfter && c.Shadow.Position != ShadowBefore {
+		return fmt.Errorf("shadow.position must be after or before")
+	}
+	if c.Names.Resolver != "" {
+		if _, err := netip.ParseAddrPort(c.Names.Resolver); err != nil {
+			return fmt.Errorf("names.resolver must be an address and port, such as 10.10.0.1:53")
+		}
+	}
+	if c.Names.Cache <= 0 {
+		return fmt.Errorf("names.cache must be a duration, such as 10m")
+	}
 	return nil
 }
 
@@ -427,4 +472,16 @@ stats:
     hour: 90d      # hourly figures and top lists
     day: 730d      # daily figures and top lists
   top_n: 100       # entries kept per top list, per hour and day
+
+# Shadow mode (docs/shadow.md): Housecarl's ruleset is loaded beside the
+# firewall you have now, but blocks, rejects and translates nothing. Its
+# counters, log and tracer show what it would do. No change is on
+# probation, and takeovers and closing connections are refused. housecarld
+# reads this when it starts: restart it to switch.
+shadow:
+  enabled: false
+  # after: Housecarl's filter chains run after the other firewall's, and
+  # see only what it lets through: what Housecarl would drop is what
+  # switching over would break. before: they see every packet.
+  position: after
 `

internal/config/config_test.go

@@ -35,6 +35,7 @@ func TestErrors(t *testing.T) {
 		"{tls: {acme: {domain: '*.example.com'}}, jarl: {server: https://j, token: x}}":                            "full name",
 		"{tls: {acme: {domain: d.example.com, directory: http://x}}, jarl: {server: https://j, token: x}}":         "https://",
 		"ruleset: relative.nft": "ruleset",
+		"shadow: {position: x}": "shadow.position",
 	}
 	for yml, want := range cases {
 		path := filepath.Join(t.TempDir(), "s.yaml")

internal/engine/engine.go

@@ -191,6 +191,7 @@ func (e *Engine) commit(ctx context.Context, p *auth.Principal, description stri
 	// Only report warnings the change brings; old ones are on the status
 	// page.
 	res.Warnings = newProblems(validate.Check(e.state, e.env()), problems)
+	res.Warnings = append(res.Warnings, quotaRestarts(e.state, next)...)
 	if e.Firewall != nil {
 		if err := e.Firewall.Check(ctx, next); err != nil {
 			return res, &Error{Kind: KindFirewall, Message: "nftables refused the ruleset: " + err.Error()}
@@ -230,6 +231,23 @@ func (e *Engine) commit(ctx context.Context, p *auth.Principal, description stri
 	return res, nil
 }
 
+// quotaRestarts warns of the rules whose quota starts again because they
+// now count it in a way what they used can't be carried to.
+func quotaRestarts(before, after *model.State) []validate.Problem {
+	was := map[int64]*model.Rule{}
+	for _, r := range before.Rules {
+		was[r.ID] = r
+	}
+	var out []validate.Problem
+	for _, r := range after.Rules {
+		if carry, why := model.QuotaCarry(was[r.ID], r); carry == model.QuotaRestarts {
+			out = append(out, validate.Problem{Severity: validate.Warning, Code: "quota_restarts", Object: r.Label(),
+				Message: "its quota starts again: " + why})
+		}
+	}
+	return out
+}
+
 func newProblems(before, after []validate.Problem) []validate.Problem {
 	seen := map[string]bool{}
 	for _, p := range before {
@@ -491,6 +509,12 @@ func Normalize(st *model.State) {
 		normMatch(&x.Match)
 		x.ToPort = normPorts(x.ToPort)
 	}
+	for _, b := range st.BanLists {
+		b.Name = strings.TrimSpace(b.Name)
+		b.Scope = strings.ToLower(strings.TrimSpace(b.Scope))
+		b.Hosts = normAddrs(b.Hosts)
+		b.Never = normAddrs(b.Never)
+	}
 	// Positions are fractions after insertions between rules; renumber
 	// only when two have come too close to put another between.
 	st.Sort()

internal/engine/engine_test.go

@@ -173,3 +173,49 @@ func TestPositionsStayStableUntilCrowded(t *testing.T) {
 		t.Errorf("order %v", order)
 	}
 }
+
+// TestQuotaRestartsWarns: a change that counts a quota so what it used
+// can't carry over warns; one that can (to a total, wider groups) doesn't.
+func TestQuotaRestartsWarns(t *testing.T) {
+	e := newEngine(t)
+	ctx := context.Background()
+	var id int64
+	if _, err := e.Apply(ctx, admin, "", Options{}, func(st *model.State, newID func() int64) error {
+		id = newID()
+		st.Rules = append(st.Rules, &model.Rule{ID: id, Chain: "forward", Action: "drop", Quota: &model.Quota{Bytes: 1 << 30, Over: true, PerSource: true}})
+		return nil
+	}); err != nil {
+		t.Fatal(err)
+	}
+	change := func(f func(r *model.Rule)) []string {
+		res, err := e.Apply(ctx, admin, "", Options{DryRun: true}, func(st *model.State, _ func() int64) error {
+			f(st.ByID(model.KindRule, id).(*model.Rule))
+			return nil
+		})
+		if err != nil {
+			t.Fatal(err)
+		}
+		var codes []string
+		for _, w := range res.Warnings {
+			if w.Code == "quota_restarts" {
+				codes = append(codes, w.Message)
+			}
+		}
+		return codes
+	}
+	if w := change(func(r *model.Rule) { r.Quota.PerSource, r.Quota.PerDestination = false, true }); len(w) != 1 || !strings.Contains(w[0], "per source can't be counted per destination") {
+		t.Errorf("per source to per destination: %v", w)
+	}
+	if w := change(func(r *model.Rule) { r.Quota.PerSource = false }); len(w) != 0 {
+		t.Errorf("to a total (summed): %v", w)
+	}
+	if w := change(func(r *model.Rule) { r.GroupV6 = 56 }); len(w) != 0 {
+		t.Errorf("wider groups: %v", w)
+	}
+	if w := change(func(r *model.Rule) { r.GroupV6 = 128 }); len(w) != 1 {
+		t.Errorf("finer groups: %v", w)
+	}
+	if w := change(func(r *model.Rule) { r.Quota.Bytes = 2 << 30 }); len(w) != 0 {
+		t.Errorf("another amount: %v", w)
+	}
+}

internal/importer/audit_test.go

@@ -0,0 +1,98 @@
+package importer
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"strings"
+	"testing"
+
+	"housecarl/internal/render"
+	"housecarl/internal/validate"
+)
+
+// TestImportAudit: rulesets the importer must not change the meaning of
+// (testdata/audit.nft, as nft 1.1.3 lists it).
+func TestImportAudit(t *testing.T) {
+	js, _ := os.ReadFile("testdata/audit.json")
+	txt, _ := os.ReadFile("testdata/audit.txt")
+	defs, _ := os.ReadFile("testdata/audit.nft")
+	res, err := Import(Input{JSON: js, Text: string(txt), Defines: string(defs), Zones: map[string]string{"enp2s0.20": "iot", "eth9": "x"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	st := res.State
+	notes := strings.Join(res.Notes, "\n")
+	text := render.Render(st, render.Options{}).Text
+	has := func(what, s string) {
+		t.Helper()
+		if !strings.Contains(text, s) {
+			t.Errorf("%s: no %q in\n%s", what, s, text)
+		}
+	}
+	// Two chains on the input hook: their rules once each, in priority
+	// order; established stays a rule after the first chain's rule.
+	if n := strings.Count(text, "192.0.2.66"); n != 1 {
+		t.Errorf("the early input chain's rule %d times", n)
+	}
+	if strings.Index(text, "192.0.2.66") > strings.Index(text, "ct state established,related counter accept comment \"hc:r") ||
+		!strings.Contains(notes, "both hook input") {
+		t.Errorf("chains on one hook:\n%s", text)
+	}
+	// A second match on a field, and != on the hop limit: raw, as written.
+	has("two source matches", "ip saddr 10.0.0.0/8 ip saddr != 10.0.10.0/24 tcp dport 22 drop")
+	has("hop limit", "ip6 hoplimit != 255 drop")
+	// A jump: a rule within it is the rule's own; one outside it, or raw,
+	// keeps the jump's match in front.
+	has("within the jump", `ip saddr 10.9.0.5 tcp dport 22 counter accept comment "hc:r`)
+	has("outside the jump", "ip saddr 10.9.0.0/24 ip saddr 10.8.0.1 tcp dport 23 accept")
+	has("raw under the jump", "ip saddr 10.9.0.0/24 meta mark 0x00000001 accept")
+	// Sets: a timeout set is dynamic; an interface set is written out in
+	// the raw rule that names it.
+	if s := st.Set("blocklist"); s == nil || !s.Dynamic || s.Timeout != 3600 {
+		t.Errorf("blocklist: %+v", s)
+	}
+	has("dynamic set", "ip saddr @blocklist_v4 counter drop")
+	has("interface set", `iifname { "enp2s0", "enp3s0" } accept`)
+	// An accept to a forwarded host that isn't only for forwarded
+	// connections stays a rule; a DNAT of two ports, two forwards.
+	has("not only forwarded", `ip daddr 10.0.10.37 tcp dport 443 counter accept comment "hc:r`)
+	if len(st.Forwards) != 2 || st.Forwards[0].Name != "web-80" || st.Forwards[1].Name != "web-443" {
+		t.Errorf("forwards %+v", st.Forwards)
+	}
+	// Zones: as given (a VLAN), from defines, and why.
+	if z := st.Zone("iot"); z == nil || z.Interfaces[0] != "enp2s0.20" {
+		t.Errorf("zone iot: %+v", st.Zones)
+	}
+	if st.Zone("lan") == nil || st.Zone("wan") == nil {
+		t.Errorf("zones %+v", st.Zones)
+	}
+	for _, want := range []string{"iot (enp2s0.20: as given)", "lan (enp2s0: the define lan_if)", "--zone eth9=x: no rule names"} {
+		if !strings.Contains(notes, want) {
+			t.Errorf("no note %q in\n%s", want, notes)
+		}
+	}
+	if ps := validate.Check(st, nil); validate.HasErrors(ps) {
+		t.Errorf("problems: %+v", validate.Errors(ps))
+	}
+	// And nftables takes it.
+	cmd := strings.Fields(os.Getenv("HOUSECARL_NFT"))
+	if len(cmd) == 0 {
+		return
+	}
+	c := exec.CommandContext(context.Background(), cmd[0], append(cmd[1:], "-c", "-f", "/dev/stdin")...)
+	c.Stdin = strings.NewReader(text)
+	if out, err := c.CombinedOutput(); err != nil {
+		t.Errorf("nft -c: %v\n%s\n%s", err, out, text)
+	}
+}
+
+// TestValidIface: interface names a define may hold, VLANs too.
+func TestValidIface(t *testing.T) {
+	for s, want := range map[string]bool{"enp2s0": true, "wan": true, "enp2s0.20": true, "br0.4094": true, "veth*": true,
+		"10.0.10.5": false, "example.com": false, "a.b.c": false, "enp2s0.20.30": false, "averyveryverylongname": false} {
+		if validIface(s) != want {
+			t.Errorf("validIface(%q) = %v", s, !want)
+		}
+	}
+}

internal/importer/importer.go

@@ -20,6 +20,7 @@ import (
 	"strings"
 
 	"housecarl/internal/model"
+	"housecarl/internal/render"
 )
 
 // Input is what to import.
@@ -34,6 +35,9 @@ type Input struct {
 	// Table is the table to import ("inet main"); empty: the only table
 	// that isn't Housecarl's.
 	Table string
+	// Zones names zones by interface ("enp2s0": "lan"), over the names
+	// the importer would choose; interfaces given one name share a zone.
+	Zones map[string]string
 }
 
 // Result is the imported configuration and what to know about it.
@@ -81,6 +85,8 @@ type nftSet struct {
 	Table   string            `json:"table"`
 	Name    string            `json:"name"`
 	Type    json.RawMessage   `json:"type"`
+	Flags   []string          `json:"flags"`
+	Timeout int               `json:"timeout"`
 	Elem    []json.RawMessage `json:"elem"`
 }
 
@@ -121,6 +127,8 @@ type importer struct {
 	meters    map[string]bool   // sets used as meters
 	noted     map[string]bool
 	positions map[string]float64
+	// zoneFrom says where each zone's name came from, for the note.
+	zoneFrom map[string]string
 }
 
 func (im *importer) note(format string, args ...any) {
@@ -136,7 +144,7 @@ func Import(in Input) (*Result, error) {
 		return nil, fmt.Errorf("the input isn't nft's JSON (nft -j list ruleset): %v", err)
 	}
 	im := &importer{in: in, chains: map[string]nftChain{}, rules: map[string][]nftRule{}, sets: map[string]nftSet{}, maps: map[string]nftMap{},
-		text: map[int]string{}, st: model.NewState(), used: map[string]bool{}, noted: map[string]bool{}, positions: map[string]float64{}}
+		text: map[int]string{}, st: model.NewState(), used: map[string]bool{}, noted: map[string]bool{}, positions: map[string]float64{}, zoneFrom: map[string]string{}}
 	var tables []nftTable
 	for _, item := range doc.Nftables {
 		if raw, ok := item["table"]; ok {
@@ -207,6 +215,7 @@ func Import(in Input) (*Result, error) {
 	im.importSets()
 	im.run()
 	im.nameForwards()
+	im.noteZones()
 	im.st.Renumber()
 	return &Result{State: im.st, Table: im.table.Family + " " + im.table.Name, Notes: im.notes, Raw: im.raw}, nil
 }
@@ -309,10 +318,13 @@ func key(vals []string) string {
 	return strings.Join(s, ",")
 }
 
-var ifaceRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_.@-]{0,14}\*?$`)
+var ifaceRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_@-]{0,14}\*?$`)
+
+// vlanRe is a VLAN sub-interface: enp2s0.20.
+var vlanRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_@-]*\.[0-9]{1,4}$`)
 
 func validIface(s string) bool {
-	return ifaceRe.MatchString(s) && !strings.ContainsAny(s, ".:") || s == "lo"
+	return len(s) <= 15 && (ifaceRe.MatchString(s) || vlanRe.MatchString(s))
 }
 
 // ---- sets
@@ -360,12 +372,22 @@ func (im *importer) importSets() {
 		for _, e := range s.Elem {
 			entries = append(entries, values(e)...)
 		}
-		if existing := im.st.Set(n); existing != nil {
+		name := setName(n)
+		if contains(s.Flags, "timeout") || contains(s.Flags, "dynamic") {
+			// Filled at run time (by a program, or a rule): a dynamic set.
+			// Its entries now are passing state, not configuration.
+			im.st.Sets = append(im.st.Sets, &model.AddressSet{ID: im.newID(), Name: name, Dynamic: true, Timeout: s.Timeout})
+			im.used[name] = true
+			im.note("set %s has entries that come and go (flags %s): it became a dynamic set, without its %d entries now; whatever fills it must now fill Housecarl's (hc set entries add %s ADDRESS, the API, or nft add element inet housecarl %s_v4 / %s_v6)",
+				n, strings.Join(s.Flags, ","), len(entries), name, name, name)
+			continue
+		}
+		if existing := im.st.Set(name); existing != nil {
 			existing.Entries = append(existing.Entries, entries...)
 			continue
 		}
-		im.st.Sets = append(im.st.Sets, &model.AddressSet{ID: im.newID(), Name: setName(n), Entries: entries})
+		im.st.Sets = append(im.st.Sets, &model.AddressSet{ID: im.newID(), Name: name, Entries: entries})
-		im.used[n] = true
+		im.used[name] = true
 	}
 }
 
@@ -397,7 +419,10 @@ func (im *importer) namedSet(entries []string) string {
 	return name
 }
 
-// zone returns the zone of an interface, making it on first use.
+// zone returns the zone of an interface, making it on first use. Its
+// name: the one given for it (Input.Zones); else the name of a define
+// holding the interface ("define lan_if = enp2s0": lan); else the
+// interface's own.
 func (im *importer) zone(iface string) string {
 	for _, z := range im.st.Zones {
 		for _, i := range z.Interfaces {
@@ -406,21 +431,78 @@ func (im *importer) zone(iface string) string {
 			}
 		}
 	}
-	name := iface
+	if given, ok := im.in.Zones[iface]; ok {
+		name := setName(given)
+		// Interfaces given one name share its zone.
+		if z := im.st.Zone(name); z != nil && im.zoneFrom[name] == "given" {
+			z.Interfaces = append(z.Interfaces, iface)
+			return name
+		}
+		if im.st.Zone(name) == nil {
+			return im.newZone(name, iface, "given")
+		}
+		im.note("--zone %s=%s: the zone %s is taken (by %s), so %s is named as below", iface, given, name, strings.Join(im.st.Zone(name).Interfaces, ", "), iface)
+	}
+	name, from := iface, "its interface's name"
 	if d, ok := im.ifNames[iface]; ok {
-		name = d
+		name, from = d, "the define "+d
-		for _, suf := range []string{"_interface", "_iface", "_int", "_if", "_dev"} {
+		for _, suf := range []string{"_interface", "_ifname", "_iface", "_device", "_int", "_nic", "_if", "_dev"} {
 			name = strings.TrimSuffix(name, suf)
 		}
 	}
 	name = setName(strings.TrimSuffix(name, "*"))
 	if im.st.Zone(name) != nil {
-		name = setName(iface + "_zone")
+		name, from = setName(iface+"_zone"), "its interface's name (the other was taken)"
 	}
+	return im.newZone(name, iface, from)
+}
+
+// zone2 is the zone an interface is in, or "" (making none).
+func (im *importer) zoneOfIface(iface string) string {
+	for _, z := range im.st.Zones {
+		if contains(z.Interfaces, iface) {
+			return z.Name
+		}
+	}
+	return ""
+}
+
+func (im *importer) newZone(name, iface, from string) string {
 	im.st.Zones = append(im.st.Zones, &model.Zone{ID: im.newID(), Name: name, Interfaces: []string{iface}})
+	im.zoneFrom[name] = from
 	return name
 }
 
+// noteZones says how each zone was named, and how to change it.
+func (im *importer) noteZones() {
+	ifaces := make([]string, 0, len(im.in.Zones))
+	for i := range im.in.Zones {
+		ifaces = append(ifaces, i)
+	}
+	sort.Strings(ifaces)
+	for _, i := range ifaces {
+		if im.zoneOfIface(i) == "" {
+			im.note("--zone %s=%s: no rule names the interface %s, so there is no zone for it", i, im.in.Zones[i], i)
+		}
+	}
+	if len(im.st.Zones) == 0 {
+		return
+	}
+	var parts []string
+	for _, z := range im.st.Zones {
+		from := im.zoneFrom[z.Name]
+		if from == "given" {
+			from = "as given"
+		}
+		ext := ""
+		if z.External {
+			ext = ", external"
+		}
+		parts = append(parts, fmt.Sprintf("%s (%s%s: %s)", z.Name, strings.Join(z.Interfaces, ", "), ext, from))
+	}
+	im.note("zones: %s; name them yourself with --zone INTERFACE=NAME (lan, iot, guest: what the network is for), or later with hc zone update NAME --name NEW", strings.Join(parts, "; "))
+}
+
 func (im *importer) external(iface string) {
 	if z := im.st.Zone(im.zone(iface)); z != nil && !z.External {
 		z.External = true
@@ -522,6 +604,11 @@ type parsed struct {
 	// name: a port forward's name, when the rule's comment won't do
 	// (one of a map's elements).
 	name string
+	// inJump: the rule came from a chain jumped to; jumpText is what the
+	// jumps to it match, as nftables text (none when it isn't known), put
+	// in front of the rule's own text if it stays raw.
+	inJump   bool
+	jumpText string
 }
 
 // mapUse is a lookup in a map: "dnat ip to tcp dport map @forwards" or
@@ -550,7 +637,11 @@ func parse(r nftRule) *parsed {
 	return p
 }
 
-func (p *parsed) bad(what string) { p.unsupported = append(p.unsupported, what) }
+func (p *parsed) bad(what string) {
+	if !contains(p.unsupported, what) {
+		p.unsupported = append(p.unsupported, what)
+	}
+}
 
 func (p *parsed) expr(k string, v json.RawMessage) {
 	switch k {
@@ -839,19 +930,24 @@ func (p *parsed) match(m match) {
 	if meta, ok := left["meta"]; ok {
 		var x struct{ Key string }
 		json.Unmarshal(meta, &x)
-		switch x.Key {
+		switch {
-		case "iifname":
+		case (x.Key == "iifname" || x.Key == "oifname") && anySet(right):
+			p.bad("meta " + x.Key + " in a named set")
+		case x.Key == "iifname" && len(p.inIf) > 0, x.Key == "oifname" && len(p.outIf) > 0:
+			// Both must hold; Housecarl's rule has one list: kept raw.
+			p.bad("two matches on meta " + x.Key)
+		case x.Key == "iifname":
 			p.inIf, p.notIn = right, not
-		case "oifname":
+		case x.Key == "oifname":
 			p.outIf, p.notOut = right, not
-		case "nfproto":
+		case x.Key == "nfproto":
-			if len(right) == 1 && !not {
+			if len(right) == 1 && !not && (p.family == "" || p.family == right[0]) {
 				p.family = right[0]
 			} else {
 				p.bad("meta nfproto")
 			}
-		case "l4proto":
+		case x.Key == "l4proto":
-			if len(right) == 1 && !not {
+			if len(right) == 1 && !not && (p.proto == "" || p.proto == right[0]) {
 				p.proto = right[0]
 			} else {
 				p.bad("meta l4proto with several protocols")
@@ -864,23 +960,29 @@ func (p *parsed) match(m match) {
 	if pl, ok := left["payload"]; ok {
 		var x struct{ Protocol, Field string }
 		json.Unmarshal(pl, &x)
+		fam := map[string]string{"ip": "ipv4", "ip6": "ipv6"}[x.Protocol]
 		switch {
+		// A second match on a field Housecarl keeps one list for (both
+		// must hold): the rule stays raw, as written.
+		case x.Field == "saddr" && len(p.src) > 0, x.Field == "daddr" && len(p.dst) > 0,
+			x.Field == "dport" && len(p.dports) > 0, x.Field == "type" && len(p.icmp) > 0:
+			p.bad("two matches on " + x.Protocol + " " + x.Field)
+		case fam != "" && p.family != "" && p.family != fam && (x.Field == "saddr" || x.Field == "daddr"):
+			p.bad("both IPv4 and IPv6 addresses")
 		case (x.Protocol == "ip" || x.Protocol == "ip6") && (x.Field == "saddr" || x.Field == "daddr"):
-			if p.family == "" {
+			p.family = fam
-				p.family = map[string]string{"ip": "ipv4", "ip6": "ipv6"}[x.Protocol]
-			}
 			if x.Field == "saddr" {
 				p.src, p.notSrc = right, not
 			} else {
 				p.dst, p.notDst = right, not
 			}
-		case (x.Protocol == "tcp" || x.Protocol == "udp" || x.Protocol == "sctp") && x.Field == "dport" && !not:
+		case (x.Protocol == "tcp" || x.Protocol == "udp" || x.Protocol == "sctp") && x.Field == "dport" && !not && (p.proto == "" || p.proto == x.Protocol):
 			p.proto, p.dports = x.Protocol, right
-		case x.Protocol == "tcp" && x.Field == "flags" && len(right) == 1 && right[0] == "syn":
+		case x.Protocol == "tcp" && x.Field == "flags" && len(right) == 1 && right[0] == "syn" && !not:
 			p.tcpSyn = true
 		case (x.Protocol == "icmp" || x.Protocol == "icmpv6") && x.Field == "type" && !not:
 			p.icmpProto, p.icmp = x.Protocol, right
-		case x.Protocol == "ip6" && x.Field == "hoplimit" && len(right) == 1 && right[0] == "255":
+		case x.Protocol == "ip6" && x.Field == "hoplimit" && len(right) == 1 && right[0] == "255" && !not:
 			p.hop255 = true
 		default:
 			p.bad(x.Protocol + " " + x.Field)
@@ -891,9 +993,9 @@ func (p *parsed) match(m match) {
 		var x struct{ Key string }
 		json.Unmarshal(ct, &x)
 		switch {
-		case x.Key == "state" && !not:
+		case x.Key == "state" && !not && len(p.ctState) == 0:
 			p.ctState = right
-		case x.Key == "status" && !not:
+		case x.Key == "status" && !not && len(p.ctStatus) == 0:
 			p.ctStatus = right
 		default:
 			p.bad("ct " + x.Key)
@@ -905,40 +1007,80 @@ func (p *parsed) match(m match) {
 	}
 }
 
+// anySet reports whether a match's values name a set ("@name").
+func anySet(vals []string) bool {
+	for _, v := range vals {
+		if strings.HasPrefix(v, "@") {
+			return true
+		}
+	}
+	return false
+}
+
 // ---- converting
 
 func (im *importer) run() {
 	type base struct {
 		name, kind string
+		prio       int
 	}
 	var bases []base
-	for name, c := range im.chains {
+	names := make([]string, 0, len(im.chains))
+	for name := range im.chains {
+		names = append(names, name)
+	}
+	sort.Strings(names)
+	policies := map[string][]string{}
+	for _, name := range names {
+		c := im.chains[name]
 		switch {
 		case c.Type == "filter" && c.Hook == "prerouting":
-			bases = append(bases, base{name, model.ChainEarly})
+			bases = append(bases, base{name, model.ChainEarly, c.Prio})
+			if c.Prio > -200 {
+				im.note("chain %s hooks prerouting at priority %d, after connection tracking; Housecarl's early chain runs before it (raw), where connection states aren't known yet: check its rules", name, c.Prio)
+			}
 		case c.Type == "filter" && (c.Hook == "input" || c.Hook == "forward" || c.Hook == "output"):
-			bases = append(bases, base{name, c.Hook})
+			bases = append(bases, base{name, c.Hook, c.Prio})
+			policies[c.Hook] = append(policies[c.Hook], c.Policy)
 		case c.Type == "nat" && c.Hook == "prerouting":
-			bases = append(bases, base{name, "dstnat"})
+			bases = append(bases, base{name, "dstnat", c.Prio})
 		case c.Type == "nat" && c.Hook == "postrouting":
-			bases = append(bases, base{name, "srcnat"})
+			bases = append(bases, base{name, "srcnat", c.Prio})
 		case c.Hook != "":
 			im.note("chain %s (%s %s) isn't imported: Housecarl doesn't manage that hook", name, c.Type, c.Hook)
 		}
+	}
+	// Several chains on one hook: a packet must get past each (an accept
+	// ends only its own chain), and any policy drop drops it.
 	set := &im.st.Settings
-		switch {
+	for hook, ps := range policies {
-		case c.Hook == "input" && c.Type == "filter":
+		policy := model.PolicyAccept
-			set.InputPolicy = c.Policy
+		if contains(ps, model.PolicyDrop) {
-		case c.Hook == "forward" && c.Type == "filter":
+			policy = model.PolicyDrop
-			set.ForwardPolicy = c.Policy
+		}
-		case c.Hook == "output" && c.Type == "filter":
+		switch hook {
-			set.OutputPolicy = c.Policy
+		case "input":
+			set.InputPolicy = policy
+		case "forward":
+			set.ForwardPolicy = policy
+		case "output":
+			set.OutputPolicy = policy
 		}
 	}
 	// NAT first: the filter chains' port-forward accepts are folded into
-	// the port forwards.
+	// the port forwards. Chains on the same hook in priority order.
 	order := map[string]int{"dstnat": 0, "srcnat": 1, model.ChainEarly: 2, model.ChainInput: 3, model.ChainForward: 4, model.ChainOutput: 5}
-	sort.SliceStable(bases, func(i, j int) bool { return order[bases[i].kind] < order[bases[j].kind] })
+	sort.SliceStable(bases, func(i, j int) bool {
+		if bases[i].kind != bases[j].kind {
+			return order[bases[i].kind] < order[bases[j].kind]
+		}
+		return bases[i].prio < bases[j].prio
+	})
+	for i := 1; i < len(bases); i++ {
+		if b, a := bases[i], bases[i-1]; b.kind == a.kind && b.kind != "dstnat" && b.kind != "srcnat" {
+			im.note("chains %s and %s both hook %s: a packet must get past each, but Housecarl has one %s chain, so their rules follow each other (by priority); that only does the same if what %s accepts, %s would accept too: check it", a.name, b.name, b.kind, b.kind, a.name, b.name)
+		}
+	}
 	im.st.Settings.Builtins = model.Builtins{}
 	found := map[string]map[string]string{} // builtin → chain → "prefix", "later" or "demoted"
 	heads := map[string][]headRule{}        // chain → the built-ins its rules start with, in order
@@ -956,8 +1098,12 @@ func (im *importer) run() {
 				im.srcnat(p)
 			}
 		default:
+			if !contains(chains, b.kind) {
 				chains = append(chains, b.kind)
-			prefix := true
+			}
+			// Built-ins come first only at the head of the hook's first
+			// chain (or of one after a chain of nothing but built-ins).
+			prefix := len(rest[b.kind]) == 0
 			for _, p := range rules {
 				if bi := im.builtinOf(b.kind, p); bi != "" {
 					if found[bi] == nil {
@@ -1016,6 +1162,7 @@ func (im *importer) flatten(chain string, prefix *parsed, depth int) []*parsed {
 			}
 			jumpMatch := *p
 			jumpMatch.verdict, jumpMatch.target = "", ""
+			jumpMatch.jumpText = joinText(p.jumpText, im.condText(p))
 			out = append(out, im.flatten(p.target, &jumpMatch, depth+1)...)
 			if p.verdict == "goto" {
 				im.note("the goto to %s became the same rules followed by the rest of %s: check the order", p.target, chain)
@@ -1147,36 +1294,153 @@ func hasMatch(p, prefix *parsed) bool {
 		len(p.inIf) != len(prefix.inIf) || len(p.outIf) != len(prefix.outIf) || len(p.ctState) != len(prefix.ctState) || len(p.icmp) != len(prefix.icmp)
 }
 
-// merge puts a jump's matches in front of a jumped-to rule's. When both
+// condText is a jump rule's text without its verdict, counter and
-// match the same field, the rule's own (narrower, usually) is kept and
+// comment: what it matches ("ip saddr 10.50.50.0/24"); "" when there is
-// noted.
+// no text for it.
+var verdictRe = regexp.MustCompile(`\s*\b(jump|goto)\s+\S+\s*$`)
+
+func (im *importer) condText(p *parsed) string {
+	text, ok := im.text[p.handle]
+	if !ok {
+		return ""
+	}
+	if i := strings.LastIndex(text, ` comment "`); i >= 0 {
+		text = text[:i]
+	}
+	text = verdictRe.ReplaceAllString(strings.TrimSpace(text), "")
+	return strings.Join(strings.Fields(strings.ReplaceAll(" "+text+" ", " counter ", " ")), " ")
+}
+
+func joinText(a, b string) string {
+	return strings.TrimSpace(a + " " + b)
+}
+
+// merge puts a jump's matches in front of a jumped-to rule's: the rule
+// applies where both match. When both match the same field, that holds as
+// one list only when one's values lie within the other's (the narrower is
+// kept); otherwise the rule is kept raw, after the jump's text.
 func (im *importer) merge(prefix, p *parsed) *parsed {
 	q := *p
+	q.inJump, q.jumpText = true, prefix.jumpText
 	q.unsupported = append(append([]string{}, prefix.unsupported...), p.unsupported...)
-	fill := func(what string, dst *[]string, not *bool, src []string, srcNot bool) {
+	both := func(what string) { q.bad("both the jump to it and the rule match " + what) }
-		if len(*dst) == 0 && len(src) > 0 {
+	addrs := func(what string, dst *[]string, not *bool, src []string, srcNot bool) {
+		switch {
+		case len(src) == 0:
+		case len(*dst) == 0:
 			*dst, *not = src, srcNot
-		} else if len(*dst) > 0 && len(src) > 0 && key(*dst) != key(src) {
+		case *not || srcNot:
-			im.note("rule handle %d: both the jump to it and the rule match %s; the rule's own (%s) are kept, the jump's (%s) dropped: check it",
+			if key(*dst) != key(src) || *not != srcNot {
-				p.handle, what, strings.Join(*dst, ", "), strings.Join(src, ", "))
+				both(what)
+			}
+		case im.within(*dst, src):
+			// The rule's own are narrower: they say it all.
+		case im.within(src, *dst):
+			*dst = src
+		default:
+			both(what)
 		}
 	}
-	fill("interfaces in", &q.inIf, &q.notIn, prefix.inIf, prefix.notIn)
+	same := func(what string, dst *[]string, not *bool, src []string, srcNot bool) {
-	fill("interfaces out", &q.outIf, &q.notOut, prefix.outIf, prefix.notOut)
+		switch {
-	fill("sources", &q.src, &q.notSrc, prefix.src, prefix.notSrc)
+		case len(src) == 0:
-	fill("destinations", &q.dst, &q.notDst, prefix.dst, prefix.notDst)
+		case len(*dst) == 0:
-	if q.family == "" {
+			*dst, *not = src, srcNot
+		case key(*dst) != key(src) || *not != srcNot:
+			both(what)
+		}
+	}
+	var noNot bool
+	same("interfaces in", &q.inIf, &q.notIn, prefix.inIf, prefix.notIn)
+	same("interfaces out", &q.outIf, &q.notOut, prefix.outIf, prefix.notOut)
+	addrs("sources", &q.src, &q.notSrc, prefix.src, prefix.notSrc)
+	addrs("destinations", &q.dst, &q.notDst, prefix.dst, prefix.notDst)
+	same("connection states", &q.ctState, &noNot, prefix.ctState, false)
+	same("connection status", &q.ctStatus, &noNot, prefix.ctStatus, false)
+	switch {
+	case prefix.family == "":
+	case q.family == "":
 		q.family = prefix.family
+	case q.family != prefix.family:
+		both("an address family")
 	}
-	if len(q.ctState) == 0 {
+	switch {
-		q.ctState = prefix.ctState
+	case prefix.proto == "":
-	}
+	case q.proto == "":
-	if q.proto == "" {
 		q.proto, q.dports = prefix.proto, prefix.dports
+	case q.proto != prefix.proto:
+		both("a protocol")
+	default:
+		same("ports", &q.dports, &noNot, prefix.dports, false)
+	}
+	switch {
+	case prefix.icmpProto == "":
+	case q.icmpProto == "":
+		q.icmpProto, q.icmp = prefix.icmpProto, prefix.icmp
+	case q.icmpProto != prefix.icmpProto || key(q.icmp) != key(prefix.icmp):
+		both("ICMP types")
+	}
+	q.hop255 = q.hop255 || prefix.hop255
+	q.tcpSyn = q.tcpSyn || prefix.tcpSyn
+	if prefix.limit != nil || prefix.quota != nil || prefix.connLimit > 0 || prefix.log {
+		// Once in the jump, these count or log each packet once; on every
+		// rule they would act once per rule.
+		q.bad("a jump with a limit, quota or log")
 	}
 	return &q
 }
 
+// within reports whether every address of inner lies within outer's
+// (literal addresses and prefixes, or sets of the table).
+func (im *importer) within(inner, outer []string) bool {
+	resolve := func(list []string) ([]netip.Prefix, bool) {
+		var out []netip.Prefix
+		for _, e := range list {
+			vals := []string{e}
+			if strings.HasPrefix(e, "@") {
+				s, ok := im.sets[e[1:]]
+				if !ok {
+					return nil, false
+				}
+				vals = nil
+				for _, el := range s.Elem {
+					vals = append(vals, values(el)...)
+				}
+			}
+			for _, v := range vals {
+				pf, err := netip.ParsePrefix(v)
+				if err != nil {
+					a, err := netip.ParseAddr(v)
+					if err != nil {
+						return nil, false // a range, or something else
+					}
+					pf = netip.PrefixFrom(a, a.BitLen())
+				}
+				out = append(out, pf.Masked())
+			}
+		}
+		return out, len(out) > 0
+	}
+	in, ok1 := resolve(inner)
+	out, ok2 := resolve(outer)
+	if !ok1 || !ok2 {
+		return false
+	}
+	for _, a := range in {
+		found := false
+		for _, b := range out {
+			if b.Bits() <= a.Bits() && b.Contains(a.Addr()) {
+				found = true
+				break
+			}
+		}
+		if !found {
+			return false
+		}
+	}
+	return true
+}
+
 // ---- built-ins
 
 var essentials6 = map[string]bool{"destination-unreachable": true, "packet-too-big": true, "time-exceeded": true, "parameter-problem": true, "echo-request": true, "echo-reply": true}
@@ -1402,8 +1666,19 @@ func (im *importer) zones(ifaces []string) []string {
 	return out
 }
 
+// checkSets marks a rule naming a set that wasn't imported (not of
+// addresses, or a meter) unsupported: it stays raw.
+func (im *importer) checkSets(p *parsed) {
+	for _, e := range append(append([]string{}, p.src...), p.dst...) {
+		if n, ok := strings.CutPrefix(e, "@"); ok && (im.meters[n] || im.st.Set(setName(n)) == nil) {
+			p.bad("the set @" + n + ", which isn't an address set Housecarl imported")
+		}
+	}
+}
+
 // toMatch builds a Match; ok is false when something can't be said.
 func (im *importer) toMatch(p *parsed) (model.Match, bool) {
+	im.checkSets(p)
 	svcs, ok := im.services(p)
 	if !ok || len(p.unsupported) > 0 {
 		return model.Match{}, false
@@ -1441,6 +1716,10 @@ func (im *importer) rule(chain string, p *parsed) {
 	default:
 		p.bad(p.verdict)
 	}
+	if p.log && !im.noted["log"] {
+		im.noted["log"] = true
+		im.note("rules that log now log to Housecarl's log (hc log, Traffic → Log), not the kernel log or the journal")
+	}
 	m, ok := im.toMatch(p)
 	if r.CountsTraffic() && r.Action == model.ActionAccept {
 		// Housecarl checks these before established connections are
@@ -1471,6 +1750,14 @@ func (im *importer) rawRule(r *model.Rule, p *parsed) {
 	if i := strings.LastIndex(text, ` comment "`); i >= 0 {
 		text = strings.TrimSpace(text[:i])
 	}
+	// From a chain jumped to: only where the jumps matched.
+	if p.inJump {
+		if p.jumpText == "" {
+			im.note("rule handle %d of chain %s couldn't be expressed (%s), and the jump to it has no text to keep it raw with: add it by hand: %s", p.handle, r.Chain, strings.Join(p.unsupported, ", "), text)
+			return
+		}
+		text = p.jumpText + " " + text
+	}
 	// A named map isn't carried over: a raw rule naming it would make
 	// nftables refuse the whole ruleset.
 	for _, ref := range mapRefRe.FindAllStringSubmatch(text, -1) {
@@ -1480,6 +1767,12 @@ func (im *importer) rawRule(r *model.Rule, p *parsed) {
 			return
 		}
 	}
+	text, bad := im.setRefs(text)
+	if bad != "" {
+		im.note("rule handle %d of chain %s uses the set @%s, which Housecarl's table doesn't have (%s): left out, add it by hand: %s",
+			p.handle, r.Chain, bad, strings.Join(p.unsupported, ", "), text)
+		return
+	}
 	if r.Action == "" {
 		r.Action = model.ActionAccept
 	}
@@ -1491,41 +1784,86 @@ func (im *importer) rawRule(r *model.Rule, p *parsed) {
 		why = "a match Housecarl has no field for"
 	}
 	im.raw = append(im.raw, fmt.Sprintf("%s: %s (%s)", r.Chain, text, why))
-	if strings.Contains(text, "@") {
+}
-		im.note("raw rule %q names a set of the old table: Housecarl's sets are NAME_v4 and NAME_v6", text)
+
+var addrSetRefRe = regexp.MustCompile(`\b(ip6?) (saddr|daddr) (!= )?@([A-Za-z0-9_]+)`)
+
+// setRefs makes a raw rule's set references work in Housecarl's table:
+// an imported address set is NAME_v4 or NAME_v6 there, by the family the
+// rule matches; another set of the old table (interface names, ports) is
+// written out as an anonymous set of its elements. It returns the name of
+// a set it couldn't resolve (a meter, an empty or unknown one).
+func (im *importer) setRefs(text string) (string, string) {
+	text = addrSetRefRe.ReplaceAllStringFunc(text, func(m string) string {
+		sm := addrSetRefRe.FindStringSubmatch(m)
+		name := setName(sm[4])
+		if im.meters[sm[4]] || im.st.Set(name) == nil {
+			return m
+		}
+		return fmt.Sprintf("%s %s %s@%s", sm[1], sm[2], sm[3], render.SetName(name, sm[1] == "ip6"))
+	})
+	bad := ""
+	text = mapRefRe.ReplaceAllStringFunc(text, func(m string) string {
+		n := m[1:]
+		if strings.HasSuffix(n, "_v4") || strings.HasSuffix(n, "_v6") {
+			if im.st.Set(strings.TrimSuffix(strings.TrimSuffix(n, "_v4"), "_v6")) != nil {
+				return m // rewritten above
 			}
 		}
+		set, ok := im.sets[n]
+		if !ok || im.meters[n] || len(set.Elem) == 0 || contains(set.Flags, "timeout") || contains(set.Flags, "dynamic") {
+			bad = n
+			return m
+		}
+		var typ string
+		json.Unmarshal(set.Type, &typ)
+		var elems []string
+		for _, e := range set.Elem {
+			for _, v := range values(e) {
+				if typ == "ifname" || typ == "string" {
+					v = quote(v)
+				}
+				elems = append(elems, v)
+			}
+		}
+		return "{ " + strings.Join(elems, ", ") + " }"
+	})
+	return text, bad
+}
+
+func quote(s string) string { return `"` + strings.ReplaceAll(s, `"`, "") + `"` }
 
 // ---- NAT
 
 func (im *importer) dstnat(p *parsed) {
+	im.checkSets(p)
+	if a, err := netip.ParseAddr(p.dnatAddr); err == nil && a.Is6() {
+		p.bad("a DNAT to an IPv6 address (Housecarl's port forwards translate IPv4; IPv6 gets a pinhole)")
+	}
+	if p.notIn || len(p.outIf) > 0 || p.limit != nil || p.quota != nil || p.connLimit > 0 || p.log || len(p.ctState) > 0 && key(p.ctState) != "new" || len(p.icmp) > 0 {
+		p.bad("a DNAT with matches a port forward has no field for")
+	}
 	switch {
-	case p.dnatAddr != "" && len(p.unsupported) == 0 && p.proto != "" && len(p.dports) == 1 && !p.notDst && !p.notSrc:
+	case p.dnatAddr != "" && len(p.unsupported) == 0 && p.proto != "" && len(p.dports) > 0 && !p.notDst && !p.notSrc:
-		ports, err := model.NormalizePorts(p.dports[0])
+		var ports []string
+		for _, d := range p.dports {
+			n, err := model.NormalizePorts(d)
 			if err != nil {
+				p.bad("the port " + d)
 				break
 			}
-		f := &model.PortForward{Protocols: []string{p.proto}, Ports: ports, To: p.dnatAddr, ToPort: p.dnatPort, Comment: p.comment, Name: forwardName(p.comment)}
+			ports = append(ports, n)
-		if p.name != "" {
-			f.Name = strings.ToLower(nameRe.ReplaceAllString(p.name, "-"))
 		}
-		if f.ToPort == ports {
+		if len(p.unsupported) > 0 {
-			f.ToPort = ""
+			break
+		}
+		// "tcp dport { 80, 443 } dnat to X": a forward for each port.
+		for _, port := range ports {
+			im.portForward(p, port, len(ports) > 1)
 		}
-		f.PublicAddresses = im.addrRefs(p.dst)
-		f.Sources = im.addrRefs(p.src)
 		for _, i := range p.inIf {
 			im.external(i)
 		}
-		// The same forward for another protocol: one forward, both.
-		for _, o := range im.st.Forwards {
-			if o.Ports == f.Ports && o.To == f.To && o.ToPort == f.ToPort && key(o.PublicAddresses) == key(f.PublicAddresses) && key(o.Sources) == key(f.Sources) {
-				o.Protocols = append(o.Protocols, p.proto)
-				return
-			}
-		}
-		f.ID, f.Position = im.newID(), float64(len(im.st.Forwards)+1)
-		im.st.Forwards = append(im.st.Forwards, f)
 		return
 	case p.redirect != "" && p.redirect != "?" && len(p.unsupported) == 0:
 		m, ok := im.toMatch(p)
@@ -1534,7 +1872,35 @@ func (im *importer) dstnat(p *parsed) {
 			return
 		}
 	}
-	im.note("the dstnat rule handle %d couldn't be imported (%s): %s", p.handle, strings.Join(p.unsupported, ", "), im.text[p.handle])
+	im.note("the dstnat rule handle %d couldn't be imported (%s): add it by hand: %s", p.handle, strings.Join(p.unsupported, ", "), im.text[p.handle])
+}
+
+// portForward makes the port forward for one public port of a DNAT rule
+// (several: the port goes into its name).
+func (im *importer) portForward(p *parsed, ports string, several bool) {
+	f := &model.PortForward{Protocols: []string{p.proto}, Ports: ports, To: p.dnatAddr, ToPort: p.dnatPort, Comment: p.comment, Name: forwardName(p.comment)}
+	if p.name != "" {
+		f.Name = strings.ToLower(nameRe.ReplaceAllString(p.name, "-"))
+	}
+	if several && f.Name != "" {
+		f.Name += "-" + ports
+	}
+	if f.ToPort == ports {
+		f.ToPort = ""
+	}
+	f.PublicAddresses = im.addrRefs(p.dst)
+	f.Sources = im.addrRefs(p.src)
+	// The same forward for another protocol: one forward, both.
+	for _, o := range im.st.Forwards {
+		if o.Ports == f.Ports && o.To == f.To && o.ToPort == f.ToPort && key(o.PublicAddresses) == key(f.PublicAddresses) && key(o.Sources) == key(f.Sources) {
+			if !contains(o.Protocols, p.proto) {
+				o.Protocols = append(o.Protocols, p.proto)
+			}
+			return
+		}
+	}
+	f.ID, f.Position = im.newID(), float64(len(im.st.Forwards)+1)
+	im.st.Forwards = append(im.st.Forwards, f)
 }
 
 var forwardWords = regexp.MustCompile(`(?i)^\s*port\s*forward(ing)?\s*[:-]?\s*`)
@@ -1570,6 +1936,7 @@ func (im *importer) nameForwards() {
 }
 
 func (im *importer) srcnat(p *parsed) {
+	im.checkSets(p)
 	if len(p.unsupported) > 0 {
 		im.note("the srcnat rule handle %d couldn't be imported (%s): %s", p.handle, strings.Join(p.unsupported, ", "), im.text[p.handle])
 		return
@@ -1618,7 +1985,8 @@ func (im *importer) srcnat(p *parsed) {
 // absorbForward folds a port forward's own forward-chain accept (and an
 // IPv6 pinhole on its port) into the port forward.
 func (im *importer) absorbForward(p *parsed) bool {
-	if p.verdict != "accept" || len(p.unsupported) > 0 || len(p.dst) != 1 || p.notDst || len(p.dports) != 1 || p.proto == "" || len(p.src) > 0 || len(p.outIf) > 0 {
+	if p.verdict != "accept" || len(p.unsupported) > 0 || len(p.dst) != 1 || p.notDst || len(p.dports) != 1 || p.proto == "" || len(p.src) > 0 || len(p.outIf) > 0 ||
+		p.notIn || p.limit != nil || p.quota != nil || p.connLimit > 0 || p.log || len(p.ctState) > 0 && key(p.ctState) != "new" {
 		return false
 	}
 	port, err := model.NormalizePorts(p.dports[0])
@@ -1627,6 +1995,12 @@ func (im *importer) absorbForward(p *parsed) bool {
 	}
 	dst := p.dst[0]
 	if p.family == "ipv4" || !strings.Contains(dst, ":") {
+		// The forward's own accept takes only what was port forwarded: an
+		// accept of anything to that host and port stays a rule (inside
+		// clients reach the host directly through it).
+		if key(p.ctStatus) != "dnat" {
+			return false
+		}
 		for _, f := range im.st.Forwards {
 			inside := f.ToPort
 			if inside == "" {
@@ -1653,7 +2027,8 @@ func (im *importer) absorbForward(p *parsed) bool {
 			cands = append(cands, f)
 		}
 	}
-	if len(cands) == 1 && len(p.ctStatus) == 0 {
+	// The pinhole lets in from everywhere: only an accept that did too.
+	if len(cands) == 1 && len(p.ctStatus) == 0 && len(p.inIf) == 0 && len(p.ctState) == 0 {
 		cands[0].To6 = dst
 		im.note("the IPv6 rule to %s port %s became %s's IPv6 pinhole", dst, port, cands[0].Label())
 		return true

internal/importer/importer_test.go

@@ -115,9 +115,16 @@ func TestImportRouter(t *testing.T) {
 	if !wgDNS {
 		t.Errorf("wg0_input not inlined")
 	}
-	// The jump's and the rule's sources both: the rule's are kept, noted.
+	// The jump's and the rule's sources both: the rule's lie within the
-	if !strings.Contains(strings.Join(res.Notes, "\n"), "the rule's own (10.50.50.3, 10.50.50.6) are kept") {
+	// jump's, so they say it all, without a note.
-		t.Errorf("no note on the overlapping sources: %q", res.Notes)
+	var wgSSH bool
+	for _, r := range st.Rules {
+		if r.Chain == model.ChainInput && strings.Join(r.Services, ",") == "tcp/22" && strings.Join(r.Sources, ",") == "10.50.50.3,10.50.50.6" {
+			wgSSH = true
+		}
+	}
+	if !wgSSH || strings.Contains(strings.Join(res.Notes, "\n"), "jump") {
+		t.Errorf("the WireGuard SSH rule: found %v, notes %q", wgSSH, res.Notes)
 	}
 	// The imported configuration is valid, and renders.
 	if ps := validate.Check(st, nil); validate.HasErrors(ps) {

internal/importer/testdata/audit.json

@@ -0,0 +1 @@
+{"nftables": [{"metainfo": {"version": "1.1.3", "release_name": "Commodore Bullmoose #4", "json_schema_version": 1}}, {"table": {"family": "inet", "name": "audit", "handle": 1}}, {"chain": {"family": "inet", "table": "audit", "name": "early", "handle": 1, "type": "filter", "hook": "input", "prio": -10, "policy": "accept"}}, {"chain": {"family": "inet", "table": "audit", "name": "input", "handle": 2, "type": "filter", "hook": "input", "prio": 0, "policy": "drop"}}, {"chain": {"family": "inet", "table": "audit", "name": "admin", "handle": 3}}, {"chain": {"family": "inet", "table": "audit", "name": "forward", "handle": 4, "type": "filter", "hook": "forward", "prio": 0, "policy": "drop"}}, {"chain": {"family": "inet", "table": "audit", "name": "prerouting", "handle": 5, "type": "nat", "hook": "prerouting", "prio": -100, "policy": "accept"}}, {"set": {"family": "inet", "name": "blocklist", "table": "audit", "type": "ipv4_addr", "handle": 6, "flags": ["timeout"], "timeout": 3600}}, {"set": {"family": "inet", "name": "lans", "table": "audit", "type": "ifname", "handle": 7, "elem": ["enp2s0", "enp3s0"]}}, {"rule": {"family": "inet", "table": "audit", "chain": "early", "handle": 8, "comment": "an early input chain", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": "192.0.2.66"}}, {"drop": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 9, "expr": [{"match": {"op": "in", "left": {"ct": {"key": "state"}}, "right": ["established", "related"]}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 10, "comment": "dynamic", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": "@blocklist"}}, {"drop": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 11, "comment": "an interface set", "expr": [{"match": {"op": "==", "left": {"meta": {"key": "iifname"}}, "right": "@lans"}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 12, "comment": "two source matches", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": {"prefix": {"addr": "10.0.0.0", "len": 8}}}}, {"match": {"op": "!=", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": {"prefix": {"addr": "10.0.10.0", "len": 24}}}}, {"match": {"op": "==", "left": {"payload": {"protocol": "tcp", "field": "dport"}}, "right": 22}}, {"drop": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 13, "comment": "hop limit not 255", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "icmpv6", "field": "type"}}, "right": "nd-neighbor-solicit"}}, {"match": {"op": "!=", "left": {"payload": {"protocol": "ip6", "field": "hoplimit"}}, "right": 255}}, {"drop": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 14, "comment": "DNS from the IoT VLAN", "expr": [{"match": {"op": "==", "left": {"meta": {"key": "iifname"}}, "right": "enp2s0.20"}}, {"match": {"op": "==", "left": {"payload": {"protocol": "udp", "field": "dport"}}, "right": 53}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "input", "handle": 15, "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": {"prefix": {"addr": "10.9.0.0", "len": 24}}}}, {"jump": {"target": "admin"}}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "admin", "handle": 16, "comment": "within the jump", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": "10.9.0.5"}}, {"match": {"op": "==", "left": {"payload": {"protocol": "tcp", "field": "dport"}}, "right": 22}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "admin", "handle": 17, "comment": "outside the jump", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "saddr"}}, "right": "10.8.0.1"}}, {"match": {"op": "==", "left": {"payload": {"protocol": "tcp", "field": "dport"}}, "right": 23}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "admin", "handle": 18, "comment": "raw under the jump", "expr": [{"match": {"op": "==", "left": {"meta": {"key": "mark"}}, "right": 1}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "forward", "handle": 19, "expr": [{"match": {"op": "in", "left": {"ct": {"key": "state"}}, "right": ["established", "related"]}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "forward", "handle": 20, "comment": "not only forwarded", "expr": [{"match": {"op": "==", "left": {"payload": {"protocol": "ip", "field": "daddr"}}, "right": "10.0.10.37"}}, {"match": {"op": "==", "left": {"payload": {"protocol": "tcp", "field": "dport"}}, "right": 443}}, {"accept": null}]}}, {"rule": {"family": "inet", "table": "audit", "chain": "forward", "hanLine truncated

internal/importer/testdata/audit.nft

@@ -0,0 +1,53 @@
+#! /sbin/nft -f
+# Cases the importer must not get wrong (importer_test.go TestImportAudit).
+
+define wan_if = "wan"
+define lan_if = "enp2s0"
+define iot_if = "enp2s0.20"
+define admins = { 10.9.0.0/24 }
+
+table inet audit {
+	set blocklist {
+		type ipv4_addr
+		flags timeout
+		timeout 1h
+	}
+	set lans {
+		type ifname
+		elements = { "enp2s0", "enp3s0" }
+	}
+
+	chain early {
+		type filter hook input priority -10; policy accept;
+		ip saddr 192.0.2.66 drop comment "an early input chain"
+	}
+
+	chain input {
+		type filter hook input priority filter; policy drop;
+		ct state established,related accept
+		ip saddr @blocklist drop comment "dynamic"
+		iifname @lans accept comment "an interface set"
+		ip saddr 10.0.0.0/8 ip saddr != 10.0.10.0/24 tcp dport 22 drop comment "two source matches"
+		icmpv6 type nd-neighbor-solicit ip6 hoplimit != 255 drop comment "hop limit not 255"
+		iifname $iot_if udp dport 53 accept comment "DNS from the IoT VLAN"
+		ip saddr $admins jump admin
+	}
+
+	chain admin {
+		ip saddr 10.9.0.5 tcp dport 22 accept comment "within the jump"
+		ip saddr 10.8.0.1 tcp dport 23 accept comment "outside the jump"
+		meta mark 0x00000001 accept comment "raw under the jump"
+	}
+
+	chain forward {
+		type filter hook forward priority filter; policy drop;
+		ct state established,related accept
+		ip daddr 10.0.10.37 tcp dport 443 accept comment "not only forwarded"
+		iifname $lan_if oifname $wan_if accept
+	}
+
+	chain prerouting {
+		type nat hook prerouting priority dstnat; policy accept;
+		ip daddr 203.0.113.1 tcp dport { 80, 443 } dnat ip to 10.0.10.37 comment "web"
+	}
+}

internal/importer/testdata/audit.txt

@@ -0,0 +1,46 @@
+table inet audit { # handle 1
+	set blocklist { # handle 6
+		type ipv4_addr
+		timeout 1h
+	}
+
+	set lans { # handle 7
+		type ifname
+		elements = { "enp2s0",
+			     "enp3s0" }
+	}
+
+	chain early { # handle 1
+		type filter hook input priority filter - 10; policy accept;
+		ip saddr 192.0.2.66 drop comment "an early input chain" # handle 8
+	}
+
+	chain input { # handle 2
+		type filter hook input priority filter; policy drop;
+		ct state established,related accept # handle 9
+		ip saddr @blocklist drop comment "dynamic" # handle 10
+		iifname @lans accept comment "an interface set" # handle 11
+		ip saddr 10.0.0.0/8 ip saddr != 10.0.10.0/24 tcp dport 22 drop comment "two source matches" # handle 12
+		icmpv6 type nd-neighbor-solicit ip6 hoplimit != 255 drop comment "hop limit not 255" # handle 13
+		iifname "enp2s0.20" udp dport 53 accept comment "DNS from the IoT VLAN" # handle 14
+		ip saddr 10.9.0.0/24 jump admin # handle 15
+	}
+
+	chain admin { # handle 3
+		ip saddr 10.9.0.5 tcp dport 22 accept comment "within the jump" # handle 16
+		ip saddr 10.8.0.1 tcp dport 23 accept comment "outside the jump" # handle 17
+		meta mark 0x00000001 accept comment "raw under the jump" # handle 18
+	}
+
+	chain forward { # handle 4
+		type filter hook forward priority filter; policy drop;
+		ct state established,related accept # handle 19
+		ip daddr 10.0.10.37 tcp dport 443 accept comment "not only forwarded" # handle 20
+		iifname "enp2s0" oifname "wan" accept # handle 21
+	}
+
+	chain prerouting { # handle 5
+		type nat hook prerouting priority dstnat; policy accept;
+		ip daddr 203.0.113.1 tcp dport { 80, 443 } dnat ip to 10.0.10.37 comment "web" # handle 23
+	}
+}

internal/matrix/matrix.go

@@ -9,6 +9,7 @@ package matrix
 import (
 	"fmt"
 	"net/netip"
+	"regexp"
 	"strings"
 
 	"housecarl/internal/api"
@@ -63,6 +64,50 @@ func Cell(st *model.State, o Options, from, to string) (api.MatrixCell, bool) {
 }
 
 func cell(st *model.State, o Options, chain, from, to string) api.MatrixCell {
+	c := walkCell(st, o, chain, from, to)
+	sum(&c)
+	if from == to && chain == model.ChainForward {
+		c.SameZone, c.Note = true, sameZoneNote(st.Zone(from))
+	}
+	return c
+}
+
+// sum fills in the overview: the summary, the exceptions counted by
+// verdict, and what the allowing ones let through.
+func sum(c *api.MatrixCell) {
+	c.Open = []string{}
+	seen := map[string]bool{}
+	for _, e := range c.Exceptions {
+		switch e.Verdict {
+		case "allowed":
+			c.Allowed++
+			if !seen[e.Short] {
+				seen[e.Short] = true
+				c.Open = append(c.Open, e.Short)
+			}
+		case "blocked":
+			c.Blocked++
+		}
+	}
+	c.Summary = c.Verdict
+	if c.Verdict == "blocked" && c.Allowed > 0 || c.Verdict == "allowed" && c.Blocked > 0 {
+		c.Summary = "partly"
+	}
+}
+
+// sameZoneNote says what of a zone's traffic to itself the firewall sees:
+// only what it routes between the zone's interfaces.
+func sameZoneNote(z *model.Zone) string {
+	if z == nil {
+		return ""
+	}
+	if len(z.Interfaces) == 1 && !strings.HasSuffix(z.Interfaces[0], "*") {
+		return fmt.Sprintf("traffic within %s stays on its own network: it passes the firewall only if routed back out of %s", z.Name, z.Interfaces[0])
+	}
+	return fmt.Sprintf("only traffic routed between %s's interfaces (%s) passes the firewall; within one of them it stays on its own network", z.Name, strings.Join(z.Interfaces, ", "))
+}
+
+func walkCell(st *model.State, o Options, chain, from, to string) api.MatrixCell {
 	c := api.MatrixCell{From: from, To: to, Chain: chain, Exceptions: []api.MatrixException{}}
 	b := st.Settings.Builtins
 	fromZone := st.Zone(from)
@@ -71,7 +116,7 @@ func cell(st *model.State, o Options, chain, from, to string) api.MatrixCell {
 	if chain != model.ChainOutput {
 		if b.Antispoof && external {
 			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: model.ChainEarly, Label: "built-in: antispoof", Verdict: "blocked",
-				What: "sources that can't come from outside (private ranges, your own prefixes)"})
+				What: "sources that can't come from outside (private ranges, your own prefixes)", Short: "spoofed sources", Marks: []string{"b-antispoof", "b-antispoof6"}})
 		}
 		if decided := walk(st, o, &c, model.ChainEarly, from, ""); decided {
 			return c
@@ -81,22 +126,26 @@ func cell(st *model.State, o Options, chain, from, to string) api.MatrixCell {
 	switch chain {
 	case model.ChainInput:
 		if b.ICMPv6 {
-			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: icmpv6", Verdict: "allowed", What: "ICMPv6 errors, echo, neighbour discovery, MLD"})
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: icmpv6", Verdict: "allowed", What: "ICMPv6 errors, echo, neighbour discovery, MLD",
+				Short: "ICMPv6", Marks: []string{"b-icmpv6", "b-icmpv6-nd", "b-icmpv6-mld"}})
 		}
 		if b.PingInternal && !external {
-			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: ping", Verdict: "allowed", What: "ping (ICMP echo)"})
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: ping", Verdict: "allowed", What: "ping (ICMP echo)", Short: "ping", Marks: []string{"b-ping"}})
 		}
 		if b.DHCPv6Client && external {
-			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: dhcpv6-client", Verdict: "allowed", What: "DHCPv6 replies to the firewall"})
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: dhcpv6-client", Verdict: "allowed", What: "DHCPv6 replies to the firewall",
+				Short: "DHCPv6", Marks: []string{"b-dhcpv6-client"}})
 		}
 	case model.ChainForward:
-		if n := forwards(st); n > 0 && external && !st.Zone(to).External {
+		if marks := forwards(st); len(marks) > 0 && external && !st.Zone(to).External {
+			n := fmt.Sprintf("%d port forward%s", len(marks), map[bool]string{true: "", false: "s"}[len(marks) == 1])
 			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "port forwards", Verdict: "allowed",
-				What: fmt.Sprintf("%d port forward%s to inside hosts", n, map[bool]string{true: "", false: "s"}[n == 1])})
+				What: n + " to inside hosts", Short: n, Marks: marks})
 		}
 	case model.ChainOutput:
 		if st.Settings.OutputPolicy == model.PolicyDrop && b.ICMPv6 {
-			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: icmpv6", Verdict: "allowed", What: "ICMPv6 errors, echo, neighbour discovery, MLD"})
+			c.Exceptions = append(c.Exceptions, api.MatrixException{Chain: chain, Label: "built-in: icmpv6", Verdict: "allowed", What: "ICMPv6 errors, echo, neighbour discovery, MLD",
+				Short: "ICMPv6", Marks: []string{"b-icmpv6"}})
 		}
 	}
 	if walk(st, o, &c, chain, from, to) {
@@ -104,17 +153,22 @@ func cell(st *model.State, o Options, chain, from, to string) api.MatrixCell {
 	}
 	policy := map[string]string{model.ChainInput: st.Settings.InputPolicy, model.ChainForward: st.Settings.ForwardPolicy, model.ChainOutput: st.Settings.OutputPolicy}[chain]
 	c.Verdict, c.DecidedBy = verdictOf(policy), chain+" policy"
+	if policy == model.PolicyDrop {
+		c.Marks = []string{"p-" + chain} // only a drop policy is counted
+	}
 	return c
 }
 
-func forwards(st *model.State) int {
+// forwards returns the counters' marks of the port forwards that send
-	n := 0
+// connections inside.
+func forwards(st *model.State) []string {
+	var marks []string
 	for _, f := range st.Forwards {
 		if !f.Disabled && f.To != "" {
-			n++
+			marks = append(marks, fmt.Sprintf("f%d", f.ID))
 		}
 	}
-	return n
+	return marks
 }
 
 // walk goes through a chain's rules for the pair, as nftables checks
@@ -130,8 +184,22 @@ func walk(st *model.State, o Options, c *api.MatrixCell, chain, from, to string)
 		}
 	}
 	for _, r := range rules {
+		mark := []string{fmt.Sprintf("r%d", r.ID)}
 		if r.Raw != "" {
-			c.Exceptions = append(c.Exceptions, api.MatrixException{RuleID: r.ID, Chain: chain, Label: "rule " + r.Label(), Verdict: "unknown", What: "a raw rule: " + r.Raw})
+			rr := readRaw(r.Raw)
+			switch {
+			case !rr.ok:
+				c.Exceptions = append(c.Exceptions, api.MatrixException{RuleID: r.ID, Chain: chain, Label: "rule " + r.Label(), Verdict: "unknown", What: "a raw rule: " + r.Raw,
+					Short: "a raw rule", Marks: mark})
+			case !rr.forNew || chain == model.ChainEarly && rr.verdict == "allowed":
+				// Not for new connections; an early accept decides nothing.
+			case rr.match == "":
+				c.Verdict, c.DecidedBy, c.RuleID, c.Marks = rr.verdict, "rule "+r.Label(), r.ID, mark
+				return true
+			default:
+				c.Exceptions = append(c.Exceptions, api.MatrixException{RuleID: r.ID, Chain: chain, Label: "rule " + r.Label(), Verdict: rr.verdict, What: "raw: " + rr.match,
+					Short: rr.short, Marks: mark})
+			}
 			continue
 		}
 		if !zones(r, from, to) || !forNew(r) {
@@ -151,10 +219,11 @@ func walk(st *model.State, o Options, c *api.MatrixCell, chain, from, to string)
 			if what == "" {
 				what = "everything"
 			}
-			c.Exceptions = append(c.Exceptions, api.MatrixException{RuleID: r.ID, Chain: chain, Label: "rule " + r.Label(), Verdict: v, What: what})
+			c.Exceptions = append(c.Exceptions, api.MatrixException{RuleID: r.ID, Chain: chain, Label: "rule " + r.Label(), Verdict: v, What: what,
+				Short: short(r, what), Marks: mark})
 			continue
 		}
-		c.Verdict, c.DecidedBy, c.RuleID = v, "rule "+r.Label(), r.ID
+		c.Verdict, c.DecidedBy, c.RuleID, c.Marks = v, "rule "+r.Label(), r.ID, mark
 		return true
 	}
 	return false
@@ -265,6 +334,160 @@ func forNew(r *model.Rule) bool {
 	return false
 }
 
+// raw is what the matrix can read of a raw rule.
+type raw struct {
+	ok      bool   // read: it applies to every pair of its chain
+	forNew  bool   // it can match a new connection
+	verdict string // allowed or blocked
+	match   string // what it matches ("" for everything)
+	short   string
+}
+
+// rawStops are words that make a raw rule's reach or effect unclear to
+// the matrix: interfaces (which zones?), jumps, NAT, changes, verdict maps.
+var rawStops = map[string]bool{
+	"iifname": true, "oifname": true, "iif": true, "oif": true, "iiftype": true, "oiftype": true,
+	"jump": true, "goto": true, "return": true, "vmap": true, "queue": true, "continue": true,
+	"dnat": true, "snat": true, "masquerade": true, "redirect": true, "tproxy": true,
+	"set": true, "notrack": true, "dup": true, "fwd": true, "synproxy": true, "flow": true,
+}
+
+// readRaw reads a raw rule that is a match and then accept, drop or
+// reject: the verdict is the last word (but a reject's "with …"); counter
+// and log only count and log.
+func readRaw(text string) raw {
+	toks := fields(text)
+	for i, t := range toks {
+		if t == "comment" {
+			toks = toks[:i]
+			break
+		}
+	}
+	verdict := -1
+	for i, t := range toks {
+		if strings.HasPrefix(t, `"`) {
+			continue
+		}
+		w := strings.Trim(t, ",{}")
+		if rawStops[w] {
+			return raw{}
+		}
+		if w == "accept" || w == "drop" || w == "reject" {
+			verdict = i
+		}
+	}
+	if verdict < 0 || verdict != len(toks)-1 && !(toks[verdict] == "reject" && toks[verdict+1] == "with") {
+		return raw{}
+	}
+	rr := raw{ok: true, forNew: true, verdict: verdictOf(toks[verdict])}
+	var match []string
+	for i := 0; i < verdict; i++ {
+		switch toks[i] {
+		case "counter":
+			continue
+		case "log":
+			// Its options run up to the next match.
+			for i+1 < verdict && map[string]bool{"prefix": true, "level": true, "group": true, "flags": true}[toks[i+1]] {
+				i += 2
+			}
+			continue
+		}
+		match = append(match, toks[i])
+	}
+	rr.match = strings.Join(match, " ")
+	// ct state without new: not for new connections. The states are one
+	// word ("established,related") or a set ("{ established, related }").
+	for i := 0; i+2 < len(match); i++ {
+		if match[i] != "ct" || match[i+1] != "state" {
+			continue
+		}
+		list := match[i+2]
+		if list == "{" || strings.HasPrefix(list, "{") {
+			var b []string
+			for j := i + 2; j < len(match); j++ {
+				b = append(b, match[j])
+				if strings.HasSuffix(match[j], "}") {
+					break
+				}
+			}
+			list = strings.Join(b, " ")
+		}
+		rr.forNew = false
+		for _, st := range strings.FieldsFunc(list, func(c rune) bool { return c == ',' || c == ' ' || c == '{' || c == '}' }) {
+			rr.forNew = rr.forNew || st == "new"
+		}
+	}
+	rr.short = rr.match
+	if m := icmpRe.FindStringSubmatch(rr.match); m != nil {
+		rr.short = m[1]
+	}
+	if len(rr.short) > 40 {
+		rr.short = rr.short[:39] + "…"
+	}
+	return rr
+}
+
+// fields splits a rule into words, a quoted string being one.
+func fields(s string) []string {
+	var out []string
+	var cur strings.Builder
+	quoted := false
+	for _, c := range s {
+		switch {
+		case c == '"':
+			quoted = !quoted
+			cur.WriteRune(c)
+		case !quoted && (c == ' ' || c == '\t'):
+			if cur.Len() > 0 {
+				out = append(out, cur.String())
+				cur.Reset()
+			}
+		default:
+			cur.WriteRune(c)
+		}
+	}
+	if cur.Len() > 0 {
+		out = append(out, cur.String())
+	}
+	return out
+}
+
+// icmpRe finds an ICMP type match in a raw rule: shown as just icmp or
+// icmpv6.
+var icmpRe = regexp.MustCompile(`\b(icmpv6|icmp) type\b`)
+
+// short sums up an exception for the overview: its services (ICMP types
+// as just icmp or icmpv6), with the addresses it is limited to, which
+// matter as much ("tcp/22 from trusted"); times and rates are left to
+// the details.
+func short(r *model.Rule, what string) string {
+	var parts []string
+	if len(r.Services) > 0 {
+		var svcs []string
+		seen := map[string]bool{}
+		for _, s := range r.Services {
+			if p, _, ok := strings.Cut(s, "/"); ok && (p == "icmp" || p == "icmpv6") {
+				s = p
+			}
+			if !seen[s] {
+				seen[s] = true
+				svcs = append(svcs, s)
+			}
+		}
+		parts = append(parts, strings.Join(svcs, ", "))
+	}
+	for _, c := range strings.Split(what, " · ") {
+		if strings.HasPrefix(c, "from ") || strings.HasPrefix(c, "to ") {
+			parts = append(parts, c)
+		}
+	}
+	if len(parts) == 0 {
+		first, _, _ := strings.Cut(what, " · ")
+		return first
+	}
+	return strings.Join(parts, " ")
+}
+
 // conditions says what limits a rule to part of a pair's traffic ("" for
 // none: it decides the pair).
 func conditions(st *model.State, r *model.Rule) string {

internal/matrix/matrix_test.go

@@ -110,3 +110,90 @@ func TestConnected(t *testing.T) {
 		t.Errorf("guest → lan: %v", ids(guestLan)) // not "to a guest": it is in guest, not lan
 	}
 }
+
+// TestSummary: each pair at a glance, what its exceptions let through,
+// and the counters behind each line.
+func TestSummary(t *testing.T) {
+	st := state()
+	st.Settings.InputPolicy = model.PolicyDrop
+	m := Build(st)
+	for _, c := range []struct {
+		from, to, summary string
+		allowed, blocked  int
+		open              string
+	}{
+		{"lan", "wan", "allowed", 0, 0, ""},
+		{"guest", "wan", "partly", 1, 0, "http, https"},
+		{"wan", "lan", "partly", 1, 1, "1 port forward"},
+		{"lan", "guest", "blocked", 0, 0, ""},
+		{"lan", Firewall, "partly", 4, 0, "ICMPv6|ping|ssh from admins|dns"},
+	} {
+		got := find(m, c.from, c.to)
+		if got.Summary != c.summary || got.Allowed != c.allowed || got.Blocked != c.blocked || strings.Join(got.Open, "|") != c.open {
+			t.Errorf("%s → %s: %s +%d -%d %q; want %s +%d -%d %q", c.from, c.to, got.Summary, got.Allowed, got.Blocked, got.Open, c.summary, c.allowed, c.blocked, c.open)
+		}
+	}
+	if c := find(m, "lan", "lan"); !c.SameZone || !strings.Contains(c.Note, "out of eth1") {
+		t.Errorf("same zone: %+v", c)
+	}
+	if c := find(m, "lan", "wan"); c.SameZone || c.Note != "" {
+		t.Errorf("not the same zone: %+v", c)
+	}
+	wl := find(m, "wan", "lan")
+	if strings.Join(wl.Exceptions[0].Marks, ",") != "b-antispoof,b-antispoof6" || strings.Join(wl.Exceptions[1].Marks, ",") != "f30" || strings.Join(wl.Marks, ",") != "r15" {
+		t.Errorf("marks: %+v", wl)
+	}
+	icmp := &model.Rule{Match: model.Match{Services: []string{"icmp/echo-request", "icmp/echo-reply", "icmpv6/echo-request", "ssh"}}}
+	if s := short(icmp, "x"); s != "icmp, icmpv6, ssh" {
+		t.Errorf("short: %q", s)
+	}
+	ssh := &model.Rule{Match: model.Match{Sources: []string{"trusted"}, Services: []string{"ssh"}}, Days: []string{"mon"}}
+	if s := short(ssh, conditions(st, ssh)); s != "ssh from trusted" {
+		t.Errorf("short with a source: %q", s)
+	}
+	if s := short(&model.Rule{Days: []string{"mon"}}, "mon"); s != "mon" {
+		t.Errorf("short of a time: %q", s)
+	}
+	if c := find(m, "lan", Firewall); strings.Join(c.Marks, ",") != "p-input" {
+		t.Errorf("a drop policy's mark: %v", c.Marks)
+	}
+}
+
+// TestRaw: raw rules that are a match and a verdict are read; others stay
+// unknown.
+func TestRaw(t *testing.T) {
+	for _, c := range []struct {
+		raw, verdict, short string
+		ok, forNew          bool
+	}{
+		{"icmpv6 type { nd-router-solicit, nd-neighbor-advert } ip6 hoplimit 255 accept", "allowed", "icmpv6", true, true},
+		{`tcp dport 22 counter log prefix "ssh: " drop comment "x"`, "blocked", "tcp dport 22", true, true},
+		{"ip saddr 10.0.0.1 reject with icmpx type admin-prohibited", "blocked", "ip saddr 10.0.0.1", true, true},
+		{"ct state established,related accept", "allowed", "ct state established,related", true, false},
+		{"ct state { new, untracked } tcp dport 80 accept", "allowed", "ct state { new, untracked } tcp dport 80", true, true},
+		{"counter accept", "allowed", "", true, true},
+		{`iifname "wan" tcp dport 22 accept`, "", "", false, false},
+		{"tcp dport vmap { 22 : accept }", "", "", false, false},
+		{"meta mark set 1 accept", "", "", false, false},
+		{"tcp dport 22 counter", "", "", false, false},
+	} {
+		rr := readRaw(c.raw)
+		if rr.ok != c.ok || c.ok && (rr.verdict != c.verdict || rr.short != c.short || rr.forNew != c.forNew) {
+			t.Errorf("%q: %+v", c.raw, rr)
+		}
+	}
+	// In a pair: an exception that lets ND in, not "unknown".
+	st := state()
+	st.Rules = append(st.Rules, &model.Rule{ID: 90, Chain: "input", Position: 0.5, Raw: "icmpv6 type { nd-router-solicit } ip6 hoplimit 255 accept"})
+	st.Sort()
+	e := find(Build(st), "wan", Firewall).Exceptions
+	found := false
+	for _, x := range e {
+		if x.RuleID == 90 {
+			found = x.Verdict == "allowed" && x.Short == "icmpv6" && strings.HasPrefix(x.What, "raw: icmpv6 type")
+		}
+	}
+	if !found {
+		t.Errorf("the raw ND rule: %+v", e)
+	}
+}

internal/model/addr.go

@@ -32,6 +32,34 @@ func (a Address) String() string {
 	return a.Prefix.String()
 }
 
+// Contains reports whether the entry holds an address.
+func (a Address) Contains(x netip.Addr) bool {
+	x = x.Unmap()
+	if a.First.IsValid() {
+		return x.Is4() == a.First.Is4() && !x.Less(a.First) && !a.End.Less(x)
+	}
+	return a.Prefix.Contains(x)
+}
+
+// Overlaps reports whether the entry and a prefix share an address.
+func (a Address) Overlaps(p netip.Prefix) bool {
+	if !a.First.IsValid() {
+		return a.Prefix.Overlaps(p)
+	}
+	if a.First.Is4() != p.Addr().Is4() {
+		return false
+	}
+	// The prefix's first and last addresses against the range.
+	first := p.Masked().Addr()
+	last := first
+	b := last.AsSlice()
+	for i := p.Bits(); i < len(b)*8; i++ {
+		b[i/8] |= 0x80 >> (i % 8)
+	}
+	last, _ = netip.AddrFromSlice(b)
+	return !last.Less(a.First) && !a.End.Less(first)
+}
+
 // ParseAddress reads an address, a prefix (host bits must be zero) or a
 // range, IPv4 or IPv6.
 func ParseAddress(s string) (Address, error) {

internal/model/auth.go

@@ -7,9 +7,10 @@ import (
 
 // Grant gives access to one resource (see internal/auth).
 type Grant struct {
-	Resource string `json:"resource"` // config, sets, log, audit, *
+	Resource string `json:"resource"` // config, sets, bans, log, audit, *
 	Access   string `json:"access"`   // read or write (write implies read)
-	// Sets limits a "sets" grant to these dynamic sets (none: all).
+	// Sets limits a "sets" grant to these dynamic sets, and a "bans"
+	// grant to these ban lists (none: all).
 	Sets []string `json:"sets,omitempty"`
 }
 

internal/model/bans.go

@@ -0,0 +1,53 @@
+package model
+
+import (
+	"net/netip"
+	"time"
+)
+
+// Ban is an address a ban list keeps out (docs/bans.md): of every port,
+// or of one protocol and port. Bans are kept apart from the
+// configuration, like a dynamic set's entries: not in the history, and
+// not on probation.
+type Ban struct {
+	ID     int64  `json:"id"`
+	ListID int64  `json:"list_id"`
+	List   string `json:"list"`
+	// Address is what is banned: an address, or an IPv6 prefix of the
+	// list's grouping ("2001:db8:1:2::/64").
+	Address  string `json:"address"`
+	Protocol string `json:"protocol,omitempty"` // tcp or udp; "" every port
+	Port     int    `json:"port,omitempty"`
+	// Expires is when the ban ends (nil: it doesn't); Timeout the
+	// seconds it has left then.
+	Expires *time.Time `json:"expires,omitempty"`
+	Timeout int        `json:"timeout,omitempty"`
+	Created time.Time  `json:"created"`
+	// Who banned it (the user or token's name), Source the program's own
+	// name for why ("guard/ssh", "fail2ban/sshd"), Reason free text (a
+	// log line).
+	Who    string `json:"who"`
+	Source string `json:"source,omitempty"`
+	Reason string `json:"reason,omitempty"`
+}
+
+// Prefix is the banned address as a prefix.
+func (b *Ban) Prefix() (netip.Prefix, error) {
+	if p, err := netip.ParsePrefix(b.Address); err == nil {
+		return p, nil
+	}
+	a, err := netip.ParseAddr(b.Address)
+	if err != nil {
+		return netip.Prefix{}, err
+	}
+	return netip.PrefixFrom(a, a.BitLen()), nil
+}
+
+// BanAddress writes a banned prefix: an address alone when it is one
+// address, else the prefix.
+func BanAddress(p netip.Prefix) string {
+	if p.IsSingleIP() {
+		return p.Addr().String()
+	}
+	return p.String()
+}

internal/model/changeset.go

@@ -54,6 +54,9 @@ type Op struct {
 	// Data is the object (create), the fields to change (update,
 	// settings) or where to move it (move).
 	Data json.RawMessage `json:"data,omitempty"`
+	// Place is where a create puts the object in its order (a move's
+	// data); none: at the end.
+	Place   json.RawMessage `json:"place,omitempty"`
 	Summary string          `json:"summary"`
 	AddedAt time.Time       `json:"added_at"`
 }

internal/model/objects.go

@@ -274,6 +274,45 @@ func (r *Rule) Groups() (v4, v6 int) {
 	return v4, v6
 }
 
+// What becomes of what a rule's quota used when the rule changes
+// (QuotaCarry).
+const (
+	QuotaKept      = ""          // counted as before: carried as it is
+	QuotaSummed    = "summed"    // per address to a total: the addresses' sum
+	QuotaRegrouped = "regrouped" // per address in wider groups: added up per group
+	QuotaRestarts  = "restarts"  // can't be carried: starts again
+)
+
+// QuotaCarry says what becomes of what old's quota used when the rule
+// becomes new (both the same rule, with quotas), and for QuotaRestarts
+// why. The amount, over and the resets don't matter: only how it counts.
+func QuotaCarry(old, new *Rule) (carry, why string) {
+	if old == nil || new == nil || old.Quota == nil || new.Quota == nil {
+		return QuotaKept, ""
+	}
+	was, is := old.Quota.PerAddress(), new.Quota.PerAddress()
+	words := map[string]string{"saddr": "per source", "daddr": "per destination", "": "in total"}
+	switch {
+	case was == "" && is == "":
+		return QuotaKept, ""
+	case is == "":
+		return QuotaSummed, ""
+	case was == "":
+		return QuotaRestarts, "a total can't be split by address, so each address starts at nothing"
+	case was != is:
+		return QuotaRestarts, "what was counted " + words[was] + " can't be counted " + words[is]
+	}
+	o4, o6 := old.Groups()
+	n4, n6 := new.Groups()
+	switch {
+	case o4 == n4 && o6 == n6:
+		return QuotaKept, ""
+	case n4 > o4 || n6 > o6:
+		return QuotaRestarts, "addresses are grouped more finely now, and a group's use can't be split"
+	}
+	return QuotaRegrouped, ""
+}
+
 // PortForward publishes a service on an inside host: incoming connections
 // to a public address and port are sent on (DNAT) to To[:ToPort], let
 // through the forward chain, and, with Hairpin, also work from inside.
@@ -353,3 +392,103 @@ func (r *Redirect) Label() string {
 	}
 	return fmt.Sprintf("redirect #%d", r.ID)
 }
+
+// Ban list scopes: what a list's bans keep the banned addresses from.
+const (
+	// BanScopeFirewall: the firewall itself (the input chain).
+	BanScopeFirewall = "firewall"
+	// BanScopeHosts: the list's hosts, behind the firewall (the forward
+	// chain, where port forwards' addresses are already translated).
+	BanScopeHosts = "hosts"
+	// BanScopeEverything: the firewall and everything behind it.
+	BanScopeEverything = "everything"
+)
+
+// BanScopes lists the scopes.
+var BanScopes = []string{BanScopeFirewall, BanScopeHosts, BanScopeEverything}
+
+// Ban lists' defaults.
+const (
+	DefaultBanTime    = 3600
+	DefaultBanMaxTime = 7 * 24 * 3600
+	DefaultBanEntries = 10000
+	DefaultBanGroupV6 = 64
+)
+
+// BanList is where a program such as Guard, or fail2ban, bans addresses
+// (docs/bans.md). The program names only the address, and perhaps a
+// protocol and ports; the list says what the address is kept from. The
+// bans themselves are kept apart from the configuration, like a dynamic
+// set's entries.
+type BanList struct {
+	ID    int64  `json:"id"`
+	Name  string `json:"name"`
+	Scope string `json:"scope"`
+	// Hosts are set names or addresses: what scope hosts protects.
+	Hosts []string `json:"hosts,omitempty"`
+	// Never are set names or addresses the list never bans.
+	Never []string `json:"never,omitempty"`
+	// DefaultTime is a ban's length in seconds when none is asked for
+	// (0: an hour); MaxTime the longest (0: a week; -1: no limit, so
+	// bans may be permanent).
+	DefaultTime int `json:"default_time,omitempty"`
+	MaxTime     int `json:"max_time,omitempty"`
+	// MaxEntries limits the bans the list holds (0: 10000).
+	MaxEntries int `json:"max_entries,omitempty"`
+	// GroupV6 is the IPv6 prefix length banned for an address (0: /64,
+	// what one customer usually gets); IPv4 addresses are banned alone.
+	GroupV6  int    `json:"group_v6,omitempty"`
+	Comment  string `json:"comment,omitempty"`
+	Disabled bool   `json:"disabled,omitempty"`
+}
+
+func (b *BanList) GetID() int64  { return b.ID }
+func (b *BanList) Label() string { return b.Name }
+
+// BanTime is the default ban length in seconds.
+func (b *BanList) BanTime() int {
+	if b.DefaultTime > 0 {
+		return b.DefaultTime
+	}
+	return DefaultBanTime
+}
+
+// BanMaxTime is the longest ban in seconds, 0 for no limit.
+func (b *BanList) BanMaxTime() int {
+	switch {
+	case b.MaxTime < 0:
+		return 0
+	case b.MaxTime > 0:
+		return b.MaxTime
+	}
+	return DefaultBanMaxTime
+}
+
+// Entries is the most bans the list holds.
+func (b *BanList) Entries() int {
+	if b.MaxEntries > 0 {
+		return b.MaxEntries
+	}
+	return DefaultBanEntries
+}
+
+// V6Group is the IPv6 prefix length an address is banned with.
+func (b *BanList) V6Group() int {
+	if b.GroupV6 > 0 {
+		return b.GroupV6
+	}
+	return DefaultBanGroupV6
+}
+
+// Covers reports whether the list's bans apply to a chain.
+func (b *BanList) Covers(chain string) bool {
+	switch b.Scope {
+	case BanScopeFirewall:
+		return chain == ChainInput
+	case BanScopeHosts:
+		return chain == ChainForward
+	case BanScopeEverything:
+		return chain == ChainInput || chain == ChainForward
+	}
+	return false
+}

internal/model/quota_test.go

@@ -0,0 +1,26 @@
+package model
+
+import "testing"
+
+func TestQuotaCarry(t *testing.T) {
+	rule := func(per string, v4, v6 int) *Rule {
+		return &Rule{GroupV4: v4, GroupV6: v6, Quota: &Quota{Bytes: 1, PerSource: per == "src", PerDestination: per == "dst"}}
+	}
+	for _, c := range []struct {
+		old, new *Rule
+		want     string
+	}{
+		{rule("", 0, 0), rule("", 0, 0), QuotaKept},
+		{rule("src", 0, 0), rule("src", 32, 64), QuotaKept}, // the defaults, written out
+		{rule("src", 0, 0), rule("", 0, 0), QuotaSummed},
+		{rule("", 0, 0), rule("src", 0, 0), QuotaRestarts},
+		{rule("src", 0, 0), rule("dst", 0, 0), QuotaRestarts},
+		{rule("src", 0, 0), rule("src", 24, 56), QuotaRegrouped},
+		{rule("src", 0, 0), rule("src", 24, 128), QuotaRestarts},
+		{nil, rule("src", 0, 0), QuotaKept},
+	} {
+		if got, _ := QuotaCarry(c.old, c.new); got != c.want {
+			t.Errorf("%+v -> %+v: %q, want %q", c.old, c.new, got, c.want)
+		}
+	}
+}

internal/model/state.go

@@ -20,6 +20,7 @@ type State struct {
 	Forwards  []*PortForward `json:"forwards"`
 	SNATs     []*SourceNAT   `json:"snats"`
 	Redirects []*Redirect    `json:"redirects"`
+	BanLists  []*BanList     `json:"ban_lists"`
 }
 
 // Object kinds, for changes and the history.
@@ -32,10 +33,11 @@ const (
 	KindForward  = "forward"
 	KindSNAT     = "snat"
 	KindRedirect = "redirect"
+	KindBanList  = "banlist"
 )
 
 // Kinds lists the object kinds (not settings) in display order.
-var Kinds = []string{KindZone, KindSet, KindService, KindRule, KindForward, KindSNAT, KindRedirect}
+var Kinds = []string{KindZone, KindSet, KindService, KindRule, KindForward, KindSNAT, KindRedirect, KindBanList}
 
 // Object is anything in the State with an ID.
 type Object interface {
@@ -73,6 +75,9 @@ func (st *State) Fill() {
 	if st.Redirects == nil {
 		st.Redirects = []*Redirect{}
 	}
+	if st.BanLists == nil {
+		st.BanLists = []*BanList{}
+	}
 }
 
 // Objects returns the objects of a kind.
@@ -107,6 +112,10 @@ func (st *State) Objects(kind string) []Object {
 		for _, o := range st.Redirects {
 			out = append(out, o)
 		}
+	case KindBanList:
+		for _, o := range st.BanLists {
+			out = append(out, o)
+		}
 	}
 	return out
 }
@@ -141,6 +150,8 @@ func (st *State) Put(kind string, id int64, data json.RawMessage) error {
 		return put(&st.SNATs, id, data)
 	case KindRedirect:
 		return put(&st.Redirects, id, data)
+	case KindBanList:
+		return put(&st.BanLists, id, data)
 	}
 	return &unknownKind{kind}
 }
@@ -187,6 +198,7 @@ func (st *State) Sort() {
 	sort.SliceStable(st.Zones, func(i, j int) bool { return st.Zones[i].Name < st.Zones[j].Name })
 	sort.SliceStable(st.Sets, func(i, j int) bool { return st.Sets[i].Name < st.Sets[j].Name })
 	sort.SliceStable(st.Services, func(i, j int) bool { return st.Services[i].Name < st.Services[j].Name })
+	sort.SliceStable(st.BanLists, func(i, j int) bool { return st.BanLists[i].Name < st.BanLists[j].Name })
 	sort.SliceStable(st.Rules, func(i, j int) bool {
 		a, b := st.Rules[i], st.Rules[j]
 		if a.Chain != b.Chain {
@@ -262,7 +274,7 @@ func (st *State) ByID(kind string, id int64) Object {
 }
 
 // Find finds an object of a kind by ID or name (zones, sets, services,
-// port forwards) or by ID alone (rules, source NAT, redirects).
+// port forwards, ban lists) or by ID alone (rules, source NAT, redirects).
 func (st *State) Find(kind, ref string) Object {
 	for _, o := range st.Objects(kind) {
 		if itoa(o.GetID()) == ref {
@@ -290,6 +302,20 @@ func (st *State) Find(kind, ref string) Object {
 				return f
 			}
 		}
+	case KindBanList:
+		if b := st.BanList(ref); b != nil {
+			return b
+		}
+	}
+	return nil
+}
+
+// BanList finds a ban list by name.
+func (st *State) BanList(name string) *BanList {
+	for _, b := range st.BanLists {
+		if strings.EqualFold(b.Name, name) {
+			return b
+		}
 	}
 	return nil
 }

internal/names/names.go

@@ -0,0 +1,152 @@
+// Package names finds the names of addresses by reverse lookups, for
+// showing them beside the addresses (connections, the log, quotas,
+// traffic per device). Names are cached and never stored: statistics stay
+// keyed by address.
+package names
+
+import (
+	"context"
+	"errors"
+	"net"
+	"net/netip"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Lookups running at once, and the most names kept.
+const (
+	parallel = 16
+	maxCache = 20000
+)
+
+// Resolver looks up and caches addresses' names.
+type Resolver struct {
+	lookup func(ctx context.Context, addr string) ([]string, error)
+	// ttl is how long a name (or its absence) is kept; a lookup that
+	// failed (a timeout, a server error) is tried again after retry.
+	ttl, retry time.Duration
+	timeout    time.Duration
+	now        func() time.Time
+
+	mu      sync.Mutex
+	cache   map[netip.Addr]entry
+	pending map[netip.Addr]chan struct{}
+	sem     chan struct{}
+}
+
+type entry struct {
+	name    string
+	expires time.Time
+}
+
+// New makes a resolver asking server ("10.10.0.1:53"; "" for the
+// system's, /etc/hosts then /etc/resolv.conf), keeping names for ttl.
+func New(server string, ttl time.Duration) *Resolver {
+	res := &net.Resolver{PreferGo: true}
+	if server != "" {
+		res.Dial = func(ctx context.Context, network, _ string) (net.Conn, error) {
+			var d net.Dialer
+			return d.DialContext(ctx, network, server)
+		}
+	}
+	return NewFunc(res.LookupAddr, ttl)
+}
+
+// NewFunc makes a resolver with its own lookup (tests).
+func NewFunc(lookup func(ctx context.Context, addr string) ([]string, error), ttl time.Duration) *Resolver {
+	return &Resolver{lookup: lookup, ttl: ttl, retry: time.Minute, timeout: 2 * time.Second, now: time.Now,
+		cache: map[netip.Addr]entry{}, pending: map[netip.Addr]chan struct{}{}, sem: make(chan struct{}, parallel)}
+}
+
+// Names returns the names of addrs: those cached, and those looked up
+// within wait. Lookups still running then carry on and are cached for
+// the next call. An address without a name is left out.
+func (r *Resolver) Names(ctx context.Context, addrs []netip.Addr, wait time.Duration) map[netip.Addr]string {
+	out := map[netip.Addr]string{}
+	var waiting []netip.Addr
+	var done []chan struct{}
+	now := r.now()
+	r.mu.Lock()
+	for _, a := range addrs {
+		if e, ok := r.cache[a]; ok && now.Before(e.expires) {
+			if e.name != "" {
+				out[a] = e.name
+			}
+			continue
+		}
+		ch, ok := r.pending[a]
+		if !ok {
+			ch = make(chan struct{})
+			r.pending[a] = ch
+			go r.resolve(a, ch)
+		}
+		waiting = append(waiting, a)
+		done = append(done, ch)
+	}
+	r.mu.Unlock()
+	if len(waiting) == 0 {
+		return out
+	}
+	t := time.NewTimer(wait)
+	defer t.Stop()
+	for _, ch := range done {
+		select {
+		case <-ch:
+		case <-t.C:
+			return r.collect(out, waiting)
+		case <-ctx.Done():
+			return r.collect(out, waiting)
+		}
+	}
+	return r.collect(out, waiting)
+}
+
+// collect adds what the lookups found for addrs.
+func (r *Resolver) collect(out map[netip.Addr]string, addrs []netip.Addr) map[netip.Addr]string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	for _, a := range addrs {
+		if e, ok := r.cache[a]; ok && e.name != "" {
+			out[a] = e.name
+		}
+	}
+	return out
+}
+
+func (r *Resolver) resolve(a netip.Addr, done chan struct{}) {
+	r.sem <- struct{}{}
+	ctx, cancel := context.WithTimeout(context.Background(), r.timeout)
+	ns, err := r.lookup(ctx, a.String())
+	cancel()
+	<-r.sem
+	e := entry{expires: r.now().Add(r.ttl)}
+	var dnsErr *net.DNSError
+	switch {
+	case err == nil && len(ns) > 0:
+		e.name = strings.TrimSuffix(ns[0], ".")
+	case err != nil && !(errors.As(err, &dnsErr) && dnsErr.IsNotFound):
+		e.expires = r.now().Add(r.retry)
+	}
+	r.mu.Lock()
+	if len(r.cache) >= maxCache {
+		r.prune()
+	}
+	r.cache[a] = e
+	delete(r.pending, a)
+	r.mu.Unlock()
+	close(done)
+}
+
+// prune drops expired names, or all of them when none has expired.
+func (r *Resolver) prune() {
+	now := r.now()
+	for a, e := range r.cache {
+		if !now.Before(e.expires) {
+			delete(r.cache, a)
+		}
+	}
+	if len(r.cache) >= maxCache {
+		clear(r.cache)
+	}
+}

internal/names/names_test.go

@@ -0,0 +1,57 @@
+package names
+
+import (
+	"context"
+	"errors"
+	"net"
+	"net/netip"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+func TestNames(t *testing.T) {
+	var calls atomic.Int32
+	slow := make(chan struct{})
+	r := NewFunc(func(ctx context.Context, addr string) ([]string, error) {
+		calls.Add(1)
+		switch addr {
+		case "10.0.0.1":
+			return []string{"nas.example.net."}, nil
+		case "10.0.0.2":
+			return nil, &net.DNSError{Err: "no such host", Name: addr, IsNotFound: true}
+		case "10.0.0.3":
+			return nil, errors.New("server failure")
+		case "10.0.0.4":
+			<-slow
+			return []string{"slow.example.net."}, nil
+		}
+		return nil, nil
+	}, time.Hour)
+	a := netip.MustParseAddr
+	all := []netip.Addr{a("10.0.0.1"), a("10.0.0.2"), a("10.0.0.3"), a("10.0.0.4")}
+
+	got := r.Names(context.Background(), all, 200*time.Millisecond)
+	if len(got) != 1 || got[a("10.0.0.1")] != "nas.example.net" {
+		t.Fatalf("first call: %v", got)
+	}
+	// The slow lookup finishes later and is there next time; the others
+	// come from the cache, the failure too until it is retried.
+	close(slow)
+	time.Sleep(50 * time.Millisecond)
+	got = r.Names(context.Background(), all, 200*time.Millisecond)
+	if len(got) != 2 || got[a("10.0.0.4")] != "slow.example.net" {
+		t.Fatalf("second call: %v", got)
+	}
+	if n := calls.Load(); n != 4 {
+		t.Fatalf("%d lookups, want 4 (cached)", n)
+	}
+
+	// A failure is tried again after retry, a missing name after ttl.
+	now := time.Now().Add(2 * time.Minute)
+	r.now = func() time.Time { return now }
+	r.Names(context.Background(), all, 200*time.Millisecond)
+	if n := calls.Load(); n != 5 {
+		t.Fatalf("%d lookups, want 5 (the failure again)", n)
+	}
+}

internal/nft/nft.go

@@ -15,6 +15,7 @@ import (
 	"errors"
 	"fmt"
 	"net"
+	"net/netip"
 	"os"
 	"os/exec"
 	"path/filepath"
@@ -189,6 +190,30 @@ func (r *Runner) DeleteElement(ctx context.Context, set, addr string) error {
 	return err
 }
 
+// AddBan puts a ban in force: added, or if it is there already, given
+// its new time (nftables keeps an element's old timeout when it is added
+// again, so it is added, deleted and added in one transaction).
+func (r *Runner) AddBan(ctx context.Context, b render.Ban) error {
+	set := render.BanSet(b.List, b.Addr.Addr().Is6(), b.Proto != "")
+	e := render.BanElement(b)
+	t := ""
+	if b.Timeout > 0 {
+		t = fmt.Sprintf(" timeout %ds", b.Timeout)
+	}
+	script := fmt.Sprintf("add element inet %[1]s %[2]s { %[3]s }\ndelete element inet %[1]s %[2]s { %[3]s }\nadd element inet %[1]s %[2]s { %[3]s%[4]s }\n", render.Table, set, e, t)
+	_, err := r.run(ctx, script, "-f", "/dev/stdin")
+	return err
+}
+
+// DeleteBan lifts a ban. One already gone (it ran out) isn't an error.
+func (r *Runner) DeleteBan(ctx context.Context, b render.Ban) error {
+	set := render.BanSet(b.List, b.Addr.Addr().Is6(), b.Proto != "")
+	e := render.BanElement(b)
+	script := fmt.Sprintf("add element inet %[1]s %[2]s { %[3]s }\ndelete element inet %[1]s %[2]s { %[3]s }\n", render.Table, set, e)
+	_, err := r.run(ctx, script, "-f", "/dev/stdin")
+	return err
+}
+
 // ---- reading JSON
 
 // Table is what Housecarl reads back from its table: counters by mark
@@ -366,7 +391,7 @@ func ParseTable(data []byte) (*Table, error) {
 				continue
 			}
 			name := strings.TrimSuffix(strings.TrimSuffix(s.Name, "_v4"), "_v6")
-			if name == s.Name || strings.HasPrefix(s.Name, "meter_") || strings.HasPrefix(s.Name, "conn_") {
+			if name == s.Name || strings.HasPrefix(s.Name, "meter_") || strings.HasPrefix(s.Name, "conn_") || render.IsBanSet(s.Name) {
 				continue
 			}
 			for _, e := range s.Elem {
@@ -409,6 +434,10 @@ type Firewall struct {
 	// BootFile is where the confirmed ruleset is saved for the boot unit
 	// (housecarl-ruleset.service; "" for none).
 	BootFile string
+	// Shadow is shadow mode: the ruleset changes nothing, beside another
+	// firewall (render.Options; docs/shadow.md). ShadowBefore runs its
+	// filter chains before that one's.
+	Shadow, ShadowBefore bool
 
 	// Replacing the table resets its counters: base keeps the totals of
 	// the tables replaced since housecarld started.
@@ -418,6 +447,14 @@ type Firewall struct {
 	// restore is the quota usage saved before housecarld (or the machine)
 	// stopped, put back by the next apply.
 	restore *QuotaUsage
+	// adjust are quota changes the next apply makes (AdjustQuota).
+	adjust []QuotaAdjust
+	// quotaRules are the rules with a quota in the ruleset in force, by
+	// ID: what their usage was counted by (nil before the first apply).
+	quotaRules map[int64]*model.Rule
+	// Bans returns the active bans, which every apply puts back (the
+	// database keeps them; nil: none).
+	Bans func() []render.Ban
 	// flowDevs are the flowtable's devices in the ruleset in force, by
 	// name, with their interface index then (0: left out, missing).
 	flowDevs map[string]int
@@ -458,6 +495,119 @@ func (f *Firewall) RestoreQuotas(u *QuotaUsage) {
 	f.restore = u
 }
 
+// QuotaAdjust takes Give bytes off what a quota used (never below zero):
+// the named quota Name, or the entries of the per-address sets Name_v4
+// and Name_v6, Addr's alone if given (the set's key, as listed).
+type QuotaAdjust struct {
+	Name string
+	Addr string
+	Give int64
+}
+
+// AdjustQuota has the next apply change what a quota used.
+func (f *Firewall) AdjustQuota(a QuotaAdjust) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.adjust = append(f.adjust, a)
+}
+
+// apply takes what a's say off u. An entry with nothing left used is
+// dropped, so its quota starts again with its next packet.
+func (u *QuotaUsage) apply(adj []QuotaAdjust) {
+	less := func(used, give int64) int64 {
+		if give >= used {
+			return 0
+		}
+		return used - give
+	}
+	for _, a := range adj {
+		if used, ok := u.Quotas[a.Name]; ok {
+			u.Quotas[a.Name] = less(used, a.Give)
+		}
+		for _, set := range []string{a.Name + "_v4", a.Name + "_v6"} {
+			var kept []render.Element
+			for _, e := range u.QuotaSets[set] {
+				if a.Addr == "" || e.Addr == a.Addr {
+					e.Used = less(e.Used, a.Give)
+				}
+				if e.Used > 0 {
+					kept = append(kept, e)
+				}
+			}
+			if _, ok := u.QuotaSets[set]; ok {
+				u.QuotaSets[set] = kept
+			}
+		}
+	}
+}
+
+// carry keeps what quotas used when their rules now count differently
+// (model.QuotaCarry): per address to a total, the sum; to wider groups,
+// the groups' sums; what can't be carried starts again. was are the
+// rules in force before (nil: nothing to compare with).
+func (u *QuotaUsage) carry(was map[int64]*model.Rule, st *model.State) {
+	for _, r := range st.Rules {
+		old := was[r.ID]
+		carry, _ := model.QuotaCarry(old, r)
+		if carry == model.QuotaKept {
+			continue
+		}
+		name := render.QuotaName(r.ID)
+		sets := []string{name + "_v4", name + "_v6"}
+		switch carry {
+		case model.QuotaSummed:
+			var sum int64
+			for _, set := range sets {
+				for _, e := range u.QuotaSets[set] {
+					sum += e.Used
+				}
+			}
+			if u.Quotas == nil {
+				u.Quotas = map[string]int64{}
+			}
+			u.Quotas[name] = sum
+		case model.QuotaRegrouped:
+			v4, v6 := r.Groups()
+			for _, set := range sets {
+				byKey := map[string]int64{}
+				var order []string
+				for _, e := range u.QuotaSets[set] {
+					a, err := netip.ParseAddr(e.Addr)
+					if err != nil {
+						continue
+					}
+					bits := v4
+					if a.Is6() {
+						bits = v6
+					}
+					p, _ := a.Prefix(bits)
+					k := p.Addr().String()
+					if _, ok := byKey[k]; !ok {
+						order = append(order, k)
+					}
+					byKey[k] += e.Used
+				}
+				if len(order) == 0 {
+					continue
+				}
+				var es []render.Element
+				for _, k := range order {
+					es = append(es, render.Element{Addr: k, Used: byKey[k]})
+				}
+				u.QuotaSets[set] = es
+			}
+			continue
+		}
+		// What was counted the other way goes: a total, or the entries.
+		if carry != model.QuotaSummed {
+			delete(u.Quotas, name)
+		}
+		for _, set := range sets {
+			delete(u.QuotaSets, set)
+		}
+	}
+}
+
 // ResetQuota starts a rule's quota again: the named quota, or the
 // per-address quota sets (both families).
 func (f *Firewall) ResetQuota(ctx context.Context, ruleID int64, perAddress bool) error {
@@ -478,7 +628,7 @@ func (f *Firewall) ResetQuota(ctx context.Context, ruleID int64, perAddress bool
 }
 
 func (f *Firewall) options() render.Options {
-	var o render.Options
+	o := render.Options{Shadow: f.Shadow, ShadowBefore: f.ShadowBefore}
 	if f.Interfaces != nil {
 		o.Interfaces = f.Interfaces()
 	}
@@ -488,9 +638,26 @@ func (f *Firewall) options() render.Options {
 // Render builds the ruleset for st as Apply would.
 func (f *Firewall) Render(st *model.State) *render.Ruleset { return render.Render(st, f.options()) }
 
-// Check implements engine.Firewall.
+// RenderEnforcing builds the ruleset for st as Apply would outside shadow
+// mode.
+func (f *Firewall) RenderEnforcing(st *model.State) *render.Ruleset {
+	o := f.options()
+	o.Shadow, o.ShadowBefore = false, false
+	return render.Render(st, o)
+}
+
+// Check implements engine.Firewall. In shadow mode it checks the
+// enforcing ruleset too, so that leaving shadow mode can't fail.
 func (f *Firewall) Check(ctx context.Context, st *model.State) error {
-	return f.Runner.Check(ctx, render.Render(st, f.options()))
+	if err := f.Runner.Check(ctx, render.Render(st, f.options())); err != nil {
+		return err
+	}
+	if f.Shadow {
+		if err := f.Runner.Check(ctx, f.RenderEnforcing(st)); err != nil {
+			return fmt.Errorf("the ruleset for when shadow mode is off: %w", err)
+		}
+	}
+	return nil
 }
 
 // Apply implements engine.Firewall.
@@ -510,7 +677,13 @@ func (f *Firewall) Apply(ctx context.Context, st *model.State) error {
 		usage = old.QuotaUsage
 	}
 	usage.merge(f.restore)
+	usage.carry(f.quotaRules, st)
+	usage.apply(f.adjust)
+	f.adjust = nil
 	opt.Quotas, opt.QuotaSets = usage.Quotas, usage.QuotaSets
+	if f.Bans != nil {
+		opt.Bans = f.Bans()
+	}
 	if old != nil {
 		opt.DeviceSets = old.DeviceSets
 	}
@@ -518,6 +691,14 @@ func (f *Firewall) Apply(ctx context.Context, st *model.State) error {
 		return err
 	}
 	f.restore = nil
+	f.quotaRules = map[int64]*model.Rule{}
+	for _, r := range st.Rules {
+		if r.Quota != nil {
+			c, q := *r, *r.Quota // only the quota and the grouping are read
+			c.Quota = &q
+			f.quotaRules[r.ID] = &c
+		}
+	}
 	f.flowDevs = map[string]int{}
 	for _, d := range st.Settings.FlowtableDevices {
 		if opt.Interfaces == nil || opt.Interfaces[d] {
@@ -535,14 +716,54 @@ func (f *Firewall) Apply(ctx context.Context, st *model.State) error {
 			f.base[k] = b
 		}
 	}
+	pruneCounters(f.base, st)
 	return nil
 }
 
+// pruneCounters forgets the totals of objects st no longer has (deleted,
+// or replaced by an import with new IDs), so they aren't listed beside
+// the current ones. A disabled object keeps its totals for when it comes
+// back; built-ins and policies are always kept.
+func pruneCounters(base map[string]Counter, st *model.State) {
+	ids := map[string]bool{}
+	for _, r := range st.Rules {
+		ids[fmt.Sprintf("r%d", r.ID)] = true
+	}
+	for _, x := range st.Forwards {
+		ids[fmt.Sprintf("f%d", x.ID)] = true
+	}
+	for _, x := range st.SNATs {
+		ids[fmt.Sprintf("s%d", x.ID)] = true
+	}
+	for _, x := range st.Redirects {
+		ids[fmt.Sprintf("d%d", x.ID)] = true
+	}
+	for _, x := range st.BanLists {
+		ids[render.BanMark(x.ID)] = true
+	}
+	for mark := range base {
+		if strings.HasPrefix(mark, "ban") && strings.Trim(mark[3:], "0123456789") == "" {
+			if !ids[mark] {
+				delete(base, mark)
+			}
+			continue
+		}
+		obj, _, _ := strings.Cut(mark, "-") // f3-hairpin, f3-6 are f3's
+		if len(obj) < 2 || !strings.ContainsRune("rfsd", rune(obj[0])) || strings.Trim(obj[1:], "0123456789") != "" {
+			continue // b-established, p-input
+		}
+		if !ids[obj] {
+			delete(base, mark)
+		}
+	}
+}
+
 // SaveBoot writes the ruleset of st for the boot unit, and the fallback
 // it loads if that fails (FallbackFile), atomically. The boot ruleset has
 // no flowtable: it loads before the network comes up, when its devices
 // (ppp0, wg0) don't exist yet and nftables would refuse the whole file;
-// housecarld adds it once they do.
+// housecarld adds it once they do. In shadow mode both are shadow mode's:
+// the fallback then filters nothing.
 func (f *Firewall) SaveBoot(st *model.State) error {
 	if f.BootFile == "" {
 		return nil
@@ -550,10 +771,14 @@ func (f *Firewall) SaveBoot(st *model.State) error {
 	if err := os.MkdirAll(filepath.Dir(f.BootFile), 0o750); err != nil {
 		return err
 	}
-	if err := writeAtomic(FallbackFile(f.BootFile), render.Fallback(st)); err != nil {
+	fallback := render.Fallback(st)
+	if f.Shadow {
+		fallback = render.ShadowFallback()
+	}
+	if err := writeAtomic(FallbackFile(f.BootFile), fallback); err != nil {
 		return err
 	}
-	rs := render.Render(st, render.Options{Interfaces: map[string]bool{}})
+	rs := render.Render(st, render.Options{Interfaces: map[string]bool{}, Shadow: f.Shadow, ShadowBefore: f.ShadowBefore})
 	return writeAtomic(f.BootFile, rs.Text)
 }
 
@@ -605,6 +830,14 @@ func (f *Firewall) AddElement(ctx context.Context, set, addr string, timeout int
 	return f.Runner.AddElement(ctx, set, addr, timeout)
 }
 
+// AddBan puts a ban in force (Runner.AddBan).
+func (f *Firewall) AddBan(ctx context.Context, b render.Ban) error { return f.Runner.AddBan(ctx, b) }
+
+// DeleteBan lifts a ban (Runner.DeleteBan).
+func (f *Firewall) DeleteBan(ctx context.Context, b render.Ban) error {
+	return f.Runner.DeleteBan(ctx, b)
+}
+
 // DeleteElement removes an entry from a dynamic set.
 func (f *Firewall) DeleteElement(ctx context.Context, set, addr string) error {
 	return f.Runner.DeleteElement(ctx, set, addr)

internal/nft/nft_test.go

@@ -3,9 +3,12 @@ package nft
 import (
 	"context"
 	"fmt"
+	"net/netip"
 	"os"
 	"os/exec"
 	"path/filepath"
+	"reflect"
+	"sort"
 	"strings"
 	"testing"
 
@@ -185,6 +188,31 @@ func TestQuotasInContainer(t *testing.T) {
 	if t2, d2 := used(); t2 < total || d2 < dev {
 		t.Errorf("after a reload: total %d (was %d), per destination %d (was %d)", t2, total, d2, dev)
 	}
+	// Some given back, and one address started again, by the next apply.
+	total, dev = used()
+	fw.AdjustQuota(QuotaAdjust{Name: "quota_r1", Give: 1000})
+	fw.AdjustQuota(QuotaAdjust{Name: "quota_r2", Addr: "127.0.0.3", Give: 1 << 62})
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatal(err)
+	}
+	if t2, d2 := used(); t2 > total-1000 || t2 < total-2000 || d2 != 0 {
+		t.Errorf("after giving back: total %d (was %d, gave 1000), per destination %d (was %d, reset)", t2, total, d2, dev)
+	}
+	// Counted in total now: what the addresses used carries over.
+	ping("127.0.0.3")
+	_, dev = used()
+	st.Rules[1].Quota.PerDestination = false
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatal(err)
+	}
+	tb, _ := fw.Table(ctx)
+	if tb.Quotas["quota_r2"] < dev || dev == 0 {
+		t.Errorf("per destination to a total: %d, was %d", tb.Quotas["quota_r2"], dev)
+	}
+	st.Rules[1].Quota.PerDestination = true
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatal(err)
+	}
 	// Reset by hand.
 	if err := fw.ResetQuota(ctx, 1, false); err != nil {
 		t.Fatal(err)
@@ -199,7 +227,7 @@ func TestQuotasInContainer(t *testing.T) {
 	// saved usage comes back.
 	ping("127.0.0.2")
 	ping("127.0.0.3")
-	tb, _ := fw.Table(ctx)
+	tb, _ = fw.Table(ctx)
 	saved := tb.QuotaUsage
 	exec.Command("docker", "exec", name, "nft", "delete", "table", "inet", "housecarl").Run()
 	fw.RestoreQuotas(&saved)
@@ -211,6 +239,62 @@ func TestQuotasInContainer(t *testing.T) {
 	}
 }
 
+// TestQuotaAdjust: giving back takes off what was used, never below
+// zero; an entry with nothing left goes.
+func TestQuotaAdjust(t *testing.T) {
+	u := QuotaUsage{Quotas: map[string]int64{"quota_r1": 5000, "quota_r9": 10},
+		QuotaSets: map[string][]render.Element{"quota_r2_v4": {{Addr: "10.0.0.1", Used: 3000}, {Addr: "10.0.0.2", Used: 4000}},
+			"quota_r2_v6": {{Addr: "2001:db8::", Used: 100}}}}
+	u.apply([]QuotaAdjust{{Name: "quota_r1", Give: 2000}, {Name: "quota_r9", Give: 2000}, {Name: "quota_r2", Addr: "10.0.0.2", Give: 1500},
+		{Name: "quota_r2", Give: 100}})
+	if u.Quotas["quota_r1"] != 3000 || u.Quotas["quota_r9"] != 0 {
+		t.Errorf("named: %v", u.Quotas)
+	}
+	want := map[string][]render.Element{"quota_r2_v4": {{Addr: "10.0.0.1", Used: 2900}, {Addr: "10.0.0.2", Used: 2400}}, "quota_r2_v6": nil}
+	if !reflect.DeepEqual(u.QuotaSets, want) {
+		t.Errorf("sets: %+v", u.QuotaSets)
+	}
+}
+
+// TestQuotaCarry: usage follows a rule counting its quota differently.
+func TestQuotaCarry(t *testing.T) {
+	perSource := &model.Rule{ID: 2, Quota: &model.Quota{Bytes: 1 << 30, PerSource: true}}
+	usage := func() QuotaUsage {
+		return QuotaUsage{Quotas: map[string]int64{}, QuotaSets: map[string][]render.Element{
+			"quota_r2_v4": {{Addr: "10.0.0.1", Used: 100}, {Addr: "10.0.0.2", Used: 50}},
+			"quota_r2_v6": {{Addr: "2001:db8:0:1::", Used: 7}, {Addr: "2001:db8:0:2::", Used: 3}}}}
+	}
+	was := map[int64]*model.Rule{2: perSource}
+	with := func(f func(r *model.Rule)) *model.State {
+		r := *perSource
+		q := *perSource.Quota
+		r.Quota = &q
+		f(&r)
+		return &model.State{Rules: []*model.Rule{&r}}
+	}
+	u := usage()
+	u.carry(was, with(func(r *model.Rule) { r.Quota.PerSource = false }))
+	if u.Quotas["quota_r2"] != 160 || len(u.QuotaSets["quota_r2_v4"]) != 0 {
+		t.Errorf("to a total: %+v", u)
+	}
+	u = usage()
+	u.carry(was, with(func(r *model.Rule) { r.GroupV4, r.GroupV6 = 24, 48 }))
+	want := map[string][]render.Element{"quota_r2_v4": {{Addr: "10.0.0.0", Used: 150}}, "quota_r2_v6": {{Addr: "2001:db8::", Used: 10}}}
+	if !reflect.DeepEqual(u.QuotaSets, want) {
+		t.Errorf("wider groups: %+v", u.QuotaSets)
+	}
+	u = usage()
+	u.carry(was, with(func(r *model.Rule) { r.Quota.PerSource, r.Quota.PerDestination = false, true }))
+	if len(u.QuotaSets) != 0 {
+		t.Errorf("per destination now: %+v", u.QuotaSets)
+	}
+	u = usage()
+	u.carry(nil, with(func(r *model.Rule) { r.Quota.PerSource = false }))
+	if len(u.QuotaSets["quota_r2_v4"]) != 2 || u.Quotas["quota_r2"] != 0 {
+		t.Errorf("nothing to compare with (the first apply): %+v", u)
+	}
+}
+
 // TestDeviceCountsInContainer: the per-device counters are read back and
 // survive a reload.
 func TestDeviceCountsInContainer(t *testing.T) {
@@ -365,3 +449,161 @@ func TestBootInContainer(t *testing.T) {
 		t.Errorf("the fallback stayed after an apply: %+v %v", tb, err)
 	}
 }
+
+// TestShadowInContainer: in shadow mode a drop counts but doesn't drop,
+// the ruleset for leaving shadow mode is checked too, and the boot files
+// are shadow mode's.
+func TestShadowInContainer(t *testing.T) {
+	image := os.Getenv("HOUSECARL_NFT_IMAGE")
+	if image == "" {
+		t.Skip("HOUSECARL_NFT_IMAGE not set")
+	}
+	name := "housecarl-nft-shadow-test"
+	exec.Command("docker", "rm", "-f", name).Run()
+	if out, err := exec.Command("docker", "run", "-d", "--name", name, "--cap-add", "NET_ADMIN", image, "sleep", "300").CombinedOutput(); err != nil {
+		t.Fatalf("starting the container: %v %s", err, out)
+	}
+	t.Cleanup(func() { exec.Command("docker", "rm", "-f", name).Run() })
+	dir := t.TempDir()
+	fw := &Firewall{Runner: &Runner{Command: []string{"docker", "exec", "-i", name, "nft"}}, Shadow: true, BootFile: filepath.Join(dir, "ruleset.nft")}
+	ctx := context.Background()
+	st := model.NewState()
+	st.Settings.OutputPolicy = model.PolicyDrop
+	st.Settings.InputPolicy = model.PolicyAccept
+	st.Settings.Builtins.Loopback = false // it would accept the ping first
+	st.Rules = []*model.Rule{{ID: 3, Chain: model.ChainOutput, Position: 1, Action: model.ActionReject, Match: model.Match{Destinations: []string{"127.0.0.2"}}}}
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatalf("apply: %v", err)
+	}
+	ping := func() error {
+		return exec.Command("docker", "exec", name, "ping", "-c", "2", "-W", "1", "127.0.0.2").Run()
+	}
+	if err := ping(); err != nil {
+		t.Errorf("shadow mode blocked a ping: %v", err)
+	}
+	tb, err := fw.Table(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if tb.Counters["r3"].Packets == 0 {
+		t.Errorf("what would be rejected isn't counted: %+v", tb.Counters)
+	}
+	// A rule shadow mode leaves out, which nftables refuses when enforcing.
+	broken := st.Clone()
+	broken.Rules = append(broken.Rules, &model.Rule{ID: 4, Chain: model.ChainInput, Position: 1, Raw: "tcp dport 22 jump nowhere", Comment: "jump"})
+	if err := fw.Check(ctx, broken); err == nil || !strings.Contains(err.Error(), "shadow mode is off") || !strings.Contains(err.Error(), "input #4 (jump)") {
+		t.Errorf("the ruleset for leaving shadow mode wasn't checked: %v", err)
+	}
+	// Enforcing, the same rule rejects.
+	enforcing := &Firewall{Runner: fw.Runner}
+	if err := enforcing.Apply(ctx, st); err != nil {
+		t.Fatal(err)
+	}
+	if ping() == nil {
+		t.Error("enforcing didn't reject")
+	}
+	// The boot files: both load, and filter nothing.
+	if err := fw.SaveBoot(st); err != nil {
+		t.Fatal(err)
+	}
+	for _, f := range []string{"ruleset.nft", "ruleset-fallback.nft"} {
+		b, _ := os.ReadFile(filepath.Join(dir, f))
+		c := exec.Command("docker", "exec", "-i", name, "nft", "-f", "/dev/stdin")
+		c.Stdin = strings.NewReader(string(b))
+		if out, err := c.CombinedOutput(); err != nil {
+			t.Fatalf("%s doesn't load: %v %s", f, err, out)
+		}
+		if err := ping(); err != nil {
+			t.Errorf("%s blocked a ping: %v", f, err)
+		}
+	}
+	if tb, err := fw.Table(ctx); err != nil || !tb.Fallback {
+		t.Errorf("the shadow fallback isn't recognised: %+v %v", tb, err)
+	}
+}
+
+// TestPruneCounters: the totals of objects gone (an import with --replace
+// gives every object a new ID) are forgotten; built-ins, policies and
+// disabled objects keep theirs.
+func TestPruneCounters(t *testing.T) {
+	base := map[string]Counter{}
+	for _, m := range []string{"r1", "r2", "r3", "f4", "f4-hairpin", "f4-6", "f5-hairpin", "s6", "d7", "d8", "b-established", "p-input", "fallback-input"} {
+		base[m] = Counter{Packets: 1}
+	}
+	st := model.NewState()
+	st.Rules = []*model.Rule{{ID: 1}, {ID: 3, Disabled: true}}
+	st.Forwards = []*model.PortForward{{ID: 4}}
+	st.Redirects = []*model.Redirect{{ID: 8}}
+	pruneCounters(base, st)
+	var got []string
+	for m := range base {
+		got = append(got, m)
+	}
+	sort.Strings(got)
+	if strings.Join(got, " ") != "b-established d8 f4 f4-6 f4-hairpin fallback-input p-input r1 r3" {
+		t.Errorf("kept %v", got)
+	}
+}
+
+// TestBansInContainer: a ban goes in, a second one renews its time, a
+// reload keeps the bans the database has, and lifting one already gone
+// isn't an error.
+func TestBansInContainer(t *testing.T) {
+	image := os.Getenv("HOUSECARL_NFT_IMAGE")
+	if image == "" {
+		t.Skip("HOUSECARL_NFT_IMAGE not set")
+	}
+	name := "housecarl-nft-bans"
+	exec.Command("docker", "rm", "-f", name).Run()
+	if out, err := exec.Command("docker", "run", "-d", "--name", name, "--cap-add", "NET_ADMIN", image, "sleep", "300").CombinedOutput(); err != nil {
+		t.Fatalf("starting the container: %v %s", err, out)
+	}
+	t.Cleanup(func() { exec.Command("docker", "rm", "-f", name).Run() })
+	var active []render.Ban
+	fw := &Firewall{Runner: &Runner{Command: []string{"docker", "exec", "-i", name, "nft"}}, Bans: func() []render.Ban { return active }}
+	ctx := context.Background()
+	st := model.NewState()
+	st.Zones = []*model.Zone{{ID: 1, Name: "wan", Interfaces: []string{"eth0"}, External: true}}
+	st.BanLists = []*model.BanList{{ID: 2, Name: "router", Scope: model.BanScopeFirewall}}
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatalf("apply: %v", err)
+	}
+	list := func(set string) string {
+		out, _ := exec.Command("docker", "exec", name, "nft", "list", "set", "inet", "housecarl", set).CombinedOutput()
+		return string(out)
+	}
+	b := render.Ban{List: 2, Addr: netip.MustParsePrefix("203.0.113.9/32"), Timeout: 60}
+	if err := fw.AddBan(ctx, b); err != nil {
+		t.Fatal(err)
+	}
+	b.Timeout = 7200
+	if err := fw.AddBan(ctx, b); err != nil {
+		t.Fatalf("renewing: %v", err)
+	}
+	if got := list("ban_2_v4"); !strings.Contains(got, "203.0.113.9 timeout 2h") {
+		t.Errorf("renewed ban: %s", got)
+	}
+	svc := render.Ban{List: 2, Addr: netip.MustParsePrefix("2001:db8:1:2::/64"), Proto: "tcp", Port: 22}
+	if err := fw.AddBan(ctx, svc); err != nil {
+		t.Fatal(err)
+	}
+	active = []render.Ban{svc}
+	if err := fw.Apply(ctx, st); err != nil {
+		t.Fatalf("reapply: %v", err)
+	}
+	if got := list("ban_2_v6_svc"); !strings.Contains(got, "2001:db8:1:2:: . 6 . 22") {
+		t.Errorf("after a reload, the database's ban: %s", got)
+	}
+	if got := list("ban_2_v4"); strings.Contains(got, "203.0.113.9") {
+		t.Errorf("after a reload, a ban the database doesn't have: %s", got)
+	}
+	if err := fw.DeleteBan(ctx, svc); err != nil {
+		t.Fatal(err)
+	}
+	if err := fw.DeleteBan(ctx, svc); err != nil {
+		t.Errorf("lifting a ban already gone: %v", err)
+	}
+	if got := list("ban_2_v6_svc"); strings.Contains(got, "2001:db8") {
+		t.Errorf("lifted ban: %s", got)
+	}
+}

internal/render/bans.go

@@ -0,0 +1,139 @@
+package render
+
+import (
+	"fmt"
+	"net/netip"
+	"strings"
+
+	"housecarl/internal/model"
+)
+
+// Ban is an active ban, for rendering (Options.Bans): its list, the
+// address banned (an IPv4 address, or an IPv6 prefix of the list's
+// grouping), and for a ban of one port its protocol and port. Timeout is
+// the seconds it has left (0: it doesn't end).
+type Ban struct {
+	List    int64
+	Addr    netip.Prefix
+	Proto   string // "tcp", "udp", or "" for every port
+	Port    int
+	Timeout int
+}
+
+// BanSet is the name of a ban list's set for a family, and with svc
+// true its set of bans of one port.
+func BanSet(list int64, v6, svc bool) string {
+	name := fmt.Sprintf("ban_%d_v4", list)
+	if v6 {
+		name = fmt.Sprintf("ban_%d_v6", list)
+	}
+	if svc {
+		name += "_svc"
+	}
+	return name
+}
+
+// BanMark is the mark (comment "hc:ban3") of a ban list's rules.
+func BanMark(list int64) string { return fmt.Sprintf("ban%d", list) }
+
+// BanElement is the set element of a ban: "203.0.113.9", or
+// "203.0.113.9 . tcp . 22".
+func BanElement(b Ban) string {
+	e := b.Addr.Addr().String()
+	if b.Proto != "" {
+		e += fmt.Sprintf(" . %s . %d", b.Proto, b.Port)
+	}
+	return e
+}
+
+// IsBanSet reports whether a set is a ban list's.
+func IsBanSet(name string) bool { return strings.HasPrefix(name, "ban_") }
+
+// banSets declares the ban lists' sets.
+func (r *renderer) banSets() {
+	for _, b := range r.st.BanLists {
+		if b.Disabled {
+			continue
+		}
+		obj := "ban list " + b.Name
+		for _, v6 := range []bool{false, true} {
+			r.w.emit(obj, "\tset %s { type %s; flags timeout; }", BanSet(b.ID, v6, false), addrType(v6))
+			r.w.emit(obj, "\tset %s { type %s . inet_proto . inet_service; flags timeout; }", BanSet(b.ID, v6, true), addrType(v6))
+		}
+	}
+}
+
+// banRules writes the drops of the ban lists that cover a chain, ahead
+// of everything that accepts: a banned address's established connections
+// are cut too.
+func (r *renderer) banRules(chain string) {
+	for _, b := range r.st.BanLists {
+		if b.Disabled || !b.Covers(chain) {
+			continue
+		}
+		obj := "ban list " + b.Name
+		d4, d6 := r.st.SplitRefs(b.Hosts)
+		for _, fam := range []string{"v4", "v6"} {
+			dsts := []string{""}
+			if b.Scope == model.BanScopeHosts {
+				refs := d4
+				if fam == "v6" {
+					refs = d6
+				}
+				var ok bool
+				if dsts, ok = addrAlts(fam, "daddr", refs, true, false); !ok {
+					continue
+				}
+			}
+			key := ipKeyword(fam) + " saddr"
+			if fam == "v6" && b.V6Group() < 128 {
+				key += " and " + v6Mask(b.V6Group())
+			}
+			v6 := fam == "v6"
+			for _, d := range dsts {
+				r.banDrop(obj, b.ID, strings.Join(nonEmpty([]string{d, key + " @" + BanSet(b.ID, v6, false)}), " "))
+				r.banDrop(obj, b.ID, strings.Join(nonEmpty([]string{d, key + " . meta l4proto . th dport @" + BanSet(b.ID, v6, true)}), " "))
+			}
+		}
+	}
+}
+
+// banDrop writes one drop of a ban list, or in shadow mode counts it.
+func (r *renderer) banDrop(obj string, list int64, match string) {
+	if r.opt.Shadow {
+		r.w.emit(obj, "\t\t%s counter accept comment \"hc:%s %s\"", match, BanMark(list), Would("drop"))
+		return
+	}
+	r.w.emit(obj, "\t\t%s counter drop comment \"hc:%s\"", match, BanMark(list))
+}
+
+// v6Mask is the mask of an IPv6 prefix length: 64 → ffff:ffff:ffff:ffff::.
+func v6Mask(bits int) string {
+	mask := make([]byte, 16)
+	for i := 0; i < bits; i++ {
+		mask[i/8] |= 0x80 >> (i % 8)
+	}
+	m, _ := netip.AddrFromSlice(mask)
+	return m.String()
+}
+
+// banElements adds the active bans, with the time each has left, so a
+// reload keeps them.
+func (r *renderer) banElements() {
+	for _, ban := range r.opt.Bans {
+		var list *model.BanList
+		for _, b := range r.st.BanLists {
+			if b.ID == ban.List && !b.Disabled {
+				list = b
+			}
+		}
+		if list == nil || !ban.Addr.IsValid() {
+			continue
+		}
+		timeout := ""
+		if ban.Timeout > 0 {
+			timeout = fmt.Sprintf(" timeout %ds", ban.Timeout)
+		}
+		r.w.emit("ban list "+list.Name, "add element inet %s %s { %s%s }", Table, BanSet(list.ID, ban.Addr.Addr().Is6(), ban.Proto != ""), BanElement(ban), timeout)
+	}
+}

internal/render/render.go

@@ -8,6 +8,14 @@
 // counter and a comment "hc:<kind><id>", which is how counters are read
 // back, and Ruleset.Lines maps each line to its object, for nftables'
 // error messages.
+//
+// In shadow mode (Options.Shadow, docs/shadow.md) the same rules are
+// written so that they change nothing: drops and rejects accept instead
+// (an accept only ends Housecarl's chain; the packet goes on to other
+// tables' chains, which decide), policies accept, NAT only counts, and
+// what alters packets (MSS clamping, the flowtable) is left out. What
+// would be dropped, rejected or translated carries a marker in its
+// comment ("[would drop]"), which the tracer reads.
 package render
 
 import (
@@ -42,8 +50,20 @@ type Options struct {
 	// DeviceSets are the per-device traffic counters' entries
 	// (stats_up_v4 …), so a reload keeps them.
 	DeviceSets map[string][]Element
+	// Bans are the ban lists' active bans, added to their sets.
+	Bans []Ban
+	// Shadow renders for shadow mode: nothing is blocked, rejected,
+	// translated or changed (see the package comment).
+	Shadow bool
+	// ShadowBefore runs the filter chains just before other tables' on
+	// their hooks (priority - 10) instead of just after (+ 10).
+	ShadowBefore bool
 }
 
+// Would marks, in shadow mode, a rule's comment with what it would do:
+// "[would drop]".
+func Would(action string) string { return "[would " + action + "]" }
+
 // Element is a dynamic set's entry with its remaining lifetime, or a
 // per-address quota's entry with what it used.
 type Element struct {
@@ -112,10 +132,14 @@ type renderer struct {
 func (r *renderer) render() {
 	w := r.w
 	w.emit("", "# Built by Housecarl from its database; changes made here are lost.")
+	if r.opt.Shadow {
+		w.emit("", "# Shadow mode: nothing here drops, rejects or translates; other tables decide.")
+	}
 	w.emit("", "table inet %s {}", Table)
 	w.emit("", "delete table inet %s", Table)
 	w.emit("", "table inet %s {", Table)
 	r.sets()
+	r.banSets()
 	r.deviceSets()
 	devices := r.flowtableDevices()
 	if len(devices) > 0 {
@@ -144,6 +168,7 @@ func (r *renderer) render() {
 	w.line = base + chains.line
 	w.emit("", "}")
 	r.elements()
+	r.banElements()
 }
 
 // ---- sets
@@ -265,6 +290,10 @@ func addrType(v6 bool) string {
 }
 
 func (r *renderer) flowtableDevices() []string {
+	if r.opt.Shadow {
+		// Offloaded connections skip every table's forward chain.
+		return nil
+	}
 	var out []string
 	for _, d := range r.st.Settings.FlowtableDevices {
 		if r.opt.Interfaces == nil || r.opt.Interfaces[d] {
@@ -277,8 +306,19 @@ func (r *renderer) flowtableDevices() []string {
 // ---- chains
 
 func (r *renderer) chain(name, hook, prio, policy string) {
+	nat := prio == "dstnat" || prio == "srcnat"
+	if r.opt.Shadow {
+		// NAT chains are asked in turn only until one translates: to
+		// count anything, Housecarl's must come first.
+		if nat || r.opt.ShadowBefore {
+			prio += " - 10"
+		} else {
+			prio += " + 10"
+		}
+		policy = model.PolicyAccept
+	}
 	r.w.emit("", "\tchain %s {", name)
-	r.w.emit("", "\t\ttype %s hook %s priority %s; policy %s;", map[bool]string{true: "nat", false: "filter"}[prio == "dstnat" || prio == "srcnat"], hook, prio, policy)
+	r.w.emit("", "\t\ttype %s hook %s priority %s; policy %s;", map[bool]string{true: "nat", false: "filter"}[nat], hook, prio, policy)
 }
 
 func (r *renderer) end() { r.w.emit("", "\t}") }
@@ -287,6 +327,16 @@ func (r *renderer) builtin(name, format string, args ...any) {
 	r.w.emit("built-in: "+name, "\t\t"+format+" comment \"hc:b-%s\"", append(args, name)...)
 }
 
+// builtinDrop writes a built-in that drops what it matches, or in shadow
+// mode counts it.
+func (r *renderer) builtinDrop(name, format string, args ...any) {
+	if r.opt.Shadow {
+		r.w.emit("built-in: "+name, "\t\t"+format+" counter accept comment \"hc:b-%s %s\"", append(args, name, Would("drop"))...)
+		return
+	}
+	r.builtin(name, format+" counter drop", args...)
+}
+
 func (r *renderer) early() {
 	st := r.st
 	r.chain("early", "prerouting", "raw", "accept")
@@ -302,10 +352,10 @@ func (r *renderer) early() {
 			}
 		}
 		if len(v4) > 0 {
-			r.builtin("antispoof", "iifname %s ip saddr { %s } counter drop", ifaces(ext), strings.Join(v4, ", "))
+			r.builtinDrop("antispoof", "iifname %s ip saddr { %s }", ifaces(ext), strings.Join(v4, ", "))
 		}
 		if len(v6) > 0 {
-			r.builtin("antispoof6", "iifname %s ip6 saddr { %s } counter drop", ifaces(ext), strings.Join(v6, ", "))
+			r.builtinDrop("antispoof6", "iifname %s ip6 saddr { %s }", ifaces(ext), strings.Join(v6, ", "))
 		}
 	}
 	r.rules(model.ChainEarly, true)
@@ -325,7 +375,7 @@ func (r *renderer) connBuiltins() {
 		r.builtin("established", "ct state established,related counter accept")
 	}
 	if b.DropInvalid {
-		r.builtin("invalid", "ct state invalid counter drop")
+		r.builtinDrop("invalid", "ct state invalid")
 	}
 }
 
@@ -336,6 +386,7 @@ func (r *renderer) input() {
 	if b.Loopback {
 		r.builtin("loopback", "iifname \"lo\" accept")
 	}
+	r.banRules(model.ChainInput)
 	// Rules that count traffic see every packet: before the established
 	// connections are accepted.
 	r.rules(model.ChainInput, true)
@@ -365,8 +416,10 @@ func (r *renderer) forward(flowtable bool) {
 	st := r.st
 	b := st.Settings.Builtins
 	r.chain("forward", "forward", "filter", st.Settings.ForwardPolicy)
+	r.banRules(model.ChainForward)
 	ext := st.ExternalInterfaces()
-	if b.MSSClamp && len(ext) > 0 {
+	// Shadow mode changes no packet.
+	if b.MSSClamp && len(ext) > 0 && !r.opt.Shadow {
 		r.builtin("mss-clamp", "oifname %s tcp flags syn tcp option maxseg size set rt mtu", ifaces(ext))
 	}
 	r.rules(model.ChainForward, true)
@@ -404,15 +457,21 @@ func (r *renderer) output() {
 	r.end()
 }
 
-// policyTail counts (and optionally logs) what reaches the policy.
+// policyTail counts (and optionally logs) what reaches the policy. In
+// shadow mode, where the policy accepts, that is what it would drop, and
+// it is always logged: the log is how shadow mode shows it.
 func (r *renderer) policyTail(chain, policy string) {
 	if policy != model.PolicyDrop {
 		return
 	}
 	obj := "policy: " + chain
-	if r.st.Settings.LogPolicyDrops && chain != "output" {
+	if r.opt.Shadow || r.st.Settings.LogPolicyDrops && chain != "output" {
 		r.w.emit(obj, "\t\tlimit rate 10/second burst 20 packets log prefix %s group %d", quote("hc "+chain+" policy drop: "), LogGroup)
 	}
+	if r.opt.Shadow {
+		r.w.emit(obj, "\t\tcounter comment \"hc:p-%s %s\"", chain, Would("drop"))
+		return
+	}
 	r.w.emit(obj, "\t\tcounter comment \"hc:p-%s\"", chain)
 }
 
@@ -428,7 +487,15 @@ func (r *renderer) rules(chain string, traffic bool) {
 		obj := rule.Label()
 		mark := fmt.Sprintf("r%d", rule.ID)
 		if rule.Raw != "" {
-			r.w.emit(obj, "\t\t%s", rule.Raw)
+			raw := rule.Raw
+			if r.opt.Shadow {
+				var ok bool
+				if raw, ok = ShadowRaw(raw); !ok {
+					r.w.emit(obj, "\t\t# left out in shadow mode: %s", rule.Raw)
+					continue
+				}
+			}
+			r.w.emit(obj, "\t\t%s", raw)
 			continue
 		}
 		var tail []string
@@ -478,17 +545,22 @@ func (r *renderer) verdict(rule *model.Rule, mark string) string {
 		}
 		v = append(v, fmt.Sprintf("log prefix %s group %d", quote(prefix), LogGroup))
 	}
-	switch rule.Action {
+	text := rule.Comment
-	case model.ActionAccept, model.ActionDrop:
+	switch {
+	case r.opt.Shadow && (rule.Action == model.ActionDrop || rule.Action == model.ActionReject):
+		// It ends Housecarl's chain as the drop would, and no more.
+		v = append(v, model.ActionAccept)
+		text = Would(rule.Action) + " " + text
+	case rule.Action == model.ActionAccept, rule.Action == model.ActionDrop:
 		v = append(v, rule.Action)
-	case model.ActionReject:
+	case rule.Action == model.ActionReject:
 		if rule.RejectWith == "tcp-reset" {
 			v = append(v, "reject with tcp reset")
 		} else {
 			v = append(v, "reject")
 		}
 	}
-	v = append(v, "comment "+comment(mark, rule.Comment))
+	v = append(v, "comment "+comment(mark, text))
 	return strings.Join(v, " ")
 }
 
@@ -781,7 +853,7 @@ func (r *renderer) dstnat() {
 					}
 				}
 			}
-			r.w.emit(obj, "\t\t%s counter redirect to :%s comment %s", strings.Join(nonEmpty(line.parts), " "), to, comment(mark, x.Comment))
+			r.w.emit(obj, "\t\t%s counter %s", strings.Join(nonEmpty(line.parts), " "), r.nat("redirect to :"+to, mark, x.Comment))
 		}
 	}
 	ext := st.ExternalInterfaces()
@@ -811,7 +883,7 @@ func (r *renderer) dstnat() {
 		for _, wh := range where {
 			for _, sa := range srcAlts {
 				for _, p := range f.Protocols {
-					r.w.emit(obj, "\t\t%s counter dnat ip to %s comment %s", strings.Join(nonEmpty([]string{wh, sa, p + " dport " + f.Ports}), " "), to, comment(mark, f.Name+" "+f.Comment))
+					r.w.emit(obj, "\t\t%s counter %s", strings.Join(nonEmpty([]string{wh, sa, p + " dport " + f.Ports}), " "), r.nat("dnat ip to "+to, mark, f.Name+" "+f.Comment))
 				}
 			}
 		}
@@ -885,7 +957,7 @@ func (r *renderer) srcnat() {
 		}
 		for _, sa := range srcAlts {
 			for _, p := range f.Protocols {
-				r.w.emit(obj, "\t\t%s counter masquerade comment %s", strings.Join(nonEmpty([]string{sa, "ip daddr " + f.To, p + " dport " + ports, "ct status dnat"}), " "), comment(mark, f.Name))
+				r.w.emit(obj, "\t\t%s counter %s", strings.Join(nonEmpty([]string{sa, "ip daddr " + f.To, p + " dport " + ports, "ct status dnat"}), " "), r.nat("masquerade", mark, f.Name))
 			}
 		}
 	}
@@ -917,12 +989,21 @@ func (r *renderer) srcnat() {
 					action = "snat ip to " + x.To
 				}
 			}
-			r.w.emit(obj, "\t\t%s counter %s comment %s", strings.Join(nonEmpty(line.parts), " "), action, comment(mark, x.Comment))
+			r.w.emit(obj, "\t\t%s counter %s", strings.Join(nonEmpty(line.parts), " "), r.nat(action, mark, x.Comment))
 		}
 	}
 	r.end()
 }
 
+// nat writes a NAT rule's statement and comment; in shadow mode only the
+// comment, saying what it would do ("[would dnat]").
+func (r *renderer) nat(action, mark, text string) string {
+	if r.opt.Shadow {
+		return "comment " + comment(mark, Would(strings.Fields(action)[0])+" "+text)
+	}
+	return action + " comment " + comment(mark, text)
+}
+
 // ---- helpers
 
 // hours writes a time window. nftables can't take a range past midnight,

internal/render/render_test.go

@@ -2,6 +2,8 @@ package render
 
 import (
 	"context"
+	"fmt"
+	"net/netip"
 	"os"
 	"os/exec"
 	"strings"
@@ -94,10 +96,28 @@ func router() *model.State {
 	st.Redirects = []*model.Redirect{
 		{ID: next(), Position: 1, Match: model.Match{InZones: []string{"lan"}, NotDests: true, Destinations: []string{"10.0.10.5"}, Services: []string{"udp/53"}}, Comment: "force DNS"},
 	}
+	st.BanLists = []*model.BanList{
+		{ID: next(), Name: "router", Scope: model.BanScopeFirewall, Never: []string{"trusted"}},
+		{ID: next(), Name: "web", Scope: model.BanScopeHosts, Hosts: []string{"10.0.10.57", "2a02:8013:7f3d:10::57", "dns_servers"}},
+		{ID: next(), Name: "edge", Scope: model.BanScopeEverything, GroupV6: 48},
+		{ID: next(), Name: "off", Scope: model.BanScopeEverything, Disabled: true},
+	}
 	st.Sort()
 	return st
 }
 
+// banOptions are active bans in each ban list, of every port and of one.
+func banOptions(st *model.State) []Ban {
+	var out []Ban
+	for _, b := range st.BanLists {
+		out = append(out,
+			Ban{List: b.ID, Addr: netip.MustParsePrefix("203.0.113.9/32"), Timeout: 3000},
+			Ban{List: b.ID, Addr: netip.MustParsePrefix("2001:db8:1:2::/64"), Proto: "tcp", Port: 22},
+			Ban{List: b.ID, Addr: netip.MustParsePrefix("198.51.100.7/32"), Proto: "udp", Port: 53, Timeout: 60})
+	}
+	return out
+}
+
 // quotaOptions carries quota usage, as a reload does: the guest quota's,
 // and two devices' of the per-device quota.
 func quotaOptions(st *model.State) (map[string]int64, map[string][]Element) {
@@ -254,15 +274,80 @@ func TestRenderedRulesetPassesNft(t *testing.T) {
 	vlan.Settings.FlowtableDevices = []string{"lo", "enp2s0.20"}
 	for name, st := range map[string]*model.State{"router": router(), "empty": model.NewState(), "stats": stats, "vlan": vlan} {
 		q, qs := quotaOptions(st)
-		rs := Render(st, Options{Elements: map[string][]Element{"guests": {{Addr: "10.0.30.5", Timeout: 1700}, {Addr: "2a02:8013:7f3d:30::5"}}}, Quotas: q, QuotaSets: qs,
+		// Enforcing, and in shadow mode after and before other tables.
-			DeviceSets: map[string][]Element{"stats_up_v4": {{Addr: "10.0.10.5", Timeout: 3000, Packets: 7, Bytes: 7000}}, "stats_down_v6": {{Addr: "2001:db8:1:2::", Packets: 1, Bytes: 100}}}})
+		for _, mode := range []Options{{}, {Shadow: true}, {Shadow: true, ShadowBefore: true}} {
+			opt := Options{Elements: map[string][]Element{"guests": {{Addr: "10.0.30.5", Timeout: 1700}, {Addr: "2a02:8013:7f3d:30::5"}}}, Quotas: q, QuotaSets: qs, Bans: banOptions(st),
+				DeviceSets: map[string][]Element{"stats_up_v4": {{Addr: "10.0.10.5", Timeout: 3000, Packets: 7, Bytes: 7000}}, "stats_down_v6": {{Addr: "2001:db8:1:2::", Packets: 1, Bytes: 100}}},
+				Shadow:     mode.Shadow, ShadowBefore: mode.ShadowBefore}
+			nftCheck(t, cmd, fmt.Sprintf("%s (shadow %v, before %v)", name, mode.Shadow, mode.ShadowBefore), Render(st, opt).Text)
+		}
+	}
+	nftCheck(t, cmd, "shadow fallback", ShadowFallback())
+}
+
+func nftCheck(t *testing.T, cmd []string, name, text string) {
+	t.Helper()
 	c := exec.CommandContext(context.Background(), cmd[0], append(cmd[1:], "-c", "-f", "/dev/stdin")...)
-		c.Stdin = strings.NewReader(rs.Text)
+	c.Stdin = strings.NewReader(text)
 	if out, err := c.CombinedOutput(); err != nil {
 		t.Errorf("%s: nft -c: %v\n%s", name, err, out)
-			for i, l := range strings.Split(rs.Text, "\n") {
+		for i, l := range strings.Split(text, "\n") {
 			t.Logf("%3d %s", i+1, l)
 		}
 	}
 }
+
+func TestRenderBans(t *testing.T) {
+	st := router()
+	id := map[string]int64{}
+	for _, b := range st.BanLists {
+		id[b.Name] = b.ID
+	}
+	text := Render(st, Options{Bans: banOptions(st)}).Text
+	f := func(format string, args ...any) string { return fmt.Sprintf(format, args...) }
+	r, w, e, off := id["router"], id["web"], id["edge"], id["off"]
+	for _, want := range []string{
+		f("set ban_%d_v4 { type ipv4_addr; flags timeout; }", r),
+		f("set ban_%d_v6_svc { type ipv6_addr . inet_proto . inet_service; flags timeout; }", r),
+		// The firewall: in input, every port and one port, /64 for IPv6.
+		f(`ip saddr @ban_%d_v4 counter drop comment "hc:ban%d"`, r, r),
+		f(`ip saddr . meta l4proto . th dport @ban_%d_v4_svc counter drop comment "hc:ban%d"`, r, r),
+		f(`ip6 saddr and ffff:ffff:ffff:ffff:: @ban_%d_v6 counter drop comment "hc:ban%d"`, r, r),
+		f(`ip6 saddr and ffff:ffff:ffff:ffff:: . meta l4proto . th dport @ban_%d_v6_svc counter drop comment "hc:ban%d"`, r, r),
+		// Hosts: in forward, first, to the hosts (literals and sets).
+		f(`ip daddr 10.0.10.57 ip saddr @ban_%d_v4 counter drop`, w),
+		f(`ip daddr @dns_servers_v4 ip saddr . meta l4proto . th dport @ban_%d_v4_svc counter drop`, w),
+		f(`ip6 daddr 2a02:8013:7f3d:10::57 ip6 saddr and ffff:ffff:ffff:ffff:: @ban_%d_v6 counter drop`, w),
+		// Everything: input and forward, /48.
+		f(`ip6 saddr and ffff:ffff:ffff:: @ban_%d_v6 counter drop`, e),
+		// The bans, with their time.
+		f("add element inet housecarl ban_%d_v4 { 203.0.113.9 timeout 3000s }", r),
+		f("add element inet housecarl ban_%d_v6_svc { 2001:db8:1:2:: . tcp . 22 }", w),
+		f("add element inet housecarl ban_%d_v4_svc { 198.51.100.7 . udp . 53 timeout 60s }", e),
+	} {
+		if !strings.Contains(text, want) {
+			t.Errorf("no %q in\n%s", want, text)
+		}
+	}
+	if n := strings.Count(text, f(`@ban_%d_v4 counter drop`, e)); n != 2 {
+		t.Errorf("scope everything: %d rules of every port in IPv4, want 2 (input and forward)", n)
+	}
+	if n := strings.Count(text, f(`@ban_%d_v4 counter drop`, w)); n != 2 {
+		t.Errorf("scope hosts: %d IPv4 rules of every port, want 2 (forward: the literal host and the set)", n)
+	}
+	// Ahead of the accepts: the first rules after loopback, and of forward.
+	in, _, _ := strings.Cut(text[strings.Index(text, "chain input"):], "established")
+	fw, _, _ := strings.Cut(text[strings.Index(text, "chain forward"):], "maxseg")
+	for _, part := range []string{in, fw} {
+		if strings.Contains(part, " accept comment \"hc:r") {
+			t.Errorf("a rule accepts before the bans:\n%s", part)
+		}
+	}
+	if strings.Contains(text, f("ban_%d_", off)) {
+		t.Errorf("a disabled list is rendered")
+	}
+	shadow := Render(st, Options{Shadow: true, Bans: banOptions(st)}).Text
+	if want := f(`ip saddr @ban_%d_v4 counter accept comment "hc:ban%d [would drop]"`, r, r); !strings.Contains(shadow, want) {
+		t.Errorf("shadow: no %q", want)
+	}
 }

internal/render/shadow.go

@@ -0,0 +1,127 @@
+package render
+
+import (
+	"fmt"
+	"strings"
+
+	"housecarl/internal/model"
+)
+
+// effects are the words of a raw rule's statements that act on packets
+// (or jump away). In shadow mode a raw rule with any of them is left out,
+// unless its only one is a final drop or reject, which becomes an accept.
+var effects = map[string]bool{
+	"drop": true, "reject": true, "queue": true,
+	"dnat": true, "snat": true, "masquerade": true, "redirect": true, "tproxy": true,
+	"notrack": true, "dup": true, "fwd": true, "synproxy": true, "flow": true, "offload": true,
+	"set": true, "reset": true, "jump": true, "goto": true, "return": true,
+}
+
+// ShadowRaw rewrites a raw rule for shadow mode: as written when it only
+// matches, counts, logs or accepts; with its final drop or reject made an
+// accept (marked "[would drop]" in its comment); ok is false when it does
+// anything else to packets (NAT, setting a mark or header field, a jump,
+// a verdict map…), and must be left out.
+func ShadowRaw(raw string) (out string, ok bool) {
+	toks := tokens(raw)
+	verdict := -1
+	for i := 0; i < len(toks); i++ {
+		w := word(toks[i])
+		if w == "comment" {
+			i++ // its text
+			continue
+		}
+		if !effects[w] {
+			continue
+		}
+		if verdict >= 0 || (w != "drop" && w != "reject") {
+			return "", false
+		}
+		verdict = i
+		// What follows a verdict, up to the comment, must be a reject's
+		// "with …".
+		j := i + 1
+		for j < len(toks) && word(toks[j]) != "comment" {
+			j++
+		}
+		if j > i+1 && (w == "drop" || word(toks[i+1]) != "with") {
+			return "", false
+		}
+		i = j - 1
+	}
+	if verdict < 0 {
+		return raw, true
+	}
+	mark := Would(word(toks[verdict]))
+	head := append(append([]string{}, toks[:verdict]...), "accept")
+	for i := verdict + 1; i < len(toks); i++ {
+		if word(toks[i]) != "comment" {
+			continue
+		}
+		if i+1 < len(toks) {
+			text := strings.Trim(toks[i+1], `"`)
+			return strings.Join(append(head, "comment", quote(strings.TrimSpace(mark+" "+text))), " "), true
+		}
+	}
+	return strings.Join(append(head, "comment", quote(mark)), " "), true
+}
+
+// tokens splits a rule into words, a quoted string being one.
+func tokens(s string) []string {
+	var out []string
+	var cur strings.Builder
+	quoted := false
+	for _, c := range s {
+		switch {
+		case c == '"':
+			quoted = !quoted
+			cur.WriteRune(c)
+		case !quoted && (c == ' ' || c == '\t'):
+			if cur.Len() > 0 {
+				out = append(out, cur.String())
+				cur.Reset()
+			}
+		default:
+			cur.WriteRune(c)
+		}
+	}
+	if cur.Len() > 0 {
+		out = append(out, cur.String())
+	}
+	return out
+}
+
+// word is a token as a keyword: without the punctuation of sets and maps
+// around it; a quoted string is no keyword.
+func word(t string) string {
+	if strings.HasPrefix(t, `"`) {
+		return ""
+	}
+	return strings.ToLower(strings.Trim(t, ",{}:()"))
+}
+
+// ShadowLeftOut lists the raw rules shadow mode leaves out (ShadowRaw).
+func ShadowLeftOut(st *model.State) []*model.Rule {
+	var out []*model.Rule
+	for _, r := range st.Rules {
+		if r.Raw == "" || r.Disabled {
+			continue
+		}
+		if _, ok := ShadowRaw(r.Raw); !ok {
+			out = append(out, r)
+		}
+	}
+	return out
+}
+
+// ShadowFallback is the boot fallback in shadow mode: the other firewall
+// protects the machine, so it is an empty table that filters nothing,
+// with a rule outside every hook that tells housecarld it is in force.
+func ShadowFallback() string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "# Housecarl's fallback in shadow mode: loaded at boot only if ruleset.nft\n")
+	fmt.Fprintf(&b, "# fails to load. It filters nothing; the other firewall decides.\n")
+	fmt.Fprintf(&b, "table inet %s {}\ndelete table inet %s\n", Table, Table)
+	fmt.Fprintf(&b, "table inet %s {\n\tchain fallback {\n\t\tcounter comment \"hc:%s-shadow\"\n\t}\n}\n", Table, FallbackMark)
+	return b.String()
+}

internal/render/shadow_test.go

@@ -0,0 +1,91 @@
+package render
+
+import (
+	"regexp"
+	"strings"
+	"testing"
+
+	"housecarl/internal/model"
+)
+
+// TestShadow: in shadow mode nothing in the ruleset drops, rejects,
+// translates or changes a packet, and what would is counted and marked.
+func TestShadow(t *testing.T) {
+	st := router()
+	st.Rules = append(st.Rules,
+		&model.Rule{ID: 950, Chain: model.ChainInput, Position: 99, Raw: `tcp dport 9999 counter reject with tcp reset comment "old"`},
+		&model.Rule{ID: 951, Chain: model.ChainForward, Position: 99, Raw: "meta mark set 0x1 accept"})
+	st.Settings.OutputPolicy = model.PolicyDrop
+	text := Render(st, Options{Shadow: true}).Text
+	// What acts on packets: verdicts, NAT, changes, the flowtable.
+	acts := regexp.MustCompile(`(?m)(\b(drop|reject|dnat|snat|masquerade|redirect)\b|maxseg size set|flow add|flowtable|policy drop)[^"\n]*$`)
+	for _, l := range strings.Split(text, "\n") {
+		code, _, _ := strings.Cut(l, "comment ")
+		code = strings.ReplaceAll(code, "ct status dnat", "") // a match
+		if strings.HasPrefix(strings.TrimSpace(l), "#") {
+			continue
+		}
+		if m := acts.FindString(code); m != "" && !strings.Contains(code, "log prefix") {
+			t.Errorf("acts on packets in shadow mode: %s", l)
+		}
+	}
+	for _, want := range []string{
+		"type filter hook input priority filter + 10; policy accept;",
+		"type filter hook prerouting priority raw + 10; policy accept;",
+		"type nat hook prerouting priority dstnat - 10; policy accept;",
+		"type nat hook postrouting priority srcnat - 10; policy accept;",
+		`ct state invalid counter accept comment "hc:b-invalid [would drop]"`,
+		`counter comment "hc:p-input [would drop]"`,
+		`counter comment "hc:p-output [would drop]"`,
+		`log prefix "hc output policy drop: "`,
+		`[would reject]`,
+		`[would dnat] jellyfin`,
+		`[would masquerade]`,
+		`[would redirect]`,
+		`tcp dport 9999 counter accept comment "[would reject] old"`,
+		"# left out in shadow mode: meta mark set 0x1 accept",
+		"# Shadow mode:",
+	} {
+		if !strings.Contains(text, want) {
+			t.Errorf("no %q in\n%s", want, text)
+		}
+	}
+	before := Render(st, Options{Shadow: true, ShadowBefore: true}).Text
+	for _, want := range []string{"priority filter - 10;", "priority raw - 10;", "priority dstnat - 10;"} {
+		if !strings.Contains(before, want) {
+			t.Errorf("before: no %q", want)
+		}
+	}
+	// Enforcing is unchanged by it.
+	if strings.Contains(Render(st, Options{}).Text, "would") {
+		t.Error("shadow markers when enforcing")
+	}
+	if got := ShadowLeftOut(st); len(got) != 1 || got[0].ID != 951 {
+		t.Errorf("left out: %v", got)
+	}
+}
+
+func TestShadowRaw(t *testing.T) {
+	cases := []struct{ in, out string }{
+		{"meta skuid 0 counter accept", "meta skuid 0 counter accept"},
+		{"tcp dport 22 drop", `tcp dport 22 accept comment "[would drop]"`},
+		{`tcp dport 22 counter drop comment "no ssh"`, `tcp dport 22 counter accept comment "[would drop] no ssh"`},
+		{"ip saddr 10.0.0.1 reject with icmpx type admin-prohibited", `ip saddr 10.0.0.1 accept comment "[would reject]"`},
+		{"tcp dport 22 reject with tcp reset", `tcp dport 22 accept comment "[would reject]"`},
+		{`log prefix "drop me: " group 5 accept`, `log prefix "drop me: " group 5 accept`},
+		{"ip saddr { 10.0.0.1, 10.0.0.2 } counter", "ip saddr { 10.0.0.1, 10.0.0.2 } counter"},
+		{"tcp dport vmap { 22 : drop, 80 : accept }", ""},
+		{"meta mark set 1", ""},
+		{"ip daddr 1.2.3.4 dnat ip to 10.0.0.1", ""},
+		{"tcp dport 22 jump other", ""},
+		{"drop accept", ""},
+		{"meta nftrace set 1", ""},
+		{"ct state new notrack", ""},
+	}
+	for _, c := range cases {
+		out, ok := ShadowRaw(c.in)
+		if ok != (c.out != "") || out != c.out {
+			t.Errorf("%q: got %q %v, want %q", c.in, out, ok, c.out)
+		}
+	}
+}

internal/server/authn.go

@@ -102,6 +102,8 @@ func (s *Server) authenticate(r *http.Request) (*auth.Principal, *model.Session,
 		p, err := s.tokenPrincipal(r.Context(), strings.TrimSpace(tok), ip, r.UserAgent())
 		if err != nil {
 			s.limiter.fail("ip:" + ip)
+			// For Guard and fail2ban: the address first, so a name can't fake it.
+			s.log.Warn("token refused", "ip", ip, "reason", err.Error())
 			s.eng.Audit(r.Context(), &auth.Principal{IP: ip, Method: "token"}, "auth.token_failed", "", "failure", err.Error())
 			return nil, nil, &httpError{http.StatusUnauthorized, "unauthenticated", "invalid API token: " + err.Error()}
 		}
@@ -246,6 +248,8 @@ func (s *Server) login(w http.ResponseWriter, r *http.Request) error {
 		if err == nil && u.Disabled && ok {
 			reason = "account disabled"
 		}
+		// For Guard and fail2ban: the address first, so a name can't fake it.
+		s.log.Warn("login failed", "ip", ip, "user", name, "reason", reason)
 		s.eng.Audit(r.Context(), who, "auth.login_failed", "user "+name, "failure", reason)
 		return &httpError{http.StatusUnauthorized, "login_failed", "wrong username or password"}
 	}
@@ -275,7 +279,7 @@ func (s *Server) logout(w http.ResponseWriter, r *http.Request, p *auth.Principa
 }
 
 func (s *Server) whoamiOf(p *auth.Principal, se *model.Session) api.Whoami {
-	out := api.Whoami{User: p.User, Admin: p.IsAdmin(), Method: p.Method, Permissions: p.Permissions(), Build: version.Info()}
+	out := api.Whoami{User: p.User, Admin: p.IsAdmin(), Method: p.Method, Permissions: p.Permissions(), Build: version.Info(), Mode: s.mode()}
 	if se != nil {
 		out.CSRFToken = se.CSRFToken
 	}

internal/server/bans_test.go

@@ -0,0 +1,154 @@
+package server
+
+import (
+	"bytes"
+	"context"
+	"log/slog"
+	"strings"
+	"testing"
+
+	"housecarl/internal/api"
+	"housecarl/internal/model"
+	"housecarl/internal/store"
+)
+
+// TestBans: a program with a token for one ban list bans addresses: of
+// a service's ports, every port, or numbers; renewed when banned again;
+// IPv6 by /64; never the inside, the firewall's own, or the list's
+// never; times cut to the list's longest; the address's connections the
+// ban drops are closed; and lifted by ID or address.
+func TestBans(t *testing.T) {
+	e := newEnv(t)
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "wan", Interfaces: []string{"wan"}, External: true}, nil)
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "lan", Interfaces: []string{"eth1"}}, nil)
+	for _, b := range []model.BanList{
+		{Name: "web", Scope: model.BanScopeHosts, Hosts: []string{"10.0.10.57"}, Never: []string{"198.51.100.0/24"}, MaxTime: 3600},
+		{Name: "router", Scope: model.BanScopeFirewall},
+	} {
+		if code := e.call(e.token, "POST", "/banlists", b, nil); code != 201 {
+			t.Fatalf("ban list %s: %d", b.Name, code)
+		}
+	}
+	fc := &fakeConns{list: []api.Connection{
+		{Family: "ipv4", Protocol: "tcp", Source: "203.0.113.9", SourcePort: 40000, Destination: "217.155.8.194", DestinationPort: 2222, DNATTo: "10.0.10.57", DNATPort: 22},
+		{Family: "ipv4", Protocol: "tcp", Source: "203.0.113.9", SourcePort: 40001, Destination: "10.0.10.58", DestinationPort: 22},
+		{Family: "ipv4", Protocol: "tcp", Source: "203.0.113.9", SourcePort: 40002, Destination: "217.155.8.194", DestinationPort: 22},
+	}}
+	e.srv.SetConntrack(fc)
+	var tok api.Token
+	if code := e.call(e.token, "POST", "/me/tokens", api.TokenInput{Name: "guard", Grants: []string{"bans:write:web"}}, &tok); code != 201 {
+		t.Fatalf("token: %d", code)
+	}
+	ban := func(in api.BanInput, want int) api.BanResult {
+		t.Helper()
+		var out api.BanResult
+		var eb api.ErrorBody
+		code := e.call(tok.Secret, "POST", "/banlists/web/bans", in, &out)
+		if code != want {
+			e.call(tok.Secret, "POST", "/banlists/web/bans", in, &eb)
+			t.Errorf("ban %+v: %d, want %d (%+v)", in, code, want, eb)
+		}
+		return out
+	}
+	out := ban(api.BanInput{Address: "203.0.113.9", Ports: []byte(`"ssh"`), Time: 600, Source: "guard/ssh", Reason: "Failed password"}, 201)
+	if len(out.Bans) != 1 || out.Bans[0].Protocol != "tcp" || out.Bans[0].Port != 22 || out.Bans[0].Existing || out.Bans[0].Timeout != 600 {
+		t.Errorf("ssh ban: %+v", out)
+	}
+	if out.Closed != 1 || len(fc.closed) != 1 || fc.closed[0].DNATTo != "10.0.10.57" {
+		t.Errorf("closed %d: %+v (only the one to the list's host)", out.Closed, fc.closed)
+	}
+	if _, ok := e.fw.bans[`ban_`+itoa(out.Bans[0].ListID)+`_v4_svc 203.0.113.9 . tcp . 22`]; !ok {
+		t.Errorf("not in force: %+v", e.fw.bans)
+	}
+	if again := ban(api.BanInput{Address: "203.0.113.9", Ports: []byte(`22`), Protocol: "tcp"}, 201); len(again.Bans) != 1 || !again.Bans[0].Existing {
+		t.Errorf("banned again: %+v", again)
+	}
+	if v6 := ban(api.BanInput{Address: "2001:db8:1:2::77", Time: 99999}, 201); len(v6.Bans) != 1 || v6.Bans[0].Address != "2001:db8:1:2::/64" ||
+		v6.Bans[0].Protocol != "" || v6.Bans[0].Timeout != 3600 {
+		t.Errorf("IPv6, every port, cut to an hour: %+v", v6)
+	}
+	if n := ban(api.BanInput{Address: "::ffff:203.0.113.10", Ports: []byte(`53`)}, 201); len(n.Bans) != 2 || n.Bans[0].Address != "203.0.113.10" {
+		t.Errorf("a number without a protocol is tcp and udp: %+v", n)
+	}
+	if n := ban(api.BanInput{Address: "203.0.113.11", Protocol: "tcp", Ports: []byte(`["http", "https"]`)}, 201); len(n.Bans) != 2 {
+		t.Errorf("two services: %+v", n)
+	}
+	if n := ban(api.BanInput{Address: "203.0.113.12", Protocol: "tcp", Ports: []byte(`"0:65535"`)}, 201); len(n.Bans) != 1 || n.Bans[0].Protocol != "" {
+		t.Errorf("fail2ban's every port: %+v", n)
+	}
+	ban(api.BanInput{Address: "203.0.113.13", Ports: []byte(`"bogus"`)}, 400)
+	ban(api.BanInput{Address: "203.0.113.13", Protocol: "tcp"}, 400)
+	ban(api.BanInput{Address: "not-an-address"}, 400)
+	for _, a := range []string{"198.51.100.4", "10.0.10.99", "217.155.8.194", "127.0.0.1", "fe80::1", "192.168.5.5"} {
+		var eb api.ErrorBody
+		if code := e.call(tok.Secret, "POST", "/banlists/web/bans", api.BanInput{Address: a}, &eb); code != 422 || eb.Error.Code != "never_ban" {
+			t.Errorf("never ban %s: %d %+v", a, code, eb)
+		}
+	}
+	if code := e.call(tok.Secret, "POST", "/banlists/router/bans", api.BanInput{Address: "203.0.113.9"}, nil); code != 403 {
+		t.Errorf("another list: %d", code)
+	}
+	// The program may read its list, and only its list.
+	var bl model.BanList
+	if code := e.call(tok.Secret, "GET", "/banlists/web", nil, &bl); code != 200 || bl.V6Group() != 64 {
+		t.Errorf("reading its list: %d %+v", code, bl)
+	}
+	var lists []model.BanList
+	if e.call(tok.Secret, "GET", "/banlists", nil, &lists); len(lists) != 1 || lists[0].Name != "web" {
+		t.Errorf("lists: %+v", lists)
+	}
+	if code := e.call(tok.Secret, "GET", "/banlists/router", nil, nil); code != 403 {
+		t.Errorf("reading another list: %d", code)
+	}
+	var bans []model.Ban
+	e.call(tok.Secret, "GET", "/banlists/web/bans", nil, &bans)
+	if len(bans) != 7 || bans[0].List != "web" || bans[0].Who != "admin" || bans[0].Source != "guard/ssh" {
+		t.Errorf("bans: %d %+v", len(bans), bans)
+	}
+	var removed map[string]int
+	if code := e.call(tok.Secret, "DELETE", "/banlists/web/bans?address=2001:db8:1:2::5", nil, &removed); code != 200 || removed["removed"] != 1 {
+		t.Errorf("lift by address: %d %+v", code, removed)
+	}
+	if code := e.call(tok.Secret, "DELETE", "/banlists/web/bans/"+itoa(bans[0].ID), nil, &removed); code != 200 || removed["removed"] != 1 {
+		t.Errorf("lift by ID: %d %+v", code, removed)
+	}
+	if code := e.call(tok.Secret, "DELETE", "/banlists/web/bans?address=203.0.113.200", nil, nil); code != 404 {
+		t.Errorf("lifting what isn't banned: %d", code)
+	}
+	if _, ok := e.fw.bans[`ban_`+itoa(bans[0].ListID)+`_v4_svc 203.0.113.9 . tcp . 22`]; ok {
+		t.Errorf("still in force after lifting")
+	}
+	evs, _ := e.srv.store.ListAudit(context.Background(), store.AuditFilter{Action: "ban."})
+	if len(evs) < 10 || !strings.Contains(evs[len(evs)-1].Detail, "guard/ssh: Failed password") {
+		t.Errorf("audit: %d %+v", len(evs), evs[len(evs)-1])
+	}
+	// The list's name is what programs know it by.
+	if code := e.call(e.token, "PATCH", "/banlists/web", map[string]any{"name": "web2"}, nil); code != 409 {
+		t.Errorf("renaming a ban list: %d", code)
+	}
+	// Every apply puts the bans back; a deleted list's go.
+	if got := ActiveBans(context.Background(), e.srv.store); len(got) != 5 {
+		t.Errorf("active bans: %d", len(got))
+	}
+	if code := e.call(e.token, "DELETE", "/banlists/web", nil, nil); code != 200 {
+		t.Errorf("deleting the list: %d", code)
+	}
+	if got := ActiveBans(context.Background(), e.srv.store); len(got) != 0 {
+		t.Errorf("a deleted list's bans: %d", len(got))
+	}
+}
+
+// TestFailedLoginLogged: a failed login and a refused token are logged
+// with the address first, for Guard's and fail2ban's patterns.
+func TestFailedLoginLogged(t *testing.T) {
+	e := newEnv(t)
+	var buf bytes.Buffer
+	e.srv.log = slog.New(slog.NewTextHandler(&buf, nil))
+	e.call("", "POST", "/auth/login", api.Login{Username: `x ip=8.8.8.8`, Password: "wrong"}, nil)
+	e.call("housecarl_aaaaaaaaaaaaaaaa_bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "GET", "/status", nil, nil)
+	for _, want := range []string{`msg="login failed" ip=127.0.0.1 user="x ip=8.8.8.8"`, `msg="token refused" ip=127.0.0.1 reason=`} {
+		if !strings.Contains(buf.String(), want) {
+			t.Errorf("no %q in\n%s", want, buf.String())
+		}
+	}
+}

internal/server/h_bans.go

@@ -0,0 +1,555 @@
+package server
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/netip"
+	"strconv"
+	"strings"
+	"time"
+
+	"housecarl/internal/api"
+	"housecarl/internal/auth"
+	"housecarl/internal/model"
+	"housecarl/internal/render"
+	"housecarl/internal/store"
+)
+
+// Ban lists (docs/bans.md): the lists are configuration objects; their
+// bans are kept apart, like a dynamic set's entries, audited but not in
+// the history or on probation. A ban is saved first, then put in force;
+// every apply puts all of them back (ActiveBans).
+
+// banList finds a ban list the principal may read or ban into.
+func (s *Server) banList(r *http.Request, p *auth.Principal, write bool) (*model.BanList, error) {
+	ref := r.PathValue("ref")
+	o := s.eng.State().Find(model.KindBanList, ref)
+	if o == nil {
+		if !p.CanBan(ref, write) {
+			return nil, banForbidden(ref, write)
+		}
+		return nil, notFound("no ban list %s", ref)
+	}
+	b := o.(*model.BanList)
+	if !p.CanBan(b.Name, write) {
+		return nil, banForbidden(b.Name, write)
+	}
+	return b, nil
+}
+
+func banForbidden(list string, write bool) error {
+	access := "read"
+	if write {
+		access = "write"
+	}
+	return forbidden("permission denied: this needs bans:%s for %s", access, list)
+}
+
+// ActiveBans are the bans to put back with every apply (nft.Firewall's
+// Bans).
+func ActiveBans(ctx context.Context, st *store.Store) []render.Ban {
+	list, err := st.ListBans(ctx, 0)
+	if err != nil {
+		return nil
+	}
+	out := make([]render.Ban, 0, len(list))
+	for _, b := range list {
+		if rb, ok := renderBan(b); ok {
+			out = append(out, rb)
+		}
+	}
+	return out
+}
+
+func renderBan(b *model.Ban) (render.Ban, bool) {
+	p, err := b.Prefix()
+	if err != nil {
+		return render.Ban{}, false
+	}
+	return render.Ban{List: b.ListID, Addr: p, Proto: b.Protocol, Port: b.Port, Timeout: b.Timeout}, true
+}
+
+// listBans answers GET /banlists/{ref}/bans, and GET /bans: every list's
+// the principal may read.
+func (s *Server) listBans(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	st := s.eng.State()
+	var id int64
+	if r.PathValue("ref") != "" {
+		b, err := s.banList(r, p, false)
+		if err != nil {
+			return err
+		}
+		id = b.ID
+	}
+	all, err := s.store.ListBans(r.Context(), id)
+	if err != nil {
+		return err
+	}
+	out := []*model.Ban{}
+	for _, b := range all {
+		o := st.ByID(model.KindBanList, b.ListID)
+		if o == nil || !p.CanBan(o.Label(), false) {
+			continue
+		}
+		b.List = o.Label()
+		out = append(out, b)
+	}
+	writeJSON(w, http.StatusOK, out)
+	return nil
+}
+
+// addBan answers POST /banlists/{ref}/bans.
+func (s *Server) addBan(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	list, err := s.banList(r, p, true)
+	if err != nil {
+		return err
+	}
+	var in api.BanInput
+	if err := decodeJSON(r, &in); err != nil {
+		return err
+	}
+	st := s.eng.State()
+	if list.Disabled {
+		return conflict("ban list %s is disabled: enable it first", list.Name)
+	}
+	addr, err := netip.ParseAddr(strings.Trim(strings.TrimSpace(in.Address), "[]"))
+	if err != nil {
+		return badRequest("%q isn't an address", in.Address)
+	}
+	addr = addr.Unmap().WithZone("")
+	bits := 32
+	if addr.Is6() {
+		bits = list.V6Group()
+	}
+	prefix := netip.PrefixFrom(addr, bits).Masked()
+	if why := s.neverBan(r.Context(), st, list, p, addr, prefix); why != "" {
+		s.eng.Audit(r.Context(), p, "ban.add", list.Name+" "+model.BanAddress(prefix), "failure", why)
+		return &httpError{http.StatusUnprocessableEntity, "never_ban", fmt.Sprintf("%s isn't banned: %s", model.BanAddress(prefix), why)}
+	}
+	ports, err := banPorts(st, in.Protocol, in.Ports)
+	if err != nil {
+		return badRequest("%v", err)
+	}
+	var expires *time.Time
+	switch secs := banSeconds(list, in.Time); {
+	case secs > 0:
+		t := s.nowFn().Add(time.Duration(secs) * time.Second)
+		expires = &t
+	}
+	if n, err := s.store.CountBans(r.Context(), list.ID); err != nil {
+		return err
+	} else if n+len(ports) > list.Entries() {
+		existing := 0
+		for _, pp := range ports {
+			if s.banExists(r.Context(), list.ID, model.BanAddress(prefix), pp) {
+				existing++
+			}
+		}
+		if n+len(ports)-existing > list.Entries() {
+			return &httpError{http.StatusUnprocessableEntity, "list_full", fmt.Sprintf("ban list %s holds its most bans (%d, max_entries); wait for some to end, or raise it", list.Name, list.Entries())}
+		}
+	}
+	out := api.BanResult{Bans: []api.BanMade{}}
+	for _, pp := range ports {
+		b := &model.Ban{ListID: list.ID, List: list.Name, Address: model.BanAddress(prefix), Protocol: pp.proto, Port: pp.port,
+			Expires: expires, Who: p.Name(), Source: clip(in.Source, 100), Reason: clip(in.Reason, 500)}
+		existing, err := s.store.PutBan(r.Context(), b)
+		if err != nil {
+			return err
+		}
+		if expires != nil {
+			b.Timeout = int(time.Until(*expires).Round(time.Second) / time.Second)
+		}
+		rb, _ := renderBan(b)
+		if err := s.fw.AddBan(r.Context(), rb); err != nil {
+			s.log.Warn("putting a ban in force failed; the next change or restart adds it", "list", list.Name, "address", b.Address, "err", err)
+		}
+		out.Bans = append(out.Bans, api.BanMade{Ban: *b, Existing: existing})
+		detail := b.Source
+		if b.Reason != "" {
+			detail = strings.TrimSpace(detail + ": " + b.Reason)
+		}
+		s.eng.Audit(r.Context(), p, "ban.add", list.Name+" "+banLabel(b), "success", detail)
+	}
+	out.Closed = s.closeBanned(r.Context(), st, list, prefix, ports)
+	writeJSON(w, http.StatusCreated, out)
+	return nil
+}
+
+func (s *Server) banExists(ctx context.Context, list int64, addr string, pp banPort) bool {
+	bans, _ := s.store.ListBans(ctx, list)
+	for _, b := range bans {
+		if b.Address == addr && b.Protocol == pp.proto && b.Port == pp.port {
+			return true
+		}
+	}
+	return false
+}
+
+// banSeconds is a ban's length: what was asked (0 the list's default, -1
+// its longest), cut to the list's longest. 0 means it doesn't end.
+func banSeconds(list *model.BanList, asked int) int {
+	max := list.BanMaxTime()
+	secs := asked
+	switch {
+	case asked == 0:
+		secs = list.BanTime()
+	case asked < 0:
+		secs = max
+	}
+	if max > 0 && secs > max {
+		secs = max
+	}
+	return secs
+}
+
+func banLabel(b *model.Ban) string {
+	if b.Protocol == "" {
+		return b.Address
+	}
+	return fmt.Sprintf("%s %s/%d", b.Address, b.Protocol, b.Port)
+}
+
+func clip(s string, n int) string {
+	s = strings.TrimSpace(s)
+	if len(s) > n {
+		return s[:n]
+	}
+	return s
+}
+
+// banPort is one protocol and port banned; proto "" is every port.
+type banPort struct {
+	proto string
+	port  int
+}
+
+// maxBanPorts is the most ports one ban request may name: a range in a
+// service counts each port.
+const maxBanPorts = 64
+
+// banPorts reads a ban request's protocol and ports (api.BanInput).
+func banPorts(st *model.State, protocol string, raw json.RawMessage) ([]banPort, error) {
+	protocol = strings.ToLower(strings.TrimSpace(protocol))
+	if protocol != "" && protocol != "tcp" && protocol != "udp" && protocol != "all" {
+		return nil, fmt.Errorf("protocol %q: use tcp, udp, or leave it out for every port", protocol)
+	}
+	if protocol == "all" {
+		protocol = ""
+	}
+	var words []string
+	if len(raw) > 0 && string(raw) != "null" {
+		var list []json.RawMessage
+		if json.Unmarshal(raw, &list) != nil {
+			list = []json.RawMessage{raw}
+		}
+		for _, item := range list {
+			var n int
+			var str string
+			switch {
+			case json.Unmarshal(item, &n) == nil:
+				words = append(words, strconv.Itoa(n))
+			case json.Unmarshal(item, &str) == nil:
+				words = append(words, strings.FieldsFunc(str, func(c rune) bool { return c == ',' || c == ' ' })...)
+			default:
+				return nil, fmt.Errorf("ports: give numbers or service names, in a list or a string")
+			}
+		}
+	}
+	if len(words) == 0 {
+		if protocol != "" {
+			return nil, fmt.Errorf("protocol %s without ports: give the ports, or leave the protocol out for every port", protocol)
+		}
+		return []banPort{{}}, nil
+	}
+	protos := []string{"tcp", "udp"}
+	if protocol != "" {
+		protos = []string{protocol}
+	}
+	var out []banPort
+	seen := map[banPort]bool{}
+	add := func(proto string, port int) {
+		bp := banPort{proto, port}
+		if !seen[bp] {
+			seen[bp] = true
+			out = append(out, bp)
+		}
+	}
+	for _, w := range words {
+		lw := strings.ToLower(w)
+		switch lw {
+		case "all", "any", "0:65535", "1:65535", "0-65535", "1-65535":
+			return []banPort{{}}, nil
+		}
+		if n, err := strconv.Atoi(w); err == nil {
+			if n < 1 || n > 65535 {
+				return nil, fmt.Errorf("port %d: use 1 to 65535", n)
+			}
+			for _, p := range protos {
+				add(p, n)
+			}
+			continue
+		}
+		svc := st.Service(w)
+		if svc == nil {
+			return nil, fmt.Errorf("%q is neither a port nor a service (built-in or yours)", w)
+		}
+		matched := false
+		for _, e := range st.ServiceEntries([]string{svc.Name}) {
+			if e.Proto != "tcp" && e.Proto != "udp" || protocol != "" && e.Proto != protocol {
+				continue
+			}
+			for _, part := range strings.Split(e.Ports, ",") {
+				lo, hi, _ := strings.Cut(strings.TrimSpace(part), "-")
+				a, err1 := strconv.Atoi(lo)
+				b := a
+				var err2 error
+				if hi != "" {
+					b, err2 = strconv.Atoi(hi)
+				}
+				if err1 != nil || err2 != nil || b-a >= maxBanPorts {
+					return nil, fmt.Errorf("service %s has the range %s: ban every port instead", svc.Name, part)
+				}
+				for port := a; port <= b; port++ {
+					add(e.Proto, port)
+					matched = true
+				}
+			}
+		}
+		if !matched {
+			return nil, fmt.Errorf("service %s has no %s ports", svc.Name, map[bool]string{true: protocol, false: "tcp or udp"}[protocol != ""])
+		}
+	}
+	if len(out) > maxBanPorts {
+		return nil, fmt.Errorf("%d ports in one ban: ban every port instead (up to %d)", len(out), maxBanPorts)
+	}
+	return out, nil
+}
+
+// neverBan says why an address (banned as prefix) must not be banned, or
+// "": it is the firewall's, an inside network's, the client's own, in the
+// list's never, or not an address one bans.
+func (s *Server) neverBan(ctx context.Context, st *model.State, list *model.BanList, p *auth.Principal, addr netip.Addr, prefix netip.Prefix) string {
+	switch {
+	case addr.IsLoopback(), addr.IsUnspecified(), addr.IsMulticast(), addr.IsLinkLocalUnicast(), addr.IsLinkLocalMulticast():
+		return "it is a loopback, link-local, multicast or unspecified address"
+	}
+	if own, err := netip.ParseAddr(p.IP); err == nil && prefix.Contains(own.Unmap()) {
+		return "it is the address of whoever asked (" + own.Unmap().String() + ")"
+	}
+	inside := s.insideNetworks(st)
+	if inside(addr) || inside(prefix.Addr()) {
+		return "it is inside: in a network of an inside zone, or private"
+	}
+	external := map[string]bool{}
+	for _, z := range st.Zones {
+		external[z.Name] = z.External
+	}
+	for net, zone := range s.connected(st) {
+		if !external[zone] && net.Overlaps(prefix) {
+			return fmt.Sprintf("it overlaps %s, a network of zone %s", net, zone)
+		}
+	}
+	if s.ifaces != nil {
+		if all, err := s.ifaces.All(); err == nil {
+			for name, ps := range all {
+				for _, a := range ps {
+					if prefix.Contains(a.Addr()) {
+						return fmt.Sprintf("it is the firewall's own (%s on %s)", a.Addr(), name)
+					}
+				}
+			}
+		}
+	}
+	for _, o := range st.Settings.OwnPrefixes {
+		if op, err := netip.ParsePrefix(o); err == nil && op.Overlaps(prefix) {
+			return "it is in the firewall's own prefix " + o + " (settings)"
+		}
+	}
+	for _, ref := range list.Never {
+		if why := s.inNever(ctx, st, ref, prefix); why != "" {
+			return why + " (the list's never)"
+		}
+	}
+	return ""
+}
+
+// inNever: is prefix in a never entry (an address, or a set)?
+func (s *Server) inNever(ctx context.Context, st *model.State, ref string, prefix netip.Prefix) string {
+	overlaps := func(entry string) bool {
+		a, err := model.ParseAddress(entry)
+		return err == nil && a.Overlaps(prefix)
+	}
+	if model.IsLiteral(ref) {
+		if overlaps(ref) {
+			return "it is in " + ref
+		}
+		return ""
+	}
+	set := st.Set(ref)
+	if set == nil {
+		return ""
+	}
+	entries := set.Entries
+	if set.Dynamic {
+		if t, err := s.fw.Table(ctx); err == nil {
+			for _, e := range t.Elements[set.Name] {
+				entries = append(entries, e.Addr)
+			}
+		}
+	}
+	for _, e := range entries {
+		if overlaps(e) {
+			return fmt.Sprintf("it is in set %s (%s)", set.Name, e)
+		}
+	}
+	return ""
+}
+
+// closeBanned closes the banned address's connections the list's bans
+// now drop: offloaded to the flowtable, they would go on without
+// reaching the rules. Not in shadow mode, which changes no traffic.
+func (s *Server) closeBanned(ctx context.Context, st *model.State, list *model.BanList, prefix netip.Prefix, ports []banPort) int {
+	if s.cfg.Shadow || s.conns == nil {
+		return 0
+	}
+	conns, err := s.conns.List(ctx)
+	if err != nil {
+		s.log.Warn("listing connections to close a ban's failed", "err", err)
+		return 0
+	}
+	own := map[netip.Addr]bool{}
+	if s.ifaces != nil {
+		if all, err := s.ifaces.All(); err == nil {
+			for _, ps := range all {
+				for _, a := range ps {
+					own[a.Addr()] = true
+				}
+			}
+		}
+	}
+	h4, h6 := st.SplitRefs(list.Hosts)
+	hosts := append(append([]string{}, h4.Literals...), h6.Literals...)
+	for _, name := range append(h4.Sets, h6.Sets...) {
+		if set := st.Set(name); set != nil {
+			hosts = append(hosts, set.Entries...)
+		}
+	}
+	isHost := func(a netip.Addr) bool {
+		for _, h := range hosts {
+			if x, err := model.ParseAddress(h); err == nil && x.Contains(a) {
+				return true
+			}
+		}
+		return false
+	}
+	closed := 0
+	for _, c := range conns {
+		src, err := netip.ParseAddr(c.Source)
+		if err != nil || !prefix.Contains(src.Unmap()) {
+			continue
+		}
+		dst, _ := netip.ParseAddr(c.Destination)
+		port := c.DestinationPort
+		if c.DNATTo != "" {
+			dst, _ = netip.ParseAddr(c.DNATTo)
+			if c.DNATPort != 0 {
+				port = c.DNATPort
+			}
+		}
+		toFirewall := own[dst.Unmap()]
+		switch list.Scope {
+		case model.BanScopeFirewall:
+			if !toFirewall {
+				continue
+			}
+		case model.BanScopeHosts:
+			if toFirewall || !isHost(dst.Unmap()) {
+				continue
+			}
+		}
+		hit := false
+		for _, pp := range ports {
+			hit = hit || pp.proto == "" || pp.proto == c.Protocol && pp.port == port
+		}
+		if !hit {
+			continue
+		}
+		if err := s.conns.Close(ctx, c); err == nil {
+			closed++
+		}
+	}
+	return closed
+}
+
+// deleteBans answers DELETE /banlists/{ref}/bans/{id}, and DELETE
+// /banlists/{ref}/bans?address=A: every ban of that address (any of its
+// grouping), of every port and of each.
+func (s *Server) deleteBans(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	list, err := s.banList(r, p, true)
+	if err != nil {
+		return err
+	}
+	var gone []*model.Ban
+	if idText := r.PathValue("id"); idText != "" {
+		id, err := strconv.ParseInt(idText, 10, 64)
+		if err != nil {
+			return badRequest("ban ID %q isn't a number", idText)
+		}
+		b, err := s.store.GetBan(r.Context(), id)
+		if errors.Is(err, store.ErrNotFound) || err == nil && b.ListID != list.ID {
+			return notFound("no ban %d in ban list %s", id, list.Name)
+		}
+		if err != nil {
+			return err
+		}
+		gone = append(gone, b)
+	} else {
+		q := strings.Trim(strings.TrimSpace(r.URL.Query().Get("address")), "[]")
+		a, err := netip.ParseAddr(q)
+		if err != nil {
+			return badRequest("say which ban: DELETE …/bans/ID, or …/bans?address=ADDRESS")
+		}
+		a = a.Unmap()
+		bans, err := s.store.ListBans(r.Context(), list.ID)
+		if err != nil {
+			return err
+		}
+		for _, b := range bans {
+			if pre, err := b.Prefix(); err == nil && pre.Contains(a) {
+				gone = append(gone, b)
+			}
+		}
+		if len(gone) == 0 {
+			return notFound("%s isn't banned in ban list %s", a, list.Name)
+		}
+	}
+	for _, b := range gone {
+		if err := s.store.DeleteBan(r.Context(), b.ID); err != nil {
+			return err
+		}
+		if rb, ok := renderBan(b); ok {
+			if err := s.fw.DeleteBan(r.Context(), rb); err != nil {
+				s.log.Warn("lifting a ban failed; the next change or restart drops it", "list", list.Name, "address", b.Address, "err", err)
+			}
+		}
+		s.eng.Audit(r.Context(), p, "ban.remove", list.Name+" "+banLabel(b), "success", "")
+	}
+	writeJSON(w, http.StatusOK, map[string]int{"removed": len(gone)})
+	return nil
+}
+
+// pruneBans forgets the bans that ended, and those of lists st no longer
+// has.
+func (s *Server) pruneBans(ctx context.Context, st *model.State) {
+	var ids []int64
+	for _, b := range st.BanLists {
+		ids = append(ids, b.ID)
+	}
+	if _, err := s.store.PruneBans(ctx, ids); err != nil {
+		s.log.Warn("pruning bans failed", "err", err)
+	}
+}

internal/server/h_config.go

@@ -6,6 +6,7 @@ import (
 	"net/http"
 	"net/netip"
 	"sort"
+	"strconv"
 	"strings"
 
 	udiff "github.com/aymanbagabas/go-udiff"
@@ -15,6 +16,7 @@ import (
 	"housecarl/internal/engine"
 	"housecarl/internal/matrix"
 	"housecarl/internal/model"
+	"housecarl/internal/nft"
 	"housecarl/internal/store"
 )
 
@@ -106,6 +108,7 @@ func (s *Server) rulesetDiff(before, after *model.State) string {
 var kindOf = map[string]string{
 	"zones": model.KindZone, "sets": model.KindSet, "services": model.KindService, "rules": model.KindRule,
 	"forwards": model.KindForward, "snats": model.KindSNAT, "redirects": model.KindRedirect,
+	"banlists": model.KindBanList,
 }
 
 func pathKind(r *http.Request) (string, error) {
@@ -126,13 +129,16 @@ func findObject(st *model.State, kind, ref string) (model.Object, error) {
 // ---- objects
 
 func (s *Server) listObjects(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
-	if err := require(p, auth.ResConfig, false); err != nil {
-		return err
-	}
 	kind, err := pathKind(r)
 	if err != nil {
 		return err
 	}
+	if kind == model.KindBanList && !p.Can(auth.ResConfig, false) {
+		return s.listOwnBanLists(w, p)
+	}
+	if err := require(p, auth.ResConfig, false); err != nil {
+		return err
+	}
 	cur, st, draft, err := s.viewState(w, r)
 	if err != nil {
 		return err
@@ -178,13 +184,22 @@ func (s *Server) listObjects(w http.ResponseWriter, r *http.Request, p *auth.Pri
 }
 
 func (s *Server) getObject(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
-	if err := require(p, auth.ResConfig, false); err != nil {
-		return err
-	}
 	kind, err := pathKind(r)
 	if err != nil {
 		return err
 	}
+	// A program that bans into a list may read it (its grouping, times).
+	if kind == model.KindBanList && !p.Can(auth.ResConfig, false) {
+		b, err := s.banList(r, p, false)
+		if err != nil {
+			return err
+		}
+		writeJSON(w, http.StatusOK, b)
+		return nil
+	}
+	if err := require(p, auth.ResConfig, false); err != nil {
+		return err
+	}
 	_, st, _, err := s.viewState(w, r)
 	if err != nil {
 		return err
@@ -221,6 +236,8 @@ func decodeObject(kind string, data []byte) (model.Object, error) {
 		o = &model.SourceNAT{}
 	case model.KindRedirect:
 		o = &model.Redirect{}
+	case model.KindBanList:
+		o = &model.BanList{}
 	}
 	dec := json.NewDecoder(strings.NewReader(string(data)))
 	dec.DisallowUnknownFields()
@@ -272,6 +289,8 @@ func setID(o model.Object, id int64) {
 		x.ID = id
 	case *model.Redirect:
 		x.ID = id
+	case *model.BanList:
+		x.ID = id
 	}
 }
 
@@ -309,7 +328,19 @@ func (s *Server) createObject(w http.ResponseWriter, r *http.Request, p *auth.Pr
 	if _, err := decodeObject(kind, raw); err != nil {
 		return err
 	}
-	return s.change(w, r, p, model.Op{Op: model.OpCreate, Kind: kind, Data: raw})
+	op := model.Op{Op: model.OpCreate, Kind: kind, Data: raw}
+	// ?before=ID, ?after=ID or ?first=1: where it goes in its order.
+	q := r.URL.Query()
+	var place api.MoveInput
+	place.Before, _ = strconv.ParseInt(q.Get("before"), 10, 64)
+	place.After, _ = strconv.ParseInt(q.Get("after"), 10, 64)
+	if queryBool(r, "first") {
+		place.To = "start"
+	}
+	if place.Before != 0 || place.After != 0 || place.To != "" {
+		op.Place, _ = json.Marshal(place)
+	}
+	return s.change(w, r, p, op)
 }
 
 func (s *Server) updateObject(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
@@ -377,6 +408,11 @@ func usedBy(st *model.State, kind string, o model.Object) string {
 				return f.Label()
 			}
 		}
+		for _, b := range st.BanLists {
+			if has(b.Hosts) || has(b.Never) {
+				return "ban list " + b.Name
+			}
+		}
 	}
 	return ""
 }
@@ -419,6 +455,10 @@ func renameRefs(st *model.State, kind, old, now string) {
 			fix(f.PublicAddresses)
 			fix(f.HairpinSources)
 		}
+		for _, b := range st.BanLists {
+			fix(b.Hosts)
+			fix(b.Never)
+		}
 	}
 }
 
@@ -471,10 +511,39 @@ func (s *Server) zoneMatrix(w http.ResponseWriter, r *http.Request, p *auth.Prin
 	if err != nil {
 		return err
 	}
-	writeJSON(w, http.StatusOK, matrix.Build(st, matrix.Options{Connected: s.connected(st)}))
+	m := matrix.Build(st, matrix.Options{Connected: s.connected(st)})
+	if t, err := s.fw.Table(r.Context()); err == nil {
+		matrixHits(&m, t.Counters)
+	}
+	writeJSON(w, http.StatusOK, m)
 	return nil
 }
 
+// matrixHits fills in the hits of the matrix's rules, built-ins and
+// policies from the counters.
+func matrixHits(m *api.Matrix, counters map[string]nft.Counter) {
+	hits := func(marks []string) *uint64 {
+		var n uint64
+		found := false
+		for _, mk := range marks {
+			if c, ok := counters[mk]; ok {
+				n, found = n+c.Packets, true
+			}
+		}
+		if !found {
+			return nil
+		}
+		return &n
+	}
+	for i := range m.Cells {
+		c := &m.Cells[i]
+		c.Hits = hits(c.Marks)
+		for j := range c.Exceptions {
+			c.Exceptions[j].Hits = hits(c.Exceptions[j].Marks)
+		}
+	}
+}
+
 // connected maps the networks on the firewall's interfaces (link-local
 // and loopback aside) to their zones.
 func (s *Server) connected(st *model.State) map[netip.Prefix]string {
@@ -505,6 +574,10 @@ func (s *Server) ruleset(w http.ResponseWriter, r *http.Request, p *auth.Princip
 		return err
 	}
 	rs := s.fw.Render(st)
+	// In shadow mode: what would be loaded once it is off.
+	if queryBool(r, "enforcing") {
+		rs = s.fw.RenderEnforcing(st)
+	}
 	w.Header().Set("Content-Type", "text/plain; charset=utf-8")
 	w.Write([]byte(rs.Text))
 	return nil
@@ -622,3 +695,19 @@ func (s *Server) rollbackChanges(w http.ResponseWriter, r *http.Request, p *auth
 	writeJSON(w, http.StatusOK, map[string]int{"rolled_back": n})
 	return nil
 }
+
+// listOwnBanLists lists the ban lists a principal without config:read
+// may use.
+func (s *Server) listOwnBanLists(w http.ResponseWriter, p *auth.Principal) error {
+	out := []*model.BanList{}
+	for _, b := range s.eng.State().BanLists {
+		if p.CanBan(b.Name, false) {
+			out = append(out, b)
+		}
+	}
+	if len(out) == 0 && !p.Can(auth.ResBans, false) {
+		return require(p, auth.ResConfig, false)
+	}
+	writeJSON(w, http.StatusOK, out)
+	return nil
+}

internal/server/h_health.go

@@ -19,6 +19,8 @@ type health struct {
 	OK       bool   `json:"ok"`
 	Version  string `json:"version"`
 	Firewall string `json:"firewall"` // applied, unconfigured, fallback, missing
+	// Mode is enforcing, or shadow: the ruleset blocks nothing.
+	Mode string `json:"mode"`
 	// Probation: changes waiting to be confirmed, rolled back otherwise.
 	Probation int `json:"probation"`
 	// Warnings are the codes of the machine's problems (nftables_service).
@@ -49,7 +51,7 @@ func (s *Server) healthz(w http.ResponseWriter, r *http.Request) {
 }
 
 func (s *Server) checkHealth(r *http.Request) health {
-	out := health{Version: version.String(), Warnings: []string{}}
+	out := health{Version: version.String(), Warnings: []string{}, Mode: s.mode()}
 	t, err := s.fw.Table(r.Context())
 	switch {
 	case err != nil && s.eng.State().Empty():
@@ -63,7 +65,12 @@ func (s *Server) checkHealth(r *http.Request) health {
 	}
 	pr := s.probationStatus()
 	out.Probation = len(pr.Pending) + pr.Takeovers
-	for _, p := range s.hostProblems(r.Context(), nil) {
+	// The other tables matter in shadow mode (shadow_unprotected).
+	var ruleset []byte
+	if s.cfg.Shadow {
+		ruleset, _ = s.fw.ListRuleset(r.Context())
+	}
+	for _, p := range s.hostProblems(r.Context(), ruleset) {
 		out.Warnings = append(out.Warnings, p.Code)
 	}
 	return out

internal/server/h_import.go

@@ -25,7 +25,7 @@ func (s *Server) importNft(w http.ResponseWriter, r *http.Request, p *auth.Princ
 	if err := decodeJSON(r, &in); err != nil {
 		return err
 	}
-	res, err := importer.Import(importer.Input{JSON: in.JSON, Text: in.Text, Defines: in.Defines, Table: in.Table})
+	res, err := importer.Import(importer.Input{JSON: in.JSON, Text: in.Text, Defines: in.Defines, Table: in.Table, Zones: in.Zones})
 	if err != nil {
 		return badRequest("%v", err)
 	}
@@ -53,7 +53,8 @@ func (s *Server) importNft(w http.ResponseWriter, r *http.Request, p *auth.Princ
 	s.afterChange(r.Context(), p, er)
 	if er.Applied {
 		// Until it is taken over, the old table still applies too: the
-		// status warns (takeover_pending).
+		// status warns (takeover_pending; not in shadow mode, where that
+		// is the point).
 		s.store.SetMeta(r.Context(), importedTableKey, res.Table)
 	}
 	out.MutationResponse = api.MutationResponse{Applied: er.Applied, DryRun: er.DryRun, HistoryID: er.HistoryID, Changes: er.Changes, Warnings: er.Warnings}
@@ -117,6 +118,9 @@ func (s *Server) takeover(w http.ResponseWriter, r *http.Request, p *auth.Princi
 	if err := notStaged(r, "a takeover"); err != nil {
 		return err
 	}
+	if err := s.refuseInShadow("delete other tables (the firewall in force)"); err != nil {
+		return err
+	}
 	var in api.TakeoverInput
 	if err := decodeJSON(r, &in); err != nil {
 		return err

internal/server/h_status.go

@@ -27,7 +27,10 @@ func (s *Server) status(w http.ResponseWriter, r *http.Request, p *auth.Principa
 	}
 	st := s.eng.State()
 	out := api.Status{Build: version.Info(), Started: s.started, Problems: s.eng.Problems(), Probation: s.probationStatus(),
-		Interfaces: s.hostInterfaces(st), Counts: map[string]int{}, OtherTables: []string{}}
+		Interfaces: s.hostInterfaces(st), Counts: map[string]int{}, OtherTables: []string{}, Mode: s.mode()}
+	if s.cfg.Shadow {
+		out.ShadowPosition = s.cfg.ShadowPosition
+	}
 	if out.Problems == nil {
 		out.Problems = []validate.Problem{}
 	}
@@ -46,10 +49,11 @@ func (s *Server) status(w http.ResponseWriter, r *http.Request, p *auth.Principa
 	} else {
 		out.Applied = true
 	}
-	if data, err := s.fw.ListRuleset(r.Context()); err == nil {
+	data, err := s.fw.ListRuleset(r.Context())
+	if err == nil {
 		out.OtherTables = otherTables(data)
 	}
-	out.Problems = append(out.Problems, s.hostProblems(r.Context(), out.OtherTables)...)
+	out.Problems = append(out.Problems, s.hostProblems(r.Context(), data)...)
 	out.Problems = append(out.Problems, s.statsProblems()...)
 	writeJSON(w, http.StatusOK, out)
 	return nil
@@ -57,6 +61,20 @@ func (s *Server) status(w http.ResponseWriter, r *http.Request, p *auth.Principa
 
 const importedTableKey = "imported_table"
 
+// shadowProblems are shadow mode's warnings about the machine.
+func (s *Server) shadowProblems(ruleset []byte) []validate.Problem {
+	var out []validate.Problem
+	if ruleset != nil && len(inputFilters(ruleset)) == 0 {
+		out = append(out, validate.Problem{Severity: validate.Warning, Code: "shadow_unprotected", Object: "shadow mode",
+			Message: "housecarld is in shadow mode, where its ruleset blocks nothing, and no other table filters what reaches this machine (none has a filter chain on the input hook): nothing protects it; load your firewall, or leave shadow mode"})
+	}
+	for _, r := range render.ShadowLeftOut(s.eng.State()) {
+		out = append(out, validate.Problem{Severity: validate.Warning, Code: "shadow_raw_left_out", Object: r.Label(),
+			Message: "shadow mode leaves this raw rule out: it does more to packets than accept, drop or reject (NAT, setting a mark or field, a jump, a verdict map…), so its hits aren't counted and later rules see what it would have taken"})
+	}
+	return out
+}
+
 // nftablesUnitLinks are where an enabled nftables.service shows (Debian
 // enables it for sysinit.target).
 var nftablesUnitLinks = []string{
@@ -66,8 +84,15 @@ var nftablesUnitLinks = []string{
 
 // hostProblems are warnings about the machine rather than the
 // configuration: another ruleset that still loads at boot, or the table
-// an import came from, still in force beside Housecarl's.
+// an import came from, still in force beside Housecarl's. ruleset is the
-func (s *Server) hostProblems(ctx context.Context, others []string) []validate.Problem {
+// machine's (nft -j list ruleset; nil: not read). In shadow mode those
+// are expected; the warnings are that nothing else filters, and the raw
+// rules left out.
+func (s *Server) hostProblems(ctx context.Context, ruleset []byte) []validate.Problem {
+	if s.cfg.Shadow {
+		return s.shadowProblems(ruleset)
+	}
+	others := otherTables(ruleset)
 	var out []validate.Problem
 	for _, link := range s.unitLinks {
 		if _, err := os.Lstat(link); err == nil {
@@ -300,6 +325,7 @@ func (l *logRing) Latest(n int, after time.Time, keep func(api.LogEntry) bool) [
 // LogPacket records a packet the firewall logged (internal/nflog calls
 // it).
 func (s *Server) LogPacket(e api.LogEntry) {
+	e.Shadow = s.cfg.Shadow
 	s.logs.Add(e)
 	if s.stats != nil {
 		s.stats.Packet(e)

internal/server/h_traffic.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 	"net/http"
+	"net/netip"
 	"sort"
 	"strconv"
 	"strings"
@@ -88,12 +89,92 @@ func (s *Server) listConnections(w http.ResponseWriter, r *http.Request, p *auth
 	return nil
 }
 
+// lookupNames finds the names of the inside addresses asked about
+// (docs/configuration.md#names), waiting at most a second for lookups.
+func (s *Server) lookupNames(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	if err := require(p, auth.ResLog, false); err != nil {
+		return err
+	}
+	var in api.NamesInput
+	if err := decodeJSON(r, &in); err != nil {
+		return err
+	}
+	if len(in.Addresses) > maxNames {
+		return badRequest("ask for at most %d addresses at once", maxNames)
+	}
+	out := api.Names{Enabled: s.cfg.Names != nil, Names: map[string]string{}}
+	if s.cfg.Names == nil {
+		writeJSON(w, http.StatusOK, out)
+		return nil
+	}
+	inside := s.insideNetworks(s.eng.State())
+	asked := map[netip.Addr][]string{}
+	var addrs []netip.Addr
+	for _, a := range in.Addresses {
+		ad, err := netip.ParseAddr(a)
+		if err != nil || !inside(ad.Unmap()) {
+			continue
+		}
+		ad = ad.Unmap()
+		if _, ok := asked[ad]; !ok {
+			addrs = append(addrs, ad)
+		}
+		asked[ad] = append(asked[ad], a)
+	}
+	for a, n := range s.cfg.Names.Names(r.Context(), addrs, time.Second) {
+		for _, as := range asked[a] {
+			out.Names[as] = n
+		}
+	}
+	writeJSON(w, http.StatusOK, out)
+	return nil
+}
+
+// maxNames is the most addresses one POST /names may ask about.
+const maxNames = 2000
+
+// insideNetworks tells inside addresses: in a network connected to an
+// inside zone's interface, or private and not in an external zone's.
+func (s *Server) insideNetworks(st *model.State) func(netip.Addr) bool {
+	external := map[string]bool{}
+	for _, z := range st.Zones {
+		external[z.Name] = z.External
+	}
+	var in, out []netip.Prefix
+	for p, zone := range s.connected(st) {
+		if external[zone] {
+			out = append(out, p)
+		} else {
+			in = append(in, p)
+		}
+	}
+	return func(a netip.Addr) bool {
+		if a.IsLoopback() || a.IsUnspecified() || a.IsMulticast() || a.IsLinkLocalUnicast() {
+			return false
+		}
+		for _, p := range in {
+			if p.Contains(a) {
+				return true
+			}
+		}
+		for _, p := range out {
+			if p.Contains(a) {
+				return false
+			}
+		}
+		return a.IsPrivate()
+	}
+}
+
 // closeConnections deletes tracking entries: the rules check those
 // connections again with their next packet.
 func (s *Server) closeConnections(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
 	if err := require(p, auth.ResConfig, true); err != nil {
 		return err
 	}
+	if err := s.refuseInShadow("close connections"); err != nil {
+		return err
+	}
 	ct, err := s.conntrack()
 	if err != nil {
 		return err
@@ -227,6 +308,12 @@ func (s *Server) traceStatus() api.TraceStatus {
 			p.Steps[j].Label = markLabel(st, p.Steps[j])
 		}
 		p.DecidedBy.Label = markLabel(st, p.DecidedBy)
+		if p.WouldDrop != nil {
+			// A copy: the tracer still holds the step.
+			wd := *p.WouldDrop
+			wd.Label = markLabel(st, wd)
+			p.WouldDrop = &wd
+		}
 	}
 	return out
 }

internal/server/ops.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"net/http"
 	"sort"
+	"strconv"
 	"strings"
 
 	"housecarl/internal/api"
@@ -63,7 +64,16 @@ func createOp(st *model.State, op *model.Op) (int64, error) {
 		*pos = lastPosition(st, op.Kind, chainOf(o))
 	}
 	data, _ := json.Marshal(o)
-	return op.ID, st.Put(op.Kind, op.ID, data)
+	if err := st.Put(op.Kind, op.ID, data); err != nil {
+		return 0, err
+	}
+	// Placed in the same change, so it is never in force anywhere else.
+	if len(op.Place) > 0 {
+		if _, err := moveOp(st, &model.Op{Op: model.OpMove, Kind: op.Kind, Ref: strconv.FormatInt(op.ID, 10), Data: op.Place}); err != nil {
+			return 0, err
+		}
+	}
+	return op.ID, nil
 }
 
 func updateOp(st *model.State, op *model.Op) (int64, error) {
@@ -94,6 +104,9 @@ func updateOp(st *model.State, op *model.Op) (int64, error) {
 		return 0, err
 	}
 	setID(next, id)
+	if old, now := o.Label(), next.Label(); old != now && op.Kind == model.KindBanList {
+		return 0, conflict("ban list %s can't be renamed: the API tokens and programs (Guard, fail2ban) that ban into it go by its name; make a new list instead", old)
+	}
 	if old, now := o.Label(), next.Label(); old != now && isNamed(op.Kind) {
 		if set, ok := o.(*model.AddressSet); ok && set.Dynamic {
 			return 0, conflict("set %s is dynamic: its entries, and the API tokens and programs that fill it, go by its name; make a new set instead", old)

internal/server/probation.go

@@ -81,7 +81,8 @@ func (s *Server) startProbation(ctx context.Context, p *auth.Principal, historyI
 	pr := &s.probation
 	pr.mu.Lock()
 	defer pr.mu.Unlock()
-	if p.Method == "socket" && len(pr.pending) == 0 && len(pr.restores) == 0 {
+	// In shadow mode a change can't cut anyone off: it blocks nothing.
+	if s.cfg.Shadow || p.Method == "socket" && len(pr.pending) == 0 && len(pr.restores) == 0 {
 		s.saveBoot()
 		return
 	}

internal/server/quota.go

@@ -4,8 +4,11 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"math"
 	"net/http"
+	"net/netip"
 	"sort"
+	"strings"
 	"time"
 
 	"housecarl/internal/api"
@@ -237,7 +240,8 @@ func (s *Server) listQuotas(w http.ResponseWriter, r *http.Request, p *auth.Prin
 	return nil
 }
 
-// resetQuota starts a rule's quota again now.
+// resetQuota starts a rule's quota again now: all of it, or one
+// address's; or gives some of what it used back, until its next reset.
 func (s *Server) resetQuota(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
 	if err := require(p, auth.ResConfig, true); err != nil {
 		return err
@@ -250,6 +254,14 @@ func (s *Server) resetQuota(w http.ResponseWriter, r *http.Request, p *auth.Prin
 	if rule.Quota == nil {
 		return badRequest("%s has no quota", rule.Label())
 	}
+	var in api.QuotaReset
+	if err := decodeOptional(r, &in); err != nil {
+		return err
+	}
+	if in.Give < 0 {
+		return badRequest("give a number of bytes to give back, 1 or more")
+	}
+	if in.Address == "" && in.Give == 0 {
 		if err := s.fw.ResetQuota(r.Context(), rule.ID, rule.Quota.PerAddress() != ""); err != nil {
 			return err
 		}
@@ -261,3 +273,84 @@ func (s *Server) resetQuota(w http.ResponseWriter, r *http.Request, p *auth.Prin
 		writeJSON(w, http.StatusOK, map[string]any{"reset": rule.Label()})
 		return nil
 	}
+	name := render.QuotaName(rule.ID)
+	adj := nft.QuotaAdjust{Name: name, Give: in.Give}
+	if in.Address != "" {
+		if rule.Quota.PerAddress() == "" {
+			return badRequest("%s has one quota for everything it matches, not one per address: leave out the address", rule.Label())
+		}
+		key, err := quotaKey(rule, in.Address)
+		if err != nil {
+			return err
+		}
+		if !s.quotaHas(r.Context(), name, key) {
+			return notFound("%s hasn't used the quota of %s since its last reset", in.Address, rule.Label())
+		}
+		adj.Addr = key
+	}
+	if adj.Give == 0 {
+		adj.Give = math.MaxInt64 // all of it: the address starts again
+	}
+	// The next apply writes the quota's new usage into the ruleset.
+	s.fw.AdjustQuota(adj)
+	if err := s.eng.ApplyCurrent(r.Context()); err != nil {
+		return fmt.Errorf("putting the ruleset in force with the quota's new usage: %w", err)
+	}
+	s.SaveQuotaUsage(r.Context())
+	detail, out := "", map[string]any{}
+	if in.Give > 0 {
+		detail = fmt.Sprintf("gave back %d bytes", in.Give)
+		out["given"] = in.Give
+	}
+	if in.Address != "" {
+		detail = strings.TrimSpace(detail + " " + in.Address)
+		out["address"] = in.Address
+	}
+	action := "quota.reset"
+	if in.Give > 0 {
+		action = "quota.give"
+	}
+	s.eng.Audit(r.Context(), p, action, rule.Label(), "success", detail)
+	out["reset"] = rule.Label()
+	writeJSON(w, http.StatusOK, out)
+	return nil
+}
+
+// quotaKey is the entry of a per-address quota an address counts under:
+// the address, or its group's first address (10.0.10.0 for a /24; an
+// IPv6 address's /64).
+func quotaKey(rule *model.Rule, s string) (string, error) {
+	var a netip.Addr
+	if p, err := netip.ParsePrefix(s); err == nil {
+		a = p.Addr()
+	} else if a, err = netip.ParseAddr(s); err != nil {
+		return "", badRequest("%q isn't an address", s)
+	}
+	a = a.Unmap()
+	v4, v6 := rule.Groups()
+	bits := v4
+	if a.Is6() {
+		bits = v6
+	}
+	p, err := a.Prefix(bits)
+	if err != nil {
+		return "", badRequest("%q: %v", s, err)
+	}
+	return p.Addr().String(), nil
+}
+
+// quotaHas reports whether a per-address quota has an entry for key.
+func (s *Server) quotaHas(ctx context.Context, name, key string) bool {
+	t, err := s.fw.Table(ctx)
+	if err != nil {
+		return false
+	}
+	for _, set := range []string{name + "_v4", name + "_v6"} {
+		for _, e := range t.QuotaSets[set] {
+			if e.Addr == key {
+				return true
+			}
+		}
+	}
+	return false
+}

internal/server/routes.go

@@ -38,9 +38,15 @@ func (s *Server) routes() {
 	m.HandleFunc("GET "+p+"/sets/{ref}/elements", s.authed(s.listElements))
 	m.HandleFunc("POST "+p+"/sets/{ref}/elements", s.authed(s.addElement))
 	m.HandleFunc("DELETE "+p+"/sets/{ref}/elements/{addr}", s.authed(s.deleteElement))
+	m.HandleFunc("GET "+p+"/bans", s.authed(s.listBans))
+	m.HandleFunc("GET "+p+"/banlists/{ref}/bans", s.authed(s.listBans))
+	m.HandleFunc("POST "+p+"/banlists/{ref}/bans", s.authed(s.addBan))
+	m.HandleFunc("DELETE "+p+"/banlists/{ref}/bans", s.authed(s.deleteBans))
+	m.HandleFunc("DELETE "+p+"/banlists/{ref}/bans/{id}", s.authed(s.deleteBans))
 	m.HandleFunc("GET "+p+"/quotas", s.authed(s.listQuotas))
 	m.HandleFunc("GET "+p+"/connections", s.authed(s.listConnections))
 	m.HandleFunc("POST "+p+"/connections/close", s.authed(s.closeConnections))
+	m.HandleFunc("POST "+p+"/names", s.authed(s.lookupNames))
 	m.HandleFunc("GET "+p+"/trace", s.authed(s.getTrace))
 	m.HandleFunc("POST "+p+"/trace", s.authed(s.startTrace))
 	m.HandleFunc("DELETE "+p+"/trace", s.authed(s.stopTrace))

internal/server/routes_cfg.go

@@ -8,6 +8,29 @@ import (
 
 // otherTables lists the tables in an `nft -j list ruleset` other than
 // Housecarl's ("inet main").
+// inputFilters lists the tables other than Housecarl's with a filter
+// chain on the input hook: what filters the machine in shadow mode.
+func inputFilters(data []byte) []string {
+	var doc struct {
+		Nftables []map[string]json.RawMessage `json:"nftables"`
+	}
+	out := []string{}
+	if json.Unmarshal(data, &doc) != nil {
+		return out
+	}
+	for _, item := range doc.Nftables {
+		raw, ok := item["chain"]
+		if !ok {
+			continue
+		}
+		var c struct{ Family, Table, Type, Hook string }
+		if json.Unmarshal(raw, &c) == nil && c.Type == "filter" && c.Hook == "input" && !(c.Family == "inet" && (c.Table == "housecarl" || c.Table == trace.Table)) {
+			out = append(out, c.Family+" "+c.Table)
+		}
+	}
+	return out
+}
+
 func otherTables(data []byte) []string {
 	var doc struct {
 		Nftables []map[string]json.RawMessage `json:"nftables"`

internal/server/server.go

@@ -18,10 +18,12 @@ import (
 	"syscall"
 	"time"
 
+	"housecarl/internal/api"
 	"housecarl/internal/auth"
 	"housecarl/internal/certs"
 	"housecarl/internal/engine"
 	"housecarl/internal/model"
+	"housecarl/internal/names"
 	"housecarl/internal/nft"
 	"housecarl/internal/render"
 	"housecarl/internal/stats"
@@ -45,6 +47,13 @@ type Config struct {
 	// ACME keeps the certificate housecarld obtains itself (tls.acme; nil
 	// when off).
 	ACME *certs.Manager
+	// Shadow is shadow mode (docs/shadow.md): nothing goes on probation,
+	// takeovers and closing connections are refused. ShadowPosition is
+	// after or before, for the status.
+	Shadow         bool
+	ShadowPosition string
+	// Names looks up inside addresses' names (nil: names are off).
+	Names *names.Resolver
 }
 
 // Interfaces lists the host's interfaces with their addresses, and
@@ -58,9 +67,13 @@ type Interfaces interface {
 // engine (internal/nft.Firewall).
 type Firewall interface {
 	Render(st *model.State) *render.Ruleset
+	// RenderEnforcing is the ruleset outside shadow mode.
+	RenderEnforcing(st *model.State) *render.Ruleset
 	Table(ctx context.Context) (*nft.Table, error)
 	AddElement(ctx context.Context, set, addr string, timeout int) error
 	DeleteElement(ctx context.Context, set, addr string) error
+	AddBan(ctx context.Context, b render.Ban) error
+	DeleteBan(ctx context.Context, b render.Ban) error
 	ListRuleset(ctx context.Context) ([]byte, error)
 	DeleteTable(ctx context.Context, family, name string) error
 	TableText(ctx context.Context, family, name string) (string, error)
@@ -68,6 +81,8 @@ type Firewall interface {
 	SaveBoot(st *model.State) error
 	// ResetQuota starts a rule's quota again (perAddress: its sets).
 	ResetQuota(ctx context.Context, ruleID int64, perAddress bool) error
+	// AdjustQuota has the next apply change what a quota used.
+	AdjustQuota(a nft.QuotaAdjust)
 	// Trace follows packets (internal/trace) until ctx ends.
 	Trace(ctx context.Context, script string, line func(string)) error
 }
@@ -128,9 +143,36 @@ func New(cfg Config, eng *engine.Engine, fw Firewall, ifaces Interfaces, log *sl
 	s := &Server{cfg: cfg, eng: eng, fw: fw, ifaces: ifaces, store: eng.Store(), log: log, mux: http.NewServeMux(), limiter: newRateLimiter(),
 		web: web, started: time.Now(), logs: newLogRing(2000), nowFn: time.Now, unitLinks: nftablesUnitLinks}
 	s.routes()
+	// What the quotas used is saved after every change too, not only every
+	// five minutes: a change that counts a quota differently converts its
+	// usage, and a restart must not bring back the old kind.
+	prev := eng.OnChange
+	eng.OnChange = func(st *model.State) {
+		if prev != nil {
+			prev(st)
+		}
+		s.SaveQuotaUsage(context.Background())
+		s.pruneBans(context.Background(), st) // the engine is locked: not s.eng.State()
+	}
 	return s
 }
 
+// mode is housecarld's mode: enforcing, or shadow.
+func (s *Server) mode() string {
+	if s.cfg.Shadow {
+		return api.ModeShadow
+	}
+	return api.ModeEnforcing
+}
+
+// refuseInShadow refuses what would change traffic in shadow mode.
+func (s *Server) refuseInShadow(what string) error {
+	if !s.cfg.Shadow {
+		return nil
+	}
+	return conflict("housecarld is in shadow mode, where it changes nothing about traffic, so it doesn't %s; leave shadow mode first (shadow.enabled in housecarl.yaml, docs/shadow.md)", what)
+}
+
 // Handler returns the HTTP handler with all middleware.
 func (s *Server) Handler() http.Handler {
 	return s.recoverer(s.securityHeaders(s.logRequests(s.mux)))
@@ -285,6 +327,7 @@ func (s *Server) housekeeping(ctx context.Context) {
 				s.log.Warn("purging sessions failed", "err", err)
 			}
 			s.limiter.cleanup()
+			s.pruneBans(ctx, s.eng.State())
 			if s.certs != nil {
 				s.certs.reloadIfChanged()
 			}

internal/server/server_test.go

@@ -7,11 +7,13 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"math"
 	"net/http"
 	"net/http/cookiejar"
 	"net/http/httptest"
 	"net/netip"
 	"os"
+	"reflect"
 	"strings"
 	"sync"
 	"testing"
@@ -21,6 +23,7 @@ import (
 	"housecarl/internal/auth"
 	"housecarl/internal/engine"
 	"housecarl/internal/model"
+	"housecarl/internal/names"
 	"housecarl/internal/nft"
 	"housecarl/internal/render"
 	"housecarl/internal/stats"
@@ -41,9 +44,29 @@ type fakeFW struct {
 	traced     string
 	traceLines []string
 	fallback   bool // the boot fallback is in force
+	shadow     bool
+	ruleset    string // nft -j list ruleset ("": a default)
+	quotas     nft.QuotaUsage
+	adjusted   []nft.QuotaAdjust
+	bans       map[string]render.Ban // by element (set and key)
 }
 
 func (f *fakeFW) Check(_ context.Context, st *model.State) error { return nil }
+func (f *fakeFW) AddBan(_ context.Context, b render.Ban) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if f.bans == nil {
+		f.bans = map[string]render.Ban{}
+	}
+	f.bans[render.BanSet(b.List, b.Addr.Addr().Is6(), b.Proto != "")+" "+render.BanElement(b)] = b
+	return nil
+}
+func (f *fakeFW) DeleteBan(_ context.Context, b render.Ban) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	delete(f.bans, render.BanSet(b.List, b.Addr.Addr().Is6(), b.Proto != "")+" "+render.BanElement(b))
+	return nil
+}
 func (f *fakeFW) Apply(_ context.Context, st *model.State) error {
 	f.mu.Lock()
 	defer f.mu.Unlock()
@@ -58,14 +81,24 @@ func (f *fakeFW) last() *model.State {
 	defer f.mu.Unlock()
 	return f.applied[len(f.applied)-1]
 }
-func (f *fakeFW) Render(st *model.State) *render.Ruleset { return render.Render(st, render.Options{}) }
+func (f *fakeFW) Render(st *model.State) *render.Ruleset {
+	return render.Render(st, render.Options{Shadow: f.shadow})
+}
+func (f *fakeFW) RenderEnforcing(st *model.State) *render.Ruleset {
+	return render.Render(st, render.Options{})
+}
 func (f *fakeFW) Table(context.Context) (*nft.Table, error) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
 	if len(f.applied) == 0 {
 		return nil, errors.New("No such file or directory: list table inet housecarl")
 	}
-	return &nft.Table{Counters: map[string]nft.Counter{"r1": {Packets: 3, Bytes: 180}}, Elements: f.elements, Fallback: f.fallback}, nil
+	return &nft.Table{Counters: map[string]nft.Counter{"r1": {Packets: 3, Bytes: 180}}, Elements: f.elements, Fallback: f.fallback, QuotaUsage: f.quotas}, nil
+}
+func (f *fakeFW) AdjustQuota(a nft.QuotaAdjust) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.adjusted = append(f.adjusted, a)
 }
 func (f *fakeFW) AddElement(_ context.Context, set, addr string, timeout int) error {
 	f.mu.Lock()
@@ -86,6 +119,9 @@ func (f *fakeFW) DeleteElement(_ context.Context, set, addr string) error {
 	return nil
 }
 func (f *fakeFW) ListRuleset(context.Context) ([]byte, error) {
+	if f.ruleset != "" {
+		return []byte(f.ruleset), nil
+	}
 	return []byte(`{"nftables":[{"table":{"family":"inet","name":"main"}},{"table":{"family":"inet","name":"housecarl"}}]}`), nil
 }
 func (f *fakeFW) DeleteTable(_ context.Context, fam, name string) error {
@@ -142,6 +178,11 @@ type env struct {
 }
 
 func newEnv(t *testing.T) *env {
+	t.Helper()
+	return newEnvWith(t, Config{})
+}
+
+func newEnvWith(t *testing.T, cfg Config) *env {
 	t.Helper()
 	ctx := context.Background()
 	st, err := store.Open(":memory:")
@@ -159,10 +200,10 @@ func newEnv(t *testing.T) *env {
 	if err != nil {
 		t.Fatal(err)
 	}
-	fw := &fakeFW{elements: map[string][]render.Element{}}
+	fw := &fakeFW{elements: map[string][]render.Element{}, shadow: cfg.Shadow}
 	eng.Firewall = fw
 	ifs := hostIfaces{"wan": {netip.MustParsePrefix("217.155.8.194/32")}, "eth1": {netip.MustParsePrefix("10.0.10.1/24")}}
-	srv := New(Config{}, eng, fw, ifs, nil, nil)
+	srv := New(cfg, eng, fw, ifs, nil, nil)
 	ts := httptest.NewServer(srv.Handler())
 	t.Cleanup(ts.Close)
 	return &env{t: t, ts: ts, srv: srv, eng: eng, fw: fw, token: tok}
@@ -733,9 +774,53 @@ func TestQuotaResets(t *testing.T) {
 	if len(qs) != 1 || qs[0].NextReset == nil || qs[0].NextReset.Day() != 21 {
 		t.Errorf("quotas %+v", qs)
 	}
-	if code := e.call(e.token, "POST", "/rules/"+itoa(created.ID)+"/reset-quota", nil, nil); code != 200 || len(e.fw.resets) != 2 {
+	reset := "/rules/" + itoa(created.ID) + "/reset-quota"
+	if code := e.call(e.token, "POST", reset, nil, nil); code != 200 || len(e.fw.resets) != 2 {
 		t.Errorf("reset by hand: %d %v", code, e.fw.resets)
 	}
+
+	// One address started again, or some given back: the next apply
+	// (made at once) writes the new usage.
+	name := render.QuotaName(created.ID)
+	e.fw.quotas = nft.QuotaUsage{QuotaSets: map[string][]render.Element{name + "_v4": {{Addr: "10.0.10.23", Used: 6 << 30}},
+		name + "_v6": {{Addr: "2001:db8:1:2::", Used: 1 << 30}}}}
+	applies := len(e.fw.applied)
+	if code := e.call(e.token, "POST", reset, api.QuotaReset{Address: "10.0.10.23"}, nil); code != 200 {
+		t.Errorf("reset one address: %d", code)
+	}
+	if code := e.call(e.token, "POST", reset, api.QuotaReset{Address: "2001:db8:1:2:aaaa::7", Give: 2 << 30}, nil); code != 200 {
+		t.Errorf("give back to an IPv6 address (its /64): %d", code)
+	}
+	if code := e.call(e.token, "POST", reset, api.QuotaReset{Give: 1 << 30}, nil); code != 200 {
+		t.Errorf("give back to everyone: %d", code)
+	}
+	want3 := []nft.QuotaAdjust{{Name: name, Addr: "10.0.10.23", Give: math.MaxInt64}, {Name: name, Addr: "2001:db8:1:2::", Give: 2 << 30}, {Name: name, Give: 1 << 30}}
+	if !reflect.DeepEqual(e.fw.adjusted, want3) || len(e.fw.applied) != applies+3 || len(e.fw.resets) != 2 {
+		t.Errorf("adjusted %+v, applied %d more, resets %v", e.fw.adjusted, len(e.fw.applied)-applies, e.fw.resets)
+	}
+	for _, c := range []struct {
+		in   api.QuotaReset
+		code int
+	}{{api.QuotaReset{Address: "10.0.10.99"}, 404}, {api.QuotaReset{Address: "nonsense"}, 400}, {api.QuotaReset{Give: -5}, 400}} {
+		if code := e.call(e.token, "POST", reset, c.in, nil); code != c.code {
+			t.Errorf("%+v: %d, want %d", c.in, code, c.code)
+		}
+	}
+}
+
+// TestQuotaUsageSavedOnChange: a change saves what the quotas used at
+// once (a restart must not bring back usage counted the old way).
+func TestQuotaUsageSavedOnChange(t *testing.T) {
+	e := newEnv(t)
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "lan", Interfaces: []string{"eth1"}}, nil)
+	e.fw.mu.Lock()
+	e.fw.quotas = nft.QuotaUsage{Quotas: map[string]int64{"quota_r7": 4242}}
+	e.fw.mu.Unlock()
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "dmz", Interfaces: []string{"eth2"}}, nil)
+	v, err := e.eng.Store().GetMeta(context.Background(), QuotaUsageKey)
+	if err != nil || !strings.Contains(v, `"quota_r7":4242`) {
+		t.Errorf("saved usage %q %v", v, err)
+	}
 }
 
 // TestAutoConfirm: the client that made a change confirms it by itself:
@@ -876,6 +961,13 @@ func TestMatrixEndpoint(t *testing.T) {
 	if c := cell("lan", "wan"); c.Verdict != "allowed" {
 		t.Errorf("lan → wan with the changeset: %+v", c)
 	}
+	// Hits: summed over an object's counters; none without a counter.
+	hm := api.Matrix{Cells: []api.MatrixCell{{Marks: []string{"p-forward"}, Exceptions: []api.MatrixException{
+		{Marks: []string{"b-antispoof", "b-antispoof6"}}, {Marks: []string{"r9"}}}}}}
+	matrixHits(&hm, map[string]nft.Counter{"p-forward": {Packets: 4}, "b-antispoof": {Packets: 2}, "b-antispoof6": {Packets: 3}})
+	if c := hm.Cells[0]; c.Hits == nil || *c.Hits != 4 || c.Exceptions[0].Hits == nil || *c.Exceptions[0].Hits != 5 || c.Exceptions[1].Hits != nil {
+		t.Errorf("hits: %+v", c)
+	}
 }
 
 // TestHostProblems: an enabled nftables.service, and an imported table
@@ -960,6 +1052,41 @@ func TestConnections(t *testing.T) {
 	}
 }
 
+// TestNames: names only for inside addresses (an inside zone's network,
+// or private outside the external zones'), under the address as asked.
+func TestNames(t *testing.T) {
+	var asked []string
+	var mu sync.Mutex
+	e := newEnvWith(t, Config{Names: names.NewFunc(func(_ context.Context, a string) ([]string, error) {
+		mu.Lock()
+		asked = append(asked, a)
+		mu.Unlock()
+		return []string{"host-" + strings.ReplaceAll(a, ".", "-") + ".example.net."}, nil
+	}, time.Hour)})
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "wan", Interfaces: []string{"wan"}, External: true}, nil)
+	e.call(e.token, "POST", "/zones", model.Zone{Name: "lan", Interfaces: []string{"eth1"}}, nil)
+	var out api.Names
+	e.call(e.token, "POST", "/names", api.NamesInput{Addresses: []string{"10.0.10.5", "::ffff:10.0.10.5", "192.168.7.7", "217.155.8.194", "8.8.8.8", "fe80::1", "nonsense"}}, &out)
+	want := map[string]string{"10.0.10.5": "host-10-0-10-5.example.net", "::ffff:10.0.10.5": "host-10-0-10-5.example.net", "192.168.7.7": "host-192-168-7-7.example.net"}
+	if !out.Enabled || !reflect.DeepEqual(out.Names, want) {
+		t.Errorf("names: %+v", out)
+	}
+	if len(asked) != 2 {
+		t.Errorf("looked up %v, want the two inside addresses once each", asked)
+	}
+	var tok api.Token
+	e.call(e.token, "POST", "/me/tokens", api.TokenInput{Name: "monitor", Role: "log"}, &tok)
+	var one, none api.Names
+	if code := e.call(tok.Secret, "POST", "/names", api.NamesInput{Addresses: []string{"10.0.10.5"}}, &one); code != 200 || len(one.Names) != 1 {
+		t.Errorf("with log:read: %d %+v", code, one)
+	}
+	off := newEnv(t)
+	off.call(off.token, "POST", "/names", api.NamesInput{Addresses: []string{"10.0.10.5"}}, &none)
+	if none.Enabled || len(none.Names) != 0 {
+		t.Errorf("names off: %+v", none)
+	}
+}
+
 // TestTrace: a trace runs with the filter's table, collects packets named
 // after Housecarl's objects, and stops when asked; one at a time.
 func TestTrace(t *testing.T) {
@@ -1060,3 +1187,151 @@ func TestStatsEndpoints(t *testing.T) {
 		}
 	}
 }
+
+// TestShadowMode: in shadow mode changes aren't on probation, what would
+// change traffic is refused, and the status, health and log say so.
+func TestShadowMode(t *testing.T) {
+	e := newEnvWith(t, Config{Shadow: true, ShadowPosition: "after"})
+	e.srv.SetConntrack(&fakeConns{})
+	link := t.TempDir() + "/nftables.service"
+	os.Symlink("/usr/lib/systemd/system/nftables.service", link)
+	e.srv.unitLinks = []string{link}
+	e.srv.store.SetMeta(context.Background(), importedTableKey, "inet main")
+	var mr api.MutationResponse
+	if code := e.call(e.token, "POST", "/zones", model.Zone{Name: "wan", Interfaces: []string{"wan"}, External: true}, &mr); code != 201 || mr.ConfirmToken != "" || mr.ConfirmBy != nil {
+		t.Errorf("a change: %d %+v", code, mr)
+	}
+	var pr api.Probation
+	e.call(e.token, "GET", "/probation", nil, &pr)
+	if len(pr.Pending) != 0 || e.fw.boot == nil {
+		t.Errorf("on probation in shadow mode: %+v (saved for the boot: %v)", pr, e.fw.boot != nil)
+	}
+	var er api.ErrorBody
+	if code := e.call(e.token, "POST", "/takeover", api.TakeoverInput{Tables: []string{"inet main"}}, &er); code != 409 || !strings.Contains(er.Error.Message, "shadow mode") || len(e.fw.deleted) != 0 {
+		t.Errorf("takeover: %d %+v %v", code, er, e.fw.deleted)
+	}
+	if code := e.call(e.token, "POST", "/connections/close", api.CloseConnections{Search: "tcp"}, &er); code != 409 {
+		t.Errorf("closing connections: %d", code)
+	}
+	if code := e.call(e.token, "POST", "/rules", model.Rule{Chain: model.ChainInput, Raw: "meta mark set 1"}, &er); code != 201 {
+		t.Fatalf("a raw rule: %d %+v", code, er)
+	}
+	var st api.Status
+	e.call(e.token, "GET", "/status", nil, &st)
+	var codes []string
+	for _, p := range st.Problems {
+		codes = append(codes, p.Code)
+	}
+	c := strings.Join(codes, " ")
+	if st.Mode != api.ModeShadow || st.ShadowPosition != "after" {
+		t.Errorf("status mode: %q %q", st.Mode, st.ShadowPosition)
+	}
+	// Expected in shadow mode: the other firewall, and its service.
+	if strings.Contains(c, "nftables_service") || strings.Contains(c, "takeover_pending") || !strings.Contains(c, "shadow_unprotected") || !strings.Contains(c, "shadow_raw_left_out") {
+		t.Errorf("warnings: %s", c)
+	}
+	e.fw.ruleset = `{"nftables":[{"chain":{"family":"inet","table":"main","name":"input","type":"filter","hook":"input","prio":0,"policy":"drop"}}]}`
+	e.call(e.token, "GET", "/status", nil, &st)
+	for _, p := range st.Problems {
+		if p.Code == "shadow_unprotected" {
+			t.Error("unprotected while inet main filters")
+		}
+	}
+	var h health
+	resp, err := http.Get(e.ts.URL + "/healthz")
+	if err != nil {
+		t.Fatal(err)
+	}
+	json.NewDecoder(resp.Body).Decode(&h)
+	resp.Body.Close()
+	if h.Mode != api.ModeShadow || !h.OK {
+		t.Errorf("healthz: %+v", h)
+	}
+	var who api.Whoami
+	e.call(e.token, "GET", "/auth/whoami", nil, &who)
+	if who.Mode != api.ModeShadow {
+		t.Errorf("whoami: %+v", who)
+	}
+	e.srv.LogPacket(api.LogEntry{Time: time.Now(), Prefix: "hc input policy drop: ", Verdict: "drop"})
+	var logs []api.LogEntry
+	e.call(e.token, "GET", "/log", nil, &logs)
+	if len(logs) != 1 || !logs[0].Shadow {
+		t.Errorf("log: %+v", logs)
+	}
+	shadow := e.text("/ruleset")
+	enforcing := e.text("/ruleset?enforcing=1")
+	if !strings.Contains(shadow, "Shadow mode") || strings.Contains(enforcing, "Shadow mode") {
+		t.Errorf("rulesets:\n%s\n---\n%s", shadow, enforcing)
+	}
+	// Enforcing, the same change goes on probation.
+	n := newEnv(t)
+	n.call(n.token, "POST", "/zones", model.Zone{Name: "wan", Interfaces: []string{"wan"}, External: true}, nil)
+	n.call(n.token, "GET", "/probation", nil, &pr)
+	if len(pr.Pending) != 1 {
+		t.Errorf("not on probation enforcing: %+v", pr)
+	}
+	n.call(n.token, "GET", "/status", nil, &st)
+	if st.Mode != api.ModeEnforcing {
+		t.Errorf("enforcing status mode: %q", st.Mode)
+	}
+}
+
+// text GETs a plain-text answer.
+func (e *env) text(path string) string {
+	e.t.Helper()
+	req, _ := http.NewRequest("GET", e.ts.URL+api.Prefix+path, nil)
+	req.Header.Set("Authorization", "Bearer "+e.token)
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		e.t.Fatal(err)
+	}
+	defer resp.Body.Close()
+	b, _ := io.ReadAll(resp.Body)
+	return string(b)
+}
+
+// TestCreatePlaced: a rule created ?before= another is placed in the same
+// change (one history entry), also when staged in a changeset.
+func TestCreatePlaced(t *testing.T) {
+	e := newEnv(t)
+	var a, b, c api.MutationResponse
+	e.call(e.token, "POST", "/rules", model.Rule{Chain: "input", Match: model.Match{Services: []string{"ssh"}}}, &a)
+	e.call(e.token, "POST", "/rules", model.Rule{Chain: "input", Match: model.Match{Services: []string{"dns"}}}, &b)
+	var first model.Rule
+	json.Unmarshal(a.Object, &first)
+	if code := e.call(e.token, "POST", "/rules?before="+itoa(first.ID), model.Rule{Chain: "input", Match: model.Match{Services: []string{"http"}}, Comment: "placed"}, &c); code != 201 {
+		t.Fatalf("create placed: %d %+v", code, c)
+	}
+	order := func() string {
+		var out []string
+		for _, r := range e.eng.State().Rules {
+			out = append(out, strings.Join(r.Services, ""))
+		}
+		return strings.Join(out, ",")
+	}
+	if o := order(); o != "http,ssh,dns" {
+		t.Errorf("order: %s", o)
+	}
+	var hs []model.HistoryEntry
+	e.call(e.token, "GET", "/history", nil, &hs)
+	if len(hs) != 3 || len(hs[0].Changes) != 1 || hs[0].Changes[0].Action() != "create" {
+		t.Errorf("history: %d entries, newest %+v", len(hs), hs[0])
+	}
+	// Staged, the place goes with it.
+	var cs model.Changeset
+	e.call(e.token, "POST", "/changesets", nil, &cs)
+	e.call(e.token, "POST", "/rules?first=1&changeset="+itoa(cs.ID), model.Rule{Chain: "input", Match: model.Match{Services: []string{"ntp"}}}, nil)
+	if code := e.call(e.token, "POST", "/changesets/"+itoa(cs.ID)+"/apply", nil, nil); code != 200 {
+		t.Fatalf("apply: %d", code)
+	}
+	if o := order(); o != "ntp,http,ssh,dns" {
+		t.Errorf("order after the changeset: %s", o)
+	}
+	// A bad place: refused, nothing created.
+	if code := e.call(e.token, "POST", "/rules?before=99999", model.Rule{Chain: "input"}, nil); code != 400 {
+		t.Errorf("before a rule that isn't there: %d", code)
+	}
+	if n := len(e.eng.State().Rules); n != 4 {
+		t.Errorf("%d rules", n)
+	}
+}

internal/stats/collector.go

@@ -402,7 +402,10 @@ func (c *Collector) sample(ctx context.Context, minute int64) map[string]int64 {
 			for set, es := range t.QuotaSets {
 				kind := strings.TrimSuffix(strings.TrimSuffix(set, "_v4"), "_v6")
 				for _, e := range es {
-					addTop(kind, e.Addr, c.table.of(set+"/"+e.Addr, e.Used))
+					d := c.table.of(set+"/"+e.Addr, e.Used)
+					addTop(kind, e.Addr, d)
+					// A per-address quota's series is its addresses' sum.
+					add("quota:"+strings.TrimPrefix(kind, "quota_"), d)
 				}
 			}
 			for set, es := range t.DeviceSets {

internal/stats/collector_test.go

@@ -85,6 +85,10 @@ func TestCollector(t *testing.T) {
 	h.table.Counters["r1"] = nft.Counter{Packets: 10, Bytes: 1000}
 	h.table.Counters["r2"] = nft.Counter{Packets: 100, Bytes: 9000}
 	h.table.Quotas["quota_r5"] = 1 << 20
+	if h.table.QuotaSets == nil {
+		h.table.QuotaSets = map[string][]render.Element{}
+	}
+	h.table.QuotaSets["quota_r6_v4"] = []render.Element{{Addr: "10.0.10.5", Used: 1000}}
 	h.table.DeviceSets["stats_up_v4"] = []render.Element{{Addr: "10.0.10.5", Bytes: 5000}}
 
 	store, err := OpenStore(":memory:")
@@ -104,6 +108,7 @@ func TestCollector(t *testing.T) {
 	h.table.Counters["r1"] = nft.Counter{Packets: 25, Bytes: 2500}
 	h.table.Counters["p-input"] = nft.Counter{Packets: 3, Bytes: 180} // a new counter counts whole
 	h.table.Quotas["quota_r5"] = 3 << 20
+	h.table.QuotaSets["quota_r6_v4"] = []render.Element{{Addr: "10.0.10.5", Used: 2000}, {Addr: "10.0.10.6", Used: 500}} // a new entry counts whole
 	h.table.DeviceSets["stats_up_v4"] = []render.Element{{Addr: "10.0.10.5", Bytes: 8000}}
 	h.table.DeviceSets["stats_down_v6"] = []render.Element{{Addr: "2001:db8:1:2::", Bytes: 70000}}
 	for range 2 {
@@ -142,6 +147,7 @@ func TestCollector(t *testing.T) {
 	check("drop:forward", sum("drop:")["forward"], 15)
 	check("drop:input", sum("drop:")["input"], 3)
 	check("quota:r5", sum("quota:")["r5"], 2<<20)
+	check("quota:r6 (per address: the sum)", sum("quota:")["r6"], 1500)
 
 	s, err := q.Summary(ctx, time.Hour, 10)
 	if err != nil {

internal/store/bans.go

@@ -0,0 +1,135 @@
+package store
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"time"
+
+	"housecarl/internal/model"
+)
+
+const banCols = `id, list_id, address, protocol, port, expires_at, created_at, who, source, reason`
+
+func scanBan(sc interface{ Scan(...any) error }, now time.Time) (*model.Ban, error) {
+	b := &model.Ban{}
+	var expires sql.NullString
+	var created string
+	if err := sc.Scan(&b.ID, &b.ListID, &b.Address, &b.Protocol, &b.Port, &expires, &created, &b.Who, &b.Source, &b.Reason); err != nil {
+		return nil, err
+	}
+	b.Created = parseTS(created)
+	b.Expires = parseTSPtr(expires)
+	if b.Expires != nil {
+		b.Timeout = max(int(b.Expires.Sub(now).Round(time.Second)/time.Second), 1)
+	}
+	return b, nil
+}
+
+// PutBan saves a ban, or renews the same ban (list, address, protocol
+// and port) with its new end, who and why (an empty source or reason
+// keeps the old). existing tells which; b gets
+// its ID either way.
+func (s *Store) PutBan(ctx context.Context, b *model.Ban) (existing bool, err error) {
+	now := time.Now()
+	err = s.db.QueryRowContext(ctx, `SELECT id FROM bans WHERE list_id = ? AND address = ? AND protocol = ? AND port = ?
+		AND (expires_at IS NULL OR expires_at > ?)`, b.ListID, b.Address, b.Protocol, b.Port, ts(now)).Scan(&b.ID)
+	existing = err == nil
+	if err != nil && !errors.Is(err, sql.ErrNoRows) {
+		return false, err
+	}
+	b.Created = now
+	err = s.db.QueryRowContext(ctx, `INSERT INTO bans (list_id, address, protocol, port, expires_at, created_at, who, source, reason)
+		VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
+		ON CONFLICT (list_id, address, protocol, port) DO UPDATE SET expires_at = excluded.expires_at,
+			who = excluded.who, source = COALESCE(NULLIF(excluded.source, ''), bans.source),
+			reason = COALESCE(NULLIF(excluded.reason, ''), bans.reason),
+			created_at = CASE WHEN bans.expires_at IS NULL OR bans.expires_at > excluded.created_at THEN bans.created_at ELSE excluded.created_at END
+		RETURNING id, created_at`,
+		b.ListID, b.Address, b.Protocol, b.Port, tsPtr(b.Expires), ts(now), b.Who, b.Source, b.Reason).Scan(&b.ID, new(string))
+	return existing, err
+}
+
+// ListBans lists the active bans of a list (0: of all lists), oldest
+// first.
+func (s *Store) ListBans(ctx context.Context, listID int64) ([]*model.Ban, error) {
+	now := time.Now()
+	q := `SELECT ` + banCols + ` FROM bans WHERE (expires_at IS NULL OR expires_at > ?)`
+	args := []any{ts(now)}
+	if listID != 0 {
+		q += ` AND list_id = ?`
+		args = append(args, listID)
+	}
+	rows, err := s.db.QueryContext(ctx, q+` ORDER BY id`, args...)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	out := []*model.Ban{}
+	for rows.Next() {
+		b, err := scanBan(rows, now)
+		if err != nil {
+			return nil, err
+		}
+		out = append(out, b)
+	}
+	return out, rows.Err()
+}
+
+// GetBan loads an active ban.
+func (s *Store) GetBan(ctx context.Context, id int64) (*model.Ban, error) {
+	now := time.Now()
+	b, err := scanBan(s.db.QueryRowContext(ctx, `SELECT `+banCols+` FROM bans WHERE id = ? AND (expires_at IS NULL OR expires_at > ?)`, id, ts(now)), now)
+	if errors.Is(err, sql.ErrNoRows) {
+		return nil, ErrNotFound
+	}
+	return b, err
+}
+
+// DeleteBan removes a ban.
+func (s *Store) DeleteBan(ctx context.Context, id int64) error {
+	_, err := s.db.ExecContext(ctx, `DELETE FROM bans WHERE id = ?`, id)
+	return err
+}
+
+// CountBans counts a list's active bans.
+func (s *Store) CountBans(ctx context.Context, listID int64) (int, error) {
+	var n int
+	err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM bans WHERE list_id = ? AND (expires_at IS NULL OR expires_at > ?)`, listID, ts(time.Now())).Scan(&n)
+	return n, err
+}
+
+// PruneBans deletes the bans that ended, and those of lists that no
+// longer exist (lists names the ones that do).
+func (s *Store) PruneBans(ctx context.Context, lists []int64) (int64, error) {
+	res, err := s.db.ExecContext(ctx, `DELETE FROM bans WHERE expires_at <= ?`, ts(time.Now()))
+	if err != nil {
+		return 0, err
+	}
+	n, _ := res.RowsAffected()
+	rows, err := s.db.QueryContext(ctx, `SELECT DISTINCT list_id FROM bans`)
+	if err != nil {
+		return n, err
+	}
+	var gone []int64
+	keep := map[int64]bool{}
+	for _, id := range lists {
+		keep[id] = true
+	}
+	for rows.Next() {
+		var id int64
+		if rows.Scan(&id) == nil && !keep[id] {
+			gone = append(gone, id)
+		}
+	}
+	rows.Close()
+	for _, id := range gone {
+		res, err := s.db.ExecContext(ctx, `DELETE FROM bans WHERE list_id = ?`, id)
+		if err != nil {
+			return n, err
+		}
+		m, _ := res.RowsAffected()
+		n += m
+	}
+	return n, nil
+}

internal/store/store.go

@@ -179,6 +179,23 @@ CREATE TABLE changesets (
 	history_id INTEGER
 );
 CREATE INDEX changesets_status ON changesets (status);
+`,
+	// 3: ban lists' bans (the lists are objects)
+	`
+CREATE TABLE bans (
+	id INTEGER PRIMARY KEY AUTOINCREMENT,
+	list_id INTEGER NOT NULL,
+	address TEXT NOT NULL,
+	protocol TEXT NOT NULL DEFAULT '',
+	port INTEGER NOT NULL DEFAULT 0,
+	expires_at TEXT,
+	created_at TEXT NOT NULL,
+	who TEXT NOT NULL DEFAULT '',
+	source TEXT NOT NULL DEFAULT '',
+	reason TEXT NOT NULL DEFAULT '',
+	UNIQUE (list_id, address, protocol, port)
+);
+CREATE INDEX bans_expires ON bans (expires_at);
 `,
 }
 

internal/store/store_test.go

@@ -103,3 +103,60 @@ func TestUsersAndAudit(t *testing.T) {
 		t.Errorf("audit %+v", evs)
 	}
 }
+
+func TestBans(t *testing.T) {
+	ctx := context.Background()
+	s, err := Open(filepath.Join(t.TempDir(), "housecarl.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	in := func(d time.Duration) *time.Time { t := time.Now().Add(d); return &t }
+	b := &model.Ban{ListID: 7, Address: "203.0.113.9", Expires: in(time.Hour), Who: "guard", Source: "guard/ssh"}
+	if existing, err := s.PutBan(ctx, b); err != nil || existing {
+		t.Fatalf("new ban: %v %v", existing, err)
+	}
+	id := b.ID
+	again := &model.Ban{ListID: 7, Address: "203.0.113.9", Expires: in(2 * time.Hour), Who: "guard", Reason: "again"}
+	if existing, err := s.PutBan(ctx, again); err != nil || !existing || again.ID != id {
+		t.Fatalf("renewed ban: %v %v id %d, want %d", existing, err, again.ID, id)
+	}
+	port := &model.Ban{ListID: 7, Address: "2001:db8:1:2::/64", Protocol: "tcp", Port: 22}
+	if _, err := s.PutBan(ctx, port); err != nil {
+		t.Fatal(err)
+	}
+	gone := &model.Ban{ListID: 7, Address: "198.51.100.1", Expires: in(-time.Second)}
+	if _, err := s.PutBan(ctx, gone); err != nil {
+		t.Fatal(err)
+	}
+	other := &model.Ban{ListID: 8, Address: "198.51.100.2"}
+	if _, err := s.PutBan(ctx, other); err != nil {
+		t.Fatal(err)
+	}
+	list, err := s.ListBans(ctx, 7)
+	if err != nil || len(list) != 2 {
+		t.Fatalf("list 7: %v %+v", err, list)
+	}
+	if list[0].Reason != "again" || list[0].Timeout < 7100 || list[0].Timeout > 7200 || list[1].Expires != nil || list[1].Timeout != 0 {
+		t.Errorf("bans: %+v %+v", list[0], list[1])
+	}
+	if n, _ := s.CountBans(ctx, 7); n != 2 {
+		t.Errorf("count %d", n)
+	}
+	// An ended ban is renewed as new.
+	if existing, _ := s.PutBan(ctx, &model.Ban{ListID: 7, Address: "198.51.100.1", Expires: in(time.Minute)}); existing {
+		t.Error("an ended ban counted as existing")
+	}
+	if n, err := s.PruneBans(ctx, []int64{7}); err != nil || n != 1 {
+		t.Errorf("prune: %d %v (list 8's ban)", n, err)
+	}
+	if all, _ := s.ListBans(ctx, 0); len(all) != 3 {
+		t.Errorf("after pruning: %+v", all)
+	}
+	if err := s.DeleteBan(ctx, id); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := s.GetBan(ctx, id); !errors.Is(err, ErrNotFound) {
+		t.Errorf("deleted ban: %v", err)
+	}
+}

internal/tablewatch/tablewatch.go

@@ -0,0 +1,100 @@
+// Package tablewatch tells when an nftables table is deleted: by
+// `nft flush ruleset` (which a ruleset file such as /etc/nftables.conf
+// usually starts with, and which nftables.service runs when it stops), or
+// by hand. It listens to nftables' netlink notifications, as `nft monitor`
+// does, and calls back once they have been quiet for a moment, since a
+// ruleset reload brings many. In shadow mode housecarld then puts its
+// table back (docs/shadow.md).
+package tablewatch
+
+import (
+	"context"
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"os"
+	"time"
+
+	"golang.org/x/sys/unix"
+)
+
+// Settle is how long the notifications must be quiet before the callback.
+const Settle = 2 * time.Second
+
+// Watch calls deleted after a table was deleted, until ctx ends. It
+// returns an error only if it can't listen at all.
+func Watch(ctx context.Context, deleted func()) error {
+	fd, err := unix.Socket(unix.AF_NETLINK, unix.SOCK_RAW|unix.SOCK_CLOEXEC|unix.SOCK_NONBLOCK, unix.NETLINK_NETFILTER)
+	if err != nil {
+		return fmt.Errorf("opening a netlink socket: %w", err)
+	}
+	if err := unix.Bind(fd, &unix.SockaddrNetlink{Family: unix.AF_NETLINK, Groups: 1 << (unix.NFNLGRP_NFTABLES - 1)}); err != nil {
+		unix.Close(fd)
+		return fmt.Errorf("listening to nftables' changes: %w", err)
+	}
+	// A non-blocking descriptor goes through Go's poller, so closing the
+	// file ends a Read in progress.
+	f := os.NewFile(uintptr(fd), "nfnetlink")
+	go func() {
+		<-ctx.Done()
+		f.Close()
+	}()
+	events := make(chan struct{}, 1)
+	go func() {
+		buf := make([]byte, 256<<10)
+		for {
+			n, err := f.Read(buf)
+			switch {
+			case err != nil && ctx.Err() != nil:
+				close(events)
+				return
+			case errors.Is(err, unix.ENOBUFS):
+				// Notifications were lost: one may have been a deletion.
+			case err != nil:
+				time.Sleep(time.Second)
+				continue
+			case !DeletesTable(buf[:n]):
+				continue
+			}
+			select {
+			case events <- struct{}{}:
+			default:
+			}
+		}
+	}()
+	var timer <-chan time.Time
+	for {
+		select {
+		case _, ok := <-events:
+			if !ok {
+				return nil
+			}
+			timer = time.After(Settle)
+		case <-timer:
+			timer = nil
+			deleted()
+		}
+	}
+}
+
+// DeletesTable reports whether a netlink datagram holds a table being
+// deleted.
+func DeletesTable(b []byte) bool {
+	want := uint16(unix.NFNL_SUBSYS_NFTABLES<<8 | unix.NFT_MSG_DELTABLE)
+	for len(b) >= unix.NLMSG_HDRLEN {
+		size := int(binary.NativeEndian.Uint32(b[0:4]))
+		typ := binary.NativeEndian.Uint16(b[4:6])
+		if size < unix.NLMSG_HDRLEN || size > len(b) {
+			return false
+		}
+		if typ == want {
+			return true
+		}
+		size = (size + unix.NLMSG_ALIGNTO - 1) &^ (unix.NLMSG_ALIGNTO - 1)
+		if size >= len(b) {
+			return false
+		}
+		b = b[size:]
+	}
+	return false
+}

internal/tablewatch/tablewatch_test.go

@@ -0,0 +1,37 @@
+package tablewatch
+
+import (
+	"encoding/binary"
+	"testing"
+
+	"golang.org/x/sys/unix"
+)
+
+// msg is a netlink message of a type, with a few bytes of payload.
+func msg(typ uint16) []byte {
+	b := make([]byte, unix.NLMSG_HDRLEN+4)
+	binary.NativeEndian.PutUint32(b[0:4], uint32(len(b)))
+	binary.NativeEndian.PutUint16(b[4:6], typ)
+	return b
+}
+
+func TestDeletesTable(t *testing.T) {
+	nft := func(m uint16) uint16 { return unix.NFNL_SUBSYS_NFTABLES<<8 | m }
+	newRule, delTable, delChain := msg(nft(unix.NFT_MSG_NEWRULE)), msg(nft(unix.NFT_MSG_DELTABLE)), msg(nft(unix.NFT_MSG_DELCHAIN))
+	cases := []struct {
+		b    []byte
+		want bool
+	}{
+		{delTable, true},
+		{append(append([]byte{}, newRule...), delTable...), true},
+		{append(append([]byte{}, newRule...), delChain...), false},
+		{newRule, false},
+		{delTable[:10], false},
+		{nil, false},
+	}
+	for i, c := range cases {
+		if got := DeletesTable(c.b); got != c.want {
+			t.Errorf("%d: %v", i, got)
+		}
+	}
+}

internal/trace/trace.go

@@ -194,8 +194,21 @@ func (t *Tracer) Line(line string) {
 	}
 }
 
-// decide notes a packet's fate: the first drop (or reject) ends it.
+// wouldRe finds shadow mode's marker in a rule's comment (render.Would):
+// "[would drop]", "[would reject]".
+var wouldRe = regexp.MustCompile(`"hc:\S+ \[would (drop|reject)\]|"\[would (drop|reject)\]`)
+
+// decide notes a packet's fate: the first drop (or reject) ends it. In
+// shadow mode Housecarl's rules accept what they would drop: that is
+// noted apart, and decides nothing.
 func decide(p *api.TracePacket, s api.TraceStep) {
+	if s.Kind == "rule" && wouldRe.MatchString(s.Rule) {
+		if p.WouldDrop == nil && p.Verdict != "dropped" {
+			step := s
+			p.WouldDrop = &step
+		}
+		return
+	}
 	if p.Verdict == "dropped" {
 		return
 	}

internal/trace/trace_test.go

@@ -87,3 +87,34 @@ func TestScript(t *testing.T) {
 		t.Error("ICMP with a port")
 	}
 }
+
+// In shadow mode Housecarl's rules accept what they would drop: the
+// packet's fate is the other table's, and the would-be drop is noted.
+func TestShadowTrace(t *testing.T) {
+	var tr Tracer
+	for _, l := range strings.Split(`trace id 9 ip main input packet: iif "wan" ip saddr 198.51.100.7 ip daddr 203.0.113.10 tcp sport 4000 tcp dport 9999 tcp flags == syn
+trace id 9 ip main input rule tcp dport 9999 accept (verdict accept)
+trace id 9 inet housecarl input rule tcp dport 9999 counter packets 1 bytes 60 accept comment "hc:r12 [would drop] block 9999" (verdict accept)
+trace id 10 ip main input packet: iif "wan" ip saddr 198.51.100.7 ip daddr 203.0.113.10 tcp sport 4001 tcp dport 7000 tcp flags == syn
+trace id 10 ip main input rule tcp dport 7000 accept (verdict accept)
+trace id 10 inet housecarl input rule counter packets 1 bytes 60 comment "hc:p-input [would drop]" (verdict continue)
+trace id 10 inet housecarl input policy accept
+trace id 11 ip main input packet: iif "wan" ip saddr 198.51.100.7 ip daddr 203.0.113.10 tcp sport 4002 tcp dport 22 tcp flags == syn
+trace id 11 ip main input rule tcp dport 22 accept (verdict accept)
+trace id 11 inet housecarl input rule tcp dport 22 counter packets 1 bytes 60 accept comment "hc:r3 ssh" (verdict accept)`, "\n") {
+		tr.Line(l)
+	}
+	ps := tr.Packets()
+	if len(ps) != 3 {
+		t.Fatalf("packets: %+v", ps)
+	}
+	if p := ps[0]; p.Verdict != "accepted" || p.WouldDrop == nil || p.WouldDrop.Mark != "r12" || p.DecidedBy.Table != "ip main" {
+		t.Errorf("first: %+v", p)
+	}
+	if p := ps[1]; p.Verdict != "accepted" || p.WouldDrop == nil || p.WouldDrop.Mark != "p-input" {
+		t.Errorf("second: %+v", p)
+	}
+	if p := ps[2]; p.WouldDrop != nil || p.DecidedBy.Mark != "r3" {
+		t.Errorf("third: %+v", p)
+	}
+}

internal/validate/validate.go

@@ -82,6 +82,7 @@ func Check(st *model.State, env *Env) []Problem {
 	for _, x := range st.Redirects {
 		c.redirect(x)
 	}
+	c.banLists()
 	return c.out
 }
 
@@ -233,9 +234,9 @@ func (c *checker) sets() {
 	for _, s := range c.st.Sets {
 		obj := "set " + s.Name
 		c.name("set", obj, s.Name, seen)
-		for _, pre := range []string{"meter_", "conn_", "quota_", "stats_"} {
+		for _, pre := range []string{"meter_", "conn_", "quota_", "stats_", "ban_"} {
 			if strings.HasPrefix(strings.ToLower(s.Name), pre) {
-				c.add(Error, "set_name_reserved", obj, "set names starting with %s are Housecarl's own (rate limits, quotas, statistics); choose another name", pre)
+				c.add(Error, "set_name_reserved", obj, "set names starting with %s are Housecarl's own (rate limits, quotas, statistics, ban lists); choose another name", pre)
 			}
 		}
 		for _, e := range s.Entries {
@@ -612,3 +613,48 @@ func (c *checker) redirect(x *model.Redirect) {
 		}
 	}
 }
+
+func (c *checker) banLists() {
+	seen := map[string]string{}
+	for _, b := range c.st.BanLists {
+		obj := "ban list " + b.Name
+		c.name("banlist", obj, b.Name, seen)
+		known := false
+		for _, sc := range model.BanScopes {
+			known = known || b.Scope == sc
+		}
+		if !known {
+			c.add(Error, "banlist_scope", obj, "the scope is %q; use firewall (the firewall itself), hosts (the list's hosts) or everything", b.Scope)
+		}
+		switch {
+		case b.Scope == model.BanScopeHosts && len(b.Hosts) == 0:
+			c.add(Error, "banlist_hosts", obj, "a list with scope hosts needs its hosts: the addresses, or sets, of the machines it protects")
+		case b.Scope != model.BanScopeHosts && len(b.Hosts) > 0:
+			c.add(Error, "banlist_hosts", obj, "only a list with scope hosts has hosts; leave them out, or use scope hosts")
+		}
+		refs := func(what string, list []string) {
+			for _, r := range list {
+				if !model.IsLiteral(r) && c.st.Set(r) == nil {
+					if _, err := model.ParseAddress(r); strings.ContainsAny(r, "./:") {
+						c.add(Error, "banlist_address", obj, "%s: %v", what, err)
+					} else {
+						c.add(Error, "unknown_set", obj, "%s: there is no set %s", what, r)
+					}
+				}
+			}
+		}
+		refs("hosts", b.Hosts)
+		refs("never", b.Never)
+		if b.DefaultTime < 0 || b.MaxTime < -1 {
+			c.add(Error, "banlist_time", obj, "ban times are seconds: default_time 0 (an hour) or more, max_time 0 (a week), more, or -1 for no limit")
+		} else if max := b.BanMaxTime(); max > 0 && b.BanTime() > max {
+			c.add(Error, "banlist_time", obj, "the default ban time (%ds) is longer than the longest (%ds)", b.BanTime(), max)
+		}
+		if b.MaxEntries < 0 || b.MaxEntries > 1000000 {
+			c.add(Error, "banlist_entries", obj, "max_entries is %d; use 1 to 1000000, or 0 for 10000", b.MaxEntries)
+		}
+		if b.GroupV6 != 0 && (b.GroupV6 < 16 || b.GroupV6 > 128) {
+			c.add(Error, "banlist_group", obj, "group_v6 is /%d; use 16 to 128 (64 bans what one customer usually gets, 128 one address)", b.GroupV6)
+		}
+	}
+}

internal/validate/validate_test.go

@@ -25,6 +25,7 @@ func base() *model.State {
 		Forwards: []*model.PortForward{{ID: 7, Position: 1, Name: "jellyfin", Protocols: []string{"tcp"}, Ports: "8096", To: "10.0.10.57",
 			PublicAddresses: []string{"203.0.113.7"}, Hairpin: true}},
 		SNATs:    []*model.SourceNAT{{ID: 8, Position: 1, Match: model.Match{OutZones: []string{"wan"}, Sources: []string{"trusted"}}}},
+		BanLists: []*model.BanList{{ID: 9, Name: "web", Scope: model.BanScopeHosts, Hosts: []string{"10.0.10.57"}, Never: []string{"trusted"}}},
 	}
 }
 
@@ -136,6 +137,17 @@ func TestCodes(t *testing.T) {
 		{"redirect_service", func(st *model.State) {
 			st.Redirects = []*model.Redirect{{ID: 11, Position: 1, Match: model.Match{InZones: []string{"lan"}}}}
 		}},
+		{"banlist_name", func(st *model.State) { st.BanLists[0].Name = "web list" }},
+		{"banlist_scope", func(st *model.State) { st.BanLists[0].Scope = "world" }},
+		{"banlist_hosts", func(st *model.State) { st.BanLists[0].Hosts = nil }},
+		{"banlist_hosts", func(st *model.State) { st.BanLists[0].Scope = model.BanScopeFirewall }},
+		{"banlist_address", func(st *model.State) { st.BanLists[0].Never = []string{"10.0.0.300"} }},
+		{"unknown_set", func(st *model.State) { st.BanLists[0].Hosts = []string{"webservers"} }},
+		{"banlist_time", func(st *model.State) { st.BanLists[0].MaxTime = -2 }},
+		{"banlist_time", func(st *model.State) { st.BanLists[0].DefaultTime, st.BanLists[0].MaxTime = 7200, 3600 }},
+		{"banlist_entries", func(st *model.State) { st.BanLists[0].MaxEntries = -1 }},
+		{"banlist_group", func(st *model.State) { st.BanLists[0].GroupV6 = 8 }},
+		{"set_name_reserved", func(st *model.State) { st.Sets[0].Name = "ban_3" }},
 		{"redirect_to_port", func(st *model.State) {
 			st.Redirects = []*model.Redirect{{ID: 11, Position: 1, Match: model.Match{InZones: []string{"lan"}, Services: []string{"dns"}}, ToPort: "53-54"}}
 		}},

internal/web/static/assets/app.js

@@ -4,6 +4,7 @@ import { wordmark, logo } from "./brand.js";
 import { api, get, state, can, setUnauthorizedHandler, setChangeHandler, loadDraft, onDraftChange, setImmediate, useDraft } from "./api.js";
 import * as changes from "./pages/changes.js";
 import * as connections from "./pages/connections.js";
+import * as bans from "./pages/bans.js";
 import * as tracePage from "./pages/trace.js";
 import * as matrixPage from "./pages/matrix.js";
 import * as statsPage from "./pages/stats.js";
@@ -32,6 +33,7 @@ const routes = [
   { pattern: /^services$/, page: services, nav: "services" },
   { pattern: /^log$/, page: log, nav: "log" },
   { pattern: /^connections$/, page: connections, nav: "connections" },
+  { pattern: /^bans$/, page: bans, nav: "bans" },
   { pattern: /^trace$/, page: tracePage, nav: "trace" },
   { pattern: /^stats$/, page: statsPage, nav: "stats" },
   { pattern: /^settings$/, page: settings, nav: "settings" },
@@ -60,6 +62,7 @@ const navItems = [
     ["stats", "#/stats", "Statistics", "log"],
     ["log", "#/log", "Log", "log"],
     ["connections", "#/connections", "Connections", "log"],
+    ["bans", "#/bans", "Bans", "bans"],
     ["trace", "#/trace", "Trace", "log"],
   ] },
   ["changes", "#/changes", "Changes", "config"],
@@ -134,7 +137,11 @@ function layout() {
         me.admin ? h("a", { href: "#/users" }, "Users") : null,
         h("button", { type: "button", class: "menu-item", onclick: logout }, "Log out"),
       ], { right: true, active: ["account", "users"].includes(current.nav), label: "Account menu" })));
-  mount(app, h("div", { class: "shell" }, header, barEl, bannerEl, mainEl, footer(me.build)));
+  // Shadow mode (housecarl.yaml): say so on every page.
+  const shadow = me.mode === "shadow" ? h("div", { class: "probation-banner shadow-banner", role: "status" },
+    h("strong", null, "Shadow mode:"), " Housecarl's ruleset is loaded but blocks, rejects and translates nothing; the other firewall decides. ",
+    "What its counters, log, tracer and statistics call drops are what Housecarl would drop. Changes aren't on probation.") : null;
+  mount(app, h("div", { class: "shell" }, header, shadow, barEl, bannerEl, mainEl, footer(me.build)));
   renderNav();
   refreshProbation();
 }

internal/web/static/assets/charts.js

@@ -331,7 +331,7 @@ export function heatmapTable(hm, unit) {
 // so the list is its own table view.
 // extra(key), when given, adds something after an entry's label (e.g. a
 // link that excludes it).
-export function barList(entries, { total, label = (k) => k, empty = "No data yet.", href, extra, format = compact } = {}) {
+export function barList(entries, { total, label = (k) => k, empty = "No data yet.", href, extra, format = compact, share: showShare = true } = {}) {
   if (!entries.length) return h("div", { class: "empty" }, empty);
   // Entries may be in a natural order (e.g. latency buckets), not by size.
   const peak = Math.max(1, ...entries.map((e) => e.count));
@@ -345,7 +345,7 @@ export function barList(entries, { total, label = (k) => k, empty = "No data yet
     return h("li", { title: (e.name ? e.name + " (" + name + ")" : name) + ": " + format(e.count) + " (" + share.toFixed(1) + "%)" },
       h("div", { class: "bar-text" },
         link ? h("a", { href: link, class: "bar-label" }, shown) : h("span", { class: "bar-label" }, shown),
-        h("span", { class: "bar-value" }, extra ? [extra(e.key), " "] : null, format(e.count), h("span", { class: "muted" }, " · " + share.toFixed(share < 10 ? 1 : 0) + "%"))),
+        h("span", { class: "bar-value" }, extra ? [extra(e.key), " "] : null, format(e.count), showShare ? h("span", { class: "muted" }, " · " + share.toFixed(share < 10 ? 1 : 0) + "%") : null)),
       h("div", { class: "bar-track" }, h("div", { class: "bar-fill", style: { width: Math.max(0.5, (e.count / peak) * 100) + "%" } })));
   }));
 }

internal/web/static/assets/pages/bans.js

@@ -0,0 +1,146 @@
+// Ban lists, and the addresses banned in them (docs/bans.md).
+import { h, mount, table, modal, errorBox, field, input, select, checkbox, confirmDialog, badge, toast, relTime } from "../ui.js";
+import { get, api, change, can } from "../api.js";
+import { rerender, onLeave } from "../app.js";
+import { listEditor, fmtSeconds, parseSeconds, rowClass, lookupNames, named, hits, counters } from "../widgets.js";
+
+const scopes = {
+  firewall: "the firewall itself",
+  hosts: "its hosts, behind the firewall",
+  everything: "the firewall and everything behind it",
+};
+
+export async function render(el) {
+  const config = can("config", false);
+  const write = can("config", true);
+  const lists = await get("/banlists");
+  const out = h("div");
+  const draw = async () => {
+    const [bans, cs] = await Promise.all([get("/bans"), counters()]);
+    const names = await lookupNames(bans.map((b) => b.address));
+    const count = {};
+    for (const b of bans) count[b.list_id] = (count[b.list_id] || 0) + 1;
+    mount(out,
+      h("h2", null, "Ban lists"),
+      h("div", { class: "panel" }, table([
+        { label: "Name", render: (l) => h("strong", null, l.name) },
+        { label: "Keeps banned addresses from", render: (l) => [scopes[l.scope] || l.scope, l.hosts && l.hosts.length ? h("span", { class: "mono small" }, ": " + l.hosts.join(", ")) : null] },
+        { label: "Never bans", render: (l) => h("span", { class: "mono small" }, (l.never || []).join(", ") || "–") },
+        { label: "Bans", render: (l) => String(count[l.id] || 0) },
+        { label: "Dropped", render: (l) => hits(cs, "ban" + l.id) },
+        { label: "Comment", render: (l) => l.comment },
+        write ? { label: "", tdClass: "actions-cell", render: (l) => l.pending === "deleted" ? badge("deleted", "err") : [
+          h("button", { class: "small reveal", onclick: () => listDialog(l) }, "Edit"), " ",
+          h("button", { class: "small reveal", onclick: () => del(l) }, "Delete")] } : null,
+      ].filter(Boolean), lists, { empty: "No ban lists yet.", rowClass })),
+      h("h2", null, "Banned addresses"),
+      h("div", { class: "panel" }, table([
+        { label: "Address", render: (b) => h("span", { class: "mono" }, named(names, b.address)) },
+        { label: "List", render: (b) => b.list },
+        { label: "Ports", render: (b) => b.protocol ? b.protocol + "/" + b.port : "all" },
+        { label: "Ends in", render: (b) => b.timeout ? fmtSeconds(b.timeout) : "never" },
+        { label: "Banned", render: (b) => relTime(b.created) },
+        { label: "By", render: (b) => [b.who, b.source ? h("span", { class: "muted small" }, " " + b.source) : null] },
+        { label: "Reason", render: (b) => h("span", { class: "small", title: b.reason || "" }, (b.reason || "").length > 80 ? b.reason.slice(0, 79) + "…" : b.reason || "") },
+        { label: "", tdClass: "actions-cell", render: (b) => mayBan() ? h("button", { class: "small reveal", onclick: () => unban(b, draw) }, "Unban") : null },
+      ], bans, { empty: "Nothing is banned." })));
+  };
+  mount(el,
+    h("div", { class: "page-head" },
+      h("div", { class: "grow" }, h("h1", null, "Bans"), h("div", { class: "muted" },
+        "Programs such as Guard and fail2ban ban addresses into a ban list; the list says what a banned address is kept from. Bans are dropped ahead of every rule, established connections too, and end by themselves.")),
+      h("div", { class: "actions" },
+        lists.length && mayBan() ? h("button", { onclick: () => banDialog(lists, draw) }, "Ban an address…") : null,
+        write ? h("button", { class: "primary", onclick: () => listDialog(null) }, "Add ban list") : null)),
+    config || lists.length ? out : h("div", { class: "empty" }, "No ban lists you may see."));
+  await draw();
+  const t = setInterval(() => { if (!document.hidden && !document.querySelector(".modal")) draw().catch(() => {}); }, 15000);
+  onLeave(() => clearInterval(t));
+}
+
+// mayBan: a bans:write grant may name its lists; the server checks.
+function mayBan() { return can("bans", true); }
+
+async function unban(b, draw) {
+  const what = b.address + (b.protocol ? " (" + b.protocol + "/" + b.port + ")" : "");
+  if (!await confirmDialog("Unban", "Lift the ban on " + what + " in " + b.list + "?", "Unban")) return;
+  try { await api("DELETE", "/banlists/" + encodeURIComponent(b.list) + "/bans/" + b.id); toast("Unbanned " + what); draw(); } catch (e) { toast(e.message, "err"); }
+}
+
+async function del(l) {
+  if (!await confirmDialog("Delete ban list", "Delete ban list " + l.name + "? Its bans are lifted, and programs banning into it are refused.", "Delete", "danger")) return;
+  try { await change("DELETE", "/banlists/" + l.id); rerender(); } catch (e) { modal("Not deleted", errorBox(e)); }
+}
+
+async function banDialog(lists, draw) {
+  const list = select(lists.map((l) => [l.name, l.name + " (" + l.scope + ")"]), lists[0].name);
+  const addr = input({ placeholder: "203.0.113.9 or 2001:db8::9", class: "mono" });
+  const proto = select([["", "every port"], ["tcp", "TCP"], ["udp", "UDP"]], "");
+  const ports = input({ placeholder: "22, ssh, http,https" });
+  const t = input({ placeholder: "the list's default", class: "short" });
+  const reason = input({ placeholder: "why, for the record" });
+  const err = h("div");
+  await modal("Ban an address", h("div", null, h("div", { class: "form-grid" },
+    field("Ban list", list), field("Address", addr, "An IPv6 address is banned with the list's grouping: its /64 unless the list says otherwise."),
+    field("Ports", h("div", { class: "filter-row" }, proto, ports), "Every port, or a protocol's ports (numbers or service names)."),
+    field("For", t, "1h, 1d; empty: the list's default."), field("Reason", reason)), err), [
+    { label: "Cancel", value: null },
+    { label: "Ban", kind: "danger", onclick: async () => {
+      mount(err);
+      const secs = parseSeconds(t.value);
+      if (secs === null) { mount(err, h("div", { class: "msg err" }, "Write the time as 1h, 30m or 1d.")); return false; }
+      if (proto.value && !ports.value.trim()) { mount(err, h("div", { class: "msg err" }, "Give the ports, or ban every port.")); return false; }
+      try {
+        const r = await api("POST", "/banlists/" + encodeURIComponent(list.value) + "/bans", { body: { address: addr.value.trim(), protocol: proto.value,
+          ports: proto.value ? ports.value.trim() : undefined, time: secs || 0, reason: reason.value.trim(), source: "web" } });
+        toast("Banned " + r.bans.map((b) => b.address + (b.protocol ? " " + b.protocol + "/" + b.port : "")).join(", ") + (r.closed ? "; " + r.closed + " connections closed" : ""));
+        draw();
+        return true;
+      } catch (e) { mount(err, errorBox(e)); return false; }
+    } },
+  ], { wide: true });
+}
+
+async function listDialog(l) {
+  l = l || { scope: "firewall", hosts: [], never: [] };
+  const name = input({ value: l.name || "", placeholder: "web", disabled: !!l.id });
+  const scope = select(Object.entries(scopes).map(([k, v]) => [k, k + ": " + v]), l.scope);
+  const hostsEd = listEditor(l.hosts || [], { placeholder: "10.0.10.57, 2001:db8:10::57, or a set", addLabel: "Add host" });
+  const never = listEditor(l.never || [], { placeholder: "198.51.100.0/24, or a set", addLabel: "Add address" });
+  const hostFields = field("Hosts", hostsEd, "The machines this list protects: their addresses, IPv4 and IPv6, or sets. Port forwards to them are covered too.").filter(Boolean);
+  const showHosts = () => { for (const n of hostFields) n.classList.toggle("hidden", scope.value !== "hosts"); };
+  scope.addEventListener("change", showHosts);
+  showHosts();
+  const time = input({ value: l.default_time ? fmtSeconds(l.default_time) : "", placeholder: "1h", class: "short" });
+  const maxTime = input({ value: l.max_time > 0 ? fmtSeconds(l.max_time) : "", placeholder: "7d", class: "short" });
+  const forever = checkbox("no limit: bans may last for ever", l.max_time < 0);
+  const entries = input({ value: l.max_entries || "", placeholder: "10000", class: "short", type: "number" });
+  const group = input({ value: l.group_v6 || "", placeholder: "64", class: "short", type: "number" });
+  const disabled = checkbox("disabled: the list's bans aren't in force", !!l.disabled);
+  const comment = input({ value: l.comment || "" });
+  const err = h("div");
+  await modal(l.id ? "Edit ban list " + l.name : "Add ban list", h("div", null, h("div", { class: "form-grid" },
+    field("Name", name, l.id ? "Programs and tokens know the list by its name, so it can't be changed." : "Programs ban into the list by this name; give each a token for it alone (bans:write:NAME)."),
+    field("Scope", scope, "What a banned address is kept from."), ...hostFields,
+    field("Never ban", never, "Besides what is never banned anyway: inside networks, the firewall's own addresses and whoever asks."),
+    field("Ban time", time, "A ban's length when the program asks for none."),
+    field("Longest ban", maxTime, "Longer bans are cut to this."), h("label"), forever,
+    field("Most bans", entries), field("IPv6 banned by", group, "Prefix length: 64 bans what one customer usually gets, 128 one address."),
+    h("label"), disabled, field("Comment", comment)), err), [
+    { label: "Cancel", value: null },
+    { label: l.id ? "Save" : "Add ban list", kind: "primary", onclick: async () => {
+      mount(err);
+      const t = parseSeconds(time.value), m = parseSeconds(maxTime.value);
+      if (t === null || m === null) { mount(err, h("div", { class: "msg err" }, "Write times as 1h, 30m or 7d.")); return false; }
+      const body = { scope: scope.value, hosts: scope.value === "hosts" ? hostsEd.value() : [], never: never.value(),
+        default_time: t || 0, max_time: forever.input.checked ? -1 : m || 0, max_entries: parseInt(entries.value, 10) || 0,
+        group_v6: parseInt(group.value, 10) || 0, disabled: disabled.input.checked, comment: comment.value.trim() };
+      if (!l.id) body.name = name.value.trim();
+      try {
+        await change(l.id ? "PATCH" : "POST", l.id ? "/banlists/" + l.id : "/banlists", body);
+        rerender();
+        return true;
+      } catch (e) { mount(err, errorBox(e)); return false; }
+    } },
+  ], { wide: true, sticky: true });
+}

internal/web/static/assets/pages/connections.js

@@ -3,7 +3,7 @@
 import { h, mount, input, debounce, toast, confirmDialog, badge } from "../ui.js";
 import { get, api, can } from "../api.js";
 import { onLeave } from "../app.js";
-import { fmtSeconds } from "../widgets.js";
+import { fmtSeconds, lookupNames, named } from "../widgets.js";
 
 export async function render(el, params, query) {
   const search = input({ type: "search", placeholder: "address, :port, protocol, state…", value: query.get("q") || "" });
@@ -22,14 +22,15 @@ export async function render(el, params, query) {
   };
   const draw = async () => {
     shown = await get("/connections", { q: search.value.trim(), limit: 500 });
+    const names = await lookupNames(shown.flatMap((c) => [c.source, c.destination, c.dnat_to]));
     count.textContent = shown.length === 1 ? "1 connection" : shown.length + " connections";
     mount(out, shown.length ? h("div", { class: "table-wrap" }, h("table", null,
       h("thead", null, h("tr", null, ["Protocol", "From", "To", "NAT", "State", "Expires", ""].map((t) => h("th", null, t)))),
       h("tbody", null, shown.map((c) => h("tr", null,
         h("td", null, c.protocol),
-        h("td", { class: "mono" }, hostPort(c.source, c.source_port)),
+        h("td", { class: "mono" }, named(names, c.source, hostPort(c.source, c.source_port))),
-        h("td", { class: "mono" }, hostPort(c.destination, c.destination_port)),
+        h("td", { class: "mono" }, named(names, c.destination, hostPort(c.destination, c.destination_port))),
-        h("td", { class: "mono small" }, c.dnat_to ? "to " + hostPort(c.dnat_to, c.dnat_port || c.destination_port) : "", c.snat_to ? (c.dnat_to ? ", " : "") + "as " + c.snat_to : ""),
+        h("td", { class: "mono small" }, c.dnat_to ? ["to ", named(names, c.dnat_to, hostPort(c.dnat_to, c.dnat_port || c.destination_port))] : "", c.snat_to ? (c.dnat_to ? ", " : "") + "as " + c.snat_to : ""),
         h("td", null, c.state || "", c.assured ? null : [" ", badge("one way", "")]),
         h("td", { class: "small" }, fmtSeconds(c.timeout)),
         h("td", { class: "actions-cell" }, write ? h("button", { class: "small reveal", onclick: () => close([c], "this connection") }, "Close") : null)))))) :

internal/web/static/assets/pages/dashboard.js

@@ -1,9 +1,9 @@
-// Dashboard: the ruleset's state, problems, interfaces, the busiest rules
+// Dashboard: the ruleset's state, problems, interfaces, the busiest rules,
-// and the latest drops.
+// the quotas and the latest drops.
-import { h, mount, table, badge, problemsList, time } from "../ui.js";
+import { h, mount, table, badge, problemsList, time, plural } from "../ui.js";
 import { get, can } from "../api.js";
 import { onLeave } from "../app.js";
-import { describe, hits, humanCount } from "../widgets.js";
+import { describe, hits, humanCount, fmtBytes, usageBar, lookupNames, named, pct } from "../widgets.js";
 import { lineChart, chartCard, legend, timeTable, stepSeconds } from "../charts.js";
 import { bytes } from "./stats.js";
 
@@ -11,12 +11,13 @@ export async function render(el) {
   const draw = async () => {
     const st = await get("/status");
     const config = can("config", false), logs = can("log", false);
-    const [rules, cs, log, tl, zones] = await Promise.all([
+    const [rules, cs, log, tl, zones, quotas] = await Promise.all([
       config ? get("/rules") : [],
       logs ? get("/counters").then((c) => c.counters).catch(() => ({})) : {},
       logs ? get("/log", { limit: 8 }).catch(() => []) : [],
       logs ? get("/stats/timeline", { since: "24h", step: "1h", keys: "zrx:*,ztx:*,drops" }).catch(() => null) : null,
       config ? get("/zones").catch(() => []) : [],
+      logs ? get("/quotas").catch(() => []) : [],
     ]);
     const busiest = rules.filter((r) => cs["r" + r.id] && cs["r" + r.id].packets > 0)
       .sort((a, b) => cs["r" + b.id].packets - cs["r" + a.id].packets).slice(0, 8);
@@ -49,11 +50,12 @@ export async function render(el) {
         h("div", { class: "panel-head" }, h("h2", null, "Busiest rules"), h("a", { class: "button small", href: "#/rules" }, "All rules")),
         table([
           { label: "Chain", render: (r) => r.chain },
-          { label: "Rule", render: (r) => r.raw || describe(r) },
+          { label: "Rule", render: (r) => h("a", { href: "#/rules?chain=" + r.chain + "&rule=" + r.id }, "#" + r.id + " ", r.raw || describe(r)) },
           { label: "Action", render: (r) => badge(r.action, { accept: "ok", drop: "err", reject: "warn" }[r.action]) },
           { label: "Hits", render: (r) => hits(cs, "r" + r.id) },
           { label: "Comment", render: (r) => r.comment },
         ], busiest)) : null,
+      quotas.length ? quotaPanel(quotas, await lookupNames(quotas.map((q) => (q.addresses || [])[0]).filter(Boolean).map((a) => a.address))) : null,
       log.length ? h("div", { class: "panel" },
         h("div", { class: "panel-head" }, h("h2", null, "Latest logged packets"), h("a", { class: "button small", href: "#/log" }, "The log")),
         table([
@@ -69,6 +71,31 @@ export async function render(el) {
   onLeave(() => clearInterval(t));
 }
 
+// quotaPanel lists every rule's quota with what it used: the total, or
+// for a per-address quota how many addresses used theirs up and the
+// busiest (the rest are on the rule). A per-address quota's bar is all
+// its addresses together, so one address alone can't fill it.
+function quotaPanel(quotas, names) {
+  const perAddress = (q) => q.quota.per_source || q.quota.per_destination;
+  const addrs = (q) => q.addresses || [];
+  const usedUp = (q) => addrs(q).filter((a) => a.used >= q.quota.bytes).length;
+  const bar = (q) => perAddress(q)
+    ? usageBar(addrs(q).reduce((n, a) => n + Math.min(a.used, q.quota.bytes), 0), addrs(q).length * q.quota.bytes)
+    : usageBar(q.used, q.quota.bytes);
+  return h("div", { class: "panel" },
+    h("div", { class: "panel-head" }, h("h2", null, "Quotas"), h("a", { class: "button small", href: "#/stats" }, "Statistics")),
+    table([
+      { label: "Rule", render: (q) => h("a", { href: "#/rules?rule=" + q.rule_id }, q.rule) },
+      { label: "Allowance", render: (q) => fmtBytes(q.quota.bytes) + (q.quota.per_source ? " each source" : q.quota.per_destination ? " each destination" : "") },
+      { label: "Used", render: (q) => perAddress(q)
+        ? [plural(addrs(q).length, "address", "addresses"), usedUp(q) ? ", " + usedUp(q) + " used up" : "",
+          addrs(q).length ? [", busiest ", named(names, addrs(q)[0].address), " " + fmtBytes(addrs(q)[0].used) + " (" + pct(addrs(q)[0].used, q.quota.bytes) + ")"] : ""]
+        : fmtBytes(q.used) + " (" + pct(q.used, q.quota.bytes) + ")" },
+      { label: "", render: (q) => addrs(q).length || !perAddress(q) ? bar(q) : "" },
+      { label: "Next reset", render: (q) => q.next_reset ? time(q.next_reset) : h("span", { class: "muted" }, "by hand") },
+    ], quotas));
+}
+
 // statsCharts: the last 24 hours' traffic with the internet and drops,
 // per hour (the Statistics page has the rest).
 function statsCharts(tl, zones) {

internal/web/static/assets/pages/import.js

@@ -8,14 +8,21 @@ export async function render(el) {
   const text = textarea("", { rows: 4, class: "mono", placeholder: "optional, the output of: sudo nft -a list ruleset (keeps what Housecarl can't express as raw rules)" });
   const defines = textarea("", { rows: 4, class: "mono", placeholder: "optional, your /etc/nftables.conf: its defines name the sets" });
   const table = input({ placeholder: "inet main (when there are several)", class: "mono" });
+  const zones = input({ placeholder: "enp2s0=lan, wg0=vpn (optional)", class: "mono" });
   const replace = checkbox("replace Housecarl's configuration", false);
   const out = h("div");
   const run = async (dry) => {
     mount(out);
+    const zoneNames = {};
+    for (const part of zones.value.split(",").map((s) => s.trim()).filter(Boolean)) {
+      const [iface, name] = part.split("=").map((s) => (s || "").trim());
+      if (!iface || !name) { mount(out, h("div", { class: "msg err" }, "Zone names: write each as interface=name (enp2s0=lan).")); return; }
+      zoneNames[iface] = name;
+    }
     let parsed;
     try { parsed = JSON.parse(json.value); } catch { mount(out, h("div", { class: "msg err" }, "The ruleset isn't JSON: paste the output of nft -j list ruleset.")); return; }
     try {
-      const r = await api("POST", "/import/nft", { body: { json: parsed, text: text.value, defines: defines.value, table: table.value.trim(), replace: replace.input.checked },
+      const r = await api("POST", "/import/nft", { body: { json: parsed, text: text.value, defines: defines.value, table: table.value.trim(), zones: zoneNames, replace: replace.input.checked },
         query: dry ? { dry_run: 1 } : undefined });
       const st = r.state;
       mount(out, h("div", { class: "msg " + (dry ? "info" : "ok") },
@@ -34,7 +41,8 @@ export async function render(el) {
     h("div", { class: "page-head" }, h("div", { class: "grow" }, h("h1", null, "Import a ruleset"),
       h("div", { class: "muted" }, "Turns an nftables ruleset into zones, sets, services, rules, port forwards and NAT. On the firewall, 'sudo hc import --defines /etc/nftables.conf -n' does the same from the command line."))),
     h("div", { class: "panel" }, h("div", { class: "form-grid" },
-      field("Ruleset (JSON)", json), field("Ruleset (text)", text), field("Ruleset file", defines), field("Table", table), h("label"), replace),
+      field("Ruleset (JSON)", json), field("Ruleset (text)", text), field("Ruleset file", defines), field("Table", table),
+      field("Zone names", zones, "Name zones by what the network is for; interfaces given one name share a zone. Otherwise a define holding the interface names it, or the interface's own name."), h("label"), replace),
       h("div", { class: "actions form-actions" },
         h("button", { onclick: () => run(true) }, "Preview"),
         h("button", { class: "primary", onclick: () => run(false) }, "Import"))),

internal/web/static/assets/pages/log.js

@@ -1,8 +1,9 @@
 // The log: packets rules (with logging on) and the policies logged,
 // refreshed every few seconds.
-import { h, mount, input, debounce } from "../ui.js";
+import { h, mount, input, debounce, badge } from "../ui.js";
-import { get } from "../api.js";
+import { get, state } from "../api.js";
 import { onLeave } from "../app.js";
+import { lookupNames, named } from "../widgets.js";
 
 export async function render(el, params, query) {
   const search = input({ type: "search", placeholder: "address, interface, rule…", value: query.get("q") || "" });
@@ -12,15 +13,20 @@ export async function render(el, params, query) {
   const addr = (a, port) => port ? (a.includes(":") ? "[" + a + "]:" + port : a + ":" + port) : a;
   const draw = async () => {
     const es = await get("/log", { limit: 300, q: search.value.trim() });
+    const names = await lookupNames(es.flatMap((e) => [e.source, e.destination]));
     mount(out, es.length ? h("div", { class: "table-wrap" }, h("table", { class: "activity" },
       h("thead", null, h("tr", null, ["Time", "Why", "In", "Out", "From", "To", "Protocol"].map((t) => h("th", null, t)))),
       h("tbody", null, es.map((e) => h("tr", null,
         h("td", { class: "mono small" }, new Date(e.time).toLocaleTimeString(undefined, { hourCycle: "h23" })),
-        h("td", null, e.mark && e.mark.startsWith("r") ? h("a", { href: "#/rules", title: "rule " + e.mark.slice(1) }, e.prefix) : e.prefix),
+        h("td", null, e.mark && e.mark.startsWith("r") ? h("a", { href: "#/rules?rule=" + e.mark.slice(1), title: "rule #" + e.mark.slice(1) }, e.prefix) : e.prefix,
+          e.shadow && /^(drop|reject)$/.test(e.verdict) ? [" ", badge("would " + e.verdict, "warn")] : null),
         h("td", { class: "mono" }, e.in || ""), h("td", { class: "mono" }, e.out || ""),
-        h("td", { class: "mono" }, addr(e.source, e.source_port)), h("td", { class: "mono" }, addr(e.destination, e.destination_port)),
+        h("td", { class: "mono" }, named(names, e.source, addr(e.source, e.source_port))),
+        h("td", { class: "mono" }, named(names, e.destination, addr(e.destination, e.destination_port))),
         h("td", null, e.protocol, e.icmp_type ? " " + e.icmp_type : "", e.tcp_flags ? h("span", { class: "muted small" }, " " + e.tcp_flags) : null)))))) :
-      h("div", { class: "empty" }, "Nothing logged yet. Turn logging on for a rule (Edit → Log), or for what the policies drop (Settings)."));
+      h("div", { class: "empty" }, state.me.mode === "shadow"
+        ? "Nothing logged yet. In shadow mode, what the policies would drop is logged; turn logging on for a rule with Edit → Log."
+        : "Nothing logged yet. Turn logging on for a rule (Edit → Log), or for what the policies drop (Settings)."));
   };
   search.addEventListener("input", debounce(draw, 300));
   mount(el,

internal/web/static/assets/pages/matrix.js

@@ -11,7 +11,7 @@ export async function render(el) {
   const name = (z) => z === "firewall" ? h("em", null, "firewall") : z;
   mount(el,
     h("div", { class: "page-head" }, h("div", { class: "grow" }, h("h1", null, "Zone matrix"),
-      h("div", { class: "muted" }, "What happens to a new connection from a zone (rows) to a zone (columns), or to and from the firewall itself, worked out from the rules in order: allowed or blocked, and how many rules decide for part of the traffic first (some services, addresses, times). Click a cell for the rules."))),
+      h("div", { class: "muted" }, "What happens to a new connection from a zone (rows) to a zone (columns), or to and from the firewall itself, worked out from the rules in order: all allowed, all blocked, or partly (some services, addresses or times let through, the rest not). Click a cell for the rules."))),
     pending ? h("div", { class: "msg info" }, "Shown with your pending changes.") : null,
     !m.zones.length ? h("div", { class: "panel" }, h("div", { class: "empty" }, "No zones yet: add them under Objects → Zones.")) :
     h("div", { class: "panel" }, h("div", { class: "table-wrap" }, h("table", { class: "matrix" },
@@ -19,20 +19,32 @@ export async function render(el) {
       h("tbody", null, cols.map((from) => h("tr", null, h("th", null, name(from)), cols.map((to) => {
         const c = cells.get(from + "\u0000" + to);
         if (!c) return h("td", { class: "matrix-none" }, "–");
-        const n = c.exceptions.length;
+        const open = c.open || [];
-        return h("td", { class: "matrix-" + c.verdict + (n ? " matrix-some" : ""), title: c.verdict + " by " + c.decided_by + (n ? ", " + n + " exception" + (n === 1 ? "" : "s") + " first" : "") },
+        const counts = [c.allowed ? "+" + c.allowed : "", c.blocked ? "−" + c.blocked : ""].filter(Boolean).join(" ");
-          h("button", { type: "button", class: "matrix-cell", onclick: () => detail(c) }, c.verdict, n ? h("span", { class: "matrix-n" }, "+" + n) : null));
+        const title = (c.same_zone ? c.note + ". " : "") + "The rest: " + c.verdict + " by " + c.decided_by +
+          (c.allowed ? "; " + c.allowed + " rule(s) let part of it through" : "") + (c.blocked ? "; " + c.blocked + " block part of it" : "");
+        return h("td", { class: "matrix-" + c.summary + (c.same_zone ? " matrix-same" : ""), title },
+          h("button", { type: "button", class: "matrix-cell", onclick: () => detail(c) },
+            c.summary === "partly" ? "partly" : c.verdict, counts ? h("span", { class: "matrix-n" }, counts) : null,
+            open.length && !c.same_zone ? h("span", { class: "matrix-open" }, open.slice(0, 3).join(" · ") + (open.length > 3 ? " · +" + (open.length - 3) + " more" : "")) : null));
       })))))),
-      h("p", { class: "muted small" }, "Zone to zone is the forward chain; to the firewall, input; from it, output. Exceptions are rules (and built-ins such as ping or port forwards) that decide for some of the traffic before the rest falls to the deciding rule or the policy. Replies and established connections aren't shown: the built-in lets those through.")));
+      h("p", { class: "muted small" }, "Zone to zone is the forward chain; to the firewall, input; from it, output. +N: rules (and built-ins such as ping or port forwards) that let part of the traffic through before the rest falls to the deciding rule or the policy; −N: those that block part of it. Under each cell, what is let through. A zone to itself (faded): only traffic routed between its interfaces passes the firewall. Replies and established connections aren't shown: the built-in lets those through.")));
 }
 
 function detail(c) {
-  const ruleLink = (id, chain, label) => id ? h("a", { href: "#/rules?chain=" + chain, onclick: () => document.querySelector(".modal-back") && document.querySelector(".modal-back").remove() }, label) : label;
+  const ruleLink = (id, chain, label) => id ? h("a", { href: "#/rules?chain=" + chain + "&rule=" + id, onclick: () => document.querySelector(".modal-back") && document.querySelector(".modal-back").remove() }, label) : label;
   const v = (verdict) => badge(verdict, { allowed: "ok", blocked: "err", logged: "", unknown: "warn" }[verdict]);
+  const hits = (n) => h("td", { class: "small num" }, n === undefined || n === null ? "–" : n.toLocaleString());
+  const summary = { allowed: "all allowed", blocked: "all blocked", partly: "partly allowed" }[c.summary];
   modal(c.from + " → " + c.to, h("div", null,
-    h("p", { class: "muted" }, "New connections, checked in this order (the " + c.chain + " chain):"),
+    h("p", null, h("strong", null, summary), h("span", { class: "muted" }, " · new connections, checked in this order (the " + c.chain + " chain):")),
-    h("div", { class: "table-wrap" }, h("table", null, h("tbody", null,
+    c.note ? h("div", { class: "msg info" }, c.note[0].toUpperCase() + c.note.slice(1) + ".") : null,
-      c.exceptions.map((e) => h("tr", null, h("td", null, v(e.verdict)), h("td", null, e.what), h("td", { class: "small" }, ruleLink(e.rule_id, e.chain, e.label)))),
+    h("div", { class: "table-wrap" }, h("table", null,
+      h("thead", null, h("tr", null, h("th", null, ""), h("th", null, "What"), h("th", null, "By"), h("th", { class: "num" }, "Hits"))),
+      h("tbody", null,
+        c.exceptions.map((e) => h("tr", null, h("td", null, v(e.verdict)), h("td", null, e.what), h("td", { class: "small" }, ruleLink(e.rule_id, e.chain, e.label)), hits(e.hits))),
         h("tr", { class: "matrix-rest" }, h("td", null, v(c.verdict)), h("td", null, h("strong", null, c.exceptions.length ? "the rest" : "everything")),
-        h("td", { class: "small" }, ruleLink(c.rule_id, c.chain, c.decided_by))))))), [{ label: "Close", value: null }], { wide: true });
+          h("td", { class: "small" }, ruleLink(c.rule_id, c.chain, c.decided_by)), hits(c.hits))))),
+    h("p", { class: "muted small" }, "Hits: each rule's packets since housecarld started, in all its traffic: a rule can cover several pairs.")),
+    [{ label: "Close", value: null }], { wide: true });
 }

internal/web/static/assets/pages/rules.js

@@ -2,7 +2,7 @@
 import { h, mount, table, modal, errorBox, field, input, select, checkbox, confirmDialog, badge, textarea, toast } from "../ui.js";
 import { get, api, change, can } from "../api.js";
 import { rerender } from "../app.js";
-import { names, matchEditor, describe, hits, counters, listEditor, rowClass, live, fmtBytes, quotas } from "../widgets.js";
+import { names, matchEditor, describe, hits, counters, listEditor, rowClass, live, fmtBytes, quotas, usageBar, lookupNames, named, pct } from "../widgets.js";
 
 const chains = [
   ["input", "Input", "Traffic to the firewall itself."],
@@ -24,7 +24,14 @@ const builtins = {
 };
 
 export async function render(el, params, query) {
-  const chain = query.get("chain") || "input";
+  // ?rule=ID (from the matrix, the log, the tracer, statistics): show
+  // that rule, in its chain, highlighted.
+  const target = Number(query.get("rule")) || 0;
+  let chain = query.get("chain");
+  if (!chain && target) {
+    try { chain = (await get("/rules/" + target)).chain; } catch { /* gone: the default chain */ }
+  }
+  chain = chain || "input";
   const [all, settings, cs, qs] = await Promise.all([get("/rules", { chain }), get("/settings"), counters(), quotas()]);
   // As nftables checks them: the rules that count traffic first.
   const rules = [...all.filter(everyPacket), ...all.filter((r) => !everyPacket(r))];
@@ -40,7 +47,8 @@ export async function render(el, params, query) {
     h("div", { class: "panel" },
       builtinsRow(chain, settings, cs),
       table([
-        { label: "#", render: (r) => r.pending === "deleted" ? "" : String(live(rules).indexOf(r) + 1) },
+        { label: "Order", render: (r) => r.pending === "deleted" ? "" : String(live(rules).indexOf(r) + 1) },
+        { label: "ID", render: (r) => r.id ? h("span", { class: "muted mono small" }, "#" + r.id) : "" },
         { label: "Match", render: (r) => r.raw ? [badge("raw", ""), " ", h("span", { class: "mono small" }, r.raw)]
           : [everyPacket(r) ? [badge("every packet", "accent"), " "] : null, describe(r), extras(r, qs[r.id])] },
         { label: "Action", render: actionBadge },
@@ -59,10 +67,15 @@ export async function render(el, params, query) {
           h("button", { class: "small reveal", onclick: () => del(r) }, "Delete"),
         ];
         } } : null,
-      ].filter(Boolean), rules, { empty: "No rules in this chain yet.", rowClass }),
+      ].filter(Boolean), rules, { empty: "No rules in this chain yet.", rowClass: (r) => rowClass(r) + (target && r.id === target ? " row-target" : "") }),
       rules.some(everyPacket) ? h("div", { class: "muted small" }, "Rules marked “every packet” are checked before established connections are accepted: they see all of a connection's traffic, not only its first packet, so a quota counts all of it and a block stops connections already open. They can drop, reject or log, not accept.") : null,
       h("div", { class: "policy small" }, "Then the policy: ", badge(policy, policy === "drop" ? "err" : "ok"),
         policy === "drop" ? [" — ", hits(cs, "p-" + chain), " packets reached it"] : null)));
+  if (target) {
+    const row = el.querySelector("tr.row-target");
+    if (row) requestAnimationFrame(() => row.scrollIntoView({ block: "center" }));
+    else el.querySelector(".tabs").after(h("div", { class: "msg warn" }, "There is no rule #" + target + " (any more)."));
+  }
 }
 
 // everyPacket: checked before established connections are accepted (as
@@ -145,46 +158,87 @@ function extras(r, q) {
   return out;
 }
 
-// quotaLine shows a quota, and what it used: in total, or per address
+// quotaLine sums up a quota on the rule's row: what it used (in total,
-// (the busiest, and the rest in a dialog).
+// or how many addresses and the busiest); the rest is in its dialog.
 function quotaLine(r, q) {
   const qt = r.quota;
   const each = qt.per_source ? " each source" : qt.per_destination ? " each destination" : "";
-  const resets = qt.reset === "monthly" ? "monthly on day " + (qt.day || 1) : qt.reset || "by hand";
+  const resets = qt.reset === "monthly" ? "monthly" : qt.reset || "reset by hand";
-  let used = null;
+  let used = fmtBytes(qt.bytes) + each;
-  if (q && !qt.per_source && !qt.per_destination) {
+  if (q && !each) used = fmtBytes(q.used) + " of " + fmtBytes(qt.bytes) + " used (" + pct(q.used, qt.bytes) + ")";
-    used = [h("strong", null, fmtBytes(q.used)), " of " + fmtBytes(qt.bytes) + " used"];
+  else if (q) {
-  } else if (q) {
+    const a = q.addresses || [], n = a.length;
-    const n = (q.addresses || []).length;
+    const up = a.filter((x) => x.used >= qt.bytes).length;
-    used = [n === 1 ? "1 address" : n + " addresses", n ? [", busiest ", h("strong", null, fmtBytes(q.addresses[0].used))] : null, " ",
+    used += " · " + (n === 1 ? "1 address" : n + " addresses") + (up ? ", " + up + " used up" : "") + (n ? ", busiest " + fmtBytes(a[0].used) + " (" + pct(a[0].used, qt.bytes) + ")" : "");
-      n ? h("button", { class: "small", onclick: () => usageDialog(r, q) }, "Usage…") : null];
   }
-  return h("div", { class: "muted small" }, (qt.over ? "quota over " : "quota ") + fmtBytes(qt.bytes) + each + ", resets " + resets,
+  const text = "quota " + used + " · " + resets;
-    used ? [" · ", used] : null,
+  return h("div", { class: "muted small" }, q && r.pending !== "new"
-    q && q.next_reset ? " · next reset " + new Date(q.next_reset).toLocaleString() : null,
+    ? h("button", { class: "link small", title: "Usage, reset and give back", onclick: () => quotaDialog(r, q) }, text) : text);
-    can("config", true) && q && r.pending !== "new" ? [" ", h("button", { class: "small", onclick: () => resetQuota(r) }, "Reset")] : null);
 }
 
-async function usageDialog(r, q) {
+// quotaDialog shows a quota in full: what it used (each address's, for a
-  await modal("Quota usage of " + (r.comment || "rule #" + r.id), h("div", null,
+// per-address one), its resets, and resetting or giving back.
-    h("p", { class: "muted" }, "Each " + (r.quota.per_source ? "source" : "destination") + " has " + fmtBytes(r.quota.bytes) + "."),
+async function quotaDialog(r, q) {
-    table([
+  const qt = r.quota;
-      { label: "Address", render: (a) => h("span", { class: "mono" }, a.address) },
+  const perAddress = qt.per_source || qt.per_destination;
+  const write = can("config", true);
+  const resets = qt.reset === "monthly" ? "monthly on day " + (qt.day || 1) : qt.reset || "only by hand";
+  const when = (t) => t ? new Date(t).toLocaleString() : "–";
+  const names = perAddress ? await lookupNames((q.addresses || []).map((a) => a.address)) : {};
+  // A row's action closes this dialog first: it opens one of its own.
+  const then = (f) => (e) => { e.target.closest(".modal").querySelector(".buttons button:last-child").click(); f(); }; // Close
+  const v = await modal("Quota of " + (r.comment || "rule #" + r.id), h("div", null,
+    h("p", null, fmtBytes(qt.bytes) + (perAddress ? (qt.per_source ? " for each source" : " for each destination") : " in all") + "; the rule matches " +
+      (qt.over ? "once it is used up" : "while it lasts") + ". Resets " + resets + "."),
+    h("p", { class: "muted small" }, "Last reset " + when(q.last_reset) + " · next reset " + when(q.next_reset)),
+    perAddress ? table([
+      { label: "Address", render: (a) => h("span", { class: "mono" }, named(names, a.address)) },
       { label: "Used", render: (a) => fmtBytes(a.used) },
-      { label: "", render: (a) => h("div", { class: "usage" }, h("div", { class: "usage-bar" + (a.used >= r.quota.bytes ? " full" : "") }, bar(a.used / r.quota.bytes))) },
+      { label: "", class: "num", tdClass: "num", render: (a) => pct(a.used, qt.bytes) },
-    ], q.addresses)), [{ label: "Close", value: null }], { wide: true });
+      { label: "", render: (a) => usageBar(a.used, qt.bytes) },
+      write ? { label: "", render: (a) => h("span", { class: "nowrap" },
+        h("button", { class: "small", onclick: then(() => resetQuota(r, a.address)) }, "Reset"), " ",
+        h("button", { class: "small", onclick: then(() => giveBack(r, a.address)) }, "Give back…")) } : null,
+    ].filter(Boolean), q.addresses || [], { empty: "No address has used it since the last reset." })
+      : h("div", null, h("strong", null, fmtBytes(q.used)), " of " + fmtBytes(qt.bytes) + " used (" + pct(q.used, qt.bytes) + ")", usageBar(q.used, qt.bytes))),
+  [
+    ...(write ? [{ label: perAddress ? "Reset all" : "Reset", value: "reset" }, { label: perAddress ? "Give back to all…" : "Give back…", value: "give" }] : []),
+    { label: "Close", value: null },
+  ], { wide: perAddress });
+  if (v === "reset") resetQuota(r);
+  if (v === "give") giveBack(r);
 }
 
-function bar(f) {
+// giveBack takes an amount off what a quota used (one address's, or
-  const seg = h("div", { class: "seg leased" });
+// everyone's): extra allowance until its next reset.
-  seg.style.width = Math.min(100, f * 100) + "%";
+async function giveBack(r, address) {
-  return seg;
+  const amount = input({ type: "number", min: 1, value: "1", class: "short" });
+  const unit = select([["mb", "MB"], ["gb", "GB"]], "gb", { class: "inline" });
+  const perAddress = r.quota.per_source || r.quota.per_destination;
+  const who = address ? " for " + address : perAddress ? " for every address" : "";
+  await modal("Give back quota", h("div", null,
+    h("p", null, "Take this much off what " + (r.comment || "rule #" + r.id) + "'s quota used" + who + ": extra allowance until its next reset, without changing the rule. This is done at once, not staged."),
+    field("Amount", h("div", null, amount, " ", unit))), [
+    { label: "Cancel", value: null },
+    { label: "Give back", kind: "primary", onclick: async () => {
+      const n = parseInt(amount.value, 10);
+      if (!(n > 0)) { toast("Give an amount of 1 or more", "err"); return false; }
+      const give = n * (unit.value === "gb" ? 1 << 30 : 1 << 20);
+      try {
+        await api("POST", "/rules/" + r.id + "/reset-quota", { body: address ? { give, address } : { give } });
+        toast("Gave back " + fmtBytes(give)); rerender();
+      } catch (e) { modal("Not given back", errorBox(e)); return false; }
+    } },
+  ]);
 }
 
-async function resetQuota(r) {
+async function resetQuota(r, address) {
   if (!await confirmDialog("Reset the quota", "Start the quota of " + (r.comment || "rule #" + r.id) + " again now" +
-    (r.quota.per_source || r.quota.per_destination ? ", for every address" : "") + "? This is done at once, not staged.", "Reset")) return;
+    (address ? " for " + address : r.quota.per_source || r.quota.per_destination ? ", for every address" : "") + "? This is done at once, not staged.", "Reset")) return;
-  try { await api("POST", "/rules/" + r.id + "/reset-quota"); toast("Quota reset"); rerender(); } catch (e) { modal("Not reset", errorBox(e)); }
+  try {
+    await api("POST", "/rules/" + r.id + "/reset-quota", address ? { body: { address } } : {});
+    toast("Quota reset"); rerender();
+  } catch (e) { modal("Not reset", errorBox(e)); }
 }
 
 async function move(r, where) {

internal/web/static/assets/pages/stats.js

@@ -3,6 +3,7 @@
 import { h, mount, fmtTime, errorBox } from "../ui.js";
 import { get } from "../api.js";
 import { lineChart, chartCard, legend, timeTable, heatmap, heatmapTable, barList, compact, stepSeconds, endLabelWidth } from "../charts.js";
+import { withNames } from "../widgets.js";
 
 const ranges = [["1h", "1 hour"], ["6h", "6 hours"], ["24h", "24 hours"], ["7d", "7 days"], ["30d", "30 days"], ["90d", "90 days"], ["365d", "a year"]];
 const rangeSecs = { "1h": 3600, "6h": 6 * 3600, "24h": 86400, "7d": 7 * 86400, "30d": 30 * 86400, "90d": 90 * 86400, "365d": 365 * 86400 };
@@ -103,7 +104,7 @@ export async function render(el, params, query) {
     listCard("What they tried", "the logged drops' protocol and port", barList(s.blocked_ports, { empty: "No logged drops " + rangeLabel + "." })),
     listCard("Devices, received", "bytes from external zones, by inside address (IPv6 per /64)", s.devices_down.length ? barList(s.devices_down, { format: bytes, href: connHref }) : devHint),
     listCard("Devices, sent", "bytes to external zones", s.devices_up.length ? barList(s.devices_up, { format: bytes, href: connHref }) : devHint.cloneNode(true)),
-    d.quotas.length ? listCard("Quotas", "bytes counted against each rule's quota, " + rangeLabel, barList(d.quotas, { format: bytes, label: (k) => "rule #" + k.slice(1), href: () => "#/rules" })) : null,
+    d.quotas.length ? listCard("Quotas", "bytes counted against each rule's quota, " + rangeLabel, barList(d.quotas, { format: bytes, label: (k) => (d.hits.find((x) => x.key === k) || {}).name || "rule #" + k.slice(1), href: ruleHref, share: false })) : null,
     listCard("Rules that matched nothing", "in the last 30 days (or since statistics started): candidates to remove",
       unused.length ? h("div", null, h("ul", { class: "plain-list" }, unused.slice(0, 12).map((e) => h("li", null, h("a", { href: ruleHref(e.key) }, e.name || e.key)))),
         unused.length > 12 ? h("div", { class: "muted small" }, "and " + (unused.length - 12) + " more: hc stats rules --unused") : null) : h("div", { class: "empty" }, "Every rule matched something."))));
@@ -142,6 +143,7 @@ async function loadAll(since, heatSince) {
     get("/stats/timeline", { since, keys: "g:conntrack,g:conntrack_max,g:mem,g:rss,g:load,g:cpu" }),
     get("/stats/timeline", { since, keys: "quota:*" }),
   ]);
+  await withNames(summary.devices_down, summary.devices_up, summary.blocked_sources);
   return { summary, zones, traffic, drops, heat, hits, gauges,
     quotas: quotas.series.map((x) => ({ key: x.key.slice(6), count: x.values.reduce((a, v) => a + Math.max(v, 0), 0) })).filter((e) => e.count > 0).sort((a, b) => b.count - a.count) };
 }
@@ -180,7 +182,7 @@ function rangeRow(since) {
 // ruleHref: a counter's mark to its page ("r12" → the rules, "b-…" → the
 // built-ins under Settings).
 function ruleHref(mark) {
-  if (/^r\d/.test(mark)) return "#/rules";
+  if (/^r\d+$/.test(mark)) return "#/rules?rule=" + mark.slice(1);
   if (mark.startsWith("b-") || mark.startsWith("p-")) return "#/settings";
   if (mark.startsWith("f")) return "#/nat";
   return null;

internal/web/static/assets/pages/trace.js

@@ -56,9 +56,11 @@ export async function render(el, params, query) {
         h("span", { class: "muted" }, head, p.in ? " · in " + p.in : "", p.out ? " · out " + p.out : ""),
         h("span", { class: "muted small trace-time" }, new Date(p.time).toLocaleTimeString())),
       decided ? h("div", { class: "small" }, p.verdict === "dropped" ? "Dropped by " : "Accepted by ", h("strong", null, decided)) : null,
-      h("table", { class: "trace-steps" }, h("tbody", null, p.steps.map((s) => h("tr", { class: /^(drop|reject)/.test(s.verdict) ? "step-drop" : s.verdict === "accept" ? "step-accept" : "" },
+      p.would_drop ? h("div", { class: "small" }, badge("shadow mode", "warn"), " Housecarl would have dropped it: ",
+        h("strong", null, p.would_drop.label || [p.would_drop.chain, p.would_drop.mark].filter(Boolean).join(" "))) : null,
+      h("table", { class: "trace-steps" }, h("tbody", null, p.steps.map((s) => h("tr", { class: /^(drop|reject)/.test(s.verdict) || /\[would (drop|reject)\]/.test(s.rule || "") ? "step-drop" : s.verdict === "accept" ? "step-accept" : "" },
         h("td", { class: "muted small" }, s.table + " " + s.chain),
-        h("td", null, s.label ? (s.mark && s.mark.startsWith("r") ? h("a", { href: "#/rules?chain=" + s.chain }, s.label) : s.label)
+        h("td", null, s.label ? (s.mark && s.mark.startsWith("r") ? h("a", { href: "#/rules?chain=" + s.chain + "&rule=" + s.mark.slice(1).split("-")[0] }, s.label) : s.label)
           : s.kind === "policy" ? s.chain + " policy" : h("span", { class: "mono small" }, s.rule)),
         h("td", { class: "small" }, s.verdict))))));
   }

internal/web/static/assets/style.css

@@ -85,6 +85,7 @@ h1 { font-size: 20px; margin: 0 0 4px; }
 h2 { font-size: 16px; margin: 18px 0 8px; }
 h3 { font-size: 14px; margin: 14px 0 6px; }
 .muted { color: var(--muted); }
+.host-name { font-weight: 600; }
 .small { font-size: 12px; }
 .nowrap { white-space: nowrap; }
 .warn-text { color: var(--warn); }
@@ -373,6 +374,7 @@ pre.wrap { white-space: pre-wrap; word-break: break-all; font-size: 12px; }
 /* The probation banner: changes roll back unless confirmed. */
 .probation-banner { background: var(--warn-soft); color: var(--text); border-bottom: 1px solid var(--warn); padding: 10px 24px; font-size: 14px; }
 .probation-banner .banner-actions { margin-left: 12px; white-space: nowrap; }
+.shadow-banner { background: var(--panel); border-bottom-color: var(--border); }
 /* List editors (zones, addresses, services). */
 .list-rows { display: flex; flex-direction: column; gap: 6px; margin-bottom: 6px; }
 .list-row { display: flex; gap: 6px; align-items: center; }
@@ -424,10 +426,18 @@ table.matrix th, table.matrix td { text-align: center; }
 table.matrix tbody th { text-align: left; }
 table.matrix th.corner { font-weight: 400; color: var(--muted); text-transform: none; }
 table.matrix td { padding: 4px; }
-.matrix-cell { width: 100%; min-width: 7em; border: 0; background: transparent; color: inherit; font-weight: 600; padding: 6px 8px; cursor: pointer; }
+table.matrix { table-layout: fixed; width: 100%; min-width: 640px; }
+table.matrix thead th:first-child, table.matrix tbody th { width: 8em; }
+.matrix-cell { display: block; width: 100%; min-width: 7em; border: 0; background: transparent; color: inherit; font-weight: 600; padding: 6px 8px; cursor: pointer; white-space: normal; text-align: center; }
 td.matrix-allowed { background: var(--add); color: var(--add-text); }
 td.matrix-blocked { background: var(--del); color: var(--del-text); }
-.matrix-n { margin-left: 6px; font-weight: 400; font-size: 12px; color: var(--warn); }
+.matrix-n { margin-left: 6px; font-weight: 400; font-size: 12px; }
+td.matrix-partly { background: var(--warn-soft); color: var(--text); }
+td.matrix-same { opacity: 0.55; }
+/* The rule a link pointed at (?rule=ID). */
+tr.row-target td { background: var(--accent-soft); font-weight: 600; }
+tr.row-target td:first-child { box-shadow: inset 3px 0 0 var(--accent); }
+.matrix-open { display: block; font-weight: 400; font-size: 12px; opacity: 0.85; margin-top: 2px; }
 td.matrix-none { color: var(--muted); }
 tr.matrix-rest td { border-top: 2px solid var(--border); }
 

internal/web/static/assets/widgets.js

@@ -1,7 +1,7 @@
 // Widgets shared by several pages: list editors with suggestions, the
 // match editor of rules and NAT, descriptions, hit counts, durations.
 import { h, input, select, field, checkbox } from "./ui.js";
-import { get } from "./api.js";
+import { get, api } from "./api.js";
 
 // ---- durations
 
@@ -98,11 +98,16 @@ export function matchEditor(m, n, opts = {}) {
 // describe sums up a match: "in lan · from trusted · ssh, dns".
 export function describe(m) {
   const parts = [];
-  const side = (word, not, list) => { if (list && list.length) parts.push(word + (not ? " not " : " ") + list.join(", ")); };
+  const list = (not, xs) => (not ? "any but " : "") + xs.join(", ");
-  side("in", m.not_in, m.in_zones);
+  // Zones are a direction, "lan → wan"; the firewall itself is the other
-  side("out", m.not_out, m.out_zones);
+  // end of the input and output chains.
-  side("from", m.not_sources, m.sources);
+  const ins = m.in_zones || [], outs = m.out_zones || [];
-  side("to", m.not_destinations, m.destinations);
+  if (ins.length || outs.length) {
+    parts.push((m.chain === "output" ? "firewall" : ins.length ? list(m.not_in, ins) : "any") + " → " +
+      (m.chain === "input" ? "firewall" : outs.length ? list(m.not_out, outs) : "any"));
+  }
+  if (m.sources && m.sources.length) parts.push("from " + list(m.not_sources, m.sources));
+  if (m.destinations && m.destinations.length) parts.push("to " + list(m.not_destinations, m.destinations));
   if (m.services && m.services.length) parts.push(m.services.join(", "));
   if (m.family) parts.push(m.family + " only");
   return parts.length ? parts.join(" · ") : "everything";
@@ -156,6 +161,41 @@ export function fmtBytes(n) {
   return n + " bytes";
 }
 
+// pct writes used as a share of total: "25%", "<1%" for a little.
+export function pct(used, total) {
+  if (!total || used <= 0) return "0%";
+  const p = used / total * 100;
+  return p < 1 ? "<1%" : Math.floor(p) + "%";
+}
+
+// usageBar draws used out of total as a bar, red once it is all used.
+export function usageBar(used, total) {
+  const seg = h("div", { class: "seg leased" });
+  seg.style.width = Math.min(100, used / total * 100) + "%";
+  return h("div", { class: "usage" }, h("div", { class: "usage-bar" + (used >= total ? " full" : "") }, seg));
+}
+
+// lookupNames asks for inside addresses' names (docs/configuration.md):
+// {address: name}, {} when names are off or can't be had.
+export async function lookupNames(addrs) {
+  const list = [...new Set(addrs.filter(Boolean))];
+  if (!list.length) return {};
+  try { return (await api("POST", "/names", { body: { addresses: list } })).names || {}; } catch { return {}; }
+}
+
+// named shows an address (or text with it, such as address:port) after
+// its name's first label; the whole name is in the tooltip.
+export function named(names, addr, text = addr) {
+  const n = names[addr];
+  return n ? h("span", { title: n + " (" + addr + ")" }, h("span", { class: "host-name" }, n.split(".")[0]), " ", h("span", { class: "muted" }, text)) : text;
+}
+
+// withNames sets each top-list entry's name from its key (an address).
+export async function withNames(...lists) {
+  const names = await lookupNames(lists.flat().map((e) => e.key));
+  for (const e of lists.flat()) if (names[e.key]) e.name = names[e.key].split(".")[0];
+}
+
 // quotas loads what the rules' quotas used, by rule ID ({} without
 // log:read).
 export async function quotas() {