Shadow mode: Housecarl beside the firewall in force, changing nothing

shadow.enabled in housecarl.yaml loads Housecarl's ruleset so that it
blocks, rejects and translates nothing: drops and rejects count and
accept (an accept only ends Housecarl's chain), policies accept and log
what they would drop, NAT only counts, MSS clamping and the flowtable
are left out, and raw rules that do more than accept, drop or reject are
left out with a warning. Comments mark what would happen ([would drop]).
Filter chains run after the other firewall's (+10) or before (-10); NAT
chains always first. Every change is checked by nftables in both forms.
No probation; takeovers and closing connections are refused. The boot
fallback filters nothing. internal/tablewatch puts the table back after
a flush ruleset. Mode in status, healthz, whoami, the web UI, the log
and the tracer; hc ruleset --enforcing. docs/shadow.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/shadow.md

@@ -0,0 +1,171 @@
+# Shadow mode
+
+In shadow mode Housecarl runs beside the firewall you have now and
+changes nothing. Its ruleset is loaded, so its counters, log, tracer and
+statistics show what it *would* do with your real traffic, but it blocks,
+rejects and translates nothing. Your current firewall keeps deciding.
+
+Use it to:
+
+* **try Housecarl** on the real machine before you trust it with the
+  firewall: the web UI, the API, `hc`, changesets, history;
+* **check an import**: import your ruleset ([import.md](import.md)) and
+  watch whether Housecarl would drop anything your firewall lets through;
+* **develop against the API**: Steward, scripts or fail2ban can fill
+  dynamic sets and see them used, with nothing at stake.
+
+## Turning it on
+
+Shadow mode is set in `/etc/housecarl/housecarl.yaml`
+([configuration.md](configuration.md)), not through the API, so no user
+or token can switch enforcement off:
+
+```yaml
+shadow:
+  enabled: true
+  position: after     # or before; see below
+```
+
+housecarld reads the file when it starts:
+
+```sh
+sudo systemctl restart housecarl
+sudo hc status        # Mode: SHADOW …
+```
+
+The web UI shows a banner on every page, `hc status` a `Mode:` line, and
+`/healthz` `"mode": "shadow"`.
+
+## Why it can't let anything through
+
+In nftables, `accept` only ends the chain it is in. The packet then
+goes on to the next base chain on the same hook, and other tables' chains
+are among those. Only a drop is final. Housecarl's table in shadow mode
+never drops, so it can't let through anything your firewall drops, and it
+can't block anything your firewall accepts.
+
+## What changes in the ruleset
+
+The same configuration is rendered, with everything that acts on packets
+turned into counting:
+
+| | Enforcing | Shadow mode |
+|---|---|---|
+| Rules that drop or reject | `drop`, `reject` | counted, then `accept` (Housecarl's chain ends as it would have). Their comment says `[would drop]` or `[would reject]` |
+| The built-ins that drop (invalid packets, antispoofing) | `drop` | counted, `[would drop]` |
+| Policies | as configured | `accept`. What a drop policy would drop is counted and **always logged**, rate-limited, in all three chains |
+| Port forwards, source NAT, redirects | translate | counted only (`[would dnat]`, `[would masquerade]`, `[would snat]`, `[would redirect]`) |
+| MSS clamping | changes SYN packets | left out |
+| The flowtable | offloads connections | left out: offloaded connections skip *every* table's forward chain, your firewall's too |
+| Raw rules | as written | as written if they only match, count, log or accept. A final `drop` or `reject` becomes `accept`. Anything else that acts on packets (NAT, `set`, `jump`, `notrack`, a verdict map…) is left out, with a warning (`shadow_raw_left_out`) |
+| Everything else | | the same: accepts, rate limits, connection limits, quotas, dynamic sets, per-device counters, logging |
+
+`hc ruleset` prints the shadow ruleset, the one loaded now.
+`hc ruleset --enforcing` prints the ruleset that will be loaded once
+shadow mode is off.
+
+**Every change is checked by nftables twice**: the shadow ruleset, and the
+enforcing one. A change that nftables would refuse once shadow mode is off
+is refused now, so leaving shadow mode can't fail on that.
+
+## Where Housecarl's chains run
+
+Your firewall's chains are usually at priority `filter` (0), its raw
+chains at `raw`. `shadow.position` puts Housecarl's filter chains just
+next to them:
+
+* **`after`** (the default) runs them at `filter + 10` (and `raw + 10`).
+  Housecarl only sees what your firewall let through. So **every would-drop
+  is traffic that would break if you switched over**: the list to work
+  through before you do. What your firewall drops and Housecarl would
+  accept (a hole) doesn't show; compare the rules for that.
+* **`before`** runs them at `filter - 10` (and `raw - 10`). Housecarl sees
+  every packet, as it would when enforcing, and its hits and drops are the
+  numbers you would get. Its would-drops include everything your firewall
+  drops too (the internet's background noise).
+
+If your firewall uses other priorities, say `filter + 20`, "after" is
+relative to `filter`, not to it: check with `nft list ruleset`.
+
+NAT chains are different: the kernel asks NAT chains in turn only until
+one translates. Housecarl's NAT chains always run first (`dstnat - 10`,
+`srcnat - 10`), whatever the position, or they would never see a
+connection your firewall translates. They translate nothing, so your
+firewall's NAT still does the work.
+
+## What you see
+
+* **Hits** (`hc counters`, the Rules page): a drop rule's hits are what it
+  would have dropped. NAT counters count connections (only a connection's
+  first packet reaches the NAT chains).
+* **The log** (`hc log`, the Log page): every packet a drop policy would
+  have dropped, and what rules with logging would have dropped, marked
+  *would drop*. `hc log` adds "(shadow mode: would drop, let through)".
+* **The tracer** (`hc trace`, the Trace page): the verdict is your
+  firewall's, the real one. When Housecarl would have dropped the packet,
+  the trace says by which rule ("shadow mode: Housecarl would have
+  dropped it, by …").
+* **Statistics** ([stats.md](stats.md)): drops, drops per chain and the
+  blocked sources are what Housecarl would have dropped.
+
+## What works differently
+
+* **No probation.** A change can't cut anyone off, so changes from the
+  network are applied and saved at once, with no confirmation. Changesets
+  work as usual; their ruleset diff is of the shadow ruleset.
+* **No takeover, no closing connections.** `hc takeover` (deleting your
+  firewall's tables) and `hc connections close` are refused (409,
+  `hc` exits with 5): both would change your traffic.
+* **The boot.** `housecarl-ruleset.service` loads the shadow ruleset at
+  boot. Its fallback ([operations.md](operations.md#if-the-ruleset-fails-to-load-at-boot))
+  is an empty table that filters nothing, since your firewall protects the
+  machine.
+* **Your firewall reloading.** A ruleset file such as `/etc/nftables.conf`
+  usually starts with `flush ruleset`, and Debian's `nftables.service`
+  runs `nft flush ruleset` when it stops or restarts. Both delete
+  Housecarl's table too. housecarld watches nftables' notifications and
+  puts its table back within a few seconds (the journal says "Housecarl's
+  table was deleted … put it back"). Dynamic sets lose their entries, and
+  quotas go back to what was last saved (every five minutes).
+* **Warnings.** `nftables_service` and `takeover_pending` aren't raised:
+  another firewall beside Housecarl is the point. Instead:
+  `shadow_unprotected` when no other table has a filter chain on the input
+  hook (nothing protects the machine), and `shadow_raw_left_out` for each
+  raw rule left out ([validation.md](validation.md#the-machine)).
+* **Quotas count** what passes in shadow mode, and keep counting when
+  you switch over.
+
+## A trial, step by step
+
+From a machine whose firewall is `/etc/nftables.conf` with `table inet main`:
+
+1. Install Housecarl and create the administrator
+   ([install.md](install.md)).
+2. Turn shadow mode on in `/etc/housecarl/housecarl.yaml`, and start
+   housecarld: `sudo systemctl enable --now housecarl`.
+3. Import your ruleset: `sudo hc import --defines /etc/nftables.conf`
+   ([import.md](import.md)). Leave `nftables.service` as it is.
+4. Use your network for a while, and watch `hc log -f`, `hc counters` and
+   the dashboard. With `position: after`, each would-drop is something
+   your firewall allows and Housecarl wouldn't: fix the rule in Housecarl,
+   or decide it doesn't matter. `hc status` lists the raw rules left out.
+5. When nothing you need would be dropped, leave shadow mode.
+
+## Leaving shadow mode
+
+1. Look at what will be loaded: `sudo hc ruleset --enforcing`.
+2. Set `shadow.enabled: false` (or remove the section), and
+   `sudo systemctl restart housecarl`. Housecarl's ruleset is now in
+   force **alongside** your firewall: a packet must be accepted by both.
+   The restart applies it at once, not on probation, but your old firewall
+   is still there and still decides too.
+3. Take over: `sudo hc takeover "inet main"` deletes the old table (on
+   probation over the network), and `sudo systemctl disable nftables`
+   (without `--now`) retires it ([import.md](import.md#3-take-over)).
+
+## Going back
+
+Set `shadow.enabled: true` and restart housecarld. If Housecarl was your
+only firewall, **the machine is now unprotected**: load your other
+firewall first. Until one filters the input hook, `hc status` and
+`/healthz` warn (`shadow_unprotected`).