housecarl/docs/shadow.md

# Shadow mode

In shadow mode Housecarl runs beside the firewall you have now and
changes nothing. Its ruleset is loaded, so its counters, log, tracer and
statistics show what it *would* do with your real traffic, but it blocks,
rejects and translates nothing. Your current firewall keeps deciding.

Use it to:

* **try Housecarl** on the real machine before you trust it with the
  firewall: the web UI, the API, `hc`, changesets, history;
* **check an import**: import your ruleset ([import.md](import.md)) and
  watch whether Housecarl would drop anything your firewall lets through;
* **develop against the API**: Steward, scripts or fail2ban can fill
  dynamic sets and see them used, with nothing at stake.

## Turning it on

Shadow mode is set in `/etc/housecarl/housecarl.yaml`
([configuration.md](configuration.md)), not through the API, so no user
or token can switch enforcement off:

```yaml
shadow:
  enabled: true
  position: after     # or before; see below
```

housecarld reads the file when it starts:

```sh
sudo systemctl restart housecarl
sudo hc status        # Mode: SHADOW …
```

The web UI shows a banner on every page, `hc status` a `Mode:` line, and
`/healthz` `"mode": "shadow"`.

## Why it can't let anything through

In nftables, `accept` only ends the chain it is in. The packet then
goes on to the next base chain on the same hook, and other tables' chains
are among those. Only a drop is final. Housecarl's table in shadow mode
never drops, so it can't let through anything your firewall drops, and it
can't block anything your firewall accepts.

## What changes in the ruleset

The same configuration is rendered, with everything that acts on packets
turned into counting:

| | Enforcing | Shadow mode |
|---|---|---|
| Rules that drop or reject | `drop`, `reject` | counted, then `accept` (Housecarl's chain ends as it would have). Their comment says `[would drop]` or `[would reject]` |
| The built-ins that drop (invalid packets, antispoofing) | `drop` | counted, `[would drop]` |
| Policies | as configured | `accept`. What a drop policy would drop is counted and **always logged**, rate-limited, in all three chains |
| Port forwards, source NAT, redirects | translate | counted only (`[would dnat]`, `[would masquerade]`, `[would snat]`, `[would redirect]`) |
| MSS clamping | changes SYN packets | left out |
| The flowtable | offloads connections | left out: offloaded connections skip *every* table's forward chain, your firewall's too |
| Raw rules | as written | as written if they only match, count, log or accept. A final `drop` or `reject` becomes `accept`. Anything else that acts on packets (NAT, `set`, `jump`, `notrack`, a verdict map…) is left out, with a warning (`shadow_raw_left_out`) |
| Everything else | | the same: accepts, rate limits, connection limits, quotas, dynamic sets, per-device counters, logging |

`hc ruleset` prints the shadow ruleset, the one loaded now.
`hc ruleset --enforcing` prints the ruleset that will be loaded once
shadow mode is off.

**Every change is checked by nftables twice**: the shadow ruleset, and the
enforcing one. A change that nftables would refuse once shadow mode is off
is refused now, so leaving shadow mode can't fail on that.

## Where Housecarl's chains run

Your firewall's chains are usually at priority `filter` (0), its raw
chains at `raw`. `shadow.position` puts Housecarl's filter chains just
next to them:

* **`after`** (the default) runs them at `filter + 10` (and `raw + 10`).
  Housecarl only sees what your firewall let through. So **every would-drop
  is traffic that would break if you switched over**: the list to work
  through before you do. What your firewall drops and Housecarl would
  accept (a hole) doesn't show; compare the rules for that.
* **`before`** runs them at `filter - 10` (and `raw - 10`). Housecarl sees
  every packet, as it would when enforcing, and its hits and drops are the
  numbers you would get. Its would-drops include everything your firewall
  drops too (the internet's background noise).

If your firewall uses other priorities, say `filter + 20`, "after" is
relative to `filter`, not to it: check with `nft list ruleset`.

NAT chains are different: the kernel asks NAT chains in turn only until
one translates. Housecarl's NAT chains always run first (`dstnat - 10`,
`srcnat - 10`), whatever the position, or they would never see a
connection your firewall translates. They translate nothing, so your
firewall's NAT still does the work.

## What you see

* **Hits** (`hc counters`, the Rules page): a drop rule's hits are what it
  would have dropped. NAT counters count connections (only a connection's
  first packet reaches the NAT chains).
* **The log** (`hc log`, the Log page): every packet a drop policy would
  have dropped, and what rules with logging would have dropped, marked
  *would drop*. `hc log` adds "(shadow mode: would drop, let through)".
* **The tracer** (`hc trace`, the Trace page): the verdict is your
  firewall's, the real one. When Housecarl would have dropped the packet,
  the trace says by which rule ("shadow mode: Housecarl would have
  dropped it, by …").
* **Statistics** ([stats.md](stats.md)): drops, drops per chain and the
  blocked sources are what Housecarl would have dropped.

## What works differently

* **No probation.** A change can't cut anyone off, so changes from the
  network are applied and saved at once, with no confirmation. Changesets
  work as usual; their ruleset diff is of the shadow ruleset.
* **No takeover, no closing connections.** `hc takeover` (deleting your
  firewall's tables) and `hc connections close` are refused (409,
  `hc` exits with 5): both would change your traffic.
* **The boot.** `housecarl-ruleset.service` loads the shadow ruleset at
  boot. Its fallback ([operations.md](operations.md#if-the-ruleset-fails-to-load-at-boot))
  is an empty table that filters nothing, since your firewall protects the
  machine.
* **Your firewall reloading.** A ruleset file such as `/etc/nftables.conf`
  usually starts with `flush ruleset`, and Debian's `nftables.service`
  runs `nft flush ruleset` when it stops or restarts. Both delete
  Housecarl's table too. housecarld watches nftables' notifications and
  puts its table back within a few seconds (the journal says "Housecarl's
  table was deleted … put it back"). Dynamic sets lose their entries, and
  quotas go back to what was last saved (every five minutes).
* **Warnings.** `nftables_service` and `takeover_pending` aren't raised:
  another firewall beside Housecarl is the point. Instead:
  `shadow_unprotected` when no other table has a filter chain on the input
  hook (nothing protects the machine), and `shadow_raw_left_out` for each
  raw rule left out ([validation.md](validation.md#the-machine)).
* **Quotas count** what passes in shadow mode, and keep counting when
  you switch over.

## A trial, step by step

From a machine whose firewall is `/etc/nftables.conf` with `table inet main`:

1. Install Housecarl and create the administrator
   ([install.md](install.md)).
2. Turn shadow mode on in `/etc/housecarl/housecarl.yaml`, and start
   housecarld: `sudo systemctl enable --now housecarl`.
3. Import your ruleset: `sudo hc import --defines /etc/nftables.conf`
   ([import.md](import.md)). Leave `nftables.service` as it is.
4. Use your network for a while, and watch `hc log -f`, `hc counters` and
   the dashboard. With `position: after`, each would-drop is something
   your firewall allows and Housecarl wouldn't: fix the rule in Housecarl,
   or decide it doesn't matter. `hc status` lists the raw rules left out.
5. When nothing you need would be dropped, leave shadow mode.

## Leaving shadow mode

1. Look at what will be loaded: `sudo hc ruleset --enforcing`.
2. Set `shadow.enabled: false` (or remove the section), and
   `sudo systemctl restart housecarl`. Housecarl's ruleset is now in
   force **alongside** your firewall: a packet must be accepted by both.
   The restart applies it at once, not on probation, but your old firewall
   is still there and still decides too.
3. Take over: `sudo hc takeover "inet main"` deletes the old table (on
   probation over the network), and `sudo systemctl disable nftables`
   (without `--now`) retires it ([import.md](import.md#3-take-over)).

## Going back

Set `shadow.enabled: true` and restart housecarld. If Housecarl was your
only firewall, **the machine is now unprotected**: load your other
firewall first. Until one filters the input hook, `hc status` and
`/healthz` warn (`shadow_unprotected`).