housecarl/docs/shadow.md

Shadow mode

In shadow mode Housecarl runs beside the firewall you have now and changes nothing. Its ruleset is loaded, so its counters, log, tracer and statistics show what it would do with your real traffic, but it blocks, rejects and translates nothing. Your current firewall keeps deciding.

Use it to:

• try Housecarl on the real machine before you trust it with the firewall: the web UI, the API, hc, changesets, history;
• check an import: import your ruleset (import.md) and watch whether Housecarl would drop anything your firewall lets through;
• develop against the API: Steward, scripts or fail2ban can fill dynamic sets and see them used, with nothing at stake.

Turning it on

Shadow mode is set in /etc/housecarl/housecarl.yaml (configuration.md), not through the API, so no user or token can switch enforcement off:

shadow:
  enabled: true
  position: after     # or before; see below

housecarld reads the file when it starts:

sudo systemctl restart housecarl
sudo hc status        # Mode: SHADOW …

The web UI shows a banner on every page, hc status a Mode: line, and /healthz "mode": "shadow".

Why it can't let anything through

In nftables, accept only ends the chain it is in. The packet then goes on to the next base chain on the same hook, and other tables' chains are among those. Only a drop is final. Housecarl's table in shadow mode never drops, so it can't let through anything your firewall drops, and it can't block anything your firewall accepts.

What changes in the ruleset

The same configuration is rendered, with everything that acts on packets turned into counting:

	Enforcing	Shadow mode
Rules that drop or reject	drop, reject	counted, then accept (Housecarl's chain ends as it would have). Their comment says [would drop] or [would reject]
The built-ins that drop (invalid packets, antispoofing)	drop	counted, [would drop]
Policies	as configured	accept. What a drop policy would drop is counted and always logged, rate-limited, in all three chains
Port forwards, source NAT, redirects	translate	counted only ([would dnat], [would masquerade], [would snat], [would redirect])
MSS clamping	changes SYN packets	left out
The flowtable	offloads connections	left out: offloaded connections skip every table's forward chain, your firewall's too
Raw rules	as written	as written if they only match, count, log or accept. A final drop or reject becomes accept. Anything else that acts on packets (NAT, set, jump, notrack, a verdict map…) is left out, with a warning (shadow_raw_left_out)
Everything else		the same: accepts, rate limits, connection limits, quotas, dynamic sets, per-device counters, logging

hc ruleset prints the shadow ruleset, the one loaded now. hc ruleset --enforcing prints the ruleset that will be loaded once shadow mode is off.

Every change is checked by nftables twice: the shadow ruleset, and the enforcing one. A change that nftables would refuse once shadow mode is off is refused now, so leaving shadow mode can't fail on that.

Where Housecarl's chains run

Your firewall's chains are usually at priority filter (0), its raw chains at raw. shadow.position puts Housecarl's filter chains just next to them:

• after (the default) runs them at filter + 10 (and raw + 10). Housecarl only sees what your firewall let through. So every would-drop is traffic that would break if you switched over: the list to work through before you do. What your firewall drops and Housecarl would accept (a hole) doesn't show; compare the rules for that.
• before runs them at filter - 10 (and raw - 10). Housecarl sees every packet, as it would when enforcing, and its hits and drops are the numbers you would get. Its would-drops include everything your firewall drops too (the internet's background noise).

If your firewall uses other priorities, say filter + 20, "after" is relative to filter, not to it: check with nft list ruleset.

NAT chains are different: the kernel asks NAT chains in turn only until one translates. Housecarl's NAT chains always run first (dstnat - 10, srcnat - 10), whatever the position, or they would never see a connection your firewall translates. They translate nothing, so your firewall's NAT still does the work.

What you see

• Hits (hc counters, the Rules page): a drop rule's hits are what it would have dropped. NAT counters count connections (only a connection's first packet reaches the NAT chains).
• The log (hc log, the Log page): every packet a drop policy would have dropped, and what rules with logging would have dropped, marked would drop. hc log adds "(shadow mode: would drop, let through)".
• The tracer (hc trace, the Trace page): the verdict is your firewall's, the real one. When Housecarl would have dropped the packet, the trace says by which rule ("shadow mode: Housecarl would have dropped it, by …").
• Statistics (stats.md): drops, drops per chain and the blocked sources are what Housecarl would have dropped.

What works differently

• No probation. A change can't cut anyone off, so changes from the network are applied and saved at once, with no confirmation. Changesets work as usual; their ruleset diff is of the shadow ruleset.
• No takeover, no closing connections. hc takeover (deleting your firewall's tables) and hc connections close are refused (409, hc exits with 5): both would change your traffic.
• The boot. housecarl-ruleset.service loads the shadow ruleset at boot. Its fallback (operations.md) is an empty table that filters nothing, since your firewall protects the machine.
• Your firewall reloading. A ruleset file such as /etc/nftables.conf usually starts with flush ruleset, and Debian's nftables.service runs nft flush ruleset when it stops or restarts. Both delete Housecarl's table too. housecarld watches nftables' notifications and puts its table back within a few seconds (the journal says "Housecarl's table was deleted … put it back"). Dynamic sets lose their entries, and quotas go back to what was last saved (every five minutes).
• Warnings. nftables_service and takeover_pending aren't raised: another firewall beside Housecarl is the point. Instead: shadow_unprotected when no other table has a filter chain on the input hook (nothing protects the machine), and shadow_raw_left_out for each raw rule left out (validation.md).
• Quotas count what passes in shadow mode, and keep counting when you switch over.

A trial, step by step

From a machine whose firewall is /etc/nftables.conf with table inet main:

1. Install Housecarl and create the administrator (install.md).
2. Turn shadow mode on in /etc/housecarl/housecarl.yaml, and start housecarld: sudo systemctl enable --now housecarl.
3. Import your ruleset: sudo hc import --defines /etc/nftables.conf (import.md). Leave nftables.service as it is.
4. Use your network for a while, and watch hc log -f, hc counters and the dashboard. With position: after, each would-drop is something your firewall allows and Housecarl wouldn't: fix the rule in Housecarl, or decide it doesn't matter. hc status lists the raw rules left out.
5. When nothing you need would be dropped, leave shadow mode.

Leaving shadow mode

1. Look at what will be loaded: sudo hc ruleset --enforcing.
2. Set shadow.enabled: false (or remove the section), and sudo systemctl restart housecarl. Housecarl's ruleset is now in force alongside your firewall: a packet must be accepted by both. The restart applies it at once, not on probation, but your old firewall is still there and still decides too.
3. Take over: sudo hc takeover "inet main" deletes the old table (on probation over the network), and sudo systemctl disable nftables (without --now) retires it (import.md).

Going back

Set shadow.enabled: true and restart housecarld. If Housecarl was your only firewall, the machine is now unprotected: load your other firewall first. Until one filters the input hook, hc status and /healthz warn (shadow_unprotected).