Shadow mode: Housecarl beside the firewall in force, changing nothing
shadow.enabled in housecarl.yaml loads Housecarl's ruleset so that it blocks, rejects and translates nothing: drops and rejects count and accept (an accept only ends Housecarl's chain), policies accept and log what they would drop, NAT only counts, MSS clamping and the flowtable are left out, and raw rules that do more than accept, drop or reject are left out with a warning. Comments mark what would happen ([would drop]). Filter chains run after the other firewall's (+10) or before (-10); NAT chains always first. Every change is checked by nftables in both forms. No probation; takeovers and closing connections are refused. The boot fallback filters nothing. internal/tablewatch puts the table back after a flush ruleset. Mode in status, healthz, whoami, the web UI, the log and the tracer; hc ruleset --enforcing. docs/shadow.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
abrown38andClaude Opus 5.5 committed
1 parent
1e6c085ef9
commit
de075edd5d
48 files changed
+1345
-93
No files matched your search
+9
-9
@@ -31,7 +31,7 @@ user name. [permissions.md](permissions.md) has what each call needs.
|
||||
| 401 | `unauthenticated`, `session_expired`, `login_failed` | |
|
||||
| 403 | `forbidden`, `csrf` | |
|
||||
| 404 | `not_found` | |
|
||||
| 409 | `conflict` | e.g. undoing a change after the objects changed again |
|
||||
| 409 | `conflict` | e.g. undoing a change after the objects changed again; a takeover or closing connections in shadow mode ([shadow.md](shadow.md)) |
|
||||
| 422 | `validation` | the change would make the configuration invalid; `problems` lists why ([validation.md](validation.md)) |
|
||||
| 422 | `firewall` | nftables refused the ruleset (`nft -c`), or applying it failed; the message has nftables' words. Nothing was changed |
|
||||
| 429 | `rate_limited` | |
|
||||
@@ -50,7 +50,7 @@ Calls that change the configuration answer:
|
||||
|
||||
* `confirm_by`: the change is on probation, and is rolled back at that
|
||||
time unless `POST /confirm` comes first. Missing for changes made
|
||||
through the socket.
|
||||
through the socket, and in shadow mode ([shadow.md](shadow.md)).
|
||||
* `confirm_token`: the client that made the change can confirm it by
|
||||
itself with `POST /confirm {"token"}`, from the same address, as the
|
||||
same user and token, over a new connection (the answer closes this
|
||||
@@ -74,19 +74,19 @@ Creating answers 201, other changes 200.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `GET /healthz` | `ok`, `version`, `firewall` (`applied`, `unconfigured`, `fallback`, `missing`), `probation` (changes waiting to be confirmed), `warnings` (codes); HTTP 503 unless `ok`. Without authentication; read at most every five seconds ([operations.md](operations.md#monitoring)) |
|
||||
| `GET /status` | `build`, `started`, `applied` (the ruleset is in force), `apply_error`, `problems`, `probation`, `interfaces`, `counts` (objects by kind), `other_tables` (other nftables tables, `"inet main"`) |
|
||||
| `GET /healthz` | `ok`, `version`, `firewall` (`applied`, `unconfigured`, `fallback`, `missing`), `mode` (`enforcing`, or `shadow`: the ruleset blocks nothing, [shadow.md](shadow.md)), `probation` (changes waiting to be confirmed), `warnings` (codes); HTTP 503 unless `ok`. Without authentication; read at most every five seconds ([operations.md](operations.md#monitoring)) |
|
||||
| `GET /status` | `build`, `started`, `mode` (`enforcing` or `shadow`), `shadow_position` (`after` or `before`, in shadow mode), `applied` (the ruleset is in force), `apply_error`, `problems`, `probation`, `interfaces`, `counts` (objects by kind), `other_tables` (other nftables tables, `"inet main"`) |
|
||||
| `GET /interfaces` | the machine's interfaces: `name`, `up`, `addresses`, `zone` |
|
||||
| `GET /matrix[?changeset=ID]` | the zone matrix: `zones`, `cells` (`from`, `to` (a zone, or `firewall`), `chain`, `verdict` (allowed, blocked), `decided_by`, `rule_id`, `exceptions`: `rule_id`, `chain`, `label`, `verdict` (allowed, blocked, logged, unknown), `what`) ([rules.md](rules.md#the-zone-matrix)) |
|
||||
| `GET /ruleset` | the nftables ruleset Housecarl builds, as text |
|
||||
| `GET /ruleset[?enforcing=1]` | the nftables ruleset Housecarl builds, as text. In shadow mode it is the shadow ruleset; `enforcing=1` gives the one loaded once shadow mode is off |
|
||||
| `GET /counters` | `{"since", "counters": {"r12": {"packets", "bytes"}, ...}}`, by mark ([design.md](design.md#its-own-table-only)); totals since housecarld started |
|
||||
| `GET /quotas` | each rule's quota: `rule_id`, `rule`, `quota`, `used` (bytes), `addresses` (`[{"address", "used"}]` for a per-address quota), `last_reset`, `next_reset` |
|
||||
| `GET /connections[?q=&limit=]` | tracked connections: `family`, `protocol`, `source`, `source_port`, `destination`, `destination_port` (or `icmp_id`, `icmp_type`, `icmp_code`), `dnat_to`, `dnat_port`, `snat_to`, `state`, `assured`, `timeout` (seconds left), `packets` and `bytes` when the kernel counts them. `q`: all of its words in the addresses, `:port`, protocol, state |
|
||||
| `POST /connections/close` | `{"connections": [...]}` (as listed) or `{"search": "192.168.1.50"}`: delete their tracking entries (`config:write`, audited); answers `{"closed": N}` |
|
||||
| `POST /connections/close` | `{"connections": [...]}` (as listed) or `{"search": "192.168.1.50"}`: delete their tracking entries (`config:write`, audited); answers `{"closed": N}`. 409 in shadow mode |
|
||||
| `POST /trace` | `{"address", "port", "protocol", "seconds"}` (5-600, default 60): start tracing (`config:write`, audited); 409 while one runs |
|
||||
| `GET /trace` | the trace running, or the last: `running`, `input`, `started`, `ends`, `error`, `packets`: each with `time`, `in`, `out`, `protocol`, addresses and ports, `tcp_flags`, `icmp_type`, `steps` (`table`, `chain`, `kind` rule or policy, `rule`, `mark`, `label`, `verdict`), `verdict` (accepted, dropped) and `decided_by` |
|
||||
| `GET /trace` | the trace running, or the last: `running`, `input`, `started`, `ends`, `error`, `packets`: each with `time`, `in`, `out`, `protocol`, addresses and ports, `tcp_flags`, `icmp_type`, `steps` (`table`, `chain`, `kind` rule or policy, `rule`, `mark`, `label`, `verdict`), `verdict` (accepted, dropped) and `decided_by`; in shadow mode `would_drop`, the step where Housecarl would have dropped it |
|
||||
| `DELETE /trace` | stop it |
|
||||
| `GET /log[?limit=&after=&q=]` | logged packets, newest first: `time`, `prefix`, `in`, `out`, `family`, `protocol`, `source`, `source_port`, `destination`, `destination_port`, `icmp_type`, `tcp_flags`, `length`, `mark` (the rule's). `after` (a time) for those since; `q` searches |
|
||||
| `GET /log[?limit=&after=&q=]` | logged packets, newest first: `time`, `prefix`, `in`, `out`, `family`, `protocol`, `source`, `source_port`, `destination`, `destination_port`, `icmp_type`, `tcp_flags`, `length`, `mark` (the rule's), `verdict`, `shadow` (logged in shadow mode: a drop or reject is what would have been done). `after` (a time) for those since; `q` searches |
|
||||
| `GET /catalog` | built-in services, chains, actions, roles, the log group |
|
||||
|
||||
### Settings
|
||||
@@ -211,7 +211,7 @@ history, but are audited.
|
||||
| `GET /history/{id}` | one, with each object before and after |
|
||||
| `POST /history/{id}/revert` | undo it; 409 if an object it touched has changed since, or it was undone already |
|
||||
| `POST /import/nft` | `{"json": NFT_JSON, "text": "nft -a list ruleset output", "defines": "ruleset file", "table": "inet main", "zones": {"enp2s0": "lan"}, "replace": false}` ([import.md](import.md)); answers with `state` (the imported configuration), `notes` and `raw` too |
|
||||
| `POST /takeover` | `{"tables": ["inet main"]}`: delete these tables (administrators); on probation over the network |
|
||||
| `POST /takeover` | `{"tables": ["inet main"]}`: delete these tables (administrators); on probation over the network; 409 in shadow mode |
|
||||
|
||||
### Certificate
|
||||
|
||||
|
||||
+4
-4
@@ -131,13 +131,13 @@ by name.
|
||||
|
||||
| Command | |
|
||||
|---|---|
|
||||
| `status` | the ruleset (in force or not), changes on probation, problems, interfaces, other tables |
|
||||
| `status` | the mode (in shadow mode), the ruleset (in force or not), changes on probation, problems, interfaces, other tables |
|
||||
| `interfaces` | the machine's interfaces, their addresses and zones |
|
||||
| `ruleset` | the nftables ruleset Housecarl builds |
|
||||
| `ruleset [--enforcing]` | the nftables ruleset Housecarl builds; in shadow mode, `--enforcing` shows the one loaded once it is off ([shadow.md](shadow.md)) |
|
||||
| `counters` | the hits of every rule, NAT rule, built-in and policy, by mark (`r12`, `f3`, `b-established`, `p-input`) |
|
||||
| `quotas` | what each rule's quota used (each address's, for per-address quotas), and when it resets |
|
||||
| `log [-f] [--search TEXT] [--limit N]` | the packets logged by rules and policies ([operations.md](operations.md#the-log)) |
|
||||
| `connections [--search TEXT] [--limit N]`, `connections close --search TEXT [-y]` | tracked connections; closing makes the rules check them again ([operations.md](operations.md#connections)) |
|
||||
| `connections [--search TEXT] [--limit N]`, `connections close --search TEXT [-y]` | tracked connections; closing makes the rules check them again ([operations.md](operations.md#connections)); not in shadow mode |
|
||||
| `stats [--since 24h]` | traffic per zone, drops, the busiest rules, blocked sources, devices; `stats status`, `stats traffic`, `stats top devices\|blocked\|tried\|quota ID`, `stats rules [--unused]`, `stats connections` ([stats.md](stats.md#the-cli)) |
|
||||
| `matrix [FROM TO]` | what the rules do to new connections between zones; with FROM and TO, that pair's rules in order ([rules.md](rules.md#the-zone-matrix)) |
|
||||
| `trace [--address A] [--port N] [--protocol P] [--seconds S]` | follow packets through every table's rules ([operations.md](operations.md#tracing)) |
|
||||
@@ -220,7 +220,7 @@ See [nat.md](nat.md).
|
||||
| `history show ID` | each object before and after |
|
||||
| `history undo ID` | apply the change backwards (refused if something it touched has changed since) |
|
||||
| `import [FILE\|-] [--text FILE] [--defines FILE] [--table T] [--zone IFACE=NAME]... [--replace]` | [import.md](import.md) |
|
||||
| `takeover TABLE... [-y]` | delete tables Housecarl's ruleset replaces (administrators) |
|
||||
| `takeover TABLE... [-y]` | delete tables Housecarl's ruleset replaces (administrators); not in shadow mode |
|
||||
|
||||
### cert
|
||||
|
||||
|
||||
@@ -30,9 +30,14 @@ reads the file when it starts.
|
||||
| `stats.database` | `/var/lib/housecarl/stats.db` | the statistics' own database; deleting it is safe |
|
||||
| `stats.retention.minute`, `.hour`, `.day` | `7d`, `90d`, `730d` | how long per-minute, hourly and daily figures are kept |
|
||||
| `stats.top_n` | `100` | entries kept per top list, per hour and per day |
|
||||
| `shadow.enabled` | `false` | shadow mode: Housecarl's ruleset is loaded beside the firewall you have, but blocks, rejects and translates nothing; its counters, log and tracer show what it would do. No change is on probation, and takeovers and closing connections are refused ([shadow.md](shadow.md)) |
|
||||
| `shadow.position` | `after` | where Housecarl's filter chains run in shadow mode: `after` the other firewall's (`filter + 10`: it sees only what that one lets through, so what it would drop is what switching would break) or `before` them (`filter - 10`: it sees every packet) |
|
||||
|
||||
Durations are written `15s`, `12h` or `30d`.
|
||||
|
||||
Shadow mode is here, and not a firewall setting, so that no user or API
|
||||
token can switch enforcement off; restart housecarld to switch.
|
||||
|
||||
The probation's length is a firewall setting, not here: `hc settings set
|
||||
--probation 5m`. So is counting traffic per device for the statistics:
|
||||
`hc settings set --stats-per-device on`.
|
||||
|
||||
@@ -100,6 +100,32 @@ Every rule carries a comment `hc:<mark>`, such as `hc:r12` for rule 12,
|
||||
`hc:f3` for port forward 3, `hc:b-established` for a built-in. Counters
|
||||
and log lines are traced back to their object through it.
|
||||
|
||||
## Shadow mode
|
||||
|
||||
With `shadow.enabled` in the configuration file ([shadow.md](shadow.md)),
|
||||
the same configuration is rendered so that it changes nothing (render's
|
||||
`Options.Shadow`): drops and rejects accept instead, policies accept,
|
||||
NAT only counts, and what alters packets (MSS clamping, the flowtable)
|
||||
and raw rules that do more than accept, drop or reject are left out.
|
||||
This rests on how nftables runs base chains: an `accept` only ends its
|
||||
own chain, and the packet goes on to the next one on the hook, other
|
||||
tables' included; only a drop is final. Another table, the real
|
||||
firewall, decides.
|
||||
|
||||
What would be dropped, rejected or translated is counted under the same
|
||||
marks, and its comment says so (`"hc:r12 [would drop] …"`), which is
|
||||
how the tracer tells it apart. The filter chains run at `filter + 10`
|
||||
(after the other firewall's: what Housecarl would drop is what switching
|
||||
over would break) or `- 10`; the NAT chains always at `- 10`, since the
|
||||
kernel asks NAT chains only until one translates.
|
||||
|
||||
Every change is checked by nftables in both forms, so leaving shadow
|
||||
mode can't fail on the ruleset. There is no probation, since nothing
|
||||
can lock anyone out; takeovers and closing connections, which would
|
||||
change traffic, are refused. A ruleset reload elsewhere (`flush
|
||||
ruleset`) deletes Housecarl's table too: housecarld listens to nftables'
|
||||
netlink notifications (internal/tablewatch) and puts it back.
|
||||
|
||||
## Not locking yourself out
|
||||
|
||||
A change made over the network is applied **on probation** (a changeset
|
||||
|
||||
+7
-2
@@ -11,6 +11,8 @@ internal/
|
||||
render/ State → the nftables ruleset text, with each line's object
|
||||
nft/ runs nft: check, apply (keeping dynamic set entries and counters), list, the boot file
|
||||
nflog/ reads logged packets from netlink log group 4171
|
||||
linkwatch/ netlink: interfaces that appear or go (the flowtable's devices)
|
||||
tablewatch/ netlink: a table deleted (shadow mode puts Housecarl's back; docs/shadow.md)
|
||||
importer/ nft -j list ruleset → State, with notes and raw rules (docs/import.md)
|
||||
hostif/ the machine's interfaces
|
||||
store/ SQLite (modernc.org/sqlite, no cgo): configuration, history, users, tokens, sessions, audit
|
||||
@@ -70,8 +72,11 @@ Newer nftables accepts things 1.1.3 doesn't (a meter with a timeout for
|
||||
`wan`, `enp2s0` and `wg0` inside it, IPv4 and IPv6;
|
||||
* "the internet" on the WAN side, a LAN client, and a "jellyfin" host.
|
||||
|
||||
It imports a copy of the maintainer's ruleset (lab addresses), takes
|
||||
over the old table, and checks with real traffic: port forwards with
|
||||
It imports a copy of the maintainer's ruleset (lab addresses) with
|
||||
housecarld in shadow mode beside it (traffic unchanged, a would-be drop
|
||||
logged and let through, the table back after the old ruleset is
|
||||
reloaded), restarts it enforcing, takes over the old table, and checks
|
||||
with real traffic: port forwards with
|
||||
hairpin NAT and the IPv6 pinhole, masquerading, antispoofing, the drop
|
||||
log, a lockout change that rolls back by itself, a restart, the boot
|
||||
ruleset after a flush, and that other tables are left alone. It makes
|
||||
|
||||
@@ -58,6 +58,12 @@ over half of the forwarding cost, which shows up as speed only when the
|
||||
CPU was the limit, and otherwise as a cooler, quieter, more responsive
|
||||
machine.
|
||||
|
||||
In shadow mode ([shadow.md](shadow.md)) Housecarl leaves its flowtable
|
||||
out: an offloaded connection skips every table's forward chain, the
|
||||
other firewall's too. If that firewall has a flowtable of its own, the
|
||||
same goes for Housecarl's forward chain: it only sees each connection's
|
||||
first packets.
|
||||
|
||||
## When not to use it
|
||||
|
||||
**When the CPU keeps up anyway.** If the speed test already reaches your
|
||||
|
||||
@@ -159,6 +159,11 @@ Housecarl's table is now in force **alongside** the old one: a packet
|
||||
must be accepted by both. Nothing should change for your traffic. If
|
||||
Housecarl already has a configuration, `--replace` replaces it.
|
||||
|
||||
To be sure first, import in **shadow mode** ([shadow.md](shadow.md)):
|
||||
Housecarl's ruleset is then loaded but drops nothing, and its log shows
|
||||
each packet the old ruleset lets through that Housecarl would drop. When
|
||||
there are none you mind, leave shadow mode and take over.
|
||||
|
||||
Imported over the network (not through the local socket), the import is
|
||||
on probation like any change: confirm it.
|
||||
|
||||
|
||||
@@ -85,6 +85,10 @@ you are changing the firewall you are connected through.
|
||||
sudo systemctl disable nftables.service
|
||||
```
|
||||
|
||||
To try Housecarl beside your firewall first, turn on **shadow mode**
|
||||
before the import: it loads Housecarl's ruleset but blocks nothing,
|
||||
and shows what it would do ([shadow.md](shadow.md)).
|
||||
|
||||
**Or start from scratch**, through the local socket (no probation).
|
||||
The first change puts the ruleset in force, and the input and forward
|
||||
policies drop what no rule accepts, so let yourself in first:
|
||||
|
||||
@@ -7,6 +7,10 @@ public port are sent to the host, and let through the forward chain.
|
||||
One object does what takes three rules by hand (the DNAT, the forward
|
||||
accept, and the hairpin masquerade).
|
||||
|
||||
In shadow mode ([shadow.md](shadow.md)) NAT rules translate nothing:
|
||||
they count the connections they would translate, and the other
|
||||
firewall's NAT does the work.
|
||||
|
||||
```sh
|
||||
hc forward add 8443 --to 192.168.1.10:443 --public 203.0.113.10 --hairpin --name web
|
||||
hc forward add 27015-27030 --proto tcp,udp --to 192.168.1.20 --public 203.0.113.10 --hairpin --name game
|
||||
|
||||
+11
-2
@@ -28,7 +28,8 @@ edits) applies each edit at once instead. With hc, see
|
||||
## Changes on probation
|
||||
|
||||
Changes made over the network roll back unless confirmed
|
||||
([design.md](design.md#not-locking-yourself-out)). The web UI and `hc`
|
||||
([design.md](design.md#not-locking-yourself-out)); in shadow mode, where
|
||||
a change can't cut anyone off, none do ([shadow.md](shadow.md)). The web UI and `hc`
|
||||
confirm their own changes once they reach Housecarl again over a new
|
||||
connection, so normally nothing waits:
|
||||
|
||||
@@ -79,6 +80,9 @@ sudo cat /var/lib/housecarl/ruleset.nft
|
||||
sudo nft -c -f /var/lib/housecarl/ruleset.nft # check it
|
||||
```
|
||||
|
||||
In shadow mode both files are shadow mode's, and the fallback filters
|
||||
nothing ([shadow.md](shadow.md#what-works-differently)).
|
||||
|
||||
Dynamic sets start empty after a reboot. The saved ruleset has no
|
||||
flowtable, since its devices may not exist yet: housecarld adds it when
|
||||
it starts, and whenever a flowtable device appears
|
||||
@@ -177,6 +181,10 @@ hc log --search 198.51.100.7
|
||||
10:02:15 hc forward policy drop: wan eth1 198.51.100.9:443 -> 192.168.1.20:51432 tcp
|
||||
```
|
||||
|
||||
In shadow mode ([shadow.md](shadow.md)) what the policies would drop is
|
||||
always logged, in all three chains, and drops are marked as what would
|
||||
have been done: the packets went on.
|
||||
|
||||
The packets go to netlink log group 4171, where housecarld reads them;
|
||||
the last 2000 are kept in memory. They aren't written to the kernel log
|
||||
or the journal. A rule's log isn't rate-limited: on a busy rule, give it
|
||||
@@ -281,13 +289,14 @@ it is asked):
|
||||
|
||||
```sh
|
||||
$ curl -s https://firewall.example.net:8070/healthz
|
||||
{"ok":true,"version":"0.1.0","firewall":"applied","probation":0,"warnings":[]}
|
||||
{"ok":true,"version":"0.1.0","firewall":"applied","mode":"enforcing","probation":0,"warnings":[]}
|
||||
```
|
||||
|
||||
| Field | |
|
||||
|---|---|
|
||||
| `ok` | `true` with HTTP 200 when the firewall is as it should be; `false` with HTTP **503** otherwise |
|
||||
| `firewall` | `applied` (Housecarl's ruleset is in force), `unconfigured` (nothing configured yet, so nothing to apply), `fallback` (the [boot fallback](#if-the-ruleset-fails-to-load-at-boot) is in force), `missing` (no Housecarl table: flushed, or never applied) |
|
||||
| `mode` | `enforcing`, or `shadow`: Housecarl's ruleset is loaded but blocks nothing, and another firewall decides ([shadow.md](shadow.md)). `ok` doesn't depend on it; to be told when a machine isn't enforcing, also expect `"mode":"enforcing"` |
|
||||
| `probation` | changes waiting to be confirmed, rolled back unless they are |
|
||||
| `warnings` | codes of problems with the machine, as `hc status` shows them (`nftables_service`) |
|
||||
|
||||
|
||||
@@ -235,6 +235,11 @@ nftables names (`@trusted_v4`). The importer makes raw rules from
|
||||
what it can't translate. It goes into the ruleset as written: add
|
||||
`counter` to it to count its packets.
|
||||
|
||||
In shadow mode ([shadow.md](shadow.md)) a raw rule's final `drop` or
|
||||
`reject` becomes `accept`; a raw rule that does anything else to packets
|
||||
(NAT, `meta mark set`, `jump`, `notrack`, a verdict map…) is left out,
|
||||
with a warning (`shadow_raw_left_out`).
|
||||
|
||||
## Built-ins
|
||||
|
||||
Housecarl adds the rules every firewall needs before yours. Each can be
|
||||
|
||||
+171
@@ -0,0 +1,171 @@
|
||||
# Shadow mode
|
||||
|
||||
In shadow mode Housecarl runs beside the firewall you have now and
|
||||
changes nothing. Its ruleset is loaded, so its counters, log, tracer and
|
||||
statistics show what it *would* do with your real traffic, but it blocks,
|
||||
rejects and translates nothing. Your current firewall keeps deciding.
|
||||
|
||||
Use it to:
|
||||
|
||||
* **try Housecarl** on the real machine before you trust it with the
|
||||
firewall: the web UI, the API, `hc`, changesets, history;
|
||||
* **check an import**: import your ruleset ([import.md](import.md)) and
|
||||
watch whether Housecarl would drop anything your firewall lets through;
|
||||
* **develop against the API**: Steward, scripts or fail2ban can fill
|
||||
dynamic sets and see them used, with nothing at stake.
|
||||
|
||||
## Turning it on
|
||||
|
||||
Shadow mode is set in `/etc/housecarl/housecarl.yaml`
|
||||
([configuration.md](configuration.md)), not through the API, so no user
|
||||
or token can switch enforcement off:
|
||||
|
||||
```yaml
|
||||
shadow:
|
||||
enabled: true
|
||||
position: after # or before; see below
|
||||
```
|
||||
|
||||
housecarld reads the file when it starts:
|
||||
|
||||
```sh
|
||||
sudo systemctl restart housecarl
|
||||
sudo hc status # Mode: SHADOW …
|
||||
```
|
||||
|
||||
The web UI shows a banner on every page, `hc status` a `Mode:` line, and
|
||||
`/healthz` `"mode": "shadow"`.
|
||||
|
||||
## Why it can't let anything through
|
||||
|
||||
In nftables, `accept` only ends the chain it is in. The packet then
|
||||
goes on to the next base chain on the same hook, and other tables' chains
|
||||
are among those. Only a drop is final. Housecarl's table in shadow mode
|
||||
never drops, so it can't let through anything your firewall drops, and it
|
||||
can't block anything your firewall accepts.
|
||||
|
||||
## What changes in the ruleset
|
||||
|
||||
The same configuration is rendered, with everything that acts on packets
|
||||
turned into counting:
|
||||
|
||||
| | Enforcing | Shadow mode |
|
||||
|---|---|---|
|
||||
| Rules that drop or reject | `drop`, `reject` | counted, then `accept` (Housecarl's chain ends as it would have). Their comment says `[would drop]` or `[would reject]` |
|
||||
| The built-ins that drop (invalid packets, antispoofing) | `drop` | counted, `[would drop]` |
|
||||
| Policies | as configured | `accept`. What a drop policy would drop is counted and **always logged**, rate-limited, in all three chains |
|
||||
| Port forwards, source NAT, redirects | translate | counted only (`[would dnat]`, `[would masquerade]`, `[would snat]`, `[would redirect]`) |
|
||||
| MSS clamping | changes SYN packets | left out |
|
||||
| The flowtable | offloads connections | left out: offloaded connections skip *every* table's forward chain, your firewall's too |
|
||||
| Raw rules | as written | as written if they only match, count, log or accept. A final `drop` or `reject` becomes `accept`. Anything else that acts on packets (NAT, `set`, `jump`, `notrack`, a verdict map…) is left out, with a warning (`shadow_raw_left_out`) |
|
||||
| Everything else | | the same: accepts, rate limits, connection limits, quotas, dynamic sets, per-device counters, logging |
|
||||
|
||||
`hc ruleset` prints the shadow ruleset, the one loaded now.
|
||||
`hc ruleset --enforcing` prints the ruleset that will be loaded once
|
||||
shadow mode is off.
|
||||
|
||||
**Every change is checked by nftables twice**: the shadow ruleset, and the
|
||||
enforcing one. A change that nftables would refuse once shadow mode is off
|
||||
is refused now, so leaving shadow mode can't fail on that.
|
||||
|
||||
## Where Housecarl's chains run
|
||||
|
||||
Your firewall's chains are usually at priority `filter` (0), its raw
|
||||
chains at `raw`. `shadow.position` puts Housecarl's filter chains just
|
||||
next to them:
|
||||
|
||||
* **`after`** (the default) runs them at `filter + 10` (and `raw + 10`).
|
||||
Housecarl only sees what your firewall let through. So **every would-drop
|
||||
is traffic that would break if you switched over**: the list to work
|
||||
through before you do. What your firewall drops and Housecarl would
|
||||
accept (a hole) doesn't show; compare the rules for that.
|
||||
* **`before`** runs them at `filter - 10` (and `raw - 10`). Housecarl sees
|
||||
every packet, as it would when enforcing, and its hits and drops are the
|
||||
numbers you would get. Its would-drops include everything your firewall
|
||||
drops too (the internet's background noise).
|
||||
|
||||
If your firewall uses other priorities, say `filter + 20`, "after" is
|
||||
relative to `filter`, not to it: check with `nft list ruleset`.
|
||||
|
||||
NAT chains are different: the kernel asks NAT chains in turn only until
|
||||
one translates. Housecarl's NAT chains always run first (`dstnat - 10`,
|
||||
`srcnat - 10`), whatever the position, or they would never see a
|
||||
connection your firewall translates. They translate nothing, so your
|
||||
firewall's NAT still does the work.
|
||||
|
||||
## What you see
|
||||
|
||||
* **Hits** (`hc counters`, the Rules page): a drop rule's hits are what it
|
||||
would have dropped. NAT counters count connections (only a connection's
|
||||
first packet reaches the NAT chains).
|
||||
* **The log** (`hc log`, the Log page): every packet a drop policy would
|
||||
have dropped, and what rules with logging would have dropped, marked
|
||||
*would drop*. `hc log` adds "(shadow mode: would drop, let through)".
|
||||
* **The tracer** (`hc trace`, the Trace page): the verdict is your
|
||||
firewall's, the real one. When Housecarl would have dropped the packet,
|
||||
the trace says by which rule ("shadow mode: Housecarl would have
|
||||
dropped it, by …").
|
||||
* **Statistics** ([stats.md](stats.md)): drops, drops per chain and the
|
||||
blocked sources are what Housecarl would have dropped.
|
||||
|
||||
## What works differently
|
||||
|
||||
* **No probation.** A change can't cut anyone off, so changes from the
|
||||
network are applied and saved at once, with no confirmation. Changesets
|
||||
work as usual; their ruleset diff is of the shadow ruleset.
|
||||
* **No takeover, no closing connections.** `hc takeover` (deleting your
|
||||
firewall's tables) and `hc connections close` are refused (409,
|
||||
`hc` exits with 5): both would change your traffic.
|
||||
* **The boot.** `housecarl-ruleset.service` loads the shadow ruleset at
|
||||
boot. Its fallback ([operations.md](operations.md#if-the-ruleset-fails-to-load-at-boot))
|
||||
is an empty table that filters nothing, since your firewall protects the
|
||||
machine.
|
||||
* **Your firewall reloading.** A ruleset file such as `/etc/nftables.conf`
|
||||
usually starts with `flush ruleset`, and Debian's `nftables.service`
|
||||
runs `nft flush ruleset` when it stops or restarts. Both delete
|
||||
Housecarl's table too. housecarld watches nftables' notifications and
|
||||
puts its table back within a few seconds (the journal says "Housecarl's
|
||||
table was deleted … put it back"). Dynamic sets lose their entries, and
|
||||
quotas go back to what was last saved (every five minutes).
|
||||
* **Warnings.** `nftables_service` and `takeover_pending` aren't raised:
|
||||
another firewall beside Housecarl is the point. Instead:
|
||||
`shadow_unprotected` when no other table has a filter chain on the input
|
||||
hook (nothing protects the machine), and `shadow_raw_left_out` for each
|
||||
raw rule left out ([validation.md](validation.md#the-machine)).
|
||||
* **Quotas count** what passes in shadow mode, and keep counting when
|
||||
you switch over.
|
||||
|
||||
## A trial, step by step
|
||||
|
||||
From a machine whose firewall is `/etc/nftables.conf` with `table inet main`:
|
||||
|
||||
1. Install Housecarl and create the administrator
|
||||
([install.md](install.md)).
|
||||
2. Turn shadow mode on in `/etc/housecarl/housecarl.yaml`, and start
|
||||
housecarld: `sudo systemctl enable --now housecarl`.
|
||||
3. Import your ruleset: `sudo hc import --defines /etc/nftables.conf`
|
||||
([import.md](import.md)). Leave `nftables.service` as it is.
|
||||
4. Use your network for a while, and watch `hc log -f`, `hc counters` and
|
||||
the dashboard. With `position: after`, each would-drop is something
|
||||
your firewall allows and Housecarl wouldn't: fix the rule in Housecarl,
|
||||
or decide it doesn't matter. `hc status` lists the raw rules left out.
|
||||
5. When nothing you need would be dropped, leave shadow mode.
|
||||
|
||||
## Leaving shadow mode
|
||||
|
||||
1. Look at what will be loaded: `sudo hc ruleset --enforcing`.
|
||||
2. Set `shadow.enabled: false` (or remove the section), and
|
||||
`sudo systemctl restart housecarl`. Housecarl's ruleset is now in
|
||||
force **alongside** your firewall: a packet must be accepted by both.
|
||||
The restart applies it at once, not on probation, but your old firewall
|
||||
is still there and still decides too.
|
||||
3. Take over: `sudo hc takeover "inet main"` deletes the old table (on
|
||||
probation over the network), and `sudo systemctl disable nftables`
|
||||
(without `--now`) retires it ([import.md](import.md#3-take-over)).
|
||||
|
||||
## Going back
|
||||
|
||||
Set `shadow.enabled: true` and restart housecarld. If Housecarl was your
|
||||
only firewall, **the machine is now unprotected**: load your other
|
||||
firewall first. Until one filters the input hook, `hc status` and
|
||||
`/healthz` warn (`shadow_unprotected`).
|
||||
@@ -100,6 +100,10 @@ The blocked lists only see what is **logged**: the input and forward
|
||||
policies' drops with `hc settings set --log-drops on`, and rules with
|
||||
logging. The drop counts above see everything.
|
||||
|
||||
In shadow mode ([shadow.md](shadow.md)) the drops, and the blocked
|
||||
lists, are what Housecarl would have dropped: the packets went on. What
|
||||
the policies would drop is always logged there.
|
||||
|
||||
A counter that goes down (an interface that came back, a quota reset)
|
||||
counts again from zero. What happens while housecarld is stopped isn't
|
||||
counted.
|
||||
|
||||
@@ -132,3 +132,8 @@ configuration: the dashboard and `hc status` show them.
|
||||
| `nftables_service` | warning, *host* | Debian's `nftables.service` is enabled: at boot it loads `/etc/nftables.conf`, and stopping or restarting it runs `nft flush ruleset`, deleting Housecarl's table too. Disable it with `systemctl disable nftables` (without `--now`) |
|
||||
| `conntrack_full` | warning, *host* | the connection tracking table is over 80% full; once it is full, the kernel drops new connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens so many ([stats.md](stats.md#warnings)) |
|
||||
| `takeover_pending` | warning, *host* | the table you imported from is still in force beside Housecarl's, so a packet must get past both; `hc takeover TABLE` deletes it once Housecarl's rules are right |
|
||||
| `shadow_unprotected` | warning, *host* | shadow mode, and no other table has a filter chain on the input hook: Housecarl's ruleset blocks nothing, and nothing else protects the machine. Load your firewall, or leave shadow mode ([shadow.md](shadow.md)) |
|
||||
| `shadow_raw_left_out` | warning, *host* | shadow mode leaves this raw rule out: it does more to packets than accept, drop or reject (NAT, setting a mark or field, a jump, a verdict map…). Its hits aren't counted, and the rules after it see what it would have taken ([shadow.md](shadow.md#what-changes-in-the-ruleset)) |
|
||||
|
||||
In shadow mode `nftables_service` and `takeover_pending` aren't raised:
|
||||
another firewall beside Housecarl is the point there.
|
||||
Reference in new issue
Block a user