abrown38andClaude Opus 5.5
e4efdf9c78
A change made over the network answers with a one-time confirm_token and closes its connection (HTTP/2: a GOAWAY). The web UI and hc send the token straight back to /confirm, which accepts it only for the newest change, from the same user and API token, the same address, and over a new connection: proof that the client can still reach the API through the new ruleset (an established connection would get through anyway). A change that cuts its client off can't be confirmed so, and rolls back by itself. The setting manual_confirm (hc settings set --auto-confirm off) and hc --no-confirm leave confirming to a person. Changes through the local socket still need no confirmation. e2e: 136 checks, among them a lockout change rolled back without anyone confirming. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
175 lines
6.0 KiB
Markdown
175 lines
6.0 KiB
Markdown
# Running Housecarl
|
|
|
|
## The services
|
|
|
|
```sh
|
|
systemctl status housecarl # housecarld: the API, the web UI, the log
|
|
journalctl -u housecarl -f
|
|
systemctl reload housecarl # reread the TLS certificate
|
|
systemctl status housecarl-ruleset # the ruleset loaded at boot
|
|
```
|
|
|
|
The firewall itself is the kernel's: it stays in force while housecarld
|
|
is stopped, restarted or upgraded. Without housecarld there is no API,
|
|
no web UI, no log and no probation timer.
|
|
|
|
housecarld logs each change it applies, rollbacks, and what it couldn't
|
|
apply. When it starts, it applies the configuration again (rolling back
|
|
first what was on probation when it stopped).
|
|
|
|
## Reviewing before applying
|
|
|
|
In the web UI, edits are collected as pending changes (the bar under the
|
|
header), and Changes shows them with the diff of the ruleset; Apply puts
|
|
them in force together. The switch in the header (Review edits / Live
|
|
edits) applies each edit at once instead. With hc, see
|
|
[cli.md](cli.md#changesets).
|
|
|
|
## Changes on probation
|
|
|
|
Changes made over the network roll back unless confirmed
|
|
([design.md](design.md#not-locking-yourself-out)). The web UI and `hc`
|
|
confirm their own changes once they reach Housecarl again over a new
|
|
connection, so normally nothing waits:
|
|
|
|
```
|
|
$ hc rule add input --from admins --service ssh
|
|
created rule input #31
|
|
applied and confirmed (history 12): this client reached Housecarl again over a new connection
|
|
```
|
|
|
|
A change that cuts the client off can't be confirmed that way:
|
|
|
|
```
|
|
applied on probation (history 13), but couldn't confirm it: … context deadline exceeded
|
|
It is rolled back within 1m50s unless confirmed ('hc confirm')
|
|
```
|
|
|
|
By hand (with `--no-confirm`, or with Settings → "a person confirms each
|
|
change"):
|
|
|
|
```sh
|
|
hc status # Probation: 2 changes roll back in 97s unless confirmed
|
|
hc confirm
|
|
hc rollback # or undo them now
|
|
```
|
|
|
|
The web UI shows a banner with a countdown and both buttons. A rollback
|
|
is in the history, as an undo by `housecarl` via `rollback`, so it can be
|
|
undone like any change.
|
|
|
|
If you are cut off by a change: wait. When the time is up, the previous
|
|
ruleset comes back by itself. If you are cut off by a confirmed change,
|
|
use the console, or SSH from a source the rules still let in:
|
|
|
|
```sh
|
|
sudo hc history list
|
|
sudo hc history undo 42
|
|
```
|
|
|
|
## The ruleset at boot
|
|
|
|
A confirmed ruleset (or one changed through the socket) is saved to
|
|
`/var/lib/housecarl/ruleset.nft`, which `housecarl-ruleset.service`
|
|
loads before the network comes up. To see what would load:
|
|
|
|
```sh
|
|
sudo cat /var/lib/housecarl/ruleset.nft
|
|
sudo nft -c -f /var/lib/housecarl/ruleset.nft # check it
|
|
```
|
|
|
|
Dynamic sets start empty after a reboot.
|
|
|
|
Keep Debian's `nftables.service` disabled, or `/etc/nftables.conf`
|
|
without `flush ruleset`: otherwise it deletes Housecarl's table at
|
|
boot. `housecarl-ruleset.service` runs after it, which covers the
|
|
ordinary case.
|
|
|
|
## Hits
|
|
|
|
Every rule, NAT rule, built-in and policy has a counter. `hc rule list`,
|
|
`hc forward list` and the web UI show packets; `hc counters` shows them
|
|
all, with bytes. They count from housecarld's start, and carry across
|
|
changes to the ruleset.
|
|
|
|
## Quotas
|
|
|
|
`hc quotas` (and the rules page) show what each quota used and when it
|
|
resets; `hc rule reset-quota ID` starts one again. housecarld resets
|
|
quotas at midnight when their period ends (late, if it was stopped then),
|
|
and saves their usage every five minutes and when it stops; after a
|
|
reboot it puts it back. A reboot without a clean stop loses at most the
|
|
last five minutes' count.
|
|
|
|
## The log
|
|
|
|
Rules with `--log` (Log in the web UI) and, with `hc settings set
|
|
--log-drops on`, what the input and forward policies drop (at most 10
|
|
packets a second) are logged:
|
|
|
|
```sh
|
|
hc log -f
|
|
hc log --search 198.51.100.7
|
|
```
|
|
|
|
```
|
|
10:02:13 hc input #95 drop: wan - 198.51.100.7:53868 -> 203.0.113.10:22 tcp syn
|
|
10:02:15 hc forward policy drop: wan eth1 198.51.100.9:443 -> 192.168.1.20:51432 tcp
|
|
```
|
|
|
|
The packets go to netlink log group 4171, where housecarld reads them;
|
|
the last 2000 are kept in memory. They aren't written to the kernel log
|
|
or the journal. A rule's log isn't rate-limited: on a busy rule, give it
|
|
a limit, or log only new connections (`--state new`).
|
|
|
|
## When something doesn't get through
|
|
|
|
1. `hc log -f` with `--log-drops on`: is it dropped by a policy (no rule
|
|
accepted it), or by a rule (its number is in the line)?
|
|
2. `hc rule list forward` (or `input`): hits on the rule you expect?
|
|
3. `hc ruleset`: the rule as nftables sees it, with the zones' interfaces
|
|
and the sets' families.
|
|
4. `hc status`: other tables. A Docker or Incus table, or an old one,
|
|
can drop what Housecarl accepts.
|
|
5. Port forwards: `hc forward list` hits, and the inside host's own
|
|
firewall and default route (answers must go back through Housecarl).
|
|
|
|
## Backups
|
|
|
|
`housecarl-backup.timer` runs `housecarld backup` daily. It writes a
|
|
consistent copy of the database to `backup.dir`
|
|
(`/var/backups/housecarl/housecarl-<time>.db`), and deletes those older
|
|
than `backup.retention` (30 days), keeping at least
|
|
`backup.keep_at_least` (7). A backup holds everything: configuration,
|
|
history, users, tokens (hashed) and the audit log.
|
|
|
|
```sh
|
|
sudo -u housecarl housecarld backup # now
|
|
sudo -u housecarl housecarld backup /tmp/h.db # to a file of your choice
|
|
sudo systemctl disable --now housecarl-backup.timer # no daily backups
|
|
```
|
|
|
|
To **restore**, stop housecarld first; restore refuses to replace the
|
|
database under it:
|
|
|
|
```sh
|
|
sudo systemctl stop housecarl
|
|
sudo -u housecarl housecarld restore /var/backups/housecarl/housecarl-20261014T020000Z.db
|
|
sudo systemctl start housecarl
|
|
```
|
|
|
|
housecarld applies the restored configuration when it starts. The old
|
|
database is kept next to it as `housecarl.db.before-restore-<time>`.
|
|
|
|
## Upgrades
|
|
|
|
`apt upgrade` restarts a running housecarld. A new version upgrades the
|
|
database when it starts, and an older one refuses a database a newer one
|
|
has upgraded: restore a backup to go back.
|
|
|
|
## Checks
|
|
|
|
`housecarld check` validates the configuration in the database against
|
|
this machine's interfaces, the same checks as `hc status` shows
|
|
([validation.md](validation.md)).
|