housecarl/docs/operations.md

Running Housecarl

The services

systemctl status housecarl          # housecarld: the API, the web UI, the log
journalctl -u housecarl -f
systemctl reload housecarl          # reread the TLS certificate
systemctl status housecarl-ruleset  # the ruleset loaded at boot

The firewall itself is the kernel's: it stays in force while housecarld is stopped, restarted or upgraded. Without housecarld there is no API, no web UI, no log and no probation timer.

housecarld logs each change it applies, rollbacks, and what it couldn't apply. When it starts, it applies the configuration again (rolling back first what was on probation when it stopped).

Reviewing before applying

In the web UI, edits are collected as pending changes (the bar under the header), and Changes shows them with the diff of the ruleset; Apply puts them in force together. The switch in the header (Review edits / Live edits) applies each edit at once instead. With hc, see cli.md.

Changes on probation

Changes made over the network roll back unless confirmed (design.md). The web UI and hc confirm their own changes once they reach Housecarl again over a new connection, so normally nothing waits:

$ hc rule add input --from admins --service ssh
created rule input #31
applied and confirmed (history 12): this client reached Housecarl again over a new connection

A change that cuts the client off can't be confirmed that way:

applied on probation (history 13), but couldn't confirm it: … context deadline exceeded
It is rolled back within 1m50s unless confirmed ('hc confirm')

By hand (with --no-confirm, or with Settings → "a person confirms each change"):

hc status              # Probation: 2 changes roll back in 97s unless confirmed
hc confirm
hc rollback            # or undo them now

The web UI shows a banner with a countdown and both buttons. A rollback is in the history, as an undo by housecarl via rollback, so it can be undone like any change.

If you are cut off by a change: wait. When the time is up, the previous ruleset comes back by itself. If you are cut off by a confirmed change, use the console, or SSH from a source the rules still let in:

sudo hc history list
sudo hc history undo 42

The ruleset at boot

A confirmed ruleset (or one changed through the socket) is saved to /var/lib/housecarl/ruleset.nft, which housecarl-ruleset.service loads before the network comes up. To see what would load:

sudo cat /var/lib/housecarl/ruleset.nft
sudo nft -c -f /var/lib/housecarl/ruleset.nft     # check it

Dynamic sets start empty after a reboot.

Keep Debian's nftables.service disabled, or /etc/nftables.conf without flush ruleset: otherwise it deletes Housecarl's table at boot. housecarl-ruleset.service runs after it, which covers the ordinary case.

Hits

Every rule, NAT rule, built-in and policy has a counter. hc rule list, hc forward list and the web UI show packets; hc counters shows them all, with bytes. They count from housecarld's start, and carry across changes to the ruleset.

Quotas

hc quotas (and the rules page) show what each quota used and when it resets; hc rule reset-quota ID starts one again. housecarld resets quotas at midnight when their period ends (late, if it was stopped then), and saves their usage every five minutes and when it stops; after a reboot it puts it back. A reboot without a clean stop loses at most the last five minutes' count.

The log

Rules with --log (Log in the web UI) and, with hc settings set --log-drops on, what the input and forward policies drop (at most 10 packets a second) are logged:

hc log -f
hc log --search 198.51.100.7

10:02:13  hc input #95 drop:     wan  -   198.51.100.7:53868 -> 203.0.113.10:22   tcp syn
10:02:15  hc forward policy drop: wan  eth1  198.51.100.9:443 -> 192.168.1.20:51432  tcp

The packets go to netlink log group 4171, where housecarld reads them; the last 2000 are kept in memory. They aren't written to the kernel log or the journal. A rule's log isn't rate-limited: on a busy rule, give it a limit, or log only new connections (--state new).

When something doesn't get through

1. hc log -f with --log-drops on: is it dropped by a policy (no rule accepted it), or by a rule (its number is in the line)?
2. hc rule list forward (or input): hits on the rule you expect?
3. hc ruleset: the rule as nftables sees it, with the zones' interfaces and the sets' families.
4. hc status: other tables. A Docker or Incus table, or an old one, can drop what Housecarl accepts.
5. Port forwards: hc forward list hits, and the inside host's own firewall and default route (answers must go back through Housecarl).

Backups

housecarl-backup.timer runs housecarld backup daily. It writes a consistent copy of the database to backup.dir (/var/backups/housecarl/housecarl-<time>.db), and deletes those older than backup.retention (30 days), keeping at least backup.keep_at_least (7). A backup holds everything: configuration, history, users, tokens (hashed) and the audit log.

sudo -u housecarl housecarld backup                 # now
sudo -u housecarl housecarld backup /tmp/h.db       # to a file of your choice
sudo systemctl disable --now housecarl-backup.timer # no daily backups

To restore, stop housecarld first; restore refuses to replace the database under it:

sudo systemctl stop housecarl
sudo -u housecarl housecarld restore /var/backups/housecarl/housecarl-20261014T020000Z.db
sudo systemctl start housecarl

housecarld applies the restored configuration when it starts. The old database is kept next to it as housecarl.db.before-restore-<time>.

Upgrades

apt upgrade restarts a running housecarld. A new version upgrades the database when it starts, and an older one refuses a database a newer one has upgraded: restore a backup to go back.

Checks

housecarld check validates the configuration in the database against this machine's interfaces, the same checks as hc status shows (validation.md).