housecarl/docs/operations.md

# Running Housecarl

## The services

```sh
systemctl status housecarl          # housecarld: the API, the web UI, the log
journalctl -u housecarl -f
systemctl reload housecarl          # reread the TLS certificate
systemctl status housecarl-ruleset  # the ruleset loaded at boot
```

The firewall itself is the kernel's: it stays in force while housecarld
is stopped, restarted or upgraded. Without housecarld there is no API,
no web UI, no log and no probation timer.

housecarld logs each change it applies, rollbacks, and what it couldn't
apply. When it starts, it applies the configuration again (rolling back
first what was on probation when it stopped).

## Reviewing before applying

In the web UI, edits are collected as pending changes (the bar under the
header), and Changes shows them with the diff of the ruleset; Apply puts
them in force together. The switch in the header (Review edits / Live
edits) applies each edit at once instead. With hc, see
[cli.md](cli.md#changesets).

## Changes on probation

Changes made over the network roll back unless confirmed
([design.md](design.md#not-locking-yourself-out)); in shadow mode, where
a change can't cut anyone off, none do ([shadow.md](shadow.md)). The web UI and `hc`
confirm their own changes once they reach Housecarl again over a new
connection, so normally nothing waits:

```
$ hc rule add input --from admins --service ssh
created rule input #31
applied and confirmed (history 12): this client reached Housecarl again over a new connection
```

A change that cuts the client off can't be confirmed that way:

```
applied on probation (history 13), but couldn't confirm it: … context deadline exceeded
It is rolled back within 1m50s unless confirmed ('hc confirm')
```

By hand (with `--no-confirm`, or with Settings → "a person confirms each
change"):

```sh
hc status              # Probation: 2 changes roll back in 97s unless confirmed
hc confirm
hc rollback            # or undo them now
```

The web UI shows a banner with a countdown and both buttons. A rollback
is in the history, as an undo by `housecarl` via `rollback`, so it can be
undone like any change.

If you are cut off by a change: wait. When the time is up, the previous
ruleset comes back by itself. If you are cut off by a confirmed change,
use the console, or SSH from a source the rules still let in:

```sh
sudo hc history list
sudo hc history undo 42
```

## The ruleset at boot

A confirmed ruleset (or one changed through the socket) is saved to
`/var/lib/housecarl/ruleset.nft`, which `housecarl-ruleset.service`
loads before the network comes up, so before PPPoE, WireGuard or DHCP
bring up their interfaces. To see what would load:

```sh
sudo cat /var/lib/housecarl/ruleset.nft
sudo nft -c -f /var/lib/housecarl/ruleset.nft     # check it
```

In shadow mode both files are shadow mode's, and the fallback filters
nothing ([shadow.md](shadow.md#what-works-differently)).

Dynamic sets start empty after a reboot. Sets from Jarl have the
addresses they last had, in the saved ruleset and again when housecarld
starts, whether Jarl is reachable yet or not ([jarl.md](jarl.md#when-jarl-cant-be-reached)).
The saved ruleset has no
flowtable, since its devices may not exist yet: housecarld adds it when
it starts, and whenever a flowtable device appears
([flowtable.md](flowtable.md#devices-that-come-and-go)).

### If the ruleset fails to load at boot

Should `ruleset.nft` fail to load (a damaged file, an nftables or kernel
upgrade that refuses something in it), the machine would otherwise boot
with no firewall at all until housecarld starts. Instead,
`housecarl-ruleset.service` loads a **fallback** that housecarld saves
beside it, `/var/lib/housecarl/ruleset-fallback.nft`, from the same
confirmed configuration:

* **Nothing new gets in from the external zones' interfaces**, to the
  firewall or through it: no port forwards, no SSH or web UI from the
  internet.
* What the firewall and the inside need to keep working is let in:
  established connections, the ICMPv6 IPv6 needs (neighbour discovery,
  router advertisements, packet too big), and DHCPv6 replies from
  link-local addresses.
* **Everything else is let through**: the LAN, WireGuard and the
  firewall itself are reachable from the inside, so you can log in and
  fix it.
* IPv4 leaving an external zone is masqueraded (when any source NAT is
  configured), so the inside still reaches the internet. MSS clamping
  stays, if it is on.

PPPoE itself isn't affected: its discovery and session are Ethernet
frames, and the PPP link's negotiation happens before any IP, neither
of which an `inet` table sees.

The fallback only knows which interfaces face the internet from the
zones marked **external**. With none, it lets nothing but loopback
through, and the warning `no_external_zone` says so.

When the fallback loads, the unit is marked failed, and the journal says
why (as a critical message):

```sh
systemctl status housecarl-ruleset
journalctl -b -u housecarl-ruleset
```

housecarld puts the real ruleset back when it starts, and saves a fresh
`ruleset.nft`. If it can't apply it either, the fallback stays in force:
`hc status` and the dashboard say "NOT in force: the boot fallback is in
force", and `/healthz` answers 503 ([Monitoring](#monitoring)). Fix
what `hc status` or the journal reports, then restart housecarld.

Keep Debian's `nftables.service` disabled (`systemctl disable
nftables`, without `--now`): at boot it loads `/etc/nftables.conf`, and
stopping or restarting it runs `nft flush ruleset`, which deletes every
table, Housecarl's included. `housecarl-ruleset.service` runs after it
at boot, which covers the ordinary case, but not a restart of
`nftables.service` later. Housecarl never reads or writes
`/etc/nftables.conf`; its ruleset is saved in
`/var/lib/housecarl/ruleset.nft`. The dashboard and `hc status` warn
while `nftables.service` is enabled.

Other programs' tables are left alone: fail2ban's (`inet f2b-table`),
Docker's, Incus's. Housecarl replaces only its own table, so they keep
their rules through every Housecarl change and restart, and nothing
needs restarting in a particular order. A packet must get past every
table on its hook, so a drop in any of them wins.

## Hits

Every rule, NAT rule, built-in and policy has a counter. `hc rule list`,
`hc forward list` and the web UI show packets; `hc counters` shows them
all, with bytes. They count from housecarld's start, and carry across
changes to the ruleset. A deleted object's counter goes with it; an
import with `--replace` gives every object a new ID, so its counters
start again. A disabled rule keeps its count for when it is enabled.

## Quotas

`hc quotas` (and the rules page) show what each quota used and when it
resets; `hc rule reset-quota ID` starts one again (`--address` one device's,
`--give 5gbytes` a top-up until the next reset). housecarld resets
quotas at midnight when their period ends (late, if it was stopped then),
and saves their usage every five minutes, after every change and when
it stops; after a reboot it puts it back. A reboot without a clean stop
loses at most the last five minutes' count.

## The log

Rules with `--log` (Log in the web UI) and, with `hc settings set
--log-drops on`, what the input and forward policies drop (at most 10
packets a second) are logged:

```sh
hc log -f
hc log --search 198.51.100.7
```

```
10:02:13  hc input #95 drop:     wan  -   198.51.100.7:53868 -> 203.0.113.10:22   tcp syn
10:02:15  hc forward policy drop: wan  eth1  198.51.100.9:443 -> 192.168.1.20:51432  tcp
```

In shadow mode ([shadow.md](shadow.md)) what the policies would drop is
always logged, in all three chains, and drops are marked as what would
have been done: the packets went on.

The packets go to netlink log group 4171, where housecarld reads them;
the last 2000 are kept in memory. They aren't written to the kernel log
or the journal. A rule's log isn't rate-limited: on a busy rule, give it
a limit, or log only new connections (`--state new`).

## Connections

The built-in that accepts established connections lets their packets
through without checking the rules again: a rule added later doesn't
stop a connection already open. Traffic → Connections (`hc connections`)
lists what the kernel tracks, with NAT (`to 192.168.1.10:443`: a port
forward; `as 203.0.113.10`: masqueraded) and the time left:

```sh
hc connections --search 192.168.1.50
hc connections --search ":443 tcp"
hc connections close --search 192.168.1.50
```

Closing a connection deletes its tracking entry: its next packet is
checked by the rules as they are now. One they block is stopped; one they
allow carries on (TCP connections are picked up again mid-stream).

To make a block stop open connections by itself, tick "every packet" on
the rule (`--every-packet`): it is checked before established connections
are accepted, like rules with a quota ([rules.md](rules.md#rules)).

## Tracing

Traffic → Trace (`hc trace`) follows packets through the whole ruleset:
Housecarl's rules and every other table (Docker's, fail2ban's, an old
`inet main`), each rule and policy they meet, and what accepted or
dropped them:

```
$ hc trace --address 192.168.1.50 --port 443 --seconds 60
14:02:13.120  192.168.1.50:51432 -> 198.51.100.9:443  tcp syn in eth1 out wan
    inet housecarl forward: built-in: mss-clamp -> continue
    inet housecarl forward: rule forward #12 (kids at night) -> drop
    => DROPPED by rule forward #12 (kids at night)
```

It traces packets from or to the address, on the port (either way), for
5 to 600 seconds, at most 200 packets and 20 a second. It works with
nftables' own tracing: a table of its own (`inet housecarl-trace`,
hooked before everything else) marks the packets, and `nft monitor
trace` reports them; the table goes when the trace ends. One trace runs
at a time.

## When something doesn't get through

1. Trace it (above): it names the rule or policy that dropped it, in
   whichever table.
2. `hc log -f` with `--log-drops on`: is it dropped by a policy (no rule
   accepted it), or by a rule (its number is in the line)?
3. `hc rule list forward` (or `input`): hits on the rule you expect?
4. `hc ruleset`: the rule as nftables sees it, with the zones' interfaces
   and the sets' families.
5. `hc status`: other tables. A Docker or Incus table, or an old one,
   can drop what Housecarl accepts.
6. Port forwards: `hc forward list` hits, and the inside host's own
   firewall and default route (answers must go back through Housecarl).

## Backups

`housecarl-backup.timer` runs `housecarld backup` daily. It writes a
consistent copy of the database to `backup.dir`
(`/var/backups/housecarl/housecarl-<time>.db`), and deletes those older
than `backup.retention` (30 days), keeping at least
`backup.keep_at_least` (7). A backup holds everything: configuration,
history, users, tokens (hashed) and the audit log.

```sh
sudo -u housecarl housecarld backup                 # now
sudo -u housecarl housecarld backup /tmp/h.db       # to a file of your choice
sudo systemctl disable --now housecarl-backup.timer # no daily backups
```

To **restore**, stop housecarld first; restore refuses to replace the
database under it:

```sh
sudo systemctl stop housecarl
sudo -u housecarl housecarld restore /var/backups/housecarl/housecarl-20261014T020000Z.db
sudo systemctl start housecarl
```

housecarld applies the restored configuration when it starts. The old
database is kept next to it as `housecarl.db.before-restore-<time>`.

## Upgrades

`apt upgrade` restarts a running housecarld. A new version upgrades the
database when it starts, and an older one refuses a database a newer one
has upgraded: restore a backup to go back.

## Monitoring

`GET /healthz` needs no login, and answers within a few seconds of any
change (it reads the ruleset at most every five seconds, however often
it is asked):

```sh
$ curl -s https://firewall.example.net:8070/healthz
{"ok":true,"version":"0.1.0","firewall":"applied","mode":"enforcing","probation":0,"warnings":[]}
```

| Field | |
|---|---|
| `ok` | `true` with HTTP 200 when the firewall is as it should be; `false` with HTTP **503** otherwise |
| `firewall` | `applied` (Housecarl's ruleset is in force), `unconfigured` (nothing configured yet, so nothing to apply), `fallback` (the [boot fallback](#if-the-ruleset-fails-to-load-at-boot) is in force), `missing` (no Housecarl table: flushed, or never applied) |
| `mode` | `enforcing`, or `shadow`: Housecarl's ruleset is loaded but blocks nothing, and another firewall decides ([shadow.md](shadow.md)). `ok` doesn't depend on it; to be told when a machine isn't enforcing, also expect `"mode":"enforcing"` |
| `probation` | changes waiting to be confirmed, rolled back unless they are |
| `warnings` | codes of problems with the machine, as `hc status` shows them (`nftables_service`) |

For Checkmk, an HTTP check on the URL that expects status 200 (and,
if you like, the string `"firewall":"applied"`) covers housecarld being
down, the fallback and a missing ruleset. Reach it from where the input
rules let the monitor in (a rule for the Checkmk server's address and
port 8070).

## Failed logins in the journal

A failed login to the web UI or the API, and a refused API token, are
written to the journal as well as the audit log, with the address first:

```
level=WARN msg="login failed" ip=203.0.113.9 user=admin reason="wrong username or password"
level=WARN msg="token refused" ip=203.0.113.9 reason="wrong token secret"
```

Guard's `housecarl` preset, or a fail2ban filter, can ban such addresses
([bans.md](bans.md)). The address comes before the user name, which a
caller chooses, so a pattern anchored on `msg="login failed" ip=` can't
be fooled by a name. (With `log.format: json` they are JSON objects with
the same fields, in the same order.)

## Checks

`housecarld check` validates the configuration in the database against
this machine's interfaces, the same checks as `hc status` shows
([validation.md](validation.md)).