housecarl/CHANGELOG.md

# Changelog

All notable changes to Housecarl are documented here. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow
[Semantic Versioning](https://semver.org/).

## [Unreleased]

### Added

- **Sets from Jarl** ([docs/jarl.md](docs/jarl.md#sets-from-jarl)). A set
  can hold the addresses of names in Jarl instead of listing them:
  every A and AAAA record of each name, CNAMEs followed (within Jarl's
  zones, in the same view), one view per name (`NAME@VIEW`; a name more
  than one view covers must say which). housecarld reads them through
  Jarl's API every `jarl.refresh` (a minute) and after each change, and
  when they change applies the ruleset again, without a history entry
  (audited as `set.jarl`). A set uses only the addresses `within` the
  zones (their connected networks, which grow with the zone) and
  networks it names; others are left out and warned of (`jarl_outside`).
  While Jarl can't be read, sets keep the addresses they had, saved in
  the database for restarts and in the boot ruleset (`jarl_unreachable`).
  Names that give nothing say why (`jarl_name`: no record, a wildcard
  only, a CNAME out of Jarl or in a loop, below a delegation, a secondary
  zone, more than one view). `hc set add --jarl NAME@VIEW --within ZONE`,
  `hc set jarl SET [--refresh]`; the kind "from Jarl" and Addresses… on
  the Sets page; `GET /sets/{name}/jarl`, `POST /sets/{name}/jarl/refresh`.
  Housecarl's Jarl token needs `records:read` for the zones.
- An administrator can make an API token for another user in the web
  UI: New token on the user's row under Users (the CLI's `hc token create
  --user` and the API could already).
- **Ban lists** ([docs/bans.md](docs/bans.md)). A program (Guard, which
  reads logs, or fail2ban) bans an address, of every port or of some,
  for a time; the ban list says what the address is kept from: the
  firewall itself (`firewall`), some hosts behind it (`hosts`, port
  forwards to them included), or `everything`. Bans are dropped ahead of
  every rule, established connections too, and the banned address's
  connections are closed. IPv6 addresses are banned by /64 (`group_v6`).
  Never banned: inside networks, the firewall's own addresses, whoever
  asks, and the list's `never`. Bans end by themselves, are cut to the
  list's longest time, survive restarts and reboots, and are audited, but
  aren't configuration (no history, probation or changesets). `hc
  banlist`, `hc bans`, `hc ban`, `hc unban`; Traffic → Bans in the web UI;
  `/banlists` and `/banlists/{ref}/bans` in the API; a `bans` grant that
  can name lists (`bans:write:web`), in the `viewer` and `operator`
  roles.
- **A fail2ban action**, `contrib/fail2ban/action.d/housecarl.conf`
  (installed under `/usr/share/doc/housecarl/contrib/`): fail2ban's bans
  go into a ban list, with the jail's port and protocol.
- **Failed logins in the journal.** A failed login and a refused API
  token are logged (`msg="login failed" ip=… user=…`), for Guard and
  fail2ban to act on.

- **Quota usage as a share.** What a quota used shows its percentage
  (`12.3 GB of 50.0 GB used (24%)`, each address's in the quota
  dialog), on the Rules page, the dashboard and in `hc quotas`.
- **Reset one address's quota, or give some back.** `hc rule
  reset-quota ID --address ADDR` starts only that device's allowance
  again; `--give 5gbytes` takes that much off what was used instead (one
  address's, or everyone's): extra allowance until the next reset,
  without changing the rule. In the web UI, a rule's quota is one
  line (what it used, how it resets) that opens a dialog with the rest:
  usage per address, last and next reset, and Reset and Give back for
  everyone or one address. `POST /rules/{ref}/reset-quota` takes `address` and `give`.
- **Names beside inside addresses.** Connections, the log, quota usage,
  the dashboard's quotas and the statistics' devices and blocked
  sources show each inside address's name, found by a reverse lookup:
  `laptop 192.168.1.20` in the web UI, `192.168.1.20 (laptop)` in `hc`.
  Only inside addresses are looked up (an inside zone's networks, and
  private ones outside the external zones'). Names are cached for 10
  minutes and never stored. `names.enabled`, `names.resolver` and
  `names.cache` in housecarl.yaml; `hc --no-names`; `POST /names`.
- **Quotas on the dashboard.** With any rule holding a quota, the
  dashboard lists them all: the allowance, what is used (for a
  per-address quota, how many addresses and the busiest) with a bar, and
  the next reset. Each opens its rule.
- **Links to a rule open it.** A rule named in the matrix, the log, the
  tracer, statistics or the dashboard's busiest rules opens the Rules
  page on its chain, scrolled to it and highlighted (`#/rules?rule=ID`).
  The Rules page shows each rule's ID (`#410`, as `hc` and labels name
  it) beside its order.
- **A clearer zone matrix.** A cell reads allow, block, or **partly**
  (some new connections let through, the rest not), with the exceptions
  counted by what they do (`+3 -1` instead of `+4`), and what they let
  through: services with the addresses they're limited to (`ssh from
  trusted`, `5 port forwards`), under the grid in `hc matrix`, in each
  cell on the Matrix page. A zone to itself is marked (only traffic routed
  between its interfaces passes the firewall). The details list each
  rule's hits. API: `summary`, `allowed`, `blocked`, `open`,
  `same_zone`, `note`, `hits`, and `short` and `hits` per exception.
  Raw rules that are a match and then accept, drop or reject (without
  interfaces) are read instead of shown as unknown.
- **Shadow mode** (`shadow.enabled` in housecarl.yaml; docs/shadow.md):
  Housecarl's ruleset is loaded beside the firewall you have but blocks,
  rejects and translates nothing. Drops and rejects are counted and accept
  instead (an accept only ends Housecarl's chain), policies accept and
  always log what they would drop, NAT only counts, MSS clamping and the
  flowtable are left out, and raw rules that do more than accept, drop or
  reject are left out with a warning (`shadow_raw_left_out`). Comments
  mark what would happen (`[would drop]`). `shadow.position`: Housecarl's
  filter chains run after the other firewall's (`filter + 10`, the
  default: what it would drop is what switching would break) or before
  (`- 10`); its NAT chains always first. Every change is checked by
  nftables in both forms, so leaving shadow mode can't fail. No
  probation; takeovers and closing connections are refused (409). The
  boot ruleset is the shadow one, and its fallback filters nothing. When
  a reload (`flush ruleset`) deletes Housecarl's table, housecarld puts it
  back (it listens to nftables' netlink notifications). `mode` in
  `/status`, `/healthz` and `/auth/whoami`; a banner in the web UI;
  would-be drops marked in the log (`shadow`) and the tracer
  (`would_drop`); `hc ruleset --enforcing` (`GET /ruleset?enforcing=1`)
  shows the ruleset for after. `shadow_unprotected` warns when no other
  table filters the input hook; `nftables_service` and `takeover_pending`
  aren't raised in shadow mode. The e2e test imports in shadow mode
  before taking over.
- **Zone names at import:** `hc import --zone enp2s0=lan` (repeat;
  interfaces given one name share a zone; Zone names on the Import
  page), and a note listing each zone and where its name came from.
  Defines holding a VLAN sub-interface (`enp2s0.20`) name their zone,
  and `_ifname`, `_device` and `_nic` are taken off define names too
  (docs/import.md, "How things are named").
- Import: sets with `flags timeout` become dynamic sets; a DNAT of
  several ports becomes one port forward per port.

### Changed

- `hc user delete` asks first, as every other delete does, and takes
  `-y`/`--yes` not to; it deleted the user and their tokens at once.
- The ruleset's diff (Changes, `hc changeset diff`, `--dry-run`) names
  the chain each part is in on its `@@` line (`@@ -40,6 +40,8 @@ chain
  forward`), as `git diff` names the function, so a rule added deep in a
  long chain says whether it is in input, forward or output.
- **History shows the ruleset's diff.** An entry's page (and `hc history
  diff ID`, `GET /history/{id}/diff`) shows what the change did to the
  ruleset, for entries made before this version too: the configuration
  before and after it is got back from the current one and rendered by
  this version.
- Negating a match is said first: the zone, From and To fields (rules
  and NAT) start with "any of" / "any but" instead of a checkbox under
  the list, shown once the list has an entry, and an "any but" list is
  marked with a red bar. An empty list is never saved as negated (it
  matched everything anyway, but would have negated what was added to
  it later).
- docs/rules.md: "Matching everything but a few", on leaving some
  addresses out with a zone and "any but", and what to do when the range
  isn't a zone.
- The rule dialog shows only the zone fields its chain can use: no
  "Going out on" for input and early, no "Coming in on" for output
  (they could only be refused). A zone left in a field the chain hides
  isn't sent.
- History lists a changed setting by its own fields
  (`builtins.drop_invalid: false → true`), on the web page and in `hc
  history show`, instead of the whole `builtins` before and after.
- Set names starting with `ban_` are now reserved (`set_name_reserved`):
  ban lists' sets use them.
- **Rules read as a direction.** A match's zones are written `lan → wan`,
  `any but wan → any`, or with the firewall itself as the other end
  (`lan → firewall` in input, `firewall → wan` in output), instead of
  `in lan · out wan` and `in not wan`; exclusions read `from any but X`.
  On the Rules and NAT pages and in `hc`; the flags and JSON stay as
  they were.

### Fixed

- `hc forward show` shows the forward's hits and comment, and its
  hairpin as `no`, `yes` or `yes, from …` (it printed `no -`).
- A history entry's ruleset diff showed the object it changed as moved:
  stepping back put the old form at the end of its list, so updating
  one port forward (turning hairpin off) also showed its forward and
  DNAT lines moving down. The diff is now only what the change did, and
  its sides are named `before N` and `after N` instead of `now`.
- Ruleset diffs numbered the hunks after one with a moved line too early
  on the new side (`@@ -76,9 +74,9 @@` for `+76,9`), so the chain named
  after `@@` could be the one before (`chain output` for a hunk in
  `chain dstnat`). go-udiff miscounts them; Housecarl now numbers them
  itself.
- The dashboard's bar for a per-address quota showed only the busiest
  address, so one address that used up its allowance filled it (100%).
  The bar is now all the addresses together, and the dashboard and the
  rule's quota line say how many addresses used theirs up.
- Changing how a quota counts lost or muddled what it used: per address
  to a total started at nothing, per source to per destination kept the
  sources' use as if it were the destinations', and other groups left
  the old entries behind. Now a total starts at the addresses' sum,
  wider groups add up their addresses, and what can't be carried starts
  again with a warning (`quota_restarts`) on the change.
- What the quotas used is saved after every change, not only every five
  minutes: a reboot soon after a change that counts a quota differently
  could bring back its usage counted the old way.
- Statistics' Quotas card linked every entry to the Rules page, not to
  its rule; each now opens its rule, named. Per-address quotas were
  missing from it (and from the `quota:rN` series): they count their
  addresses' sum.
- `hc stats top devices` listed each device twice (once for what it
  received, once for what it sent); it is one row with both again.
- `hc rule add --before/--after/--first` made two changes (create at the
  end, then move), so two history entries, two probations, and the rule
  briefly in force at the end of its chain; staged in a changeset, the
  place was lost. The create now takes the place (`POST
  /rules?before=ID`, `after`, `first`) and makes one change.
- `hc matrix` in a terminal: the colours' escape codes threw the columns
  out of line.
- `hc counters` (`GET /counters`) kept listing the objects of a
  configuration replaced since housecarld started (`hc import
  --replace` gives every object a new ID), beside the current ones; the
  totals of objects that no longer exist are now forgotten.
- Import: two base chains on one hook had their rules imported twice,
  and the second chain's leading built-ins were moved above the first
  chain's rules; they now follow each other by priority (read in a
  fixed order), with the stricter policy, and a note.
- Import: a rule inside a jumped-to chain that stays raw kept only its
  own text and so applied to everything; the jump's condition now goes
  in front. A jump and a rule matching the same field are combined only
  when one's addresses lie within the other's; else the rule stays raw.
  A jump's ports, ICMP types, connection status and hop limit were
  dropped; a limit or log on the jump is no longer silently lost.
- Import: a second match on the same field (`ip saddr A ip saddr != B`)
  replaced the first; `!=` on the hop limit and TCP flags was read as
  `==`. Such rules now stay raw, as written.
- Import: a forward-chain accept to a forwarded host and port without
  `ct status dnat` was folded into the port forward, which only lets
  forwarded connections through; it stays a rule now. An IPv6 accept
  limited to an interface no longer becomes a pinhole open to all.
- Import: raw rules naming a set of the old table made nftables refuse
  the ruleset; address sets are now renamed `NAME_v4`/`NAME_v6`, other
  sets written out with their elements, and what can't be is left out
  with a note. Rules naming a set that isn't imported stay raw instead
  of failing validation.
- Import: a DNAT to an IPv6 address, or with matches a port forward
  can't hold (an interface excluded, a limit, a log), is left out with a
  note instead of being imported without them.

## [0.1.0] - 2026-10-19

The first version of Housecarl, a firewall manager for nftables, built
the way Jarl and Steward are.

### Added

- **Zones, address sets and services:** interfaces grouped by what they
  connect to (wildcards like `incusbr*`; external zones face the
  internet); sets of IPv4 and IPv6 addresses, prefixes and ranges, mixed;
  ports and ICMP types by name, with twenty built in (docs/rules.md).
- **Rules** in four chains (input, forward, output, and early, before
  connection tracking), in order, matching by zone, address, service,
  family, connection state and status, days and hours, a rate (also per
  source address), and a connection count; accept, drop, reject (ICMP or
  TCP reset) or log. Raw nftables rules for the rest.
- **IPv4 and IPv6 together:** one rule becomes the nftables rules each
  family needs.
- **NAT:** port forwards as one object (DNAT, the forward accept, hairpin
  NAT, and optionally an IPv6 pinhole to a host's own port); source NAT
  (masquerade or an address; NAT66 on request); redirects to the firewall
  (docs/nat.md).
- **Built-ins**, each switchable: loopback, established connections,
  invalid packets, ICMPv6 (RFC 4890), ping from internal zones, DHCPv6
  client replies, antispoofing with your own prefixes, MSS clamping, and
  a flowtable fast path.
- **Changesets, as in Jarl:** changes staged to be reviewed and applied
  together, as one history entry. Each is checked as it is staged; the
  review shows the problems, nftables' check and the diff of the ruleset.
  The web UI collects edits as pending changes by default (pending rows,
  a bar with Review and Apply now, a Changes page), with a switch for live
  edits, and Undo in History is staged like any edit; hc stages with
  `-C ID` or `HOUSECARL_CHANGESET`, and has `hc
  changeset create/list/show/diff/apply/discard/remove`. An applied
  changeset is on probation like any change. `--dry-run` shows the
  ruleset's diff too (docs/design.md, docs/cli.md).
- **Bandwidth limits and quotas:** rate limits in bytes (`--limit
  20mbytes/second,over,per-source`), per source or per destination;
  quotas, an allowance of traffic per rule or per address, reset daily,
  weekly, monthly on a chosen day, or by hand, kept across changes,
  restarts and reboots (`hc quotas`, `hc rule reset-quota`, usage on the
  rules page). Rules that count traffic are checked before established
  connections are accepted, so they see every packet. Per-address limits
  group IPv6 by /64 by default (`--group-v4`, `--group-v6`). Meters,
  byte rates and quotas in an imported ruleset carry over
  (docs/rules.md).
- **Changes checked twice and applied atomically:** validation of the
  whole configuration with stable problem codes (docs/validation.md),
  then `nft -c`, then one `nft -f` transaction that replaces Housecarl's
  own table, `inet housecarl`, and nothing else. Dynamic sets' entries
  and the counters carry across.
- **Probation:** a change made over the network rolls itself back unless
  confirmed within two minutes. The web UI and `hc` confirm their own
  changes once they reach Housecarl again over a new connection, from the
  same address (a change that cuts its client off can't, and rolls back);
  `--no-confirm` and the "a person confirms each change" setting leave it
  to `hc confirm` or the banner;
  changes through the local socket are kept at once; a restart rolls
  back what was pending. No rule is forced to keep SSH or the web UI
  open.
- **At boot:** the confirmed ruleset is loaded by
  `housecarl-ruleset.service` before the network comes up, without
  housecarld. An empty configuration applies nothing. It fails closed:
  if the saved ruleset doesn't load, a fallback saved beside it
  (`ruleset-fallback.nft`) is, until housecarld puts the real one back:
  nothing new gets in from external zones, the inside keeps working
  (masqueraded), and the unit is marked failed with a critical message.
  `hc status`, the dashboard and `/healthz` say while the fallback is in
  force; the warning `no_external_zone` says when no zone is external,
  which would make the fallback close everything (docs/operations.md).
- **The flowtable and interfaces that come and go:** the boot ruleset
  has no flowtable (nftables on Debian 13's kernel refuses one naming a
  device that doesn't exist yet, such as `ppp0` before PPPoE is up, and
  with it the whole file); housecarld watches the kernel's link messages
  and applies the ruleset again when a flowtable device appears or comes
  back as a new interface (PPPoE reconnecting). A guide to when the
  flowtable helps, and what it costs (docs/flowtable.md).
- **`GET /healthz` for monitoring:** `ok`, `firewall` (applied,
  unconfigured, fallback, missing), changes on probation and the
  machine's warnings, with HTTP 503 unless the firewall is in force;
  no login needed, read at most every five seconds.
- **Import keeps a chain's head in order:** once a would-be built-in at
  the head of a chain has to stay a rule (an invalid drop the forward
  chain needs after an exception), the built-ins after it stay rules
  too, in their place, instead of moving ahead of it: an ICMPv6 accept
  no longer jumps ahead of the invalid drop it came after.
- **Renaming a zone, set or service** changes the rules, NAT rules and
  port forwards that name it, in the same change (one history entry, one
  undo); before, a name in use couldn't change. Dynamic sets keep their
  names (their tokens and programs go by them).
- VLAN sub-interfaces (`enp2s0.20`) in zones and the flowtable, tested
  with tagged traffic (docs/rules.md).
- `hc settings set --flowtable none` (and `none` for the other lists)
  empties the list.
- **Warnings about the machine:** while `nftables.service` is enabled
  (it loads `/etc/nftables.conf` at boot, and flushes every table when
  stopped), and while the table an import came from still applies.
- **Statistics** (Traffic → Statistics, `hc stats`, `/api/v1/stats/...`):
  traffic per interface and zone, drops by chain and when they happen,
  hits per rule and the rules that matched nothing, the sources blocked
  most and what they tried, connections against the kernel's limit, each
  quota's use, and housecarld's and the host's memory and load, sampled
  every minute into a database of their own (`stats.db`, kept 7 days per
  minute, 90 days per hour, 2 years per day). On by default; traffic per
  inside device is a setting (`--stats-per-device`) that adds counters
  to the forward chain. Two charts on the dashboard; a warning when the
  connection tracking table is nearly full (docs/stats.md).
- Set names starting with `meter_`, `conn_`, `quota_` or `stats_` are
  refused (`set_name_reserved`): Housecarl's own sets use them, and a set
  of yours named so could clash with them.
- **The zone matrix** (Matrix, `hc matrix`): what the rules do to new
  connections between zones and to and from the firewall, worked out
  from the rules in order, with each pair's exceptions; rules about
  networks connected to other zones are left out of a pair.
- **Connections and tracing:** the connections the kernel tracks, with
  their NAT, and closing them so the rules check them again (Traffic →
  Connections, `hc connections`); a rule option, "every packet", for
  blocks that stop connections already open; and a packet tracer
  (Traffic → Trace, `hc trace`) that follows packets through every
  table's rules and names what accepted or dropped them.
- **Hits and the log:** packet and byte counters for every rule, NAT rule,
  built-in and policy; packets logged by rules and policies, read from
  netlink log group 4171 (`hc log -f`, the Log page).
- **Dynamic sets** whose entries expire by themselves, changed through
  their own API with tokens that may be limited to one set
  (`sets:write:guests`).
- **Import** of an existing nftables ruleset (`hc import`): rules keep
  their order, defines name the sets, DNAT becomes port forwards, what
  can't be translated stays as raw rules, with notes; `hc takeover`
  deletes the old table, on probation (docs/import.md). DNAT maps and
  verdict maps are expanded into one port forward or rule per element.
- **The `hc` CLI, the REST API and the web UI** (dashboard, rules per
  chain with hits and reordering, NAT, zones, sets, services, the log,
  settings, import, history with undo; the rule dialog keeps the less
  used parts, such as rate limits, quotas and time windows, behind "+"
  buttons; each chain's built-ins are listed, with their hits, and
  switched in a window), with users, roles, scoped API
  tokens, sessions, the audit log, and the web UI's certificate from
  Let's Encrypt through Jarl, as in Steward.
- **Packaging:** Debian packages `housecarl` and `housecarl-cli`, a
  systemd unit whose only privilege is `CAP_NET_ADMIN`, daily backups.
- **The Housecarl logo:** a carved Nordic doorway in Jarl's blue, with a
  simplified version for small sizes, lighter on dark backgrounds; and a
  HOUSECARL wordmark in Jarl's and Steward's angular letters.