housecarl/docs/development.md

Developing Housecarl

Layout

cmd/housecarld/       the daemon: serve, init, user, check, backup, restore, config
cmd/hc/               the CLI
internal/
  model/              the configuration (State: settings, zones, sets, services, rules, NAT), users, history
  validate/           whole-configuration checks with stable codes (docs/validation.md)
  render/             State → the nftables ruleset text, with each line's object
  nft/                runs nft: check, apply (keeping dynamic set entries and counters), list, the boot file
  nflog/              reads logged packets from netlink log group 4171
  linkwatch/          netlink: interfaces that appear or go (the flowtable's devices)
  tablewatch/         netlink: a table deleted (shadow mode puts Housecarl's back; docs/shadow.md)
  importer/           nft -j list ruleset → State, with notes and raw rules (docs/import.md)
  hostif/             the machine's interfaces
  store/              SQLite (modernc.org/sqlite, no cgo): configuration, history, users, tokens, sessions, audit
  engine/             configuration changes: edit a copy, diff, validate, check and apply, save with history, undo
  server/             the HTTP API and the web UI, authentication, the local socket, probation, the log ring
  api/, client/       JSON types and the Go client shared by server and CLI
  auth/               passwords (argon2id), tokens, sessions, grants and roles
  certs/, jarl/       the web UI's certificate through Jarl (docs/certificates.md); jarl/ is the Jarl API client
  jarlsets/           sets from Jarl: reads the names, follows CNAMEs, keeps what was read, within (docs/jarl.md)
  config/             housecarld's YAML configuration
  version/            what a build reports
  web/static/         the web UI: plain ES modules, no build step, embedded with go:embed
deploy/               systemd units, sysusers.d and tmpfiles.d files, Debian package scripts
scripts/e2e.sh        end-to-end test with real traffic through a router container
scripts/fakejarl/     the bit of Jarl's API sets from Jarl read, for the e2e test
scripts/mkdeb.sh      the Debian packages (make deb)
scripts/logo.py       the web UI's logo from artwork/logo.svg
scripts/banner.py     the README's banner from the web UI's logo and wordmark

design.md describes how the pieces fit.

Building and testing

make            # bin/housecarld, bin/hc, bash completion
make test       # go vet, gofmt check, go test ./...
make e2e        # scripts/e2e.sh: needs Docker
make deb        # dist/housecarl_<version>_<arch>.deb and housecarl-cli_...

Never apply a test ruleset to your own machine's nftables. Tests that need the real nft run it in a container.

Unit tests need no nftables: the API tests (internal/server) use a fake firewall. The render tests compare the ruleset with what is expected, line by line, and the importer tests import the maintainer's router ruleset (internal/importer/testdata) and check the result.

Real nftables: with HOUSECARL_NFT set to a command that runs nft somewhere harmless, the render, nft and importer tests also have nft check (and apply) their rulesets. A debian:trixie image with nftables 1.1.3 (Debian 13's) is the reference:

make e2e         # makes the housecarl-nft image, if missing
export HOUSECARL_NFT="docker run --rm -i --cap-add NET_ADMIN housecarl-nft nft"
export HOUSECARL_NFT_IMAGE=housecarl-nft
make test

Newer nftables accepts things 1.1.3 doesn't (a meter with a timeout for ct count, hour ranges past midnight): keep to what 1.1.3 takes.

The e2e test builds a small network in Docker from the same image:

• a router container (housecarld, the hc CLI) on three networks named wan, enp2s0 and wg0 inside it, IPv4 and IPv6;
• "the internet" on the WAN side, a LAN client, and a "jellyfin" host.

It imports a copy of the maintainer's ruleset (lab addresses) with housecarld in shadow mode beside it (traffic unchanged, a would-be drop logged and let through, the table back after the old ruleset is reloaded), restarts it enforcing, takes over the old table, and checks with real traffic: port forwards with hairpin NAT and the IPv6 pinhole, masquerading, antispoofing, the drop log, a lockout change that rolls back by itself, a restart, the boot ruleset after a flush, and that other tables are left alone. It makes the housecarl-nft image itself if it is missing.

KEEP=1 leaves the lab running (the web UI on 10.88.10.1:8070); DOCKER_HOST points at another Docker daemon. Root isn't needed on the host.

The systemd sandbox (capabilities, the boot unit) is only tested by installing the package in a container that runs systemd (--privileged --cgroupns=host /sbin/init).

Certificates

As in Steward: internal/certs has unit tests with a fake Jarl and an in-process DNS server; a real order is tested against Jarl's e2e environment and Pebble (Steward's docs/development.md has the steps).

The web UI

Plain JavaScript modules under internal/web/static/, built with the same helpers as Jarl's and Steward's:

• All DOM goes through h() in ui.js, never innerHTML.
• The Content-Security-Policy allows only the app's own scripts and styles, so no inline styles. Use a class in style.css, or set el.style.prop.
• The files are embedded, so rebuild housecarld after changing them.

There are no JavaScript tests: bundle assets/app.js with esbuild to catch syntax and import errors, and look at the pages in a browser against the e2e lab.

Conventions

• Every user-visible change goes into CHANGELOG.md under "Unreleased", with the docs updated in the same change.
• Errors say what is wrong and how to fix it, naming the object.
• New validation rules get a stable code, a test, and a row in validation.md.
• Configuration changes go through the engine, never straight to the store, so they are validated, checked by nftables, recorded and undoable.
• Schema changes append a migration to internal/store/store.go; never edit a shipped one.
• gofmt and go vet clean (make test).

Releasing

As for Jarl and Steward:

1. Move "Unreleased" in CHANGELOG.md under ## [<version>] - <date>. A release candidate may leave it there.
2. Put the version in VERSION, and update the README's status line.
3. Run make test && make e2e.
4. Commit as "Release X", tag it git tag -a vX -m "Housecarl X", and push.

A fix to a release candidate may go out as a build of it rather than a new candidate: 0.2.0-rc.3+git<count>-<commit>, where <commit> is the short ID of the last commit before "Release X" and <count> is git rev-list --count of it. The count orders the builds (apt compares it as a number; the ID alone would order them at random); each sorts after 0.2.0-rc.3 and before 0.2.0-rc.4. Debian writes it 0.2.0~rc.3+git<count>~<commit>.

The tag starts .gitea/workflows/release.yml, which checks the tag against VERSION and the CHANGELOG, runs the tests, builds the tarballs, the CLI for each OS and the .debs, and publishes them (scripts/gitea-release.sh). It needs the repository secret RELEASE_TOKEN (write:repository, write:package).