housecarl/docs/development.md

# Developing Housecarl

## Layout

```
cmd/housecarld/       the daemon: serve, init, user, check, backup, restore, config
cmd/hc/               the CLI
internal/
  model/              the configuration (State: settings, zones, sets, services, rules, NAT), users, history
  validate/           whole-configuration checks with stable codes (docs/validation.md)
  render/             State → the nftables ruleset text, with each line's object
  nft/                runs nft: check, apply (keeping dynamic set entries and counters), list, the boot file
  nflog/              reads logged packets from netlink log group 4171
  linkwatch/          netlink: interfaces that appear or go (the flowtable's devices)
  tablewatch/         netlink: a table deleted (shadow mode puts Housecarl's back; docs/shadow.md)
  importer/           nft -j list ruleset → State, with notes and raw rules (docs/import.md)
  hostif/             the machine's interfaces
  store/              SQLite (modernc.org/sqlite, no cgo): configuration, history, users, tokens, sessions, audit
  engine/             configuration changes: edit a copy, diff, validate, check and apply, save with history, undo
  server/             the HTTP API and the web UI, authentication, the local socket, probation, the log ring
  api/, client/       JSON types and the Go client shared by server and CLI
  auth/               passwords (argon2id), tokens, sessions, grants and roles
  certs/, jarl/       the web UI's certificate through Jarl (docs/certificates.md); jarl/ is the Jarl API client
  jarlsets/           sets from Jarl: reads the names, follows CNAMEs, keeps what was read, within (docs/jarl.md)
  config/             housecarld's YAML configuration
  version/            what a build reports
  web/static/         the web UI: plain ES modules, no build step, embedded with go:embed
deploy/               systemd units, sysusers.d and tmpfiles.d files, Debian package scripts
scripts/e2e.sh        end-to-end test with real traffic through a router container
scripts/fakejarl/     the bit of Jarl's API sets from Jarl read, for the e2e test
scripts/mkdeb.sh      the Debian packages (make deb)
scripts/logo.py       the web UI's logo from artwork/logo.svg
scripts/banner.py     the README's banner from the web UI's logo and wordmark
```

[design.md](design.md) describes how the pieces fit.

## Building and testing

```sh
make            # bin/housecarld, bin/hc, bash completion
make test       # go vet, gofmt check, go test ./...
make e2e        # scripts/e2e.sh: needs Docker
make deb        # dist/housecarl_<version>_<arch>.deb and housecarl-cli_...
```

**Never apply a test ruleset to your own machine's nftables.** Tests that
need the real `nft` run it in a container.

**Unit tests** need no nftables: the API tests (`internal/server`) use a
fake firewall. The render tests compare the ruleset
with what is expected, line by line, and the importer tests import the
maintainer's router ruleset (`internal/importer/testdata`) and check the
result.

**Real nftables**: with `HOUSECARL_NFT` set to a command that runs nft
somewhere harmless, the render, nft and importer tests also have nft
check (and apply) their rulesets. A `debian:trixie` image with nftables
1.1.3 (Debian 13's) is the reference:

```sh
make e2e         # makes the housecarl-nft image, if missing
export HOUSECARL_NFT="docker run --rm -i --cap-add NET_ADMIN housecarl-nft nft"
export HOUSECARL_NFT_IMAGE=housecarl-nft
make test
```

Newer nftables accepts things 1.1.3 doesn't (a meter with a timeout for
`ct count`, hour ranges past midnight): keep to what 1.1.3 takes.

**The e2e test** builds a small network in Docker from the same image:

* a router container (housecarld, the `hc` CLI) on three networks named
  `wan`, `enp2s0` and `wg0` inside it, IPv4 and IPv6;
* "the internet" on the WAN side, a LAN client, and a "jellyfin" host.

It imports a copy of the maintainer's ruleset (lab addresses) with
housecarld in shadow mode beside it (traffic unchanged, a would-be drop
logged and let through, the table back after the old ruleset is
reloaded), restarts it enforcing, takes over the old table, and checks
with real traffic: port forwards with
hairpin NAT and the IPv6 pinhole, masquerading, antispoofing, the drop
log, a lockout change that rolls back by itself, a restart, the boot
ruleset after a flush, and that other tables are left alone. It makes
the `housecarl-nft` image itself if it is missing.

`KEEP=1` leaves the lab running (the web UI on `10.88.10.1:8070`);
`DOCKER_HOST` points at another Docker daemon. Root isn't needed on the
host.

The systemd sandbox (capabilities, the boot unit) is only tested by
installing the package in a container that runs systemd
(`--privileged --cgroupns=host /sbin/init`).

## Certificates

As in Steward: `internal/certs` has unit tests with a fake Jarl and an
in-process DNS server; a real order is tested against Jarl's e2e
environment and Pebble (Steward's docs/development.md has the steps).

## The web UI

Plain JavaScript modules under `internal/web/static/`, built with the
same helpers as Jarl's and Steward's:

* All DOM goes through `h()` in `ui.js`, never `innerHTML`.
* The Content-Security-Policy allows only the app's own scripts and
  styles, so no inline styles. Use a class in `style.css`, or set
  `el.style.prop`.
* The files are embedded, so rebuild housecarld after changing them.

There are no JavaScript tests: bundle `assets/app.js` with esbuild to
catch syntax and import errors, and look at the pages in a browser
against the e2e lab.

## Conventions

* Every user-visible change goes into `CHANGELOG.md` under "Unreleased",
  with the docs updated in the same change.
* Errors say what is wrong and how to fix it, naming the object.
* New validation rules get a stable code, a test, and a row in
  [validation.md](validation.md).
* Configuration changes go through the engine, never straight to the
  store, so they are validated, checked by nftables, recorded and
  undoable.
* Schema changes append a migration to `internal/store/store.go`; never
  edit a shipped one.
* `gofmt` and `go vet` clean (`make test`).

## Releasing

As for Jarl and Steward:

1. Move "Unreleased" in `CHANGELOG.md` under `## [<version>] - <date>`.
   A release candidate may leave it there.
2. Put the version in `VERSION`, and update the README's status line.
3. Run `make test && make e2e`.
4. Commit as "Release X", tag it `git tag -a vX -m "Housecarl X"`, and push.

A fix to a release candidate may go out as a build of it rather than a
new candidate: `0.2.0-rc.3+git<count>-<commit>`, where `<commit>` is the
short ID of the last commit before "Release X" and `<count>` is `git
rev-list --count` of it. The count orders the builds (apt compares it as a
number; the ID alone would order them at random); each sorts after
`0.2.0-rc.3` and before `0.2.0-rc.4`. Debian writes it
`0.2.0~rc.3+git<count>~<commit>`.

The tag starts `.gitea/workflows/release.yml`, which checks the tag
against `VERSION` and the CHANGELOG, runs the tests, builds the tarballs,
the CLI for each OS and the `.deb`s, and publishes them
(`scripts/gitea-release.sh`). It needs the repository secret
`RELEASE_TOKEN` (`write:repository`, `write:package`).