Build Housecarl 0.1.0: the nftables firewall manager
Zones, address sets (IPv4 and IPv6 mixed, dynamic sets with timeouts) and services, ordered rules in the early, input, forward and output chains, port forwards with hairpin NAT and IPv6 pinholes, source NAT, redirects, and switchable built-ins (loopback, conntrack, ICMPv6, antispoofing, MSS clamping, DHCPv6 client, flowtable). Every change is validated, checked with nft -c and applied in one transaction to Housecarl's own table, inet housecarl; changes made over the network roll back unless confirmed, and the confirmed ruleset is loaded at boot by housecarl-ruleset.service. An empty configuration applies nothing. Also: hit counters and the drop log (nflog), 'hc import' of an existing nftables ruleset and 'hc takeover', the hc CLI, the REST API and the web UI, users, scoped tokens (sets:write:NAME), the audit log, the web UI's certificate through Jarl, Debian packages, docs, unit tests with real nft 1.1.3, and an e2e test with real traffic (71 checks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
abrown38andClaude Opus 5.5 committed
1 parent
54beb0bba3
commit
34292eeebb
118 files changed
+22165
-114
No files matched your search
@@ -0,0 +1,137 @@
|
||||
# Developing Housecarl
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
cmd/housecarld/ the daemon: serve, init, user, check, backup, restore, config
|
||||
cmd/hc/ the CLI
|
||||
internal/
|
||||
model/ the configuration (State: settings, zones, sets, services, rules, NAT), users, history
|
||||
validate/ whole-configuration checks with stable codes (docs/validation.md)
|
||||
render/ State → the nftables ruleset text, with each line's object
|
||||
nft/ runs nft: check, apply (keeping dynamic set entries and counters), list, the boot file
|
||||
nflog/ reads logged packets from netlink log group 4171
|
||||
importer/ nft -j list ruleset → State, with notes and raw rules (docs/import.md)
|
||||
hostif/ the machine's interfaces
|
||||
store/ SQLite (modernc.org/sqlite, no cgo): configuration, history, users, tokens, sessions, audit
|
||||
engine/ configuration changes: edit a copy, diff, validate, check and apply, save with history, undo
|
||||
server/ the HTTP API and the web UI, authentication, the local socket, probation, the log ring
|
||||
api/, client/ JSON types and the Go client shared by server and CLI
|
||||
auth/ passwords (argon2id), tokens, sessions, grants and roles
|
||||
certs/, jarl/ the web UI's certificate through Jarl (docs/certificates.md)
|
||||
config/ housecarld's YAML configuration
|
||||
version/ what a build reports
|
||||
web/static/ the web UI: plain ES modules, no build step, embedded with go:embed
|
||||
deploy/ systemd units, sysusers.d and tmpfiles.d files, Debian package scripts
|
||||
scripts/e2e.sh end-to-end test with real traffic through a router container
|
||||
scripts/mkdeb.sh the Debian packages (make deb)
|
||||
scripts/logo.py the web UI's logo from artwork/logo.svg
|
||||
scripts/banner.py the README's banner from the web UI's logo and wordmark
|
||||
```
|
||||
|
||||
[design.md](design.md) describes how the pieces fit.
|
||||
|
||||
## Building and testing
|
||||
|
||||
```sh
|
||||
make # bin/housecarld, bin/hc, bash completion
|
||||
make test # go vet, gofmt check, go test ./...
|
||||
make e2e # scripts/e2e.sh: needs Docker
|
||||
make deb # dist/housecarl_<version>_<arch>.deb and housecarl-cli_...
|
||||
```
|
||||
|
||||
**Never apply a test ruleset to your own machine's nftables.** Tests that
|
||||
need the real `nft` run it in a container.
|
||||
|
||||
**Unit tests** need no nftables: the API tests (`internal/server`) use a
|
||||
fake firewall. The render tests compare the ruleset
|
||||
with what is expected, line by line, and the importer tests import the
|
||||
maintainer's router ruleset (`internal/importer/testdata`) and check the
|
||||
result.
|
||||
|
||||
**Real nftables**: with `HOUSECARL_NFT` set to a command that runs nft
|
||||
somewhere harmless, the render, nft and importer tests also have nft
|
||||
check (and apply) their rulesets. A `debian:trixie` image with nftables
|
||||
1.1.3 (Debian 13's) is the reference:
|
||||
|
||||
```sh
|
||||
make e2e # makes the housecarl-nft image, if missing
|
||||
export HOUSECARL_NFT="docker run --rm -i --cap-add NET_ADMIN housecarl-nft nft"
|
||||
export HOUSECARL_NFT_IMAGE=housecarl-nft
|
||||
make test
|
||||
```
|
||||
|
||||
Newer nftables accepts things 1.1.3 doesn't (a meter with a timeout for
|
||||
`ct count`, hour ranges past midnight): keep to what 1.1.3 takes.
|
||||
|
||||
**The e2e test** builds a small network in Docker from the same image:
|
||||
|
||||
* a router container (housecarld, the `hc` CLI) on three networks named
|
||||
`wan`, `enp2s0` and `wg0` inside it, IPv4 and IPv6;
|
||||
* "the internet" on the WAN side, a LAN client, and a "jellyfin" host.
|
||||
|
||||
It imports a copy of the maintainer's ruleset (lab addresses), takes
|
||||
over the old table, and checks with real traffic: port forwards with
|
||||
hairpin NAT and the IPv6 pinhole, masquerading, antispoofing, the drop
|
||||
log, a lockout change that rolls back by itself, a restart, the boot
|
||||
ruleset after a flush, and that other tables are left alone. It makes
|
||||
the `housecarl-nft` image itself if it is missing.
|
||||
|
||||
`KEEP=1` leaves the lab running (the web UI on `10.88.10.1:8070`);
|
||||
`DOCKER_HOST` points at another Docker daemon. Root isn't needed on the
|
||||
host.
|
||||
|
||||
The systemd sandbox (capabilities, the boot unit) is only tested by
|
||||
installing the package in a container that runs systemd
|
||||
(`--privileged --cgroupns=host /sbin/init`).
|
||||
|
||||
## Certificates
|
||||
|
||||
As in Steward: `internal/certs` has unit tests with a fake Jarl and an
|
||||
in-process DNS server; a real order is tested against Jarl's e2e
|
||||
environment and Pebble (Steward's docs/development.md has the steps).
|
||||
|
||||
## The web UI
|
||||
|
||||
Plain JavaScript modules under `internal/web/static/`, built with the
|
||||
same helpers as Jarl's and Steward's:
|
||||
|
||||
* All DOM goes through `h()` in `ui.js`, never `innerHTML`.
|
||||
* The Content-Security-Policy allows only the app's own scripts and
|
||||
styles, so no inline styles. Use a class in `style.css`, or set
|
||||
`el.style.prop`.
|
||||
* The files are embedded, so rebuild housecarld after changing them.
|
||||
|
||||
There are no JavaScript tests: bundle `assets/app.js` with esbuild to
|
||||
catch syntax and import errors, and look at the pages in a browser
|
||||
against the e2e lab.
|
||||
|
||||
## Conventions
|
||||
|
||||
* Every user-visible change goes into `CHANGELOG.md` under "Unreleased",
|
||||
with the docs updated in the same change.
|
||||
* Errors say what is wrong and how to fix it, naming the object.
|
||||
* New validation rules get a stable code, a test, and a row in
|
||||
[validation.md](validation.md).
|
||||
* Configuration changes go through the engine, never straight to the
|
||||
store, so they are validated, checked by nftables, recorded and
|
||||
undoable.
|
||||
* Schema changes append a migration to `internal/store/store.go`; never
|
||||
edit a shipped one.
|
||||
* `gofmt` and `go vet` clean (`make test`).
|
||||
|
||||
## Releasing
|
||||
|
||||
As for Jarl and Steward:
|
||||
|
||||
1. Move "Unreleased" in `CHANGELOG.md` under `## [<version>] - <date>`.
|
||||
A release candidate may leave it there.
|
||||
2. Put the version in `VERSION`, and update the README's status line.
|
||||
3. Run `make test && make e2e`.
|
||||
4. Commit as "Release X", tag it `git tag -a vX -m "Housecarl X"`, and push.
|
||||
|
||||
The tag starts `.gitea/workflows/release.yml`, which checks the tag
|
||||
against `VERSION` and the CHANGELOG, runs the tests, builds the tarballs,
|
||||
the CLI for each OS and the `.deb`s, and publishes them
|
||||
(`scripts/gitea-release.sh`). It needs the repository secret
|
||||
`RELEASE_TOKEN` (`write:repository`, `write:package`).
|
||||
Reference in new issue
Block a user