jarl/docs/development.md

# Developing Jarl

## Layout

```
cmd/jarld/            daemon: serve, init, import, adopt, check, render, resync, backup, restore, user
cmd/jarl/             CLI
internal/
  model/              configuration objects (State: views, ACLs, keys, zones, records, options) and changesets
  dnsname/            DNS name handling: FQDNs, relative names, check-names rules, reverse zones
  rr/                 record data parsing/canonicalization (miekg/dns), TXT quoting, the record type catalog
  namedconf/          named.conf lexer, parser and printer; address match lists
  iprange/            address ranges (10.0.10.101-10.0.10.199): parsing and the exact prefix cover
  catalog/            every BIND 9.20 statement: grammar (from named.conf(5)) + ARM descriptions, embedded
  validate/           whole-configuration validation and security lints
  render/             State → named.conf, keys.conf and zone files, with a source map for error messages
  bind/               named-checkconf, named-checkzone and rndc wrappers
  engine/             changesets: expand requests into ops, replay, validate, render, check, deploy, reload, verify, revert;
                      import and adopt
  importer/           reads an existing named.conf tree and its zone files
  store/              SQLite (modernc.org/sqlite, no cgo): state, changesets, users, tokens, sessions, audit
  auth/               argon2id passwords, tokens, sessions, grants and roles
  server/             HTTP API, authentication, the web UI
  stats/              query statistics: statistics-channel poller, response-log tailer, collector, separate SQLite store
  web/static/         the web UI: plain ES modules, no build step, embedded with go:embed
  api/, client/       JSON types and the Go client shared by server and CLI
  acme/               ACME DNS-01 challenge records, shared by jarl acme and the httpreq endpoint
  backup/             database backups: write (VACUUM INTO, plus the DNSSEC key files) and check, list, prune by the time in the name
  nscheck/            DNS queries to other name servers: zone serials, the parent's delegation and DS records
  dnssec/             reads named's DNSSEC key and state files; DS records; the signing phases (docs/dnssec.md)
  porkbun/            Porkbun API client (name servers, glue, DS records) and an in-memory fake for tests
  blocking/           blocklist parser, response policy zone writers, and matching a name the way named does
                      (the downloads are in engine/blocklists.go)
  config/             jarld's YAML configuration
  certs/              certificates: Jarl as an ACME client (lego), renewal, files for named
  encdns/             encrypted DNS (DoT/DoH) setup, status and health checks
  mailcheck/          reverse DNS check of the addresses mail is sent from
  totp/               two-factor login codes and recovery codes
  version/            what a build reports (release tag, commit, commits since a tag)
  testutil/           test helpers, chiefly finding BIND's tools (JARL_BIND_ROOT)
deploy/               systemd units (jarld, the daily backup service and timer), sysusers.d and tmpfiles.d files
deploy/debian/        the Debian package's control template and postinst/prerm/postrm
contrib/acme.sh/      acme.sh DNS API hook
contrib/lego/         program for lego's exec DNS provider
scripts/e2e.sh        end-to-end test against a real named
scripts/mkdeb.sh      builds the Debian package (make deb)
scripts/logo.py       traces artwork/logo.png into the web UI's logo (logo.svg, logo.js)
scripts/banner.py     builds the README's banner from the web UI's logo and wordmark
artwork/              the logo's master image and the README's banner
```

The main flow is in `internal/engine/engine.go` (`applyOps`):
expand → replay → validate → render → check → install → reload → verify →
record. [concepts.md](concepts.md) describes it from the outside.

## Building and testing

```sh
make            # bin/jarld, bin/jarl (CGO_ENABLED=0, version from ./VERSION),
                # and bin/completion/{jarl,jarld} (bash completion)
make test       # go vet, gofmt check, go test ./...
make e2e        # scripts/e2e.sh
make deb        # dist/jarl_<version>_<arch>.deb (server) and jarl-cli_<version>_<arch>.deb (DEB_ARCH=arm64 etc. cross-builds)
```

### The Debian package

`make deb` runs `scripts/mkdeb.sh`: it builds the binaries for
`DEB_ARCH`, installs them with `make install-files` into a staging tree
with Debian's paths (`PREFIX=/usr`, the unit in `/usr/lib/systemd/system`),
adds `deploy/debian/{postinst,prerm,postrm}` and a `control` file filled in
from `deploy/debian/control.in`, and packs it with `dpkg-deb
--root-owner-group`, so no root is needed. The version comes from
`./VERSION` (a `-` becomes `~`, so `0.2.0-rc1` sorts before `0.2.0`;
override with `DEB_VERSION`), and the maintainer from your git name and
email (override with `DEB_MAINTAINER`). The package ships no files in
`/etc`: postinst writes `/etc/jarl/jarl.yaml` once, so upgrades never touch
it.

To try the package without installing it on your machine:

```sh
docker run --rm -it -v "$PWD/dist:/pkg:ro" debian:trixie \
  sh -c 'apt-get update && apt-get install -y /pkg/jarl_*_amd64.deb /pkg/jarl-cli_*_amd64.deb && bash'
```

### Shell completion

The completion scripts are generated by cobra (`jarl completion bash`).
`cmd/jarl/complete.go` adds value completion: it reads each command's usage
line (`add ZONE NAME TYPE DATA...`) and completes the placeholders it knows
(`ZONE`, `VIEW`, `ACL`, `USER`, TSIG `KEY`, `TYPE`, server and group
names), so a new command whose usage line uses these names gets completion
without extra code. Try it without a shell:
`jarl __complete record add ''`.

### Tests against real BIND

The render, engine and adopt tests run the real `named-checkconf`,
`named-checkzone`, `named`, `rndc` and `dig` when they can find them, and
skip otherwise. They are found through `JARL_BIND_ROOT` (an extracted
package tree) or on the system. To get them without root or AppArmor in the
way, extract the packages:

```sh
mkdir -p ~/bind-root/debs && cd ~/bind-root/debs
apt-get download bind9 bind9-utils bind9-libs bind9-dnsutils \
    liblmdb0 libmaxminddb0 liburcu8t64 libuv1t64 libjemalloc2 libjson-c5 libnghttp2-14
for f in *.deb; do dpkg -x "$f" ~/bind-root; done
cd -
JARL_BIND_ROOT=~/bind-root make test
JARL_BIND_ROOT=~/bind-root make e2e
```

The tests start named unprivileged on random high ports in temporary
directories. On Debian/Ubuntu, AppArmor confines the installed
`/usr/sbin/named` to BIND's own directories, so the tests that *start*
named (engine integration tests, `make e2e`) need the extracted copy. The
checker tests work with the installed `bind9-utils` too.

`scripts/e2e.sh` builds a legacy BIND setup, starts named, runs
`jarld init`, `import`, `adopt` and `serve`, then drives every CLI command
(records, batches, changesets, views, ACLs, TSIG, options, users, tokens,
ACME hooks, DNSSEC signing, backup and restore with keys, audit) and checks
the answers with `dig`. The Porkbun code is tested against
`porkbun.Fake`, an in-memory stand-in for its API; nothing calls the real
one. With Docker, certificates are tested end to end against
[Pebble](https://github.com/letsencrypt/pebble), Let's Encrypt's test CA
(`ghcr.io/letsencrypt/pebble`, host network, ports 14000 and 15000),
which checks the challenges against the test named: obtaining, HTTPS
from jarld, DNS over TLS from named, renewal. `PEBBLE=0` skips that; the
certs package's own tests use a fake DNS server and `porkbun.Fake`, and
never a real CA. `KEEP=1` leaves named
and jarld running afterwards. The output says where; set `JARL_CLI_CONFIG`
to that directory's `cli.yaml` to keep using the CLI against it.

### A test machine

`scripts/testenv.sh` turns a fresh Debian/Ubuntu VM or container into a
server to try Jarl on, the way a user would. Run it as root from the
repository:

```sh
sudo scripts/testenv.sh                         # zone example.com
sudo JARL_TEST_ZONE=lab.test scripts/testenv.sh # another zone
```

It installs BIND 9, make, git, curl and Go (from go.dev, at the version in
`go.mod`, unless a new enough one is installed). It then writes a typical
hand-made configuration for Jarl to import: an ACL, options with logging,
an `internal` view (this machine's networks and the private ranges, with
recursion) and an `external` view (everyone else, authoritative only).
The zone is in both views with different answers. There is also a reverse
zone for the machine's /24. The zone files in `/etc/bind/zones` use
comments, `$TTL`, `$ORIGIN`, `$GENERATE`, legacy `type master` and most
record types. The script checks the result with `named-checkconf -z`,
(re)starts named (in the background where there is no systemd), and asks
named a few questions with `dig`. At the end it prints the steps to build
and install the packages, then run `init`, `import` and `adopt`.

`JARL_TEST_VIEWS=no` leaves out the views, and `JARL_TEST_IP` sets the
machine's address (default: the address of the default route). The old
`/etc/bind` is saved in `/root/bind-before-testenv-<time>.tgz`. Once Jarl
has adopted the configuration, running the script again needs
`JARL_TEST_FORCE=1`. Debian 13 has BIND 9.20; Ubuntu has it from 25.04.

### The web UI

The UI is plain JavaScript modules under `internal/web/static/`, with no
framework and no build step. The logo's master is `artwork/logo.png` (blue
`#1d5287` and white on a transparent background). `scripts/logo.py` traces it
with potrace (needs `potrace` and ImageMagick) into `assets/logo.svg` (the
favicon) and `assets/logo.js` (the same paths); change the PNG and rerun the
script rather than editing those files. The header and
login page draw the logo inline from `logo.js` (`logo()` in `brand.js`), so
it appears with the page instead of popping in after a separate download.
Static files are served with `Cache-Control: no-cache` and a content ETag,
so a reload revalidates them with cheap 304 answers. The
"JARL" wordmark is drawn as SVG paths by `wordmark()` in `assets/brand.js`
(over a rule on the login page, the letters alone in the header), so the UI
needs no web font and the wordmark takes the theme's text and accent colors.
The README's banner, `artwork/banner.svg`, is the login page's logo and
wordmark, generated by `scripts/banner.py` from `logo.js` and `brand.js`
(`logo.py` runs it; run it yourself after editing the wordmark).
All DOM is built with the `h()` helper in
`ui.js`, which never uses `innerHTML`, so record data and comments can't
inject markup. The Content-Security-Policy only allows the app's own
scripts and styles. To work on it, run jarld and reload the browser: the
files are embedded at build time, so rebuild jarld after changes.

## Conventions

* **Every user-visible change goes into `CHANGELOG.md`** (under
  "Unreleased"), and the docs in `docs/` are updated in the same change.
* Errors that reach users say what is wrong and how to fix it, naming the
  object.
* New validation rules get a stable `code`, a test in
  `internal/validate/validate_test.go`, and a row in
  [validation.md](validation.md).
* New API operations go through the engine (`internal/engine/expand.go`,
  `ops.go`) so they get permissions, validation, history and revert for
  free, and are documented in [api.md](api.md).
* `gofmt` and `go vet` clean (`make vet`).

## Releasing

1. Move the "Unreleased" section of `CHANGELOG.md` under
   `## [<version>] - <date>`. (A release candidate may skip this; see
   below.)
2. Put the version in `VERSION`.
3. `make test && JARL_BIND_ROOT=... make e2e` (CI runs only `make test`;
   it has no BIND).
4. Commit, tag and push: `git tag v<version> && git push origin main v<version>`.

The tag starts `.gitea/workflows/release.yml`, which checks that the tag
matches `VERSION` and that `CHANGELOG.md` has the section, runs `make
test`, builds everything (`make dist`, `make cli-dist`, `make deb` for
amd64 and arm64) and runs `scripts/gitea-release.sh`. That creates the
Gitea release (the CHANGELOG section as notes; the tarballs, the CLI for
Linux, macOS and Windows, the `.deb`s and `SHA256SUMS` attached) and
uploads the `.deb`s to the owner's Debian registry, distribution `stable`,
component `main` ([install.md](install.md#from-the-package-registry)).

### Release candidates

A version with a hyphen (`0.2.0-rc.1`, tag `v0.2.0-rc.1`) is a
pre-release:

* **On Gitea** it is marked "Pre-Release". Gitea doesn't count it as the
  latest release: the repository page, `/releases/latest` and the API's
  latest release keep pointing at the last stable one.
* **The `.deb`s go to the `testing` distribution**, not `stable`. Their
  version is `0.2.0~rc.1`, which apt sorts after `0.1.0` but before
  `0.2.0`. So in `stable` an RC would reach every server on its next
  `apt upgrade`. Servers that follow only `stable` never see RCs. A test
  machine that also lists `testing` gets each RC, and then the final
  `0.2.0` from `stable`, which supersedes it.
* **Release notes**: its own `## [0.2.0-rc.1]` section if there is one;
  otherwise the section of the version it leads up to (`## [0.2.0]`), or
  else `## [Unreleased]`. So you can tag RCs while the changes are still
  under Unreleased, and move them under `## [0.2.0]` for the real release.
  A stable release still needs its own section.

`scripts/gitea-release.sh notes` prints the notes a release would get
(`VERSION=0.2.0-rc.1 scripts/gitea-release.sh notes`), and the workflow
runs it as a check before building.

### Which version a build reports

The footer of the web UI, `jarld version`, `jarl --version`, `jarl status`
and `jarl whoami` say what a binary was built from (the footer links to
it on Gitea). Only a release build reports a version number:

* **A release**: `make` run on the commit tagged `v<VERSION>` with no
  tracked file changed. The Makefile then sets `RELEASE_TAG`, and the
  footer reads "Powered by Jarl v0.2.0", linking to the release. The
  release workflow sets `RELEASE_TAG` itself.
* **Any other build** is a development build, named by its commit (which
  Go records from git) rather than `VERSION`: "Powered by Jarl
  1a2b3c4d5e6f", linking to the commit. With uncommitted changes it is
  marked `-modified`. The other places add where it stands relative to
  the last tag, from `git describe` (the Makefile's `DESCRIBE`):
  `dev 1a2b3c4d5e6f (3 commits after v0.2.0-rc.1)`, or
  `dev 1a2b3c4d5e6f-modified (v0.2.0-rc.1 with uncommitted changes)`.
* **No git** (a build from a source archive): `VERSION`, marked as such:
  `0.2.0 (source build)`.

`RELEASE_TAG=v0.2.0 make` forces the first case.

`make deb` versions the packages the same way. A release gets `VERSION`
with `~` for `-` (`0.2.0~rc.1`). A development build gets a version after
the last tag: 3 commits past `v0.2.0-rc.1` is `0.2.0~rc.1+git3.gca0f148`,
and uncommitted changes add the build time (`.m20261009153000`). apt sorts
these after that RC and before the next one or `0.2.0`. So installing a
newer build over an older one, or over the RC, is an upgrade: plain
`apt install ./dist/…` works, without `--reinstall`. `make deb` removes
older packages of the same architecture from `dist/` first.

Pushing the same tag again (`git tag -f`, `git push -f`) re-runs it, and
the script replaces the attachments and packages, so a failed release can
simply be retried.

Once, on Gitea:

* Actions on for the repository (Settings → Repository → Actions), and a
  runner with the `ubuntu-latest` label: act_runner's default, which runs
  jobs in `docker.gitea.com/runner-images:ubuntu-latest`. The job
  downloads Go itself (`actions/setup-go`, from `go.mod`), so the runner
  needs internet access, or a mirror of the actions and Go.
* A repository secret `RELEASE_TOKEN` (Settings → Actions → Secrets): an
  access token (User settings → Applications) with the `write:repository`
  and `write:package` scopes, from a user who can write to the repository.
  The job's automatic token can't be relied on to publish packages.

`scripts/gitea-release.sh` also runs by hand after the same builds, with
`GITEA_URL`, `GITEA_REPO` (`owner/name`) and `GITEA_TOKEN` set; it needs
`curl`, `jq` and `dpkg-deb`.