abrown38andClaude Opus 5.5
a3a1da54df
On a fresh Debian/Ubuntu machine it installs BIND and the build tools (Go from go.dev), writes a hand-made style configuration with internal and external views, a split-horizon zone (example.com or JARL_TEST_ZONE) and a reverse zone, starts named and prints the steps to install Jarl and adopt the configuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
224 lines
12 KiB
Markdown
224 lines
12 KiB
Markdown
# Developing Jarl
|
|
|
|
## Layout
|
|
|
|
```
|
|
cmd/jarld/ daemon: serve, init, import, adopt, check, render, resync, backup, restore, user
|
|
cmd/jarl/ CLI
|
|
internal/
|
|
model/ configuration objects (State: views, ACLs, keys, zones, records, options) and changesets
|
|
dnsname/ DNS name handling: FQDNs, relative names, check-names rules, reverse zones
|
|
rr/ record data parsing/canonicalization (miekg/dns), TXT quoting, the record type catalog
|
|
namedconf/ named.conf lexer, parser and printer; address match lists
|
|
catalog/ every BIND 9.20 statement: grammar (from named.conf(5)) + ARM descriptions, embedded
|
|
validate/ whole-configuration validation and security lints
|
|
render/ State → named.conf, keys.conf and zone files, with a source map for error messages
|
|
bind/ named-checkconf, named-checkzone and rndc wrappers
|
|
engine/ changesets: expand requests into ops, replay, validate, render, check, deploy, reload, verify, revert;
|
|
import and adopt
|
|
importer/ reads an existing named.conf tree and its zone files
|
|
store/ SQLite (modernc.org/sqlite, no cgo): state, changesets, users, tokens, sessions, audit
|
|
auth/ argon2id passwords, tokens, sessions, grants and roles
|
|
server/ HTTP API, authentication, the web UI
|
|
stats/ query statistics: statistics-channel poller, response-log tailer, collector, separate SQLite store
|
|
web/static/ the web UI: plain ES modules, no build step, embedded with go:embed
|
|
api/, client/ JSON types and the Go client shared by server and CLI
|
|
acme/ ACME DNS-01 challenge records, shared by jarl acme and the httpreq endpoint
|
|
backup/ database backups: write (VACUUM INTO) and check, list, prune by the time in the name
|
|
nscheck/ DNS queries to other name servers: zone serials and the parent's delegation
|
|
config/ jarld's YAML configuration
|
|
deploy/ systemd units (jarld, the daily backup service and timer), sysusers.d and tmpfiles.d files
|
|
deploy/debian/ the Debian package's control template and postinst/prerm/postrm
|
|
contrib/acme.sh/ acme.sh DNS API hook
|
|
contrib/lego/ program for lego's exec DNS provider
|
|
scripts/e2e.sh end-to-end test against a real named
|
|
scripts/mkdeb.sh builds the Debian package (make deb)
|
|
scripts/logo.py traces artwork/logo.png into the web UI's logo (logo.svg, logo.js)
|
|
scripts/banner.py builds the README's banner from the web UI's logo and wordmark
|
|
artwork/ the logo's master image and the README's banner
|
|
```
|
|
|
|
The main flow is in `internal/engine/engine.go` (`applyOps`):
|
|
expand → replay → validate → render → check → install → reload → verify →
|
|
record. [concepts.md](concepts.md) describes it from the outside.
|
|
|
|
## Building and testing
|
|
|
|
```sh
|
|
make # bin/jarld, bin/jarl (CGO_ENABLED=0, version from ./VERSION),
|
|
# and bin/completion/{jarl,jarld} (bash completion)
|
|
make test # go vet, gofmt check, go test ./...
|
|
make e2e # scripts/e2e.sh
|
|
make deb # dist/jarl_<version>_<arch>.deb (server) and jarl-cli_<version>_<arch>.deb (DEB_ARCH=arm64 etc. cross-builds)
|
|
```
|
|
|
|
### The Debian package
|
|
|
|
`make deb` runs `scripts/mkdeb.sh`: it builds the binaries for
|
|
`DEB_ARCH`, installs them with `make install-files` into a staging tree
|
|
with Debian's paths (`PREFIX=/usr`, the unit in `/usr/lib/systemd/system`),
|
|
adds `deploy/debian/{postinst,prerm,postrm}` and a `control` file filled in
|
|
from `deploy/debian/control.in`, and packs it with `dpkg-deb
|
|
--root-owner-group`, so no root is needed. The version comes from
|
|
`./VERSION` (a `-` becomes `~`, so `0.2.0-rc1` sorts before `0.2.0`;
|
|
override with `DEB_VERSION`), and the maintainer from your git name and
|
|
email (override with `DEB_MAINTAINER`). The package ships no files in
|
|
`/etc`: postinst writes `/etc/jarl/jarl.yaml` once, so upgrades never touch
|
|
it.
|
|
|
|
To try the package without installing it on your machine:
|
|
|
|
```sh
|
|
docker run --rm -it -v "$PWD/dist:/pkg:ro" debian:trixie \
|
|
sh -c 'apt-get update && apt-get install -y /pkg/jarl_*_amd64.deb /pkg/jarl-cli_*_amd64.deb && bash'
|
|
```
|
|
|
|
### Shell completion
|
|
|
|
The completion scripts are generated by cobra (`jarl completion bash`).
|
|
`cmd/jarl/complete.go` adds value completion: it reads each command's usage
|
|
line (`add ZONE NAME TYPE DATA...`) and completes the placeholders it knows
|
|
(`ZONE`, `VIEW`, `ACL`, `USER`, TSIG `KEY`, `TYPE`, server and group
|
|
names), so a new command whose usage line uses these names gets completion
|
|
without extra code. Try it without a shell:
|
|
`jarl __complete record add ''`.
|
|
|
|
### Tests against real BIND
|
|
|
|
The render, engine and adopt tests run the real `named-checkconf`,
|
|
`named-checkzone`, `named`, `rndc` and `dig` when they can find them, and
|
|
skip otherwise. They are found through `JARL_BIND_ROOT` (an extracted
|
|
package tree) or on the system. To get them without root or AppArmor in the
|
|
way, extract the packages:
|
|
|
|
```sh
|
|
mkdir -p ~/bind-root/debs && cd ~/bind-root/debs
|
|
apt-get download bind9 bind9-utils bind9-libs bind9-dnsutils \
|
|
liblmdb0 libmaxminddb0 liburcu8t64 libuv1t64 libjemalloc2 libjson-c5 libnghttp2-14
|
|
for f in *.deb; do dpkg -x "$f" ~/bind-root; done
|
|
cd -
|
|
JARL_BIND_ROOT=~/bind-root make test
|
|
JARL_BIND_ROOT=~/bind-root make e2e
|
|
```
|
|
|
|
The tests start named unprivileged on random high ports in temporary
|
|
directories. On Debian/Ubuntu, AppArmor confines the installed
|
|
`/usr/sbin/named` to BIND's own directories, so the tests that *start*
|
|
named (engine integration tests, `make e2e`) need the extracted copy. The
|
|
checker tests work with the installed `bind9-utils` too.
|
|
|
|
`scripts/e2e.sh` builds a legacy BIND setup, starts named, runs
|
|
`jarld init`, `import`, `adopt` and `serve`, then drives every CLI command
|
|
(records, batches, changesets, views, ACLs, TSIG, options, users, tokens,
|
|
ACME hooks, audit) and checks the answers with `dig`. `KEEP=1` leaves named
|
|
and jarld running afterwards. The output says where; set `JARL_CLI_CONFIG`
|
|
to that directory's `cli.yaml` to keep using the CLI against it.
|
|
|
|
### A test machine
|
|
|
|
`scripts/testenv.sh` turns a fresh Debian/Ubuntu VM or container into a
|
|
server to try Jarl on, the way a user would. Run it as root from the
|
|
repository:
|
|
|
|
```sh
|
|
sudo scripts/testenv.sh # zone example.com
|
|
sudo JARL_TEST_ZONE=lab.test scripts/testenv.sh # another zone
|
|
```
|
|
|
|
It installs BIND 9, make, git, curl and Go (from go.dev, at the version in
|
|
`go.mod`, unless a new enough one is installed). It then writes a typical
|
|
hand-made configuration for Jarl to import: an ACL, options with logging,
|
|
an `internal` view (this machine's networks and the private ranges, with
|
|
recursion) and an `external` view (everyone else, authoritative only).
|
|
The zone is in both views with different answers. There is also a reverse
|
|
zone for the machine's /24. The zone files in `/etc/bind/zones` use
|
|
comments, `$TTL`, `$ORIGIN`, `$GENERATE`, legacy `type master` and most
|
|
record types. The script checks the result with `named-checkconf -z`,
|
|
(re)starts named (in the background where there is no systemd), and asks
|
|
named a few questions with `dig`. At the end it prints the steps to build
|
|
and install the packages, then run `init`, `import` and `adopt`.
|
|
|
|
`JARL_TEST_VIEWS=no` leaves out the views, and `JARL_TEST_IP` sets the
|
|
machine's address (default: the address of the default route). The old
|
|
`/etc/bind` is saved in `/root/bind-before-testenv-<time>.tgz`. Once Jarl
|
|
has adopted the configuration, running the script again needs
|
|
`JARL_TEST_FORCE=1`. Debian 13 has BIND 9.20; Ubuntu has it from 25.04.
|
|
|
|
### The web UI
|
|
|
|
The UI is plain JavaScript modules under `internal/web/static/`, with no
|
|
framework and no build step. The logo's master is `artwork/logo.png` (blue
|
|
`#1d5287` and white on a transparent background). `scripts/logo.py` traces it
|
|
with potrace (needs `potrace` and ImageMagick) into `assets/logo.svg` (the
|
|
favicon) and `assets/logo.js` (the same paths); change the PNG and rerun the
|
|
script rather than editing those files. The header and
|
|
login page draw the logo inline from `logo.js` (`logo()` in `brand.js`), so
|
|
it appears with the page instead of popping in after a separate download.
|
|
Static files are served with `Cache-Control: no-cache` and a content ETag,
|
|
so a reload revalidates them with cheap 304 answers. The
|
|
"JARL" wordmark is drawn as SVG paths by `wordmark()` in `assets/brand.js`
|
|
(over a rule on the login page, the letters alone in the header), so the UI
|
|
needs no web font and the wordmark takes the theme's text and accent colors.
|
|
The README's banner, `artwork/banner.svg`, is the login page's logo and
|
|
wordmark, generated by `scripts/banner.py` from `logo.js` and `brand.js`
|
|
(`logo.py` runs it; run it yourself after editing the wordmark).
|
|
All DOM is built with the `h()` helper in
|
|
`ui.js`, which never uses `innerHTML`, so record data and comments can't
|
|
inject markup. The Content-Security-Policy only allows the app's own
|
|
scripts and styles. To work on it, run jarld and reload the browser: the
|
|
files are embedded at build time, so rebuild jarld after changes.
|
|
|
|
## Conventions
|
|
|
|
* **Every user-visible change goes into `CHANGELOG.md`** (under
|
|
"Unreleased"), and the docs in `docs/` are updated in the same change.
|
|
* Errors that reach users say what is wrong and how to fix it, naming the
|
|
object.
|
|
* New validation rules get a stable `code`, a test in
|
|
`internal/validate/validate_test.go`, and a row in
|
|
[validation.md](validation.md).
|
|
* New API operations go through the engine (`internal/engine/expand.go`,
|
|
`ops.go`) so they get permissions, validation, history and revert for
|
|
free, and are documented in [api.md](api.md).
|
|
* `gofmt` and `go vet` clean (`make vet`).
|
|
|
|
## Releasing
|
|
|
|
1. Move the "Unreleased" section of `CHANGELOG.md` under
|
|
`## [<version>] - <date>`.
|
|
2. Put the version in `VERSION`.
|
|
3. `make test && JARL_BIND_ROOT=... make e2e` (CI runs only `make test`;
|
|
it has no BIND).
|
|
4. Commit, tag and push: `git tag v<version> && git push origin main v<version>`.
|
|
|
|
The tag starts `.gitea/workflows/release.yml`, which checks that the tag
|
|
matches `VERSION` and that `CHANGELOG.md` has the section, runs `make
|
|
test`, builds everything (`make dist`, `make cli-dist`, `make deb` for
|
|
amd64 and arm64) and runs `scripts/gitea-release.sh`. That creates the
|
|
Gitea release (the CHANGELOG section as notes; the tarballs, the CLI for
|
|
Linux, macOS and Windows, the `.deb`s and `SHA256SUMS` attached) and
|
|
uploads the `.deb`s to the owner's Debian registry, distribution `stable`,
|
|
component `main` ([install.md](install.md#from-the-package-registry)). A
|
|
version with a hyphen (`1.0.0-rc.1`) becomes a pre-release; its packages
|
|
get version `1.0.0~rc.1`, which apt sorts before `1.0.0`.
|
|
|
|
Pushing the same tag again (`git tag -f`, `git push -f`) re-runs it, and
|
|
the script replaces the attachments and packages, so a failed release can
|
|
simply be retried.
|
|
|
|
Once, on Gitea:
|
|
|
|
* Actions on for the repository (Settings → Repository → Actions), and a
|
|
runner with the `ubuntu-latest` label: act_runner's default, which runs
|
|
jobs in `docker.gitea.com/runner-images:ubuntu-latest`. The job
|
|
downloads Go itself (`actions/setup-go`, from `go.mod`), so the runner
|
|
needs internet access, or a mirror of the actions and Go.
|
|
* A repository secret `RELEASE_TOKEN` (Settings → Actions → Secrets): an
|
|
access token (User settings → Applications) with the `write:repository`
|
|
and `write:package` scopes, from a user who can write to the repository.
|
|
The job's automatic token can't be relied on to publish packages.
|
|
|
|
`scripts/gitea-release.sh` also runs by hand after the same builds, with
|
|
`GITEA_URL`, `GITEA_REPO` (`owner/name`) and `GITEA_TOKEN` set; it needs
|
|
`curl`, `jq` and `dpkg-deb`.
|