jarl/docs/operations.md

Running Jarl

Day to day

Task	How
Is everything healthy?	jarl status, or the web UI dashboard: named reachable, adoption, drift, open changesets, configuration problems, the newest backup, tokens about to expire
Are the secondaries up to date?	jarl zone serials ZONE, or the zone's Name servers tab (below)
Back up / restore	Daily by jarl-backup.timer; by hand sudo -u jarl jarld backup; restore with jarld restore (below)
What changed and who did it?	jarl history, jarl changeset show ID (with diff), jarl audit
Undo a change	jarl changeset revert ID, review it, then jarl changeset apply NEW_ID (or History → Revert in the UI)
Check the configuration	jarl check: full validation plus named-checkconf -z
Logs	journalctl -u jarl (jarld, including msg=audit lines), journalctl -u named
Restart / reload TLS	systemctl restart jarl / systemctl reload jarl
Work on the server itself	sudo jarl ...: root uses jarld's local socket and needs no token (cli.md)
Renew or replace a token	jarl token rotate NAME --grace 1h (same grants, new secret; permissions.md). jarl token list --all --expiring shows those that expire within 14 days
Upgrade Jarl	Debian package: build the new one and sudo apt install ./dist/jarl_*.deb; otherwise make && sudo make install && sudo systemctl restart jarl. jarld upgrades its database when it starts

Restarting jarld doesn't affect DNS: named keeps serving the files it loaded. jarld is only needed to make changes.

Backups

Everything Jarl knows is in its database (/var/lib/jarl/jarl.db): zones, records, views, ACLs, TSIG keys, options, users, tokens, history and the audit log. named's files are generated from it, so the database is the one thing to back up.

The daily backup

The jarl-backup.timer systemd timer runs jarld backup once a day. The Debian package turns it on; after make install, turn it on yourself:

systemctl enable --now jarl-backup.timer
systemctl list-timers jarl-backup.timer   # when it runs next

Each run:

1. writes a consistent copy of the database with SQLite's VACUUM INTO (safe while jarld runs and changes things) to a temporary file in the backup directory;
2. checks the copy with SQLite's integrity_check, and only then gives it its name, jarl-<UTC time>.db, e.g. jarl-20261008T031512Z.db. A file with that name is always a complete, intact backup;
3. deletes backups older than retention, but always keeps the newest keep_at_least, however old they are. If backups stop for a while (the machine was off, the disk was full), the old ones aren't all deleted by the next run. Ages come from the time in the name, not the file's modification time, which copying the directory would change. Only files named jarl-<time>.db (and stats-<time>.db) are ever deleted, so anything else you put in the directory is safe;
4. records the result in the database, so the dashboard and jarl status show when the newest backup was made, and whether the last attempt failed.

A failed run exits non-zero, so it shows in systemctl --failed and journalctl -u jarl-backup.

What to keep and where is in jarl.yaml (configuration.md); when it runs is the timer's:

backup:
  dir: /var/backups/jarl
  retention: 30d       # delete backups older than this (0 keeps them all)...
  keep_at_least: 7     # ...but always keep the newest 7
  stats: false         # also back up the statistics database
  warn_after: 2d       # dashboard warning when the newest backup is older

The settings are read by jarld backup itself each time it runs, from the same file jarld uses, and checked the same way: a mistake there stops jarld from starting rather than quietly breaking the backups.

To run at a different time, override the timer, and keep warn_after above the new interval:

systemctl edit jarl-backup.timer
#   [Timer]
#   OnCalendar=
#   OnCalendar=*-*-* 02:30

The backup service runs as the jarl user in a sandbox that may only write to /var/lib/jarl and /var/backups/jarl (created by tmpfiles.d/jarl.conf, mode 0700). To keep backups elsewhere, set backup.dir, create the directory (owned by jarl, mode 0700) and allow it:

systemctl edit jarl-backup.service
#   [Service]
#   ReadWritePaths=/srv/backups/jarl

Backups on the same disk don't survive the disk. Copy /var/backups/jarl elsewhere (rsync, restic, borg, your usual backup) as well.

By hand

sudo -u jarl jarld backup                       # into backup.dir, as the timer does
sudo -u jarl jarld backup /root/before-upgrade.db  # one copy, never deleted automatically
sudo jarld restore                              # list the backups in backup.dir

Query statistics live in a separate database (/var/lib/jarl/stats.db). DNS doesn't need it, so it is backed up only with backup.stats: true, as stats-<UTC time>.db.

The database holds password and token hashes, and TSIG secrets in the clear. Backups are written with mode 0600 into a 0700 directory; keep any copies just as private.

Restoring

jarld restore replaces the database with a backup. jarld must be stopped: the restore refuses to run while jarld holds the database (and jarld won't start while a restore runs).

sudo jarld restore                  # list the backups, newest first
sudo systemctl stop jarl
sudo jarld restore latest           # or: sudo jarld restore /var/backups/jarl/jarl-20261008T031512Z.db

It checks the backup's integrity and that this Jarl can read it (a backup from a newer Jarl is refused), shows what the backup and the current database hold (zones, records, users, last change), and asks before changing anything (--yes skips the question). Then it:

• moves the current database aside as jarl.db.before-restore-<UTC time> (with its -wal and -shm files), so a restore can be undone the same way;
• copies the backup in place, upgrades it to the current schema if it is older, and records the restore in its audit log.

Then start jarld and bring named in line with the restored database:

sudo systemctl start jarl
jarl status                 # drift lists the files that differ
jarl resync --dry-run       # what would change
jarl resync --force         # write everything and reload named

Things to know:

• Zone serials. An older backup can carry lower zone serials than named last served. Your own named doesn't mind, but secondaries and caches ignore a serial that went backwards, so they keep the newer data. If you have secondaries, raise the serials after a restore: jarl zone update ZONE --serial <higher than before>, then check with jarl zone serials ZONE.
• Tokens and passwords go back to what they were when the backup was made: tokens created since are gone, and revoked ones work again. Rotate or revoke as needed.
• Statistics: to restore stats-<time>.db, stop jarld and copy it over /var/lib/jarl/stats.db (owner jarl, mode 0640).
• On a new machine: install Jarl, restore the backup, start jarld and run jarl resync --force; then adopt named as in install.md if this named doesn't include Jarl's configuration yet.

Drift: files changed outside Jarl

Jarl keeps a checksum of every file it writes. If someone edits /etc/bind/jarl/named.conf or a zone file by hand (or a restore puts old files back), jarl status and the dashboard show drift, and changes that would overwrite those files are refused with 409 drift:

error: files managed by Jarl were changed outside Jarl: /var/lib/bind/jarl/internal/adbraun.lol.zone; review them and apply with force to overwrite

Decide which version is right:

• Jarl's (the usual case): jarl resync --force rewrites every file from the database and reloads named. Any single change with --force also overwrites the drifted files.
• The hand edit: put the same change into Jarl (for example jarl zone import ZONE /var/lib/bind/jarl/internal/adbraun.lol.zone for records), then jarl resync --force.

When an apply fails

Jarl checks everything it can before it touches named, so failures are rare. When one happens, nothing is left half-done:

Stage	What you see	State afterwards
validation	422 with problems	Nothing written
named-checkconf / named-checkzone	422 validation_failed with the checker's messages in check, mapped to the zone/option that caused them	Nothing written
rndc reconfig / reload fails, or a zone doesn't load within reload_timeout	502 bind_failed	Previous files restored and named reloaded

A failed change is never recorded as applied. A staged changeset stays open, so you can fix it and apply again, and the failure is in the audit log (jarl audit --outcome failure). named's own log usually explains a load failure: journalctl -u named.

If named isn't running, changes fail at the reload step and are rolled back. Start named, then apply again.

Recovering access

Revoking a token never locks you out: web logins don't depend on tokens, and root on the server can always use sudo jarl through the local socket, without a token.

Problem	Fix (on the server)
Revoked or lost the CLI's token	sudo jarl token create cli --user admin, or Account → API tokens in the web UI
Forgot the admin password	sudo jarl user update admin --password, or (also with jarld stopped) sudo -u jarl jarld user passwd admin
Locked out by the login rate limit	wait 15 minutes, or systemctl restart jarl (the limiter is in memory)
Lost every admin account	sudo -u jarl jarld user add --admin rescue
Need a token without the web UI	sudo jarl token create cli --user admin, or sudo -u jarl jarld user token admin cli

A secondary server

A second name server runs its own named and jarld; its zones are of type secondary and copy the primary's records by zone transfer, so you only ever edit records on the primary. With both named in the CLI configuration (cli.md):

# 1. One TSIG key with the same secret on both servers.
jarl -s ns1 tsig create xfer                     # prints the secret
jarl -s ns2 tsig create xfer --secret 'THE-SECRET'

# 2. On the primary: let the key transfer the zone, and list ns2 as a
#    name server (named sends NOTIFY to the zone's NS records).
jarl -s ns1 option set allow-transfer '{ key xfer; }' --zone adbraun.lol
jarl -s ns1 record add adbraun.lol @ NS ns2.adbraun.lol.

# 3. On the secondary: the zone, copied from the primary with the key.
jarl -s ns2 zone create adbraun.lol --type secondary --primary "192.0.2.53 key xfer"

If ns2 isn't one of the zone's NS records, add also-notify { ADDRESS; }; on the primary so it learns about changes quickly. With views, each view's zones are transferred separately; match the secondary's requests to the right view, for example with match-clients { key xfer; ... };.

Things that both servers need, such as ACLs, can be changed on both at once with a group (jarl -s dns acl update ...): the change is checked on both first and applied only if both accept it.

Checking name servers and the delegation

Two checks ask other name servers about a zone over DNS. They only run when you ask: on the zone's Name servers tab in the web UI, or with the CLI. Nothing checks in the background.

Are the other servers serving the current serial?

$ jarl zone serials adbraun.lol
adbraun.lol, primary, serial 2026100814 (served by named here)

SERVER           ADDRESS       FROM         STATUS       SERIAL      TIME
ns1.adbraun.lol  192.168.1.53  NS record    this server  2026100814
ns2.adbraun.lol  192.168.1.54  NS record    BEHIND by 2  2026100812  1 ms

It works for both sides of a zone transfer:

• Primary zones: asks the servers in the zone's NS records and its also-notify targets (the secondaries) which serial they serve, and compares it with what named on this server serves. A secondary that is behind hasn't picked up the latest change: check its logs, and that allow-transfer on this server lets it transfer the zone.
• Secondary and mirror zones: asks the zone's primaries (following named remote-servers lists, with their ports and TSIG keys), to see whether this server's copy is up to date.

Name server addresses come from Jarl's own A/AAAA records in the zone's view where it has them, otherwise from a DNS lookup on the server. This server itself is recognised by its addresses and isn't asked. Statuses: in sync, behind/ahead by N (serials compare as in RFC 1982, so a wrapped serial counts as ahead), doesn't serve the zone (the server answers but isn't authoritative for it), refused, no answer, and no route from here (typically an IPv6 address on a server without IPv6, which isn't counted as a problem).

With views, other servers answer from the view that matches this server's address (or TSIG key), exactly as they do for its zone transfers and NOTIFYs; queries to primaries and also-notify targets that name a key are signed with it.

jarl zone serials exits with 2 when a server isn't serving the current serial, so it can run from cron or a monitoring system.

Does the internet see the zone the way Jarl does?

For a zone published on the internet, the parent zone (lol. for adbraun.lol, set through your registrar) must delegate it to the right name servers, with glue addresses for name servers inside the zone.

$ jarl zone delegation adbraun.lol --view external
Zone:                              adbraun.lol
Parent zone:                       lol. (found through 127.0.0.53:53)
  a.nic.lol and 3 more:            delegate to ns1.adbraun.lol, ns2.adbraun.lol
NS at the parent:                  ns1.adbraun.lol, ns2.adbraun.lol
NS in the zone:                    ns1.adbraun.lol, ns2.adbraun.lol
Glue ns1.adbraun.lol:              203.0.113.53 (matches)
...
ok:      The parent delegates adbraun.lol to the same name servers as its NS records: ...
ok:      Every delegated name server answers for the zone with the current serial.

It finds the parent zone through the server's resolver (or --resolver 9.9.9.9), asks each of the parent's servers directly, and reports:

• name servers the parent delegates to that the zone's NS records don't list, and the other way round;
• glue that is missing, or differs from the zone's A/AAAA records;
• delegated servers that don't serve the zone (a lame delegation), don't answer, or serve another serial.

A zone that is only used on your own network isn't delegated at all, and the check says so in a single note, not as a problem; it also exits with 0. It exits with 2 only for errors (a lame delegation, missing glue). With split DNS, check the zone in the view the internet sees (usually external).

Taking Jarl out of the picture

The generated files are plain BIND configuration. To stop using Jarl, disable the service. named keeps working with /etc/bind/jarl/ exactly as it is. To go back to the configuration you had before adopting:

cp /etc/bind/named.conf.pre-jarl-<date> /etc/bind/named.conf
rndc reconfig

Monitoring

• GET /healthz (no authentication) reports that jarld is up.
• On the server, a check can use the local socket and needs no token: sudo jarl status -o json.
• GET /api/v1/status with a read-only token reports named reachability, drift and configuration problems. It is suitable for a monitoring check:

  jarl --token @/etc/jarl/monitor.token status -o json | jq -e '.engine.drift == [] and .engine.adopted'
  

• The audit log records failed logins as auth.login_failed / auth.login_blocked, and rejected tokens as auth.token_failed.
• Backups: .backup.stale and .backup.last_error in jarl status -o json (administrators), and systemctl --failed for the timer's runs:

  sudo jarl status -o json | jq -e '.backup.stale == false and (.backup.last_error // "") == ""'
  

• Secondaries: jarl zone serials ZONE exits with 2 when one isn't serving the current serial.
• Tokens: jarld logs a warning once a day for every token that expires within 14 days (journalctl -u jarl | grep 'token expires'), the dashboard lists them, and jarl prints a warning on stderr whenever it uses one, so a certbot hook's log shows it well before renewals fail.