adam/jarl
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases 34 Wiki Activity

Add backups, token expiry warnings and name server checks

Browse Source 
- Daily database backups: jarl-backup.timer runs 'jarld backup', which
  writes jarl-<UTC time>.db into backup.dir, checks it before naming it,
  and prunes by the time in the name while always keeping the newest
  keep_at_least. New backup: section in jarl.yaml; the dashboard and
  'jarl status' show the newest backup and warn when it is stale or
  failed. The Debian package enables the timer.
- 'jarld restore [FILE | latest]': checks the backup, shows it next to
  the current database, keeps the current one aside and records the
  restore. 'jarld serve' holds a database lock, so a restore can't run
  under a live jarld and two jarlds can't share a database.
- Token expiry warnings from 14 days ahead: dashboard, token lists, a
  daily log line, and a CLI warning via the Jarl-Token-Expires header;
  'jarl token list --expiring'.
- 'jarl zone serials' and the zone page's Name servers tab: compare the
  serial on a primary zone's secondaries (NS records, also-notify) or a
  secondary zone's primaries with this server's.
- 'jarl zone delegation': for zones on the internet, check the parent's
  delegation and glue against the zone; an undelegated zone gets a note.
- Server: Jarl/<version> header; CLI exit code 2 for failed checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
abrown38andClaude Opus 5.5 committed 2026-10-08 19:02:34 +00:00
1 parent 1afed741bf
commit 293e6ddd33
47 files changed
+3717 -95
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/operations.md
+223 -26
View File
@@ -4,14 +4,16 @@
| Task | How |
|------|-----|
| Is everything healthy? | `jarl status`, or the web UI dashboard: named reachable, adoption, drift, open changesets, configuration problems |
| Is everything healthy? | `jarl status`, or the web UI dashboard: named reachable, adoption, drift, open changesets, configuration problems, the newest backup, tokens about to expire |
| Are the secondaries up to date? | `jarl zone serials ZONE`, or the zone's Name servers tab ([below](#checking-name-servers-and-the-delegation)) |
| Back up / restore | Daily by `jarl-backup.timer`; by hand `sudo -u jarl jarld backup`; restore with `jarld restore` ([below](#backups)) |
| What changed and who did it? | `jarl history`, `jarl changeset show ID` (with diff), `jarl audit` |
| Undo a change | `jarl changeset revert ID`, review it, then `jarl changeset apply NEW_ID` (or History → Revert in the UI) |
| Check the configuration | `jarl check`: full validation plus `named-checkconf -z` |
| Logs | `journalctl -u jarl` (jarld, including `msg=audit` lines), `journalctl -u named` |
| Restart / reload TLS | `systemctl restart jarl` / `systemctl reload jarl` |
| Work on the server itself | `sudo jarl ...`: root uses jarld's local socket and needs no token ([cli.md](cli.md#on-the-server-itself)) |
| Renew or replace a token | `jarl token rotate NAME --grace 1h` (same grants, new secret; [permissions.md](permissions.md#rotating-a-token)) |
| Renew or replace a token | `jarl token rotate NAME --grace 1h` (same grants, new secret; [permissions.md](permissions.md#rotating-a-token)). `jarl token list --all --expiring` shows those that expire within 14 days |
| Upgrade Jarl | Debian package: build the new one and `sudo apt install ./dist/jarl_*.deb`; otherwise `make && sudo make install && sudo systemctl restart jarl`. jarld upgrades its database when it starts |
Restarting jarld doesn't affect DNS: named keeps serving the files it
@@ -19,46 +21,151 @@ loaded. jarld is only needed to make changes.
## Backups
Everything Jarl knows is in its database. The BIND files can be generated
from it again at any time.
Everything Jarl knows is in its database (`/var/lib/jarl/jarl.db`): zones,
records, views, ACLs, TSIG keys, options, users, tokens, history and the
audit log. named's files are generated from it, so the database is the one
thing to back up.
### The daily backup
The `jarl-backup.timer` systemd timer runs `jarld backup` once a day. The
Debian package turns it on; after `make install`, turn it on yourself:
```sh
sudo -u jarl jarld backup /var/backups/jarl/jarl-$(date +%F).db # consistent copy, safe while jarld runs
systemctl enable --now jarl-backup.timer
systemctl list-timers jarl-backup.timer # when it runs next
```
For example, as a daily timer or cron job:
Each run:
1. writes a consistent copy of the database with SQLite's `VACUUM INTO`
(safe while jarld runs and changes things) to a temporary file in the
backup directory;
2. checks the copy with SQLite's `integrity_check`, and only then gives it
its name, `jarl-<UTC time>.db`, e.g. `jarl-20261008T031512Z.db`. A file
with that name is always a complete, intact backup;
3. deletes backups older than `retention`, but always keeps the newest
`keep_at_least`, however old they are. If backups stop for a while (the
machine was off, the disk was full), the old ones aren't all deleted by
the next run. Ages come from the time in the name, not the file's
modification time, which copying the directory would change. Only files
named `jarl-<time>.db` (and `stats-<time>.db`) are ever deleted, so
anything else you put in the directory is safe;
4. records the result in the database, so the dashboard and `jarl status`
show when the newest backup was made, and whether the last attempt
failed.
A failed run exits non-zero, so it shows in `systemctl --failed` and
`journalctl -u jarl-backup`.
What to keep and where is in `jarl.yaml`
([configuration.md](configuration.md#backups)); when it runs is the
timer's:
```yaml
backup:
dir: /var/backups/jarl
retention: 30d # delete backups older than this (0 keeps them all)...
keep_at_least: 7 # ...but always keep the newest 7
stats: false # also back up the statistics database
warn_after: 2d # dashboard warning when the newest backup is older
```
The settings are read by `jarld backup` itself each time it runs, from the
same file jarld uses, and checked the same way: a mistake there stops
`jarld` from starting rather than quietly breaking the backups.
To run at a different time, override the timer, and keep `warn_after`
above the new interval:
```sh
# /etc/cron.daily/jarl-backup
#!/bin/sh
set -e
install -d -o jarl -g jarl -m 0700 /var/backups/jarl
sudo -u jarl jarld backup /var/backups/jarl/jarl-$(date +%F).db
find /var/backups/jarl -name 'jarl-*.db' -mtime +30 -delete
systemctl edit jarl-backup.timer
# [Timer]
# OnCalendar=
# OnCalendar=*-*-* 02:30
```
The backup service runs as the `jarl` user in a sandbox that may only
write to `/var/lib/jarl` and `/var/backups/jarl` (created by
`tmpfiles.d/jarl.conf`, mode 0700). To keep backups elsewhere, set
`backup.dir`, create the directory (owned by `jarl`, mode 0700) and allow
it:
```sh
systemctl edit jarl-backup.service
# [Service]
# ReadWritePaths=/srv/backups/jarl
```
Backups on the same disk don't survive the disk. Copy
`/var/backups/jarl` elsewhere (rsync, restic, borg, your usual backup) as
well.
### By hand
```sh
sudo -u jarl jarld backup # into backup.dir, as the timer does
sudo -u jarl jarld backup /root/before-upgrade.db # one copy, never deleted automatically
sudo jarld restore # list the backups in backup.dir
```
Query statistics live in a separate database (`/var/lib/jarl/stats.db`).
It isn't needed to restore DNS, so back it up only if you want to keep the
history.
DNS doesn't need it, so it is backed up only with `backup.stats: true`, as
`stats-<UTC time>.db`.
The database contains password and token *hashes* and the TSIG secrets in
the clear, so keep backups private.
The database holds password and token *hashes*, and TSIG secrets in the
clear. Backups are written with mode 0600 into a 0700 directory; keep any
copies just as private.
### Restoring
`jarld restore` replaces the database with a backup. jarld must be
stopped: the restore refuses to run while jarld holds the database (and
jarld won't start while a restore runs).
```sh
systemctl stop jarl
install -o jarl -g jarl -m 0600 /var/backups/jarl/jarl-2026-10-01.db /var/lib/jarl/jarl.db
rm -f /var/lib/jarl/jarl.db-wal /var/lib/jarl/jarl.db-shm
systemctl start jarl
jarl resync --dry-run # what differs between the restored database and the files
jarl resync --force # write everything and reload named
sudo jarld restore # list the backups, newest first
sudo systemctl stop jarl
sudo jarld restore latest # or: sudo jarld restore /var/backups/jarl/jarl-20261008T031512Z.db
```
An older backup can carry lower zone serials than named last served. Your
own named doesn't mind, but secondaries and caches ignore a serial that went
backwards. If you have secondaries, raise the serials after a restore:
`jarl zone update ZONE --serial <higher>`.
It checks the backup's integrity and that this Jarl can read it (a backup
from a newer Jarl is refused), shows what the backup and the current
database hold (zones, records, users, last change), and asks before
changing anything (`--yes` skips the question). Then it:
* moves the current database aside as
`jarl.db.before-restore-<UTC time>` (with its `-wal` and `-shm` files),
so a restore can be undone the same way;
* copies the backup in place, upgrades it to the current schema if it is
older, and records the restore in its audit log.
Then start jarld and bring named in line with the restored database:
```sh
sudo systemctl start jarl
jarl status # drift lists the files that differ
jarl resync --dry-run # what would change
jarl resync --force # write everything and reload named
```
Things to know:
* **Zone serials.** An older backup can carry lower zone serials than named
last served. Your own named doesn't mind, but secondaries and caches
ignore a serial that went backwards, so they keep the newer data. If you
have secondaries, raise the serials after a restore:
`jarl zone update ZONE --serial <higher than before>`, then check with
`jarl zone serials ZONE`.
* **Tokens and passwords** go back to what they were when the backup was
made: tokens created since are gone, and revoked ones work again. Rotate
or revoke as needed.
* **Statistics**: to restore `stats-<time>.db`, stop jarld and copy it over
`/var/lib/jarl/stats.db` (owner `jarl`, mode 0640).
* **On a new machine**: install Jarl, restore the backup, start jarld and
run `jarl resync --force`; then adopt named as in
[install.md](install.md) if this named doesn't include Jarl's
configuration yet.
## Drift: files changed outside Jarl
@@ -143,6 +250,85 @@ Things that both servers need, such as ACLs, can be changed on both at once
with a group (`jarl -s dns acl update ...`): the change is checked on both
first and applied only if both accept it.
## Checking name servers and the delegation
Two checks ask other name servers about a zone over DNS. They only run when
you ask: on the zone's **Name servers** tab in the web UI, or with the CLI.
Nothing checks in the background.
### Are the other servers serving the current serial?
```
$ jarl zone serials adbraun.lol
adbraun.lol, primary, serial 2026100814 (served by named here)
SERVER ADDRESS FROM STATUS SERIAL TIME
ns1.adbraun.lol 192.168.1.53 NS record this server 2026100814
ns2.adbraun.lol 192.168.1.54 NS record BEHIND by 2 2026100812 1 ms
```
It works for both sides of a zone transfer:
* **Primary zones**: asks the servers in the zone's NS records and its
`also-notify` targets (the secondaries) which serial they serve, and
compares it with what named on this server serves. A secondary that is
behind hasn't picked up the latest change: check its logs, and that
`allow-transfer` on this server lets it transfer the zone.
* **Secondary and mirror zones**: asks the zone's `primaries` (following
named `remote-servers` lists, with their ports and TSIG keys), to see
whether this server's copy is up to date.
Name server addresses come from Jarl's own A/AAAA records in the zone's
view where it has them, otherwise from a DNS lookup on the server. This
server itself is recognised by its addresses and isn't asked. Statuses:
*in sync*, *behind/ahead by N* (serials compare as in RFC 1982, so a
wrapped serial counts as ahead), *doesn't serve the zone* (the server
answers but isn't authoritative for it), *refused*, *no answer*, and *no
route from here* (typically an IPv6 address on a server without IPv6,
which isn't counted as a problem).
With views, other servers answer from the view that matches this server's
address (or TSIG key), exactly as they do for its zone transfers and
NOTIFYs; queries to primaries and also-notify targets that name a key are
signed with it.
`jarl zone serials` exits with 2 when a server isn't serving the current
serial, so it can run from cron or a monitoring system.
### Does the internet see the zone the way Jarl does?
For a zone published on the internet, the parent zone (`lol.` for
adbraun.lol, set through your registrar) must delegate it to the right name
servers, with glue addresses for name servers inside the zone.
```
$ jarl zone delegation adbraun.lol --view external
Zone: adbraun.lol
Parent zone: lol. (found through 127.0.0.53:53)
a.nic.lol and 3 more: delegate to ns1.adbraun.lol, ns2.adbraun.lol
NS at the parent: ns1.adbraun.lol, ns2.adbraun.lol
NS in the zone: ns1.adbraun.lol, ns2.adbraun.lol
Glue ns1.adbraun.lol: 203.0.113.53 (matches)
...
ok: The parent delegates adbraun.lol to the same name servers as its NS records: ...
ok: Every delegated name server answers for the zone with the current serial.
```
It finds the parent zone through the server's resolver (or `--resolver
9.9.9.9`), asks each of the parent's servers directly, and reports:
* name servers the parent delegates to that the zone's NS records don't
list, and the other way round;
* glue that is missing, or differs from the zone's A/AAAA records;
* delegated servers that don't serve the zone (a *lame delegation*), don't
answer, or serve another serial.
A zone that is only used on your own network isn't delegated at all, and
the check says so in a single note, not as a problem; it also exits with 0.
It exits with 2 only for errors (a lame delegation, missing glue). With
split DNS, check the zone in the view the internet sees (usually
`external`).
## Taking Jarl out of the picture
The generated files are plain BIND configuration. To stop using Jarl,
@@ -166,3 +352,14 @@ rndc reconfig
```
* The audit log records failed logins as `auth.login_failed` /
`auth.login_blocked`, and rejected tokens as `auth.token_failed`.
* Backups: `.backup.stale` and `.backup.last_error` in `jarl status -o
json` (administrators), and `systemctl --failed` for the timer's runs:
```sh
sudo jarl status -o json | jq -e '.backup.stale == false and (.backup.last_error // "") == ""'
```
* Secondaries: `jarl zone serials ZONE` exits with 2 when one isn't serving
the current serial.
* Tokens: jarld logs a warning once a day for every token that expires
within 14 days (`journalctl -u jarl | grep 'token expires'`), the
dashboard lists them, and `jarl` prints a warning on stderr whenever it
uses one, so a certbot hook's log shows it well before renewals fail.