Add backups, token expiry warnings and name server checks

- Daily database backups: jarl-backup.timer runs 'jarld backup', which
  writes jarl-<UTC time>.db into backup.dir, checks it before naming it,
  and prunes by the time in the name while always keeping the newest
  keep_at_least. New backup: section in jarl.yaml; the dashboard and
  'jarl status' show the newest backup and warn when it is stale or
  failed. The Debian package enables the timer.
- 'jarld restore [FILE | latest]': checks the backup, shows it next to
  the current database, keeps the current one aside and records the
  restore. 'jarld serve' holds a database lock, so a restore can't run
  under a live jarld and two jarlds can't share a database.
- Token expiry warnings from 14 days ahead: dashboard, token lists, a
  daily log line, and a CLI warning via the Jarl-Token-Expires header;
  'jarl token list --expiring'.
- 'jarl zone serials' and the zone page's Name servers tab: compare the
  serial on a primary zone's secondaries (NS records, also-notify) or a
  secondary zone's primaries with this server's.
- 'jarl zone delegation': for zones on the internet, check the parent's
  delegation and glue against the zone; an undelegated zone gets a note.
- Server: Jarl/<version> header; CLI exit code 2 for failed checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -10,6 +10,40 @@ The first version, to be released as 0.1.0.
 
 ### Added
 
+- Daily database backups: the `jarl-backup.timer` systemd timer (turned on
+  by the Debian package) runs `jarld backup`, which writes
+  `jarl-<UTC time>.db` into `backup.dir` (`/var/backups/jarl`), checks it
+  with SQLite's integrity check before giving it that name, and deletes
+  backups older than `backup.retention` (30d) while always keeping the
+  newest `backup.keep_at_least` (7). Ages come from the names, and only
+  files named like backups are ever deleted. Settings are in the new
+  `backup:` section of jarl.yaml; the schedule is the timer's. The dashboard
+  and `jarl status` show the newest backup, and warn when it is older than
+  `backup.warn_after` (2d) or the last attempt failed. `backup.stats` also
+  backs up the statistics database.
+- `jarld restore [FILE | latest]`: lists backups, or checks one, shows what
+  it holds next to the current database, and after confirmation replaces
+  the database with it, keeping the current one as
+  `jarl.db.before-restore-<time>`. It refuses while jarld runs: `jarld
+  serve` now holds a lock (`jarl.db.lock`), which also stops two jarlds from
+  using the same database. docs/operations.md explains restoring.
+- Warnings before API tokens expire, from 14 days ahead: on the dashboard,
+  in token lists ("soon", "expired"), once a day in jarld's log, and from
+  the `jarl` CLI on stderr whenever it uses such a token (responses carry a
+  `Jarl-Token-Expires` header). `jarl token list --expiring [WITHIN]` lists
+  them.
+- `jarl zone serials ZONE` and the zone page's Name servers tab: ask the
+  zone's other name servers which serial they serve (a primary zone's NS
+  records and also-notify targets, a secondary zone's primaries, with their
+  ports and TSIG keys) and compare it with this server's. Exits with 2 when
+  one is behind or doesn't answer.
+- `jarl zone delegation ZONE` and a button on the Name servers tab: for
+  zones on the internet, checks that the parent zone delegates the zone to
+  the name servers in its NS records, with matching glue, and that each of
+  them serves it. An undelegated (internal-only) zone gets one calm note.
+  Neither check runs unless asked.
+- jarld sends `Server: Jarl/<version>`.
+- CLI exit code 2: a check ran and found problems.
 - CLI: several servers in one configuration file. Name them
   (`jarl config server add ns1 URL`), pick one with `-s/--server NAME` or
   `JARL_SERVER`, set a default with `jarl config use`, and group them

Makefile

@@ -4,7 +4,7 @@
 #   make test         unit tests (integration tests run when BIND is found;
 #                     set JARL_BIND_ROOT, see docs/development.md)
 #   make e2e          end-to-end test against a real named (scripts/e2e.sh)
-#   sudo make install install binaries, bash completion, systemd unit,
+#   sudo make install install binaries, bash completion, systemd units,
 #                     sysusers/tmpfiles files and an example /etc/jarl/jarl.yaml
 #   make deb          a Debian package in dist/ (DEB_ARCH=arm64 for another
 #                     architecture); install it with 'sudo apt install ./dist/jarl_*.deb'
@@ -69,7 +69,8 @@ install: build install-files
 	@echo
 	@echo "Next (as root): systemd-sysusers jarl.conf && systemd-tmpfiles --create jarl.conf"
 	@echo "                chgrp jarl /etc/jarl /etc/jarl/jarl.yaml && systemctl daemon-reload"
-	@echo "Then follow docs/install.md (init, import, adopt, start)."
+	@echo "Then follow docs/install.md (init, import, adopt, start), and turn on the daily"
+	@echo "database backup:  systemctl enable --now jarl-backup.timer"
 
 # Everything but /etc/jarl/jarl.yaml (shared by install and deb).
 install-files:
@@ -77,8 +78,10 @@ install-files:
 	install -D -m 0755 $(JARL_BIN) $(DESTDIR)$(BINDIR)/jarl
 	install -D -m 0644 bin/completion/jarl $(DESTDIR)$(BASHCOMPDIR)/jarl
 	install -D -m 0644 bin/completion/jarld $(DESTDIR)$(BASHCOMPDIR)/jarld
-	install -D -m 0644 deploy/systemd/jarl.service $(DESTDIR)$(UNITDIR)/jarl.service
+	for u in jarl.service jarl-backup.service jarl-backup.timer; do \
-	sed -i 's#/usr/local/bin/jarld#$(BINDIR)/jarld#; s#/usr/local/share/doc/jarl#$(DOCDIR)#' $(DESTDIR)$(UNITDIR)/jarl.service
+		install -D -m 0644 deploy/systemd/$$u $(DESTDIR)$(UNITDIR)/$$u && \
+		sed -i 's#/usr/local/bin/jarld#$(BINDIR)/jarld#; s#/usr/local/share/doc/jarl#$(DOCDIR)#' $(DESTDIR)$(UNITDIR)/$$u || exit 1; \
+	done
 	install -D -m 0644 deploy/sysusers.d/jarl.conf $(DESTDIR)$(SYSUSERSDIR)/jarl.conf
 	install -D -m 0644 deploy/tmpfiles.d/jarl.conf $(DESTDIR)$(TMPFILESDIR)/jarl.conf
 	install -D -m 0644 README.md $(DESTDIR)$(DOCDIR)/README.md
@@ -91,10 +94,11 @@ install-files:
 
 uninstall:
 	rm -f $(DESTDIR)$(BINDIR)/jarld $(DESTDIR)$(BINDIR)/jarl $(DESTDIR)$(UNITDIR)/jarl.service \
+		$(DESTDIR)$(UNITDIR)/jarl-backup.service $(DESTDIR)$(UNITDIR)/jarl-backup.timer \
 		$(DESTDIR)$(BASHCOMPDIR)/jarl $(DESTDIR)$(BASHCOMPDIR)/jarld \
 		$(DESTDIR)$(SYSUSERSDIR)/jarl.conf $(DESTDIR)$(TMPFILESDIR)/jarl.conf
 	rm -rf $(DESTDIR)$(DOCDIR)
-	@echo "Kept /etc/jarl, /var/lib/jarl and the files in /etc/bind/jarl and /var/lib/bind/jarl."
+	@echo "Kept /etc/jarl, /var/lib/jarl, /var/backups/jarl and the files in /etc/bind/jarl and /var/lib/bind/jarl."
 
 # A Debian package, built with dpkg-deb (no root needed). The package's
 # scripts are in deploy/debian; scripts/mkdeb.sh explains the layout.

README.md

@@ -38,11 +38,19 @@ error: www.adbraun.lol has a CNAME, so it can't have any other records (found A)
   Serials are bumped automatically.
 * **Users and API tokens with granular permissions**, e.g. a token that may
   only write `_acme-challenge` TXT records in one zone. Tokens can be
-  rotated (a new secret, same grants), with an optional grace period.
+  rotated (a new secret, same grants), with an optional grace period, and
+  Jarl warns two weeks before one expires (dashboard, log, and the CLI
+  using it).
   Passwords use argon2id, sessions have CSRF protection, logins are rate
   limited.
 * **Audit log** of every login, change and token, with user, token, IP and
   outcome.
+* **Daily backups** of the database by a systemd timer, checked as they are
+  written, with retention that never deletes the last few; `jarld restore`
+  puts one back safely.
+* **Name server checks**: whether the secondaries serve the current serial,
+  and, for zones on the internet, whether the parent zone delegates them
+  to the right name servers with the right glue.
 * **No token needed on the server**: root's `sudo jarl ...` talks to jarld
   over a local unix socket, so a lost token or password never locks you
   out.

cmd/jarl/admin_cmds.go

@@ -17,6 +17,7 @@ import (
 
 	"jarl/internal/api"
 	"jarl/internal/client"
+	"jarl/internal/model"
 )
 
 // randomSecret returns a base64 TSIG secret as long as the algorithm's
@@ -71,6 +72,37 @@ func readNewPassword() (string, error) {
 	return pw, nil
 }
 
+// fmtExpiry shows a token's expiry, flagging one that is close or past.
+func fmtExpiry(exp, revoked *time.Time) string {
+	s := fmtTime(exp)
+	if exp == nil || revoked != nil {
+		return s
+	}
+	switch d := time.Until(*exp); {
+	case d < 0:
+		return s + " (EXPIRED)"
+	case d < model.TokenExpiryWarning:
+		return s + " (in " + fmtUntil(d) + ")"
+	}
+	return s
+}
+
+// parseDays parses a duration such as 14d, 36h or 90m.
+func parseDays(s string) (time.Duration, error) {
+	if n, ok := strings.CutSuffix(s, "d"); ok {
+		days, err := strconv.Atoi(n)
+		if err != nil || days <= 0 {
+			return 0, fmt.Errorf("invalid duration %q (use e.g. 14d or 36h)", s)
+		}
+		return time.Duration(days) * 24 * time.Hour, nil
+	}
+	d, err := time.ParseDuration(s)
+	if err != nil || d <= 0 {
+		return 0, fmt.Errorf("invalid duration %q (use e.g. 14d or 36h)", s)
+	}
+	return d, nil
+}
+
 func fmtTime(t *time.Time) string {
 	if t == nil || t.IsZero() {
 		return "-"
@@ -152,6 +184,22 @@ func statusCmd() *cobra.Command {
 				for _, d := range st.Drift {
 					fmt.Fprintf(w, "Drift:\t%s %s (fix with 'jarl resync --force' or apply with --force)\n", d.Path, d.Reason)
 				}
+				if b := resp.Backup; b != nil {
+					switch {
+					case b.LastAt == nil:
+						fmt.Fprintf(w, "Backup:\tnone recorded (see 'jarld backup --help')\n")
+					case b.Stale:
+						fmt.Fprintf(w, "Backup:\tLAST ONE %s AGO: %s (is jarl-backup.timer running?)\n", strings.ToUpper(fmtUntil(time.Since(*b.LastAt))), b.LastFile)
+					default:
+						fmt.Fprintf(w, "Backup:\t%s ago: %s\n", fmtUntil(time.Since(*b.LastAt)), b.LastFile)
+					}
+					if b.LastError != "" {
+						fmt.Fprintf(w, "Backup:\tFAILED %s: %s\n", fmtTime(b.LastErrorAt), b.LastError)
+					}
+				}
+				for _, t := range resp.ExpiringTokens {
+					fmt.Fprintf(w, "Token:\t%s/%s (id %d) expires %s\n", t.Username, t.Name, t.ID, fmtExpiry(t.ExpiresAt, nil))
+				}
 				if len(st.Problems) > 0 {
 					w.(interface{ Flush() error }).Flush()
 					fmt.Println("Problems:")
@@ -506,12 +554,23 @@ func userCmd() *cobra.Command {
 
 func tokenCmd() *cobra.Command {
 	cmd := &cobra.Command{Use: "token", Aliases: []string{"tokens"}, Short: "Manage API tokens", Long: "Manage API tokens. A token has its user's permissions, or fewer when it has\nits own grants.\n\n" + grantHelp}
-	var user string
+	var user, expiring string
 	var all, revoked bool
 	list := &cobra.Command{
 		Use:   "list",
 		Short: "List your tokens (--user or --all for others; admin)",
+		Example: `  jarl token list
+  jarl token list --all --expiring        # every token that expires within 14 days
+  jarl token list --all --expiring 60d`,
 		RunE: func(cmd *cobra.Command, args []string) error {
+			var within time.Duration
+			if expiring != "" {
+				d, err := parseDays(expiring)
+				if err != nil {
+					return fmt.Errorf("--expiring: %w", err)
+				}
+				within = d
+			}
 			c, err := newClient()
 			if err != nil {
 				return err
@@ -535,6 +594,17 @@ func tokenCmd() *cobra.Command {
 			if err := c.Get(ctx(), path, qv, &tokens); err != nil {
 				return err
 			}
+			if within > 0 {
+				// Expiring soon, or already expired (and not revoked).
+				now := time.Now()
+				kept := []api.Token{}
+				for _, t := range tokens {
+					if t.ExpiresWithin(now, within) {
+						kept = append(kept, t)
+					}
+				}
+				tokens = kept
+			}
 			return output(tokens, func(w io.Writer) {
 				fmt.Fprintln(w, "ID\tUSER\tNAME\tPREFIX\tGRANTS\tEXPIRES\tROTATED\tLAST USED\tREVOKED")
 				for _, t := range tokens {
@@ -546,7 +616,7 @@ func tokenCmd() *cobra.Command {
 					if t.PrevExpiresAt != nil && time.Now().Before(*t.PrevExpiresAt) {
 						rot += " (old secret until " + fmtTime(t.PrevExpiresAt) + ")"
 					}
-					fmt.Fprintf(w, "%d\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", t.ID, t.Username, t.Name, t.PublicID, g, fmtTime(t.ExpiresAt), rot, fmtTime(t.LastUsedAt), fmtTime(t.RevokedAt))
+					fmt.Fprintf(w, "%d\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", t.ID, t.Username, t.Name, t.PublicID, g, fmtExpiry(t.ExpiresAt, t.RevokedAt), rot, fmtTime(t.LastUsedAt), fmtTime(t.RevokedAt))
 				}
 			})
 		},
@@ -554,6 +624,8 @@ func tokenCmd() *cobra.Command {
 	list.Flags().StringVar(&user, "user", "", "tokens of this user")
 	list.Flags().BoolVar(&all, "all", false, "tokens of every user")
 	list.Flags().BoolVar(&revoked, "revoked", false, "include revoked tokens")
+	list.Flags().StringVar(&expiring, "expiring", "", "only tokens that expire within this long, or have expired (default 14d)")
+	list.Flags().Lookup("expiring").NoOptDefVal = "14d"
 	var role, expires string
 	var grants []string
 	var save bool

cmd/jarl/main.go

@@ -13,7 +13,9 @@ import (
 	"path/filepath"
 	"strconv"
 	"strings"
+	"sync"
 	"text/tabwriter"
+	"time"
 
 	"github.com/spf13/cobra"
 	"go.yaml.in/yaml/v3"
@@ -154,9 +156,45 @@ func newClient() (*client.Client, error) {
 		}
 	}
 	c.DryRun, c.Force = flagDryRun || c.Preflight, flagForce
+	if !c.Preflight {
+		c.OnTokenExpiry = tokenExpiryWarning(t.Name)
+	}
 	return c, nil
 }
 
+// tokenExpiryWarning returns a function that warns, once, that the token
+// used for server expires soon. Scripts (certbot hooks, cron jobs) show it
+// in their logs well before the token stops working.
+func tokenExpiryWarning(server string) func(time.Time) {
+	var once sync.Once
+	return func(t time.Time) {
+		once.Do(func() {
+			where := ""
+			if server != "" && server != localName {
+				where = " for server " + server
+			}
+			when := "expires " + fmtTime(&t) + " (in " + fmtUntil(time.Until(t)) + ")"
+			if time.Now().After(t) {
+				when = "has expired"
+			}
+			fmt.Fprintf(os.Stderr, "warning: the API token%s %s; give it a new lifetime with 'jarl token rotate --current' (see 'jarl token rotate --help')\n", where, when)
+		})
+	}
+}
+
+// fmtUntil is a short, rounded-down duration: "5 days", "3 hours".
+func fmtUntil(d time.Duration) string {
+	switch {
+	case d >= 48*time.Hour:
+		return fmt.Sprintf("%d days", int(d/(24*time.Hour)))
+	case d >= 2*time.Hour:
+		return fmt.Sprintf("%d hours", int(d/time.Hour))
+	case d >= time.Minute:
+		return fmt.Sprintf("%d minutes", int(d/time.Minute))
+	}
+	return "under a minute"
+}
+
 func firstNonEmpty(vals ...string) string {
 	for _, v := range vals {
 		if v != "" {
@@ -326,6 +364,10 @@ func printError(err error) {
 // authenticated or not allowed, 4 not found, 5 conflict, 6 invalid
 // request or rejected by validation, 7 server error.
 func exitCode(err error) int {
+	var cf *checkFailed
+	if errors.As(err, &cf) {
+		return 2
+	}
 	var ce *client.Error
 	if errors.As(err, &ce) {
 		switch {

cmd/jarl/nscheck.go

@@ -0,0 +1,269 @@
+package main
+
+import (
+	"fmt"
+	"io"
+	"strings"
+
+	"github.com/spf13/cobra"
+
+	"jarl/internal/api"
+	"jarl/internal/nscheck"
+)
+
+// checkFailed is returned when a check ran but found problems (exit code
+// 2), after its report was printed.
+type checkFailed struct{ msg string }
+
+func (e *checkFailed) Error() string { return e.msg }
+
+func zoneSerialsCmd() *cobra.Command {
+	var view string
+	cmd := &cobra.Command{
+		Use:   "serials ZONE",
+		Short: "Compare the zone's serial on its other name servers with this server's",
+		Long: `Asks the zone's other name servers which serial they serve and compares it
+with the one this server serves.
+
+For a primary zone those are the servers in its NS records and its
+also-notify targets (its secondaries); for a secondary or mirror zone, its
+primaries. Addresses come from Jarl's own A/AAAA records where it has them,
+otherwise from a DNS lookup. This server itself isn't asked.
+
+Exits with 2 when a server is behind, doesn't serve the zone, refuses, or
+doesn't answer, so it can run from cron or a monitoring system.`,
+		Example: `  jarl zone serials adbraun.lol
+  jarl zone serials adbraun.lol --view internal -o json`,
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			c, err := newClient()
+			if err != nil {
+				return err
+			}
+			path, qv := zonePath(args[0], view)
+			var resp api.SerialsResponse
+			if err := c.Get(ctx(), path+"/serials", qv, &resp); err != nil {
+				return err
+			}
+			bad := 0
+			for _, r := range resp.Servers {
+				if !r.OK() {
+					bad++
+				}
+			}
+			err = output(resp, func(w io.Writer) {
+				head := displayName(resp.Zone)
+				if resp.View != "" {
+					head += " [view " + resp.View + "]"
+				}
+				head += ", " + resp.Type
+				if resp.Serial != nil {
+					src := map[string]string{"named": "served by named here", "jarl": "Jarl's; named not asked"}[resp.SerialSource]
+					head += fmt.Sprintf(", serial %d (%s)", *resp.Serial, src)
+				}
+				fmt.Fprintf(w, "%s\n\n", head)
+				if len(resp.Servers) > 0 {
+					fmt.Fprintln(w, "SERVER\tADDRESS\tFROM\tSTATUS\tSERIAL\tTIME")
+					for _, r := range resp.Servers {
+						serial, rtt := "-", ""
+						if r.Serial != nil {
+							serial = fmt.Sprint(*r.Serial)
+						}
+						if r.RTTMs > 0 || r.Status == "in_sync" || r.Status == "behind" || r.Status == "ahead" {
+							rtt = fmt.Sprintf("%d ms", r.RTTMs)
+						}
+						addr := r.Address
+						if r.Port != "" {
+							addr += " port " + r.Port
+						}
+						fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\n", firstNonEmpty(displayName(r.Name), "-"), firstNonEmpty(addr, "-"), r.Source, nsStatusText(r), serial, rtt)
+					}
+				}
+				flush(w)
+				if len(resp.Notes) > 0 {
+					fmt.Println()
+				}
+				for _, n := range resp.Notes {
+					fmt.Printf("note: %s\n", n)
+				}
+			})
+			if err != nil {
+				return err
+			}
+			if bad > 0 {
+				if bad == 1 {
+					return &checkFailed{fmt.Sprintf("1 of %d name servers isn't serving the current serial", len(resp.Servers))}
+				}
+				return &checkFailed{fmt.Sprintf("%d of %d name servers aren't serving the current serial", bad, len(resp.Servers))}
+			}
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&view, "view", "", "view of the zone")
+	return cmd
+}
+
+func zoneDelegationCmd() *cobra.Command {
+	var view, resolver string
+	cmd := &cobra.Command{
+		Use:   "delegation ZONE",
+		Short: "Check how the parent zone delegates the zone (for zones on the internet)",
+		Long: `Checks the zone the way the internet sees it: finds the parent zone (for
+adbraun.lol, "lol."), asks each of its servers for the zone's delegation,
+and compares the name servers and glue addresses it hands out with the
+zone's NS and address records in Jarl. Then asks each delegated name server
+for the zone's serial.
+
+A zone used only on your own network isn't delegated; that's reported as a
+note, not a problem. Nothing runs this check unless you ask for it.
+
+The parent is found through jarld's resolver (/etc/resolv.conf on the
+server), or --resolver. Exits with 2 when the check finds an error (such as
+a name server that doesn't serve the zone, or missing glue); warnings and
+notes exit with 0.`,
+		Example: `  jarl zone delegation adbraun.lol --view external
+  jarl zone delegation adbraun.lol --resolver 9.9.9.9`,
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			c, err := newClient()
+			if err != nil {
+				return err
+			}
+			path, qv := zonePath(args[0], view)
+			if resolver != "" {
+				qv.Set("resolver", resolver)
+			}
+			var resp api.DelegationResponse
+			if err := c.Get(ctx(), path+"/delegation", qv, &resp); err != nil {
+				return err
+			}
+			errors := 0
+			for _, f := range resp.Findings {
+				if f.Level == "error" {
+					errors++
+				}
+			}
+			err = output(resp, func(w io.Writer) {
+				rep := resp.DelegationReport
+				fmt.Fprintf(w, "Zone:\t%s\n", displayName(rep.Zone))
+				if rep.Parent != "" {
+					fmt.Fprintf(w, "Parent zone:\t%s (found through %s)\n", rep.Parent, rep.Resolver)
+				}
+				// The parent's servers, grouped by what they say (the root
+				// zone alone has 13).
+				var order []string
+				groups := map[string][]string{}
+				for _, p := range rep.Parents {
+					var say string
+					switch p.Status {
+					case "delegated":
+						say = "delegate to " + strings.Join(displayNames(p.NS), ", ")
+					case "not_delegated":
+						say = "no delegation"
+					default:
+						say = p.Status + ": " + p.Error
+					}
+					if groups[say] == nil {
+						order = append(order, say)
+					}
+					groups[say] = append(groups[say], displayName(p.Name))
+				}
+				for _, say := range order {
+					who := groups[say][0]
+					if n := len(groups[say]); n > 1 {
+						who = fmt.Sprintf("%s and %d more", who, n-1)
+					}
+					fmt.Fprintf(w, "  %s:\t%s\n", who, say)
+				}
+				if rep.Delegated {
+					fmt.Fprintf(w, "NS at the parent:\t%s\n", strings.Join(displayNames(rep.ParentNS), ", "))
+					fmt.Fprintf(w, "NS in the zone:\t%s\n", strings.Join(displayNames(rep.JarlNS), ", "))
+					for _, g := range rep.Glue {
+						mark := "matches"
+						if !g.Match {
+							mark = "zone says " + firstNonEmpty(strings.Join(g.Jarl, ", "), "nothing")
+						}
+						fmt.Fprintf(w, "Glue %s:\t%s (%s)\n", displayName(g.Name), firstNonEmpty(strings.Join(g.Parent, ", "), "none"), mark)
+					}
+				}
+				flush(w)
+				if len(rep.Servers) > 0 {
+					fmt.Fprintln(w, "\nSERVER\tADDRESS\tSTATUS\tSERIAL")
+					for _, r := range rep.Servers {
+						serial := "-"
+						if r.Serial != nil {
+							serial = fmt.Sprint(*r.Serial)
+						}
+						fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", displayName(r.Name), firstNonEmpty(r.Address, "-"), nsStatusText(r), serial)
+					}
+					flush(w)
+				}
+				fmt.Println()
+				for _, f := range rep.Findings {
+					fmt.Printf("%-8s %s\n", f.Level+":", f.Message)
+				}
+				for _, n := range resp.Notes {
+					fmt.Printf("note:    %s\n", n)
+				}
+			})
+			if err != nil {
+				return err
+			}
+			if errors > 0 {
+				return &checkFailed{fmt.Sprintf("the delegation check found %d error(s)", errors)}
+			}
+			return nil
+		},
+	}
+	cmd.Flags().StringVar(&view, "view", "", "view of the zone")
+	cmd.Flags().StringVar(&resolver, "resolver", "", "resolver to find the parent zone with, e.g. 9.9.9.9 (default: the server's)")
+	return cmd
+}
+
+// nsStatusText describes a name server's answer in a few words.
+func nsStatusText(r nscheck.Result) string {
+	switch r.Status {
+	case "self":
+		return "this server"
+	case "in_sync":
+		return "in sync"
+	case "answered":
+		return "answers"
+	case "behind":
+		return fmt.Sprintf("BEHIND by %d", -r.Diff)
+	case "ahead":
+		return fmt.Sprintf("AHEAD by %d", r.Diff)
+	case "not_authoritative":
+		return "DOESN'T SERVE THE ZONE"
+	case "refused":
+		return "REFUSED"
+	case "unreachable":
+		return "NO ANSWER"
+	case "no_route":
+		return "no route from here"
+	case "no_address":
+		return "NO ADDRESS: " + r.Error
+	}
+	return strings.ToUpper(r.Status) + ": " + r.Error
+}
+
+func displayName(n string) string {
+	if n == "" || n == "." {
+		return n
+	}
+	return strings.TrimSuffix(n, ".")
+}
+
+func displayNames(ns []string) []string {
+	out := make([]string, len(ns))
+	for i, n := range ns {
+		out[i] = displayName(n)
+	}
+	return out
+}
+
+func flush(w io.Writer) {
+	if f, ok := w.(interface{ Flush() error }); ok {
+		f.Flush()
+	}
+}

cmd/jarl/zones.go

@@ -367,7 +367,7 @@ equal to the file: records not in the file are deleted.`,
 	importCmd.Flags().BoolVar(&replace, "replace", false, "delete records that are not in the file")
 	importCmd.Flags().BoolVar(&useSOA, "use-soa", false, "also take SOA timers and $TTL from the file")
 
-	cmd.AddCommand(list, show, create, update, del, fileCmd, importCmd)
+	cmd.AddCommand(list, show, create, update, del, fileCmd, importCmd, zoneSerialsCmd(), zoneDelegationCmd())
 	return cmd
 }
 

cmd/jarld/admin.go

@@ -356,30 +356,6 @@ func resyncCmd() *cobra.Command {
 	return cmd
 }
 
-func backupCmd() *cobra.Command {
-	return &cobra.Command{
-		Use:   "backup FILE",
-		Short: "Write a consistent copy of the database to FILE (safe while jarld runs)",
-		Args:  cobra.ExactArgs(1),
-		RunE: func(cmd *cobra.Command, args []string) error {
-			a, err := openApp(true)
-			if err != nil {
-				return err
-			}
-			defer a.close()
-			if _, err := os.Stat(args[0]); err == nil {
-				return fmt.Errorf("%s already exists", args[0])
-			}
-			if err := a.store.Backup(context.Background(), args[0]); err != nil {
-				return err
-			}
-			os.Chmod(args[0], 0o600)
-			fmt.Println("Database copied to", args[0])
-			return nil
-		},
-	}
-}
-
 func configCmd() *cobra.Command {
 	var example bool
 	cmd := &cobra.Command{

cmd/jarld/backup.go

@@ -0,0 +1,374 @@
+package main
+
+import (
+	"bufio"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"text/tabwriter"
+	"time"
+
+	"github.com/spf13/cobra"
+	"golang.org/x/term"
+
+	"jarl/internal/auth"
+	"jarl/internal/backup"
+	"jarl/internal/config"
+	"jarl/internal/store"
+)
+
+func backupCmd() *cobra.Command {
+	return &cobra.Command{
+		Use:   "backup [FILE]",
+		Short: "Back up the database (safe while jarld runs)",
+		Long: `Writes a consistent copy of Jarl's database and checks it.
+
+Without FILE the copy goes into backup.dir from jarl.yaml (/var/backups/jarl),
+named jarl-<UTC time>.db (plus stats-<UTC time>.db with backup.stats), and
+older backups there are deleted as backup.retention and backup.keep_at_least
+say. This is what the jarl-backup.timer systemd timer runs every day.
+
+With FILE it writes a single copy there, which is never deleted
+automatically.
+
+Every backup is recorded in the database, and the dashboard shows the newest.
+Restore one with 'jarld restore'.`,
+		Example: `  sudo -u jarl jarld backup
+  sudo -u jarl jarld backup /root/jarl-before-upgrade.db`,
+		Args: cobra.MaximumNArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			cfg, err := config.Load(configPath, true)
+			if err != nil {
+				return fmt.Errorf("loading configuration: %w", err)
+			}
+			if _, err := os.Stat(cfg.Database); err != nil {
+				return fmt.Errorf("no database at %s to back up (create it with 'jarld init')", cfg.Database)
+			}
+			// Not store.Open: a backup must never create or upgrade the
+			// database a running jarld uses.
+			st, err := store.OpenExisting(cfg.Database)
+			if err != nil {
+				return fmt.Errorf("opening %s: %w", cfg.Database, err)
+			}
+			defer st.Close()
+			// Run as root, leave no root-owned files behind for jarld.
+			defer newDirOwner(filepath.Dir(cfg.Database)).fix(filepath.Dir(cfg.Database))
+			ctx := context.Background()
+			now := time.Now()
+			fail := func(err error) error {
+				if rerr := st.RecordBackup(ctx, store.BackupRecord{At: now, Error: err.Error()}); rerr != nil {
+					fmt.Fprintln(os.Stderr, "warning: couldn't record the failure in the database:", rerr)
+				}
+				return err
+			}
+			if len(args) == 1 {
+				dst := args[0]
+				if _, err := os.Lstat(dst); err == nil {
+					return fmt.Errorf("%s already exists", dst)
+				}
+				if err := backup.Copy(ctx, cfg.Database, dst); err != nil {
+					os.Remove(dst)
+					return fail(fmt.Errorf("backing up %s: %w", cfg.Database, err))
+				}
+				if err := backup.Verify(ctx, dst); err != nil {
+					os.Remove(dst)
+					return fail(fmt.Errorf("the backup failed its check: %w", err))
+				}
+				fi, _ := os.Stat(dst)
+				st.RecordBackup(ctx, store.BackupRecord{At: now, File: dst, Size: fi.Size()})
+				fmt.Printf("Database backed up to %s (%s)\n", dst, size(fi.Size()))
+				return nil
+			}
+
+			dir := cfg.Backup.Dir
+			defer newDirOwner(dir).fix(dir)
+			f, err := backup.Write(ctx, cfg.Database, dir, backup.KindJarl, now)
+			if err != nil {
+				return fail(fmt.Errorf("backing up %s: %w", cfg.Database, err))
+			}
+			fmt.Printf("Database backed up to %s (%s)\n", f.Path, size(f.Size))
+			kinds := []string{backup.KindJarl}
+			if cfg.Backup.Stats {
+				if _, err := os.Stat(cfg.Stats.Database); err == nil {
+					sf, err := backup.Write(ctx, cfg.Stats.Database, dir, backup.KindStats, now)
+					if err != nil {
+						return fail(fmt.Errorf("backing up the statistics database %s: %w", cfg.Stats.Database, err))
+					}
+					fmt.Printf("Statistics backed up to %s (%s)\n", sf.Path, size(sf.Size))
+				}
+				kinds = append(kinds, backup.KindStats)
+			}
+			for _, kind := range kinds {
+				removed, err := backup.Prune(dir, kind, now, time.Duration(cfg.Backup.Retention), cfg.Backup.KeepAtLeast)
+				for _, r := range removed {
+					fmt.Printf("Deleted %s (older than %s)\n", r.Path, fmtDuration(time.Duration(cfg.Backup.Retention)))
+				}
+				if err != nil {
+					return fail(fmt.Errorf("deleting old backups: %w", err))
+				}
+			}
+			if err := st.RecordBackup(ctx, store.BackupRecord{At: now, File: f.Path, Size: f.Size}); err != nil {
+				fmt.Fprintln(os.Stderr, "warning: couldn't record the backup in the database:", err)
+			}
+			return nil
+		},
+	}
+}
+
+func restoreCmd() *cobra.Command {
+	var yes bool
+	cmd := &cobra.Command{
+		Use:   "restore [FILE | latest]",
+		Short: "Replace the database with a backup (jarld must be stopped)",
+		Long: `Without arguments, lists the backups in backup.dir.
+
+With FILE, or "latest" for the newest backup in backup.dir, it checks the
+backup, shows what it holds next to the current database, and after you
+confirm replaces the database with it. jarld must be stopped first. The
+current database is kept as DATABASE.before-restore-<UTC time>, and the
+restore is recorded in the restored database's audit log.
+
+Then start jarld and bring named's files in line with the restored database:
+
+  systemctl start jarl
+  jarl resync --dry-run     # what differs
+  jarl resync --force       # write everything and reload named
+
+See "Restoring" in docs/operations.md, also for zone serials and secondaries.`,
+		Example: `  sudo jarld restore
+  sudo systemctl stop jarl
+  sudo jarld restore latest
+  sudo jarld restore /var/backups/jarl/jarl-20261008T031512Z.db`,
+		Args: cobra.MaximumNArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			cfg, err := config.Load(configPath, true)
+			if err != nil {
+				return fmt.Errorf("loading configuration: %w", err)
+			}
+			ctx := context.Background()
+			if len(args) == 0 {
+				return listBackups(cfg.Backup.Dir)
+			}
+			src := args[0]
+			if src == "latest" {
+				files, err := backup.List(cfg.Backup.Dir)
+				if err != nil {
+					return err
+				}
+				src = ""
+				for _, f := range files {
+					if f.Kind == backup.KindJarl {
+						src = f.Path
+						break
+					}
+				}
+				if src == "" {
+					return fmt.Errorf("no backups in %s", cfg.Backup.Dir)
+				}
+			}
+			if err := backup.Verify(ctx, src); err != nil {
+				return fmt.Errorf("%s can't be restored: %w", src, err)
+			}
+			info, err := backup.Inspect(ctx, src)
+			if err != nil {
+				return err
+			}
+			if info.Schema > store.SchemaVersion() {
+				return fmt.Errorf("%s was made by a newer Jarl (schema %d, this one knows %d); install that version to restore it", src, info.Schema, store.SchemaVersion())
+			}
+			unlock, err := lockDatabase(cfg.Database)
+			if errors.Is(err, errInUse) {
+				return fmt.Errorf("jarld is running; stop it first (systemctl stop jarl), then run this again")
+			}
+			if err != nil {
+				return err
+			}
+			defer unlock()
+
+			w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
+			fmt.Fprintf(w, "Backup:\t%s\n", src)
+			describe(w, info, backupTime(src))
+			if cur, err := backup.Inspect(ctx, cfg.Database); err == nil {
+				fmt.Fprintf(w, "\nCurrent database:\t%s\n", cfg.Database)
+				describe(w, cur, time.Time{})
+			} else if !errors.Is(err, os.ErrNotExist) {
+				fmt.Fprintf(w, "\nCurrent database:\t%s (unreadable: %v)\n", cfg.Database, err)
+			}
+			w.Flush()
+			if !yes {
+				if !term.IsTerminal(int(os.Stdin.Fd())) {
+					return fmt.Errorf("add --yes to restore without being asked")
+				}
+				fmt.Print("\nReplace the current database with this backup? [y/N] ")
+				line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
+				if a := strings.ToLower(strings.TrimSpace(line)); a != "y" && a != "yes" {
+					return fmt.Errorf("nothing changed")
+				}
+			}
+
+			db := cfg.Database
+			defer newDirOwner(filepath.Dir(db)).fix(filepath.Dir(db))
+			aside := db + ".before-restore-" + time.Now().UTC().Format("20060102T150405Z")
+			// The database's WAL and shared-memory files belong to it:
+			// they move with it, and the restored copy starts without.
+			moved := false
+			for _, suffix := range []string{"", "-wal", "-shm"} {
+				if _, err := os.Lstat(db + suffix); err == nil {
+					if err := os.Rename(db+suffix, aside+suffix); err != nil {
+						return err
+					}
+					moved = true
+				}
+			}
+			if err := copyFile(src, db, 0o640); err != nil {
+				if moved {
+					os.Rename(aside, db)
+					os.Rename(aside+"-wal", db+"-wal")
+					os.Rename(aside+"-shm", db+"-shm")
+				}
+				return fmt.Errorf("restoring: %w (the current database was left in place)", err)
+			}
+			// Open it as jarld will: an older backup gets the current
+			// schema now, and the restore goes into its audit log.
+			a, err := openApp(true)
+			if err != nil {
+				return fmt.Errorf("the backup was copied to %s but doesn't open: %w", db, err)
+			}
+			p := &auth.Principal{Method: "local"}
+			if lp, err := a.localPrincipal(ctx, ""); err == nil {
+				p = lp
+			}
+			a.eng.Audit(ctx, p, "system.restore", "", 0, "", "success", "database restored from "+src)
+			a.close()
+
+			fmt.Printf("\nRestored %s from %s.\n", db, src)
+			if moved {
+				fmt.Printf("The previous database is in %s.\n", aside)
+			}
+			fmt.Print(`
+Next:
+  systemctl start jarl
+  jarl resync --dry-run     # what differs between the restored database and named's files
+  jarl resync --force       # write them and reload named
+
+If you have secondary servers, zone serials older than what they last saw
+are ignored by them: raise them with 'jarl zone update ZONE --serial N'
+(see "Restoring" in docs/operations.md).
+`)
+			return nil
+		},
+	}
+	cmd.Flags().BoolVarP(&yes, "yes", "y", false, "don't ask for confirmation")
+	return cmd
+}
+
+func listBackups(dir string) error {
+	files, err := backup.List(dir)
+	if err != nil {
+		return err
+	}
+	if len(files) == 0 {
+		fmt.Printf("No backups in %s.\n", dir)
+		return nil
+	}
+	w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
+	fmt.Fprintln(w, "MADE\tAGE\tKIND\tSIZE\tFILE")
+	for _, f := range files {
+		fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", f.Time.Local().Format("2006-01-02 15:04:05"), fmtDuration(time.Since(f.Time)), f.Kind, size(f.Size), f.Path)
+	}
+	w.Flush()
+	fmt.Println("\nRestore one with: jarld restore FILE   (or: jarld restore latest)")
+	return nil
+}
+
+func describe(w io.Writer, in *backup.Info, made time.Time) {
+	if !made.IsZero() {
+		fmt.Fprintf(w, "  made\t%s (%s ago)\n", made.Local().Format("2006-01-02 15:04:05"), fmtDuration(time.Since(made)))
+	}
+	fmt.Fprintf(w, "  contents\t%s, %s, %s (schema %d)\n", plural(in.Zones, "zone"), plural(in.Records, "record"), plural(in.Users, "user"), in.Schema)
+	if in.LastChange != nil {
+		fmt.Fprintf(w, "  last change\t%s\n", in.LastChange.Local().Format("2006-01-02 15:04:05"))
+	}
+}
+
+// backupTime is when a backup was made: from its name if it has a
+// backup's name, otherwise its modification time.
+func backupTime(path string) time.Time {
+	for _, f := range mustList(filepath.Dir(path)) {
+		if f.Path == filepath.Clean(path) {
+			return f.Time
+		}
+	}
+	if fi, err := os.Stat(path); err == nil {
+		return fi.ModTime()
+	}
+	return time.Time{}
+}
+
+func mustList(dir string) []backup.File {
+	files, _ := backup.List(dir)
+	return files
+}
+
+// copyFile copies src to dst through a temporary file, so dst is either
+// complete or absent.
+func copyFile(src, dst string, mode os.FileMode) error {
+	in, err := os.Open(src)
+	if err != nil {
+		return err
+	}
+	defer in.Close()
+	tmp := dst + ".restore-tmp"
+	out, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, mode)
+	if err != nil {
+		return err
+	}
+	if _, err := io.Copy(out, in); err != nil {
+		out.Close()
+		os.Remove(tmp)
+		return err
+	}
+	if err := out.Sync(); err != nil {
+		out.Close()
+		os.Remove(tmp)
+		return err
+	}
+	if err := out.Close(); err != nil {
+		os.Remove(tmp)
+		return err
+	}
+	return os.Rename(tmp, dst)
+}
+
+func size(n int64) string {
+	switch {
+	case n >= 1<<20:
+		return fmt.Sprintf("%.1f MB", float64(n)/(1<<20))
+	case n >= 1<<10:
+		return fmt.Sprintf("%.0f KB", float64(n)/(1<<10))
+	}
+	return fmt.Sprintf("%d bytes", n)
+}
+
+func fmtDuration(d time.Duration) string {
+	switch {
+	case d >= 48*time.Hour:
+		return fmt.Sprintf("%d days", int(d/(24*time.Hour)))
+	case d >= 2*time.Hour:
+		return fmt.Sprintf("%d hours", int(d/time.Hour))
+	case d >= 2*time.Minute:
+		return fmt.Sprintf("%d minutes", int(d/time.Minute))
+	}
+	return "under 2 minutes"
+}
+
+func plural(n int, word string) string {
+	if n == 1 {
+		return "1 " + word
+	}
+	return fmt.Sprintf("%d %ss", n, word)
+}

cmd/jarld/lock.go

@@ -0,0 +1,40 @@
+package main
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"syscall"
+)
+
+// errInUse means another process holds the database lock.
+var errInUse = errors.New("the database is in use by a running jarld or by 'jarld restore'")
+
+// lockDatabase takes the exclusive lock (on DATABASE.lock) that marks the
+// database as in use by a running jarld or by 'jarld restore', so a
+// restore can never replace the database under a running jarld, and two
+// jarlds can't serve the same database. Other commands (backup, user, ...)
+// don't take it: they work alongside a running jarld.
+func lockDatabase(db string) (release func(), err error) {
+	if err := os.MkdirAll(filepath.Dir(db), 0o750); err != nil {
+		return nil, err
+	}
+	path := db + ".lock"
+	// Read-only is enough for flock, and works whoever created the file.
+	f, err := os.OpenFile(path, os.O_RDONLY|os.O_CREATE, 0o640)
+	if err != nil {
+		return nil, err
+	}
+	if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
+		f.Close()
+		if errors.Is(err, syscall.EWOULDBLOCK) {
+			return nil, errInUse
+		}
+		return nil, fmt.Errorf("locking %s: %w", path, err)
+	}
+	return func() {
+		syscall.Flock(int(f.Fd()), syscall.LOCK_UN)
+		f.Close()
+	}, nil
+}

cmd/jarld/main.go

@@ -57,7 +57,7 @@ func rootCmd() *cobra.Command {
 		def = config.DefaultPath
 	}
 	root.PersistentFlags().StringVarP(&configPath, "config", "c", def, "configuration file")
-	root.AddCommand(serveCmd(), initCmd(), userCmd(), importCmd(), adoptCmd(), checkCmd(), renderCmd(), resyncCmd(), backupCmd(), configCmd(), versionCmd())
+	root.AddCommand(serveCmd(), initCmd(), userCmd(), importCmd(), adoptCmd(), checkCmd(), renderCmd(), resyncCmd(), backupCmd(), restoreCmd(), configCmd(), versionCmd())
 	return root
 }
 
@@ -226,6 +226,15 @@ func serveCmd() *cobra.Command {
 		Use:   "serve",
 		Short: "Run the daemon (web UI and API)",
 		RunE: func(cmd *cobra.Command, args []string) error {
+			cfg, err := config.Load(configPath, true)
+			if err != nil {
+				return fmt.Errorf("loading configuration: %w", err)
+			}
+			unlock, err := lockDatabase(cfg.Database)
+			if err != nil {
+				return fmt.Errorf("%s: %w", cfg.Database, err)
+			}
+			defer unlock()
 			a, err := openApp(true)
 			if err != nil {
 				return err
@@ -287,6 +296,7 @@ func serveCmd() *cobra.Command {
 				TrustedProxies: a.cfg.Proxies(), SessionLifetime: time.Duration(a.cfg.Session.Lifetime),
 				SessionIdle: time.Duration(a.cfg.Session.IdleTimeout), CookieSecure: a.cfg.Session.CookieSecure,
 				Version: version.Version, Stats: collector, Socket: a.cfg.Socket,
+				BackupDir: a.cfg.Backup.Dir, BackupWarnAfter: time.Duration(a.cfg.Backup.WarnAfter),
 				HTTPReq: server.HTTPReq{
 					Enabled:           a.cfg.ACME.HTTPReq.Enabled,
 					AllowPlainHTTP:    !a.cfg.ACME.HTTPReq.RequireHTTPS,

deploy/debian/postinst

@@ -18,8 +18,26 @@ if [ "$1" = configure ]; then
 		mv /etc/jarl/jarl.yaml.dpkg-new /etc/jarl/jarl.yaml
 	fi
 
+	# The daily database backup (jarl-backup.timer) is on by default. An
+	# administrator's 'systemctl disable jarl-backup.timer' is remembered
+	# by deb-systemd-helper and kept across upgrades. Before 'jarld init'
+	# the backup skips itself (there is no database yet).
+	if command -v deb-systemd-helper >/dev/null 2>&1; then
+		deb-systemd-helper unmask jarl-backup.timer >/dev/null || true
+		if deb-systemd-helper --quiet was-enabled jarl-backup.timer; then
+			deb-systemd-helper enable jarl-backup.timer >/dev/null || true
+		else
+			deb-systemd-helper update-state jarl-backup.timer >/dev/null || true
+		fi
+	elif [ -z "$2" ] && [ -d /run/systemd/system ]; then
+		systemctl enable jarl-backup.timer >/dev/null 2>&1 || true
+	fi
+
 	if [ -d /run/systemd/system ]; then
 		systemctl daemon-reload || true
+		if systemctl is-enabled --quiet jarl-backup.timer 2>/dev/null; then
+			systemctl start jarl-backup.timer || true
+		fi
 		# An upgrade restarts a running jarld (its database is upgraded
 		# when it starts). A first install starts nothing: Jarl needs
 		# 'jarld init' and 'jarld adopt' first.

deploy/debian/postrm

@@ -10,12 +10,27 @@ remove|purge)
 	;;
 esac
 
+# The backup timer's enabled state (see postinst): masked while the
+# package is removed, forgotten on purge.
+if command -v deb-systemd-helper >/dev/null 2>&1; then
+	case "$1" in
+	remove)
+		deb-systemd-helper mask jarl-backup.timer >/dev/null || true
+		;;
+	purge)
+		deb-systemd-helper purge jarl-backup.timer >/dev/null || true
+		deb-systemd-helper unmask jarl-backup.timer >/dev/null || true
+		;;
+	esac
+fi
+
 # Even a purge keeps Jarl's data: the database in /var/lib/jarl is the
 # source of your DNS configuration, and named.conf may still include the
 # files in /etc/bind/jarl. Remove them by hand when you no longer need them.
 if [ "$1" = purge ]; then
-	echo "jarl: kept /etc/jarl, /var/lib/jarl, /etc/bind/jarl, /var/lib/bind/jarl and"
+	echo "jarl: kept /etc/jarl, /var/lib/jarl, /var/backups/jarl, /etc/bind/jarl,"
-	echo "/var/cache/bind/jarl (named may still use them); remove them by hand if unused."
+	echo "/var/lib/bind/jarl and /var/cache/bind/jarl (named may still use some of"
+	echo "them); remove them by hand if unused."
 fi
 
 exit 0

deploy/debian/prerm

@@ -2,11 +2,15 @@
 # Jarl: runs before the package's files are removed (also on upgrades).
 set -e
 
-# On removal (not on upgrade) stop jarld and remove its boot-time link.
+# On removal (not on upgrade) stop jarld and the backup timer and remove
-# named keeps running with the configuration Jarl last wrote.
+# jarld's boot-time link (postrm deals with the timer's). named keeps
+# running with the configuration Jarl last wrote.
 if [ "$1" = remove ] && [ -d /run/systemd/system ]; then
-	systemctl stop jarl.service 2>/dev/null || true
+	systemctl stop jarl.service jarl-backup.timer jarl-backup.service 2>/dev/null || true
 	systemctl disable jarl.service 2>/dev/null || true
+	if ! command -v deb-systemd-helper >/dev/null 2>&1; then
+		systemctl disable jarl-backup.timer 2>/dev/null || true
+	fi
 fi
 
 exit 0

deploy/systemd/jarl-backup.service

@@ -0,0 +1,47 @@
+# Jarl: one database backup, run daily by jarl-backup.timer.
+# Installed as /etc/systemd/system/jarl-backup.service (see
+# docs/operations.md#backups). What to keep, and where, is in the backup:
+# section of /etc/jarl/jarl.yaml; when it runs is in jarl-backup.timer.
+[Unit]
+Description=Jarl database backup
+Documentation=file:///usr/local/share/doc/jarl/docs/operations.md
+# Nothing to back up before 'jarld init'. (If you moved the database in
+# jarl.yaml, change this with 'systemctl edit jarl-backup.service'.)
+ConditionPathExists=/var/lib/jarl/jarl.db
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/jarld --config /etc/jarl/jarl.yaml backup
+User=jarl
+Group=jarl
+UMask=0077
+Nice=10
+IOSchedulingClass=idle
+
+# The database (read, and the backup's record written) and the backup
+# directory. If you change backup.dir in jarl.yaml, allow the new one too:
+# 'systemctl edit jarl-backup.service' and add ReadWritePaths=/new/dir.
+ProtectSystem=strict
+ReadWritePaths=/var/lib/jarl -/var/backups/jarl
+ProtectHome=yes
+PrivateTmp=yes
+PrivateDevices=yes
+PrivateNetwork=yes
+NoNewPrivileges=yes
+CapabilityBoundingSet=
+AmbientCapabilities=
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+ProtectClock=yes
+ProtectHostname=yes
+RestrictNamespaces=yes
+RestrictRealtime=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_UNIX
+SystemCallArchitectures=native
+SystemCallFilter=@system-service
+SystemCallErrorNumber=EPERM

deploy/systemd/jarl-backup.timer

@@ -0,0 +1,20 @@
+# Jarl: runs jarl-backup.service (a database backup) every day.
+# Installed as /etc/systemd/system/jarl-backup.timer. Change the schedule
+# with 'systemctl edit jarl-backup.timer', e.g.
+#   [Timer]
+#   OnCalendar=
+#   OnCalendar=*-*-* 02:30
+# and keep backup.warn_after in jarl.yaml above the new interval.
+[Unit]
+Description=Daily Jarl database backup
+Documentation=file:///usr/local/share/doc/jarl/docs/operations.md
+
+[Timer]
+OnCalendar=daily
+# Spread the start over an hour, and catch up on a run missed while the
+# machine was off.
+RandomizedDelaySec=1h
+Persistent=true
+
+[Install]
+WantedBy=timers.target

deploy/tmpfiles.d/jarl.conf

@@ -14,3 +14,5 @@ d      /var/cache/bind/jarl  2770  jarl bind
 # (Debian's AppArmor profile allows /var/log/named) and jarld reads it as a
 # member of group bind.
 d      /var/log/named        0750  bind bind
+# Database backups (jarl-backup.timer; backup.dir in jarl.yaml).
+d      /var/backups/jarl     0700  jarl jarl

docs/api.md

@@ -39,6 +39,10 @@ sudo curl --unix-socket /run/jarl/jarl.sock http://localhost/api/v1/status
 **HTTP Basic**, only on the lego `httpreq` routes (below): the username is
 the token's user and the password the API token.
 
+A response to a token that expires within 14 days carries
+`Jarl-Token-Expires: <RFC 3339 time>`; the `jarl` CLI warns when it sees
+it. Every response has `Server: Jarl/<version>`.
+
 ## Errors
 
 ```http
@@ -138,6 +142,8 @@ may be the name.
 | PATCH | `/zones/{zone}` | ZoneInput (only the fields to change) |
 | DELETE | `/zones/{zone}` | deletes the zone with its records and options |
 | GET | `/zones/{zone}/file` | the generated zone file (text/plain) |
+| GET | `/zones/{zone}/serials` | asks the zone's other name servers for their serial (below) |
+| GET | `/zones/{zone}/delegation` | checks the parent zone's delegation of a primary zone (below); `?resolver=IP` |
 | POST | `/zones/{zone}/import` | `{"text": "<zone file>", "mode": "merge"\|"replace", "use_soa": false, "dry_run": false}` |
 
 ZoneInput:
@@ -162,6 +168,36 @@ ZoneInput:
 }
 ```
 
+`/serials` and `/delegation` send DNS queries from the server, and only
+when called ([operations.md](operations.md#checking-name-servers-and-the-delegation)).
+`/serials` works for primary, secondary and mirror zones:
+
+```json
+{
+  "zone": "adbraun.lol.", "view": "", "type": "primary",
+  "serial": 2026100814, "serial_source": "named",    // or "jarl" when named can't be asked
+  "servers": [
+    {"name": "ns1.adbraun.lol.", "address": "192.168.1.53", "source": "NS record", "status": "self", "serial": 2026100814},
+    {"name": "ns2.adbraun.lol.", "address": "192.168.1.54", "source": "NS record", "status": "behind",
+     "serial": 2026100812, "diff": -2, "rtt_ms": 1}
+  ],
+  "notes": [], "checked_at": "..."
+}
+```
+
+`status` is `self`, `in_sync`, `answered` (no serial here to compare),
+`behind`, `ahead`, `not_authoritative`, `refused`, `unreachable`,
+`no_route` (no route from this server, e.g. IPv6), `no_address` or
+`error` (with `error`). `source` is `NS record`, `also-notify` or
+`primaries`; `port` is set for a non-default port.
+
+`/delegation` returns `parent`, `resolver`, `delegated`, `parent_ns`,
+`jarl_ns`, `parent_servers` (each parent server's `status`: `delegated`,
+`not_delegated`, `unreachable`, `error`, with `ns` and `glue`), `glue`
+(`name`, `parent`, `jarl`, `match`), `servers` (as above, for the delegated
+name servers) and `findings`: `{"level": "ok"|"info"|"warning"|"error",
+"message": "..."}`. A zone that isn't delegated gets one `info` finding.
+
 ### Records
 
 | Method | Path | Body / query |
@@ -274,7 +310,7 @@ in it and its records.
 
 | Method | Path | Notes |
 |--------|------|-------|
-| GET | `/status` | named status, adoption, drift, open changesets, current problems, object counts |
+| GET | `/status` | named status, adoption, drift, open changesets, current problems, object counts; `expiring_tokens` (tokens that expire within 14 days or expired in the last 14: all for administrators, otherwise the caller's); `backup` (administrators: `last_at`, `last_file`, `last_size`, `last_error`, `last_error_at`, `dir`, `warn_after` in seconds, `stale`) |
 | POST | `/system/check` | validation + named-checkconf of the current configuration (`options:read`) |
 | POST | `/system/resync` | `{"dry_run": false, "force": false}`: rewrite every file from the database and reload (admin) |
 | GET | `/audit` | `?user=&action=&outcome=&object_type=&changeset=&q=&since=&until=&limit=200&before=ID` |

docs/cli.md

@@ -135,6 +135,7 @@ to stderr with `-o json` or `-o yaml`, so stdout holds just the documents.
 |------|---------|
 | 0 | Success |
 | 1 | Other error (bad arguments, connection failure, ...) |
+| 2 | A check found problems (`zone serials`, `zone delegation`); its report was printed |
 | 3 | Not authenticated, or not allowed (token lacks the permission) |
 | 4 | Not found |
 | 5 | Conflict: concurrent change, object in use, drift |
@@ -167,6 +168,8 @@ jarl zone update adbraun.lol --move-to-view internal
 jarl zone delete lab.adbraun.lol [-y]
 jarl zone file adbraun.lol                             # the generated zone file
 jarl zone import adbraun.lol db.adbraun.lol [--replace] [--use-soa]
+jarl zone serials adbraun.lol                          # are the secondaries (or primaries) on the current serial?
+jarl zone delegation adbraun.lol [--resolver 9.9.9.9]  # does the parent zone delegate it correctly?
 ```
 
 Reverse names follow the dots (the nibbles in IPv6), so a reverse zone
@@ -185,6 +188,14 @@ are deleted. Try it with `--dry-run` first. `--option "NAME VALUE"` on
 When a zone exists in several views, add `--view NAME` to commands that
 name it.
 
+`zone serials` asks the zone's other name servers which serial they serve
+(the NS records and also-notify targets of a primary zone, the primaries
+of a secondary) and exits with 2 if one isn't on the current serial.
+`zone delegation` is for zones on the internet: it checks the parent
+zone's delegation and glue against the zone, and calls a zone that isn't
+delegated (an internal-only zone) a note, not a problem. Both only run when
+you run them; see [operations.md](operations.md#checking-name-servers-and-the-delegation).
+
 ## Records
 
 ```sh
@@ -312,6 +323,7 @@ jarl user update bob --disabled                        # logs bob out, blocks hi
 jarl user delete bob -y
 
 jarl token list [--user alice | --all] [--revoked]
+jarl token list --all --expiring [60d]                 # expiring within 14 days (or 60d), or expired
 jarl token create laptop --expires 365d
 jarl token create backup-script --grant "*:read"
 jarl token create acme --user certbot --role acme
@@ -321,13 +333,18 @@ jarl token revoke 7
 
 jarl audit [--user alice] [--action auth.login_failed] [--outcome failure] [--since 2026-10-01] [--limit 100]
 jarl search 192.168.1.20
-jarl status          # named, adoption, drift, open changesets, problems
+jarl status          # named, adoption, drift, open changesets, backups, expiring tokens, problems
 jarl check           # validation + named-checkconf of the current configuration
 jarl resync [--force] [--dry-run]   # rewrite every file from the database
 ```
 
 Grant syntax, roles and token rotation are in [permissions.md](permissions.md).
 
+When the token `jarl` uses expires within 14 days, every command prints a
+warning on stderr (once), naming the expiry and how to rotate it. Scripts
+such as certbot hooks show it in their logs well before the token stops
+working.
+
 ## Query statistics
 
 ```sh

docs/configuration.md

@@ -36,6 +36,25 @@ Durations accept Go syntax (`90s`, `15m`, `12h`) and days (`30d`).
 | `staging_dir` | `/var/lib/jarl/staging` | Scratch space where candidate configurations are rendered and checked with named-checkconf before they are installed. |
 | `audit.retention_days` | `0` | Delete audit events older than this many days (0 keeps them forever). Changeset history is kept regardless. |
 
+### Backups
+
+`jarld backup` without a file name, which the `jarl-backup.timer` systemd
+timer runs daily, writes `jarl-<UTC time>.db` into `backup.dir` and then
+deletes old backups there. It reads these settings from this file each
+time it runs; when it runs is the timer's (`systemctl edit
+jarl-backup.timer`). See [operations.md](operations.md#backups), also for
+restoring.
+
+| Key | Default | Meaning |
+|-----|---------|---------|
+| `backup.dir` | `/var/backups/jarl` | Where backups go. The backup service's sandbox may only write here; to use another directory, allow it with `ReadWritePaths=` in `systemctl edit jarl-backup.service`. |
+| `backup.retention` | `30d` | Delete backups older than this, by the time in their name. At least `1d`; `0` keeps every backup. |
+| `backup.keep_at_least` | `7` | Always keep this many of the newest backups, however old, so a long gap in backups never deletes them all. At least 1. |
+| `backup.stats` | `false` | Also back up the query statistics database (`stats-<UTC time>.db`). |
+| `backup.warn_after` | `2d` | The dashboard and `jarl status` warn when the newest backup is older than this. Keep it above the timer's interval; `0` turns the warning off. |
+
+Only files named `jarl-<time>.db` and `stats-<time>.db` are ever deleted.
+
 ### Logging
 
 | Key | Default | Meaning |

docs/development.md

@@ -3,7 +3,7 @@
 ## Layout
 
 ```
-cmd/jarld/            daemon: serve, init, import, adopt, check, render, resync, backup, user
+cmd/jarld/            daemon: serve, init, import, adopt, check, render, resync, backup, restore, user
 cmd/jarl/             CLI
 internal/
   model/              configuration objects (State: views, ACLs, keys, zones, records, options) and changesets
@@ -24,8 +24,10 @@ internal/
   web/static/         the web UI: plain ES modules, no build step, embedded with go:embed
   api/, client/       JSON types and the Go client shared by server and CLI
   acme/               ACME DNS-01 challenge records, shared by jarl acme and the httpreq endpoint
+  backup/             database backups: write (VACUUM INTO) and check, list, prune by the time in the name
+  nscheck/            DNS queries to other name servers: zone serials and the parent's delegation
   config/             jarld's YAML configuration
-deploy/               systemd unit, sysusers.d and tmpfiles.d files
+deploy/               systemd units (jarld, the daily backup service and timer), sysusers.d and tmpfiles.d files
 deploy/debian/        the Debian package's control template and postinst/prerm/postrm
 contrib/acme.sh/      acme.sh DNS API hook
 contrib/lego/         program for lego's exec DNS provider

docs/install.md

@@ -43,22 +43,28 @@ for any architecture Go supports. apt also installs `bind9` (recommended)
 if it's missing. Installing the package:
 
 * puts the binaries in `/usr/bin`, bash completion in
-  `/usr/share/bash-completion/completions`, the systemd unit in
+  `/usr/share/bash-completion/completions`, the systemd units in
-  `/usr/lib/systemd/system/jarl.service`, the sysusers and tmpfiles files
+  `/usr/lib/systemd/system` (`jarl.service`, and `jarl-backup.service`
+  with its daily `jarl-backup.timer`), the sysusers and tmpfiles files
   in `/usr/lib`, and the docs and contrib files in `/usr/share/doc/jarl`;
 * creates the `jarl` user (in group `bind`) and Jarl's directories, and
   writes `/etc/jarl/jarl.yaml` from the example on the first install. No
   file in `/etc` belongs to the package, so your configuration is never
   replaced or asked about on upgrades;
+* turns on the daily database backup, `jarl-backup.timer`
+  ([operations.md](operations.md#backups)). It does nothing until the
+  database exists. If you turn it off (`systemctl disable
+  jarl-backup.timer`), upgrades leave it off;
 * doesn't start jarld: continue with [3. Configure](#3-configure), then
   `systemctl enable --now jarl` in step 8.
 
 **Upgrading**: build and install the newer package the same way. A running
 jarld is restarted, and upgrades its database as it starts.
 
-**Removing** (`sudo apt remove jarl`) stops jarld and removes the program,
+**Removing** (`sudo apt remove jarl`) stops jarld and the backup timer and
-but named keeps running with the configuration Jarl last wrote. Even
+removes the program, but named keeps running with the configuration Jarl
-`apt purge jarl` keeps `/etc/jarl`, `/var/lib/jarl` (the database) and the
+last wrote. Even `apt purge jarl` keeps `/etc/jarl`, `/var/lib/jarl` (the
+database), `/var/backups/jarl` and the
 files in `/etc/bind/jarl`, `/var/lib/bind/jarl` and `/var/cache/bind/jarl`,
 since named.conf may still include them. Remove them by hand once named no
 longer uses them.
@@ -78,8 +84,9 @@ This installs:
 | `/usr/local/bin/jarld`, `/usr/local/bin/jarl` | daemon and CLI |
 | `/usr/local/share/bash-completion/completions/jarl`, `.../jarld` | bash completion (loaded automatically by the `bash-completion` package) |
 | `/etc/systemd/system/jarl.service` | systemd unit (`Type=notify`, sandboxed) |
+| `/etc/systemd/system/jarl-backup.service`, `jarl-backup.timer` | the daily database backup ([operations.md](operations.md#backups)) |
 | `/usr/lib/sysusers.d/jarl.conf` | creates the `jarl` system user and adds it to group `bind` |
-| `/usr/lib/tmpfiles.d/jarl.conf` | creates `/etc/jarl`, `/var/lib/jarl`, `/etc/bind/jarl`, `/var/lib/bind/jarl`, `/var/cache/bind/jarl` |
+| `/usr/lib/tmpfiles.d/jarl.conf` | creates `/etc/jarl`, `/var/lib/jarl`, `/etc/bind/jarl`, `/var/lib/bind/jarl`, `/var/cache/bind/jarl`, `/var/backups/jarl` |
 | `/etc/jarl/jarl.yaml` | example configuration (only if none exists) |
 | `/usr/local/share/doc/jarl/` | these docs and the acme.sh hook |
 
@@ -90,19 +97,21 @@ systemd-sysusers jarl.conf
 systemd-tmpfiles --create jarl.conf
 chgrp jarl /etc/jarl/jarl.yaml
 systemctl daemon-reload
+systemctl enable --now jarl-backup.timer   # daily backups; idle until the database exists
 ```
 
 Without `make` (from a release tarball), the same steps are:
 
 ```sh
 install -m 0755 jarld jarl /usr/local/bin/
-install -m 0644 deploy/systemd/jarl.service /etc/systemd/system/
+install -m 0644 deploy/systemd/jarl.service deploy/systemd/jarl-backup.service deploy/systemd/jarl-backup.timer /etc/systemd/system/
 install -m 0644 deploy/sysusers.d/jarl.conf /usr/lib/sysusers.d/
 install -m 0644 deploy/tmpfiles.d/jarl.conf /usr/lib/tmpfiles.d/
 systemd-sysusers jarl.conf && systemd-tmpfiles --create jarl.conf
 jarld config --example > /etc/jarl/jarl.yaml
 chgrp jarl /etc/jarl/jarl.yaml && chmod 0640 /etc/jarl/jarl.yaml
 systemctl daemon-reload
+systemctl enable --now jarl-backup.timer
 ```
 
 ### Why the jarl user is in group bind
@@ -292,11 +301,12 @@ make && sudo make install
 sudo systemctl restart jarl
 ```
 
-Database migrations run automatically at startup. Before an upgrade, back
+Database migrations run automatically at startup. Before an upgrade, take
-up the database (it is safe while jarld runs):
+a backup (it is safe while jarld runs); `jarld restore` puts it back if
+needed:
 
 ```sh
-sudo -u jarl jarld backup /var/lib/jarl/jarl-$(date +%F).db
+sudo -u jarl jarld backup
 ```
 
 ## Uninstalling
@@ -304,10 +314,10 @@ sudo -u jarl jarld backup /var/lib/jarl/jarl-$(date +%F).db
 ```sh
 cp /etc/bind/named.conf.pre-jarl-<date> /etc/bind/named.conf   # or write a named.conf that includes Jarl's files
 rndc reconfig
-systemctl disable --now jarl
+systemctl disable --now jarl jarl-backup.timer
 make uninstall
 ```
 
 `/etc/bind/jarl` and `/var/lib/bind/jarl` contain plain named.conf and zone
-files that keep working without Jarl. `make uninstall` leaves them and
+files that keep working without Jarl. `make uninstall` leaves them,
-Jarl's database in place.
+Jarl's database and its backups in place.

docs/operations.md

@@ -4,14 +4,16 @@
 
 | Task | How |
 |------|-----|
-| Is everything healthy? | `jarl status`, or the web UI dashboard: named reachable, adoption, drift, open changesets, configuration problems |
+| Is everything healthy? | `jarl status`, or the web UI dashboard: named reachable, adoption, drift, open changesets, configuration problems, the newest backup, tokens about to expire |
+| Are the secondaries up to date? | `jarl zone serials ZONE`, or the zone's Name servers tab ([below](#checking-name-servers-and-the-delegation)) |
+| Back up / restore | Daily by `jarl-backup.timer`; by hand `sudo -u jarl jarld backup`; restore with `jarld restore` ([below](#backups)) |
 | What changed and who did it? | `jarl history`, `jarl changeset show ID` (with diff), `jarl audit` |
 | Undo a change | `jarl changeset revert ID`, review it, then `jarl changeset apply NEW_ID` (or History → Revert in the UI) |
 | Check the configuration | `jarl check`: full validation plus `named-checkconf -z` |
 | Logs | `journalctl -u jarl` (jarld, including `msg=audit` lines), `journalctl -u named` |
 | Restart / reload TLS | `systemctl restart jarl` / `systemctl reload jarl` |
 | Work on the server itself | `sudo jarl ...`: root uses jarld's local socket and needs no token ([cli.md](cli.md#on-the-server-itself)) |
-| Renew or replace a token | `jarl token rotate NAME --grace 1h` (same grants, new secret; [permissions.md](permissions.md#rotating-a-token)) |
+| Renew or replace a token | `jarl token rotate NAME --grace 1h` (same grants, new secret; [permissions.md](permissions.md#rotating-a-token)). `jarl token list --all --expiring` shows those that expire within 14 days |
 | Upgrade Jarl | Debian package: build the new one and `sudo apt install ./dist/jarl_*.deb`; otherwise `make && sudo make install && sudo systemctl restart jarl`. jarld upgrades its database when it starts |
 
 Restarting jarld doesn't affect DNS: named keeps serving the files it
@@ -19,46 +21,151 @@ loaded. jarld is only needed to make changes.
 
 ## Backups
 
-Everything Jarl knows is in its database. The BIND files can be generated
+Everything Jarl knows is in its database (`/var/lib/jarl/jarl.db`): zones,
-from it again at any time.
+records, views, ACLs, TSIG keys, options, users, tokens, history and the
+audit log. named's files are generated from it, so the database is the one
+thing to back up.
+
+### The daily backup
+
+The `jarl-backup.timer` systemd timer runs `jarld backup` once a day. The
+Debian package turns it on; after `make install`, turn it on yourself:
 
 ```sh
-sudo -u jarl jarld backup /var/backups/jarl/jarl-$(date +%F).db   # consistent copy, safe while jarld runs
+systemctl enable --now jarl-backup.timer
+systemctl list-timers jarl-backup.timer   # when it runs next
 ```
 
-For example, as a daily timer or cron job:
+Each run:
+
+1. writes a consistent copy of the database with SQLite's `VACUUM INTO`
+   (safe while jarld runs and changes things) to a temporary file in the
+   backup directory;
+2. checks the copy with SQLite's `integrity_check`, and only then gives it
+   its name, `jarl-<UTC time>.db`, e.g. `jarl-20261008T031512Z.db`. A file
+   with that name is always a complete, intact backup;
+3. deletes backups older than `retention`, but always keeps the newest
+   `keep_at_least`, however old they are. If backups stop for a while (the
+   machine was off, the disk was full), the old ones aren't all deleted by
+   the next run. Ages come from the time in the name, not the file's
+   modification time, which copying the directory would change. Only files
+   named `jarl-<time>.db` (and `stats-<time>.db`) are ever deleted, so
+   anything else you put in the directory is safe;
+4. records the result in the database, so the dashboard and `jarl status`
+   show when the newest backup was made, and whether the last attempt
+   failed.
+
+A failed run exits non-zero, so it shows in `systemctl --failed` and
+`journalctl -u jarl-backup`.
+
+What to keep and where is in `jarl.yaml`
+([configuration.md](configuration.md#backups)); when it runs is the
+timer's:
+
+```yaml
+backup:
+  dir: /var/backups/jarl
+  retention: 30d       # delete backups older than this (0 keeps them all)...
+  keep_at_least: 7     # ...but always keep the newest 7
+  stats: false         # also back up the statistics database
+  warn_after: 2d       # dashboard warning when the newest backup is older
+```
+
+The settings are read by `jarld backup` itself each time it runs, from the
+same file jarld uses, and checked the same way: a mistake there stops
+`jarld` from starting rather than quietly breaking the backups.
+
+To run at a different time, override the timer, and keep `warn_after`
+above the new interval:
 
 ```sh
-# /etc/cron.daily/jarl-backup
+systemctl edit jarl-backup.timer
-#!/bin/sh
+#   [Timer]
-set -e
+#   OnCalendar=
-install -d -o jarl -g jarl -m 0700 /var/backups/jarl
+#   OnCalendar=*-*-* 02:30
-sudo -u jarl jarld backup /var/backups/jarl/jarl-$(date +%F).db
+```
-find /var/backups/jarl -name 'jarl-*.db' -mtime +30 -delete
+
+The backup service runs as the `jarl` user in a sandbox that may only
+write to `/var/lib/jarl` and `/var/backups/jarl` (created by
+`tmpfiles.d/jarl.conf`, mode 0700). To keep backups elsewhere, set
+`backup.dir`, create the directory (owned by `jarl`, mode 0700) and allow
+it:
+
+```sh
+systemctl edit jarl-backup.service
+#   [Service]
+#   ReadWritePaths=/srv/backups/jarl
+```
+
+Backups on the same disk don't survive the disk. Copy
+`/var/backups/jarl` elsewhere (rsync, restic, borg, your usual backup) as
+well.
+
+### By hand
+
+```sh
+sudo -u jarl jarld backup                       # into backup.dir, as the timer does
+sudo -u jarl jarld backup /root/before-upgrade.db  # one copy, never deleted automatically
+sudo jarld restore                              # list the backups in backup.dir
 ```
 
 Query statistics live in a separate database (`/var/lib/jarl/stats.db`).
-It isn't needed to restore DNS, so back it up only if you want to keep the
+DNS doesn't need it, so it is backed up only with `backup.stats: true`, as
-history.
+`stats-<UTC time>.db`.
 
-The database contains password and token *hashes* and the TSIG secrets in
+The database holds password and token *hashes*, and TSIG secrets in the
-the clear, so keep backups private.
+clear. Backups are written with mode 0600 into a 0700 directory; keep any
+copies just as private.
 
 ### Restoring
 
+`jarld restore` replaces the database with a backup. jarld must be
+stopped: the restore refuses to run while jarld holds the database (and
+jarld won't start while a restore runs).
+
 ```sh
-systemctl stop jarl
+sudo jarld restore                  # list the backups, newest first
-install -o jarl -g jarl -m 0600 /var/backups/jarl/jarl-2026-10-01.db /var/lib/jarl/jarl.db
+sudo systemctl stop jarl
-rm -f /var/lib/jarl/jarl.db-wal /var/lib/jarl/jarl.db-shm
+sudo jarld restore latest           # or: sudo jarld restore /var/backups/jarl/jarl-20261008T031512Z.db
-systemctl start jarl
-jarl resync --dry-run     # what differs between the restored database and the files
-jarl resync --force       # write everything and reload named
 ```
 
-An older backup can carry lower zone serials than named last served. Your
+It checks the backup's integrity and that this Jarl can read it (a backup
-own named doesn't mind, but secondaries and caches ignore a serial that went
+from a newer Jarl is refused), shows what the backup and the current
-backwards. If you have secondaries, raise the serials after a restore:
+database hold (zones, records, users, last change), and asks before
-`jarl zone update ZONE --serial <higher>`.
+changing anything (`--yes` skips the question). Then it:
+
+* moves the current database aside as
+  `jarl.db.before-restore-<UTC time>` (with its `-wal` and `-shm` files),
+  so a restore can be undone the same way;
+* copies the backup in place, upgrades it to the current schema if it is
+  older, and records the restore in its audit log.
+
+Then start jarld and bring named in line with the restored database:
+
+```sh
+sudo systemctl start jarl
+jarl status                 # drift lists the files that differ
+jarl resync --dry-run       # what would change
+jarl resync --force         # write everything and reload named
+```
+
+Things to know:
+
+* **Zone serials.** An older backup can carry lower zone serials than named
+  last served. Your own named doesn't mind, but secondaries and caches
+  ignore a serial that went backwards, so they keep the newer data. If you
+  have secondaries, raise the serials after a restore:
+  `jarl zone update ZONE --serial <higher than before>`, then check with
+  `jarl zone serials ZONE`.
+* **Tokens and passwords** go back to what they were when the backup was
+  made: tokens created since are gone, and revoked ones work again. Rotate
+  or revoke as needed.
+* **Statistics**: to restore `stats-<time>.db`, stop jarld and copy it over
+  `/var/lib/jarl/stats.db` (owner `jarl`, mode 0640).
+* **On a new machine**: install Jarl, restore the backup, start jarld and
+  run `jarl resync --force`; then adopt named as in
+  [install.md](install.md) if this named doesn't include Jarl's
+  configuration yet.
 
 ## Drift: files changed outside Jarl
 
@@ -143,6 +250,85 @@ Things that both servers need, such as ACLs, can be changed on both at once
 with a group (`jarl -s dns acl update ...`): the change is checked on both
 first and applied only if both accept it.
 
+## Checking name servers and the delegation
+
+Two checks ask other name servers about a zone over DNS. They only run when
+you ask: on the zone's **Name servers** tab in the web UI, or with the CLI.
+Nothing checks in the background.
+
+### Are the other servers serving the current serial?
+
+```
+$ jarl zone serials adbraun.lol
+adbraun.lol, primary, serial 2026100814 (served by named here)
+
+SERVER           ADDRESS       FROM         STATUS       SERIAL      TIME
+ns1.adbraun.lol  192.168.1.53  NS record    this server  2026100814
+ns2.adbraun.lol  192.168.1.54  NS record    BEHIND by 2  2026100812  1 ms
+```
+
+It works for both sides of a zone transfer:
+
+* **Primary zones**: asks the servers in the zone's NS records and its
+  `also-notify` targets (the secondaries) which serial they serve, and
+  compares it with what named on this server serves. A secondary that is
+  behind hasn't picked up the latest change: check its logs, and that
+  `allow-transfer` on this server lets it transfer the zone.
+* **Secondary and mirror zones**: asks the zone's `primaries` (following
+  named `remote-servers` lists, with their ports and TSIG keys), to see
+  whether this server's copy is up to date.
+
+Name server addresses come from Jarl's own A/AAAA records in the zone's
+view where it has them, otherwise from a DNS lookup on the server. This
+server itself is recognised by its addresses and isn't asked. Statuses:
+*in sync*, *behind/ahead by N* (serials compare as in RFC 1982, so a
+wrapped serial counts as ahead), *doesn't serve the zone* (the server
+answers but isn't authoritative for it), *refused*, *no answer*, and *no
+route from here* (typically an IPv6 address on a server without IPv6,
+which isn't counted as a problem).
+
+With views, other servers answer from the view that matches this server's
+address (or TSIG key), exactly as they do for its zone transfers and
+NOTIFYs; queries to primaries and also-notify targets that name a key are
+signed with it.
+
+`jarl zone serials` exits with 2 when a server isn't serving the current
+serial, so it can run from cron or a monitoring system.
+
+### Does the internet see the zone the way Jarl does?
+
+For a zone published on the internet, the parent zone (`lol.` for
+adbraun.lol, set through your registrar) must delegate it to the right name
+servers, with glue addresses for name servers inside the zone.
+
+```
+$ jarl zone delegation adbraun.lol --view external
+Zone:                              adbraun.lol
+Parent zone:                       lol. (found through 127.0.0.53:53)
+  a.nic.lol and 3 more:            delegate to ns1.adbraun.lol, ns2.adbraun.lol
+NS at the parent:                  ns1.adbraun.lol, ns2.adbraun.lol
+NS in the zone:                    ns1.adbraun.lol, ns2.adbraun.lol
+Glue ns1.adbraun.lol:              203.0.113.53 (matches)
+...
+ok:      The parent delegates adbraun.lol to the same name servers as its NS records: ...
+ok:      Every delegated name server answers for the zone with the current serial.
+```
+
+It finds the parent zone through the server's resolver (or `--resolver
+9.9.9.9`), asks each of the parent's servers directly, and reports:
+
+* name servers the parent delegates to that the zone's NS records don't
+  list, and the other way round;
+* glue that is missing, or differs from the zone's A/AAAA records;
+* delegated servers that don't serve the zone (a *lame delegation*), don't
+  answer, or serve another serial.
+
+A zone that is only used on your own network isn't delegated at all, and
+the check says so in a single note, not as a problem; it also exits with 0.
+It exits with 2 only for errors (a lame delegation, missing glue). With
+split DNS, check the zone in the view the internet sees (usually
+`external`).
+
 ## Taking Jarl out of the picture
 
 The generated files are plain BIND configuration. To stop using Jarl,
@@ -166,3 +352,14 @@ rndc reconfig
   ```
 * The audit log records failed logins as `auth.login_failed` /
   `auth.login_blocked`, and rejected tokens as `auth.token_failed`.
+* Backups: `.backup.stale` and `.backup.last_error` in `jarl status -o
+  json` (administrators), and `systemctl --failed` for the timer's runs:
+  ```sh
+  sudo jarl status -o json | jq -e '.backup.stale == false and (.backup.last_error // "") == ""'
+  ```
+* Secondaries: `jarl zone serials ZONE` exits with 2 when one isn't serving
+  the current serial.
+* Tokens: jarld logs a warning once a day for every token that expires
+  within 14 days (`journalctl -u jarl | grep 'token expires'`), the
+  dashboard lists them, and `jarl` prints a warning on stderr whenever it
+  uses one, so a certbot hook's log shows it well before renewals fail.

docs/permissions.md

@@ -114,6 +114,25 @@ jarl token revoke 12
 In the web UI: Account → API tokens (your own) and Users → *user* →
 Tokens (admin).
 
+### Expiring tokens
+
+A token with an expiry stops working at that moment. So that it doesn't
+catch you out (a certbot token that expired breaks renewals silently, until
+the certificate itself expires), Jarl warns from **14 days before**:
+
+* the dashboard lists the tokens that expire soon or expired in the last 14
+  days (administrators see everyone's, others their own), and token lists
+  mark them *soon* or *expired*;
+* jarld logs a warning once a day for each of them;
+* every response to such a token carries a `Jarl-Token-Expires` header,
+  and the `jarl` CLI prints a warning on stderr when it sees it, so the
+  logs of the script using the token show it;
+* `jarl token list --all --expiring` lists them (`--expiring 60d` looks
+  further ahead).
+
+Rotating the token (below) gives it a new lifetime, and keeps its
+permissions.
+
 ### Rotating a token
 
 Rotating gives a token a **new secret** and changes nothing else: it keeps

internal/api/types.go

@@ -7,6 +7,7 @@ import (
 	"time"
 
 	"jarl/internal/model"
+	"jarl/internal/nscheck"
 )
 
 // Version is the API path prefix.
@@ -151,6 +152,34 @@ type StatusResponse struct {
 	Version string          `json:"version"`
 	Engine  json.RawMessage `json:"engine"`
 	Time    time.Time       `json:"time"`
+	// ExpiringTokens are tokens that expire within TokenExpiryWarning or
+	// expired less than that long ago (not revoked): every user's for
+	// administrators, otherwise the caller's own.
+	ExpiringTokens []Token `json:"expiring_tokens"`
+	// Backup is the state of database backups (administrators only).
+	Backup *BackupStatus `json:"backup,omitempty"`
+}
+
+// TokenExpiresHeader is sent with responses to a token that expires
+// soon (model.TokenExpiryWarning), holding its expiry time (RFC 3339). The
+// CLI warns its user when it sees it.
+const TokenExpiresHeader = "Jarl-Token-Expires"
+
+// BackupStatus describes the newest database backup, as recorded by
+// 'jarld backup'.
+type BackupStatus struct {
+	LastAt    *time.Time `json:"last_at,omitempty"`
+	LastFile  string     `json:"last_file,omitempty"`
+	LastSize  int64      `json:"last_size,omitempty"`
+	LastError string     `json:"last_error,omitempty"`
+	// LastErrorAt is when the newest failed backup ran (only reported
+	// when it is newer than the last success).
+	LastErrorAt *time.Time `json:"last_error_at,omitempty"`
+	Dir         string     `json:"dir"`
+	// WarnAfter (seconds) is how old the newest backup may get before
+	// Stale is set (0: never).
+	WarnAfter int64 `json:"warn_after"`
+	Stale     bool  `json:"stale"`
 }
 
 // SearchResult is one search hit.
@@ -215,3 +244,26 @@ type ZoneImport struct {
 	// UseSOA also takes the SOA timers and default TTL from the file.
 	UseSOA bool `json:"use_soa,omitempty"`
 }
+
+// SerialsResponse is GET /zones/{zone}/serials.
+type SerialsResponse struct {
+	Zone string `json:"zone"`
+	View string `json:"view,omitempty"`
+	Type string `json:"type"`
+	// Serial is what this server serves: named's loaded serial, or for a
+	// primary zone Jarl's when named can't be asked (SerialSource says
+	// which).
+	Serial       *uint32          `json:"serial"`
+	SerialSource string           `json:"serial_source,omitempty"`
+	Servers      []nscheck.Result `json:"servers"`
+	Notes        []string         `json:"notes"`
+	CheckedAt    time.Time        `json:"checked_at"`
+}
+
+// DelegationResponse is GET /zones/{zone}/delegation.
+type DelegationResponse struct {
+	*nscheck.DelegationReport
+	View      string    `json:"view,omitempty"`
+	Notes     []string  `json:"notes"`
+	CheckedAt time.Time `json:"checked_at"`
+}

internal/backup/backup.go

@@ -0,0 +1,256 @@
+// Package backup writes, checks, lists and prunes copies of Jarl's SQLite
+// databases ('jarld backup' and 'jarld restore').
+//
+// A backup made into the backup directory is named after its kind and the
+// UTC time it was made, e.g. jarl-20261008T031512Z.db (the configuration
+// database) or stats-20261008T031512Z.db (query statistics). Pruning goes
+// by the time in the name, not the file's modification time (which a copy
+// or a restore of the directory changes), and only ever touches files
+// named like this: anything else in the directory is left alone.
+package backup
+
+import (
+	"context"
+	"database/sql"
+	"fmt"
+	"net/url"
+	"os"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+	"syscall"
+	"time"
+
+	_ "modernc.org/sqlite" // registers the "sqlite" driver
+)
+
+// Kinds of database.
+const (
+	KindJarl  = "jarl"
+	KindStats = "stats"
+)
+
+const stamp = "20060102T150405Z"
+
+var nameRe = regexp.MustCompile(`^(jarl|stats)-([0-9]{8}T[0-9]{6}Z)\.db$`)
+
+// File is a backup in the backup directory.
+type File struct {
+	Path string    `json:"path"`
+	Kind string    `json:"kind"`
+	Time time.Time `json:"time"`
+	Size int64     `json:"size"`
+}
+
+// Name is the file name of a backup of kind made at t.
+func Name(kind string, t time.Time) string {
+	return kind + "-" + t.UTC().Format(stamp) + ".db"
+}
+
+// parse returns the kind and time in a backup's file name.
+func parse(name string) (kind string, t time.Time, ok bool) {
+	m := nameRe.FindStringSubmatch(name)
+	if m == nil {
+		return "", time.Time{}, false
+	}
+	t, err := time.Parse(stamp, m[2])
+	return m[1], t, err == nil
+}
+
+// Write makes a consistent copy of the live SQLite database src as
+// dir/Name(kind, now), and checks the copy before giving it its final
+// name: a file with a backup's name is always a complete, intact backup.
+func Write(ctx context.Context, src, dir, kind string, now time.Time) (*File, error) {
+	if _, err := os.Stat(src); err != nil {
+		return nil, err
+	}
+	if err := os.MkdirAll(dir, 0o700); err != nil {
+		return nil, err
+	}
+	final := filepath.Join(dir, Name(kind, now))
+	if _, err := os.Lstat(final); err == nil {
+		return nil, fmt.Errorf("%s already exists", final)
+	}
+	tmp := filepath.Join(dir, "."+Name(kind, now)+".tmp")
+	os.Remove(tmp)
+	if err := Copy(ctx, src, tmp); err != nil {
+		os.Remove(tmp)
+		return nil, err
+	}
+	if err := Verify(ctx, tmp); err != nil {
+		os.Remove(tmp)
+		return nil, fmt.Errorf("the new backup failed its check: %w", err)
+	}
+	if err := syncFile(tmp); err != nil {
+		os.Remove(tmp)
+		return nil, err
+	}
+	if err := os.Rename(tmp, final); err != nil {
+		os.Remove(tmp)
+		return nil, err
+	}
+	syncFile(dir)
+	fi, err := os.Stat(final)
+	if err != nil {
+		return nil, err
+	}
+	return &File{Path: final, Kind: kind, Time: now.UTC().Truncate(time.Second), Size: fi.Size()}, nil
+}
+
+// Copy writes a consistent copy of the live SQLite database src to dst
+// (which must not exist) with SQLite's VACUUM INTO, which is safe while
+// jarld uses the database. dst is created with mode 0600: the database
+// holds TSIG secrets and password hashes.
+func Copy(ctx context.Context, src, dst string) error {
+	// mode=rw: never create src if it's missing.
+	db, err := sql.Open("sqlite", "file:"+escape(src)+"?mode=rw&_pragma=busy_timeout(10000)")
+	if err != nil {
+		return err
+	}
+	defer db.Close()
+	old := syscall.Umask(0o077)
+	_, err = db.ExecContext(ctx, `VACUUM INTO ?`, dst)
+	syscall.Umask(old)
+	if err != nil {
+		return err
+	}
+	return os.Chmod(dst, 0o600)
+}
+
+// Verify checks that path is an intact SQLite database.
+func Verify(ctx context.Context, path string) error {
+	db, err := openCopy(path)
+	if err != nil {
+		return err
+	}
+	defer db.Close()
+	var res string
+	if err := db.QueryRowContext(ctx, `PRAGMA integrity_check`).Scan(&res); err != nil {
+		return err
+	}
+	if res != "ok" {
+		return fmt.Errorf("integrity check: %s", res)
+	}
+	return nil
+}
+
+// openCopy opens a backup read-only. immutable=1 tells SQLite the file
+// can't change, so it takes no locks and writes nothing next to it.
+func openCopy(path string) (*sql.DB, error) {
+	if _, err := os.Stat(path); err != nil {
+		return nil, err
+	}
+	return sql.Open("sqlite", "file:"+escape(path)+"?mode=ro&immutable=1")
+}
+
+// Info describes the contents of a configuration database (KindJarl).
+type Info struct {
+	Schema     int        `json:"schema"`
+	Zones      int        `json:"zones"`
+	Records    int        `json:"records"`
+	Users      int        `json:"users"`
+	LastChange *time.Time `json:"last_change,omitempty"`
+}
+
+// Inspect reads a configuration database's schema version and size. It
+// works on backups and, read-only, on the live database.
+func Inspect(ctx context.Context, path string) (*Info, error) {
+	db, err := openCopy(path)
+	if err != nil {
+		return nil, err
+	}
+	defer db.Close()
+	in := &Info{}
+	if err := db.QueryRowContext(ctx, `SELECT version FROM schema_version`).Scan(&in.Schema); err != nil {
+		return nil, fmt.Errorf("%s doesn't look like a Jarl database: %w", path, err)
+	}
+	db.QueryRowContext(ctx, `SELECT count(*) FROM zones`).Scan(&in.Zones)
+	db.QueryRowContext(ctx, `SELECT count(*) FROM records`).Scan(&in.Records)
+	db.QueryRowContext(ctx, `SELECT count(*) FROM users`).Scan(&in.Users)
+	var last sql.NullString
+	db.QueryRowContext(ctx, `SELECT max(applied_at) FROM changesets`).Scan(&last)
+	if t, err := time.Parse(time.RFC3339Nano, last.String); err == nil {
+		in.LastChange = &t
+	}
+	return in, nil
+}
+
+// List returns the backups in dir, newest first. A missing directory has
+// none.
+func List(dir string) ([]File, error) {
+	ents, err := os.ReadDir(dir)
+	if os.IsNotExist(err) {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	var out []File
+	for _, e := range ents {
+		kind, t, ok := parse(e.Name())
+		if !ok || !e.Type().IsRegular() {
+			continue
+		}
+		f := File{Path: filepath.Join(dir, e.Name()), Kind: kind, Time: t}
+		if fi, err := e.Info(); err == nil {
+			f.Size = fi.Size()
+		}
+		out = append(out, f)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Time.After(out[j].Time) })
+	return out, nil
+}
+
+// Prune deletes backups of kind in dir that are older than retention,
+// always keeping the newest keepAtLeast of them however old they are, so
+// a backup that stopped running for a while never leaves you with none.
+// retention 0 keeps everything. It returns the deleted files.
+func Prune(dir, kind string, now time.Time, retention time.Duration, keepAtLeast int) ([]File, error) {
+	if retention <= 0 {
+		return nil, nil
+	}
+	all, err := List(dir)
+	if err != nil {
+		return nil, err
+	}
+	var removed []File
+	n := 0
+	for _, f := range all {
+		if f.Kind != kind {
+			continue
+		}
+		n++
+		if n <= keepAtLeast || now.Sub(f.Time) <= retention {
+			continue
+		}
+		if err := os.Remove(f.Path); err != nil {
+			return removed, err
+		}
+		removed = append(removed, f)
+	}
+	// Temporary files left by a backup that was killed halfway.
+	ents, _ := os.ReadDir(dir)
+	for _, e := range ents {
+		if strings.HasPrefix(e.Name(), ".") && strings.HasSuffix(e.Name(), ".db.tmp") {
+			if fi, err := e.Info(); err == nil && now.Sub(fi.ModTime()) > 24*time.Hour {
+				os.Remove(filepath.Join(dir, e.Name()))
+			}
+		}
+	}
+	return removed, nil
+}
+
+func syncFile(path string) error {
+	f, err := os.Open(path)
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	return f.Sync()
+}
+
+// escape makes a path safe inside a file: URI.
+func escape(path string) string {
+	return (&url.URL{Path: path}).EscapedPath()
+}

internal/backup/backup_test.go

@@ -0,0 +1,151 @@
+package backup
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"jarl/internal/store"
+)
+
+func newDB(t *testing.T) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "jarl.db")
+	st, err := store.Open(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { st.Close() })
+	return path
+}
+
+func TestWriteVerifyInspect(t *testing.T) {
+	ctx := context.Background()
+	src := newDB(t)
+	dir := filepath.Join(t.TempDir(), "backups")
+	now := time.Date(2026, 10, 8, 3, 15, 12, 0, time.UTC)
+	f, err := Write(ctx, src, dir, KindJarl, now)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if filepath.Base(f.Path) != "jarl-20261008T031512Z.db" || f.Size == 0 {
+		t.Errorf("backup = %+v", f)
+	}
+	if fi, _ := os.Stat(f.Path); fi.Mode().Perm() != 0o600 {
+		t.Errorf("mode %v", fi.Mode())
+	}
+	if err := Verify(ctx, f.Path); err != nil {
+		t.Error(err)
+	}
+	in, err := Inspect(ctx, f.Path)
+	if err != nil || in.Schema != store.SchemaVersion() {
+		t.Errorf("inspect: %+v %v", in, err)
+	}
+	// The same second again is refused rather than overwritten.
+	if _, err := Write(ctx, src, dir, KindJarl, now); err == nil {
+		t.Error("overwrote a backup")
+	}
+	// No temporary files are left.
+	ents, _ := os.ReadDir(dir)
+	if len(ents) != 1 {
+		t.Errorf("files: %v", ents)
+	}
+	// A missing database is an error, and isn't created.
+	missing := filepath.Join(t.TempDir(), "nope.db")
+	if _, err := Write(ctx, missing, dir, KindJarl, now.Add(time.Second)); err == nil {
+		t.Error("backed up a missing database")
+	}
+	if _, err := os.Stat(missing); err == nil {
+		t.Error("created the missing database")
+	}
+}
+
+func TestVerifyCorrupt(t *testing.T) {
+	ctx := context.Background()
+	path := filepath.Join(t.TempDir(), "bad.db")
+	os.WriteFile(path, []byte("SQLite format 3\x00 but not really a database, just text that goes on"), 0o600)
+	if err := Verify(ctx, path); err == nil {
+		t.Error("a corrupt file passed")
+	}
+	if _, err := Inspect(ctx, path); err == nil {
+		t.Error("inspected a corrupt file")
+	}
+}
+
+func TestPrune(t *testing.T) {
+	dir := t.TempDir()
+	now := time.Date(2026, 10, 8, 3, 0, 0, 0, time.UTC)
+	touch := func(name string) {
+		os.WriteFile(filepath.Join(dir, name), []byte("x"), 0o600)
+	}
+	// Daily backups from 0 to 39 days old, plus files that aren't ours.
+	for d := 0; d < 40; d++ {
+		touch(Name(KindJarl, now.Add(-time.Duration(d)*24*time.Hour)))
+	}
+	touch(Name(KindStats, now.Add(-100*24*time.Hour)))
+	touch("jarl-before-upgrade.db")
+	touch("jarl-20200101T000000Z.db.bak")
+	touch("notes.txt")
+
+	removed, err := Prune(dir, KindJarl, now, 30*24*time.Hour, 7)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(removed) != 9 { // 31..39 days old
+		t.Errorf("removed %d: %v", len(removed), removed)
+	}
+	files, _ := List(dir)
+	jarl := 0
+	for _, f := range files {
+		if f.Kind == KindJarl {
+			jarl++
+			if now.Sub(f.Time) > 30*24*time.Hour {
+				t.Errorf("kept %s", f.Path)
+			}
+		}
+	}
+	if jarl != 31 {
+		t.Errorf("%d jarl backups left", jarl)
+	}
+	for _, name := range []string{"jarl-before-upgrade.db", "jarl-20200101T000000Z.db.bak", "notes.txt", Name(KindStats, now.Add(-100*24*time.Hour))} {
+		if _, err := os.Stat(filepath.Join(dir, name)); err != nil {
+			t.Errorf("%s was deleted", name)
+		}
+	}
+
+	// After a long outage every backup is old: the newest keepAtLeast stay.
+	removed, _ = Prune(dir, KindJarl, now.Add(365*24*time.Hour), 30*24*time.Hour, 7)
+	files, _ = List(dir)
+	n := 0
+	for _, f := range files {
+		if f.Kind == KindJarl {
+			n++
+		}
+	}
+	if n != 7 || len(removed) != 24 {
+		t.Errorf("after a year: %d left, %d removed", n, len(removed))
+	}
+	// Retention 0 keeps everything.
+	if removed, _ := Prune(dir, KindJarl, now.Add(3650*24*time.Hour), 0, 1); len(removed) != 0 {
+		t.Errorf("retention 0 removed %v", removed)
+	}
+}
+
+func TestPruneTempFiles(t *testing.T) {
+	dir := t.TempDir()
+	now := time.Now()
+	old := filepath.Join(dir, "."+Name(KindJarl, now.Add(-48*time.Hour))+".tmp")
+	fresh := filepath.Join(dir, "."+Name(KindJarl, now)+".tmp")
+	os.WriteFile(old, nil, 0o600)
+	os.WriteFile(fresh, nil, 0o600)
+	os.Chtimes(old, now.Add(-48*time.Hour), now.Add(-48*time.Hour))
+	Prune(dir, KindJarl, now, 24*time.Hour, 1)
+	if _, err := os.Stat(old); err == nil {
+		t.Error("old temporary file kept")
+	}
+	if _, err := os.Stat(fresh); err != nil {
+		t.Error("a running backup's temporary file was deleted")
+	}
+}

internal/client/client.go

@@ -38,6 +38,9 @@ type Client struct {
 	Preflight bool
 	// Mutated records that a preflight run sent a (dry-run) change.
 	Mutated bool
+	// OnTokenExpiry, if set, is called with the token's expiry time when
+	// jarld says the token expires soon (api.TokenExpiresHeader).
+	OnTokenExpiry func(time.Time)
 }
 
 // ErrNoPreflight is returned in Preflight mode for a request that would
@@ -189,6 +192,11 @@ func (c *Client) Do(ctx context.Context, method, path string, query url.Values,
 		return err
 	}
 	defer resp.Body.Close()
+	if v := resp.Header.Get(api.TokenExpiresHeader); v != "" && c.OnTokenExpiry != nil {
+		if t, err := time.Parse(time.RFC3339, v); err == nil {
+			c.OnTokenExpiry(t)
+		}
+	}
 	data, err := io.ReadAll(resp.Body)
 	if err != nil {
 		return err

internal/config/config.go

@@ -31,10 +31,28 @@ type Config struct {
 	Session    Session `yaml:"session"`
 	BIND       BIND    `yaml:"bind"`
 	Audit      Audit   `yaml:"audit"`
+	Backup     Backup  `yaml:"backup"`
 	Stats      Stats   `yaml:"stats"`
 	ACME       ACME    `yaml:"acme"`
 }
 
+// Backup configures 'jarld backup' run without a file name, which is what
+// the jarl-backup.timer systemd timer runs (daily by default; the schedule
+// is the timer's OnCalendar=). Each run writes a new backup into Dir and
+// then deletes the ones Retention and KeepAtLeast no longer want.
+type Backup struct {
+	Dir string `yaml:"dir"`
+	// Retention deletes backups older than this (0 keeps them all)...
+	Retention Duration `yaml:"retention"`
+	// ...but the newest KeepAtLeast are always kept, however old.
+	KeepAtLeast int `yaml:"keep_at_least"`
+	// Stats also backs up the query statistics database.
+	Stats bool `yaml:"stats"`
+	// WarnAfter makes the dashboard warn when the newest backup is older
+	// than this (0: never). Keep it above the timer's interval.
+	WarnAfter Duration `yaml:"warn_after"`
+}
+
 // ACME configures the ACME helpers.
 type ACME struct {
 	HTTPReq HTTPReq `yaml:"httpreq"`
@@ -197,7 +215,8 @@ func Default() *Config {
 			Retention:         StatsRetention{Minute: Duration(7 * 24 * time.Hour), Hour: Duration(90 * 24 * time.Hour), Day: Duration(730 * 24 * time.Hour)},
 			TopN:              100,
 		},
-		ACME: ACME{HTTPReq: HTTPReq{RequireHTTPS: true}},
+		ACME:   ACME{HTTPReq: HTTPReq{RequireHTTPS: true}},
+		Backup: Backup{Dir: "/var/backups/jarl", Retention: Duration(30 * 24 * time.Hour), KeepAtLeast: 7, WarnAfter: Duration(2 * 24 * time.Hour)},
 	}
 }
 
@@ -251,6 +270,9 @@ func (c *Config) check() error {
 	if err := c.Stats.check(); err != nil {
 		return fmt.Errorf("stats: %w", err)
 	}
+	if err := c.Backup.check(); err != nil {
+		return fmt.Errorf("backup: %w", err)
+	}
 	switch c.Log.Format {
 	case "text", "json":
 	default:
@@ -259,6 +281,22 @@ func (c *Config) check() error {
 	return nil
 }
 
+func (b *Backup) check() error {
+	if b.Dir == "" || !filepath.IsAbs(b.Dir) {
+		return fmt.Errorf("dir must be an absolute path")
+	}
+	if b.Retention < 0 || (b.Retention > 0 && b.Retention < Duration(24*time.Hour)) {
+		return fmt.Errorf("retention must be at least 1d (or 0 to keep every backup)")
+	}
+	if b.KeepAtLeast < 1 {
+		return fmt.Errorf("keep_at_least must be at least 1")
+	}
+	if b.WarnAfter < 0 || (b.WarnAfter > 0 && b.WarnAfter < Duration(time.Hour)) {
+		return fmt.Errorf("warn_after must be at least 1h (or 0 for no warning)")
+	}
+	return nil
+}
+
 func (s *Stats) check() error {
 	if !s.Enabled {
 		return nil
@@ -391,6 +429,19 @@ audit:
   # Delete audit events older than this many days (0 keeps them forever).
   retention_days: 0
 
+# Database backups (docs/operations.md#backups). The jarl-backup.timer
+# systemd timer runs 'jarld backup' daily; change when with
+# 'systemctl edit jarl-backup.timer'. Each run writes
+# DIR/jarl-<UTC time>.db, checks it, then deletes backups older than
+# 'retention', always keeping the newest 'keep_at_least'. Only files named
+# like that are ever deleted. Restore with 'jarld restore'.
+backup:
+  dir: /var/backups/jarl   # the timer's sandbox may write only here (see the docs to move it)
+  retention: 30d           # 0 keeps every backup
+  keep_at_least: 7
+  stats: false             # also back up the statistics database
+  warn_after: 2d           # dashboard warning when the newest backup is older (0: never)
+
 # Query statistics (docs/stats.md). Off by default. When enabled, Jarl makes
 # named publish its counters on a local statistics channel and log every
 # response, and jarld aggregates both into a separate database. Changing

internal/engine/engine.go

@@ -1190,3 +1190,17 @@ func (e *Engine) WorkingState(ctx context.Context, csID int64, p *auth.Principal
 	}
 	return work, base, nil
 }
+
+// LoadedSerial returns the serial named serves for zone z, from rndc
+// zonestatus. ok is false when named isn't asked (no_reload) or doesn't
+// report a serial (the zone isn't loaded yet).
+func (e *Engine) LoadedSerial(ctx context.Context, st *model.State, z *model.Zone) (serial uint32, ok bool, err error) {
+	if e.cfg.NoReload || e.tools == nil {
+		return 0, false, nil
+	}
+	zs, err := e.tools.ZoneStatus(ctx, z.Name, e.namedView(st, z))
+	if err != nil {
+		return 0, false, err
+	}
+	return zs.Serial, zs.HasSerial, nil
+}

internal/model/changeset.go

@@ -97,6 +97,16 @@ type Token struct {
 	PrevExpiresAt  *time.Time `json:"previous_secret_valid_until,omitempty"`
 }
 
+// TokenExpiryWarning is how long before a token expires Jarl starts to
+// warn about it: on the dashboard, in jarld's log, and to the CLI using it.
+const TokenExpiryWarning = 14 * 24 * time.Hour
+
+// ExpiresWithin reports whether the token is still in use (not revoked)
+// and expires within d from now. A token that has already expired counts.
+func (t *Token) ExpiresWithin(now time.Time, d time.Duration) bool {
+	return t.RevokedAt == nil && t.ExpiresAt != nil && t.ExpiresAt.Sub(now) < d
+}
+
 // IssuedAt is when the token's current secret was made.
 func (t *Token) IssuedAt() time.Time {
 	if t.RotatedAt != nil {

internal/nscheck/delegation.go

@@ -0,0 +1,449 @@
+package nscheck
+
+import (
+	"context"
+	"fmt"
+	"net/netip"
+	"sort"
+	"strings"
+	"sync"
+
+	"github.com/miekg/dns"
+)
+
+// Finding is one conclusion of a delegation check. Level is "ok",
+// "info", "warning" or "error".
+type Finding struct {
+	Level   string `json:"level"`
+	Message string `json:"message"`
+}
+
+// ParentAnswer is what one of the parent zone's servers says.
+type ParentAnswer struct {
+	Name    string `json:"name"`
+	Address string `json:"address,omitempty"`
+	// Status: "delegated", "not_delegated" (NXDOMAIN, or the name exists
+	// without a delegation), "unreachable" or "error".
+	Status string   `json:"status"`
+	NS     []string `json:"ns,omitempty"`
+	// Glue is the addresses the parent hands out for in-zone name
+	// servers.
+	Glue  map[string][]string `json:"glue,omitempty"`
+	Error string              `json:"error,omitempty"`
+}
+
+// GlueCheck compares the parent's glue for an in-zone name server with
+// Jarl's addresses for it.
+type GlueCheck struct {
+	Name   string   `json:"name"`
+	Parent []string `json:"parent"`
+	Jarl   []string `json:"jarl"`
+	Match  bool     `json:"match"`
+}
+
+// DelegationReport is the result of Delegation.
+type DelegationReport struct {
+	Zone     string `json:"zone"`
+	Parent   string `json:"parent,omitempty"`
+	Resolver string `json:"resolver"`
+	// Delegated: at least one parent server delegates the zone.
+	Delegated bool           `json:"delegated"`
+	ParentNS  []string       `json:"parent_ns"`
+	JarlNS    []string       `json:"jarl_ns"`
+	Parents   []ParentAnswer `json:"parent_servers"`
+	Glue      []GlueCheck    `json:"glue"`
+	// Servers are the delegated name servers' SOA answers, as resolvers
+	// on the internet would reach them.
+	Servers  []Result  `json:"servers"`
+	Findings []Finding `json:"findings"`
+}
+
+// Delegation checks how the parent zone delegates zone: it finds the
+// parent through the resolver, asks each of the parent's servers for the
+// zone's NS records and glue, compares those with jarlNS and jarlAddrs
+// (Jarl's NS records and its addresses for in-zone name servers), and
+// asks every delegated server for the zone's serial (compared with ref).
+func (c *Checker) Delegation(ctx context.Context, zone string, jarlNS []string, jarlAddrs map[string][]string, ref *uint32) *DelegationReport {
+	zone = dns.CanonicalName(zone)
+	rep := &DelegationReport{Zone: zone, JarlNS: canonAll(jarlNS), ParentNS: []string{}, Parents: []ParentAnswer{}, Glue: []GlueCheck{}, Servers: []Result{}, Findings: []Finding{}}
+	sort.Strings(rep.JarlNS)
+	add := func(level, format string, args ...any) {
+		for i, a := range args {
+			switch v := a.(type) {
+			case string:
+				args[i] = show(v)
+			case []string:
+				args[i] = showAll(v)
+			}
+		}
+		rep.Findings = append(rep.Findings, Finding{level, fmt.Sprintf(format, args...)})
+	}
+	res, err := c.resolver()
+	if err != nil {
+		add("error", "No resolver to find the parent zone with: %v", err)
+		return rep
+	}
+	rep.Resolver = res
+	if zone == "." {
+		add("info", "The root zone has no parent to delegate it.")
+		return rep
+	}
+
+	// The parent zone: the zone that holds the name just above ours.
+	parent, err := c.enclosingZone(ctx, res, parentName(zone))
+	if err != nil {
+		add("error", "Couldn't find the parent zone of %s through %s: %v", zone, res, err)
+		return rep
+	}
+	rep.Parent = parent
+	pns, err := c.nsNames(ctx, res, parent)
+	if err != nil || len(pns) == 0 {
+		add("error", "Couldn't find the name servers of the parent zone %s through %s: %v", parent, res, errOr(err, "none listed"))
+		return rep
+	}
+
+	// Ask every parent server, in parallel.
+	var wg sync.WaitGroup
+	answers := make([]ParentAnswer, len(pns))
+	for i, name := range pns {
+		wg.Add(1)
+		go func(i int, name string) {
+			defer wg.Done()
+			answers[i] = c.askParent(ctx, zone, name)
+		}(i, name)
+	}
+	wg.Wait()
+	rep.Parents = answers
+
+	parentNS := map[string]bool{}
+	glue := map[string]map[string]bool{}
+	reached, delegated, notDelegated := 0, 0, 0
+	nsSets := map[string]bool{}
+	for _, a := range answers {
+		switch a.Status {
+		case "delegated":
+			reached++
+			delegated++
+			nsSets[strings.Join(a.NS, " ")] = true
+			for _, n := range a.NS {
+				parentNS[n] = true
+			}
+			for n, addrs := range a.Glue {
+				if glue[n] == nil {
+					glue[n] = map[string]bool{}
+				}
+				for _, ad := range addrs {
+					glue[n][ad] = true
+				}
+			}
+		case "not_delegated":
+			reached++
+			notDelegated++
+		}
+	}
+	if reached == 0 {
+		add("error", "None of the parent zone's servers (%s) answered; this server may not be able to reach them.", pns)
+		return rep
+	}
+	if delegated == 0 {
+		rep.Delegated = false
+		where := "the parent zone " + show(parent)
+		if parent == "." {
+			where = "the root zone"
+		}
+		add("info", "%s isn't delegated from %s, so the internet can't see it. That's expected for a zone used only on your own network. To publish it, register the domain and give your registrar these name servers: %s.", zone, where, rep.JarlNS)
+		return rep
+	}
+	rep.Delegated = true
+	if notDelegated > 0 {
+		add("warning", "%d of the parent's %d servers don't know the delegation yet (a recent change at the registrar can take a while to reach them all).", notDelegated, reached)
+	}
+	if len(nsSets) > 1 {
+		add("warning", "The parent's servers give different name server lists for %s.", zone)
+	}
+	rep.ParentNS = keys(parentNS)
+
+	// The NS lists.
+	jarlSet := map[string]bool{}
+	for _, n := range rep.JarlNS {
+		jarlSet[n] = true
+	}
+	var missing, extra []string
+	for _, n := range rep.JarlNS {
+		if !parentNS[n] {
+			missing = append(missing, n)
+		}
+	}
+	for _, n := range rep.ParentNS {
+		if !jarlSet[n] {
+			extra = append(extra, n)
+		}
+	}
+	if len(missing) == 0 && len(extra) == 0 {
+		add("ok", "The parent delegates %s to the same name servers as its NS records: %s.", zone, rep.ParentNS)
+	}
+	if len(extra) > 0 {
+		add("warning", "The parent delegates to %s, which the zone's NS records don't list. Resolvers ask them too; remove them at the registrar if they no longer serve the zone, or add NS records for them.", extra)
+	}
+	if len(missing) > 0 {
+		add("warning", "The zone's NS records list %s, which the parent doesn't delegate to. Resolvers that start from the parent won't use them; add them at the registrar.", missing)
+	}
+
+	// Glue for name servers inside the zone.
+	for _, n := range rep.ParentNS {
+		if !dns.IsSubDomain(zone, n) {
+			continue
+		}
+		g := GlueCheck{Name: n, Parent: keys(glue[n]), Jarl: canonAddrs(jarlAddrs[n])}
+		g.Match = len(g.Parent) > 0 && strings.Join(g.Parent, " ") == strings.Join(g.Jarl, " ")
+		rep.Glue = append(rep.Glue, g)
+		switch {
+		case len(g.Parent) == 0:
+			add("error", "The parent has no glue address for %s, which is inside the zone, so resolvers can't find it. Add its address at the registrar (%s).", n, strings.Join(g.Jarl, ", "))
+		case len(g.Jarl) == 0:
+			add("warning", "The parent's glue for %s is %s, but the zone has no address record for it.", n, strings.Join(g.Parent, ", "))
+		case !g.Match:
+			add("warning", "The parent's glue for %s is %s, but the zone says %s. Update the glue at the registrar.", n, strings.Join(g.Parent, ", "), strings.Join(g.Jarl, ", "))
+		}
+	}
+
+	// Every delegated server, reached the way resolvers reach it: by its
+	// glue, or by looking its name up.
+	var targets []Target
+	for _, n := range rep.ParentNS {
+		t := Target{Name: n, Source: "delegation"}
+		if g := keys(glue[n]); len(g) > 0 {
+			t.Addresses = g
+		}
+		targets = append(targets, t)
+	}
+	// Here every address is asked, this server's too: what matters is
+	// what it answers from the outside.
+	pub := *c
+	pub.IsLocal = func(netip.Addr) bool { return false }
+	rep.Servers = pub.Serials(ctx, zone, ref, targets)
+	// One finding per server and outcome, however many addresses it has.
+	type outcome struct{ name, status, detail string }
+	var order []outcome
+	addrs := map[outcome][]string{}
+	bad := 0
+	for _, r := range rep.Servers {
+		var o outcome
+		switch r.Status {
+		case "in_sync", "answered", "no_route":
+			continue
+		case "behind", "ahead":
+			o = outcome{r.Name, r.Status, fmt.Sprintf("serves serial %d, not %d", *r.Serial, *ref)}
+		case "not_authoritative":
+			o = outcome{r.Name, r.Status, ""}
+		default:
+			o = outcome{r.Name, r.Status, describe(r)}
+		}
+		bad++
+		if addrs[o] == nil {
+			order = append(order, o)
+		}
+		addrs[o] = append(addrs[o], nonEmpty(r.Address, "no address"))
+	}
+	for _, o := range order {
+		at := strings.Join(addrs[o], ", ")
+		if len(addrs[o]) > 2 {
+			at = fmt.Sprintf("all %d addresses", len(addrs[o]))
+		}
+		if o.status == "not_authoritative" {
+			add("error", "%s (%s) is delegated the zone but doesn't serve it (a lame delegation).", o.name, at)
+		} else {
+			add("warning", "%s (%s): %s.", o.name, at, o.detail)
+		}
+	}
+	if bad == 0 && len(rep.Servers) > 0 {
+		add("ok", "Every delegated name server answers for the zone with the current serial.")
+	}
+	return rep
+}
+
+// askParent asks one of the parent's servers (by name) for zone's NS.
+func (c *Checker) askParent(ctx context.Context, zone, server string) ParentAnswer {
+	pa := ParentAnswer{Name: server}
+	addrs, err := c.lookup(ctx, server)
+	if err != nil || len(addrs) == 0 {
+		pa.Status, pa.Error = "error", "looking up its address: "+errOr(err, "no address").Error()
+		return pa
+	}
+	m := new(dns.Msg)
+	m.SetQuestion(zone, dns.TypeNS)
+	m.RecursionDesired = false
+	// IPv4 first: try each address until one answers.
+	sort.SliceStable(addrs, func(i, j int) bool { return !strings.Contains(addrs[i], ":") && strings.Contains(addrs[j], ":") })
+	var resp *dns.Msg
+	for _, a := range addrs {
+		pa.Address = a
+		resp, _, err = c.exchange(ctx, m.Copy(), a, nil)
+		if err == nil {
+			break
+		}
+	}
+	if err != nil {
+		pa.Status, pa.Error = "unreachable", err.Error()
+		return pa
+	}
+	if resp.Rcode == dns.RcodeNameError {
+		pa.Status = "not_delegated"
+		return pa
+	}
+	if resp.Rcode != dns.RcodeSuccess {
+		pa.Status, pa.Error = "error", "answered "+dns.RcodeToString[resp.Rcode]
+		return pa
+	}
+	// A referral puts the NS set in the authority section; a server that
+	// also serves the child zone answers it directly.
+	ns := map[string]bool{}
+	for _, sec := range [][]dns.RR{resp.Ns, resp.Answer} {
+		for _, rr := range sec {
+			if n, ok := rr.(*dns.NS); ok && dns.CanonicalName(n.Hdr.Name) == zone {
+				ns[dns.CanonicalName(n.Ns)] = true
+			}
+		}
+	}
+	if len(ns) == 0 {
+		pa.Status = "not_delegated"
+		return pa
+	}
+	pa.Status, pa.NS = "delegated", keys(ns)
+	for _, rr := range resp.Extra {
+		var name, addr string
+		switch v := rr.(type) {
+		case *dns.A:
+			name, addr = v.Hdr.Name, v.A.String()
+		case *dns.AAAA:
+			name, addr = v.Hdr.Name, v.AAAA.String()
+		default:
+			continue
+		}
+		name = dns.CanonicalName(name)
+		if ns[name] {
+			if pa.Glue == nil {
+				pa.Glue = map[string][]string{}
+			}
+			pa.Glue[name] = append(pa.Glue[name], addr)
+		}
+	}
+	for n := range pa.Glue {
+		pa.Glue[n] = canonAddrs(pa.Glue[n])
+	}
+	return pa
+}
+
+// enclosingZone finds the apex of the zone that holds name, from the
+// owner of the SOA record the resolver returns (in the answer for an apex,
+// in the authority section otherwise).
+func (c *Checker) enclosingZone(ctx context.Context, res, name string) (string, error) {
+	m := new(dns.Msg)
+	m.SetQuestion(name, dns.TypeSOA)
+	resp, _, err := c.exchangeHostPort(ctx, m, res, nil)
+	if err != nil {
+		return "", err
+	}
+	for _, sec := range [][]dns.RR{resp.Answer, resp.Ns} {
+		for _, rr := range sec {
+			if s, ok := rr.(*dns.SOA); ok {
+				return dns.CanonicalName(s.Hdr.Name), nil
+			}
+		}
+	}
+	return "", fmt.Errorf("no SOA record in the answer for %s (%s)", name, dns.RcodeToString[resp.Rcode])
+}
+
+// nsNames asks the resolver for a zone's NS names.
+func (c *Checker) nsNames(ctx context.Context, res, zone string) ([]string, error) {
+	m := new(dns.Msg)
+	m.SetQuestion(zone, dns.TypeNS)
+	resp, _, err := c.exchangeHostPort(ctx, m, res, nil)
+	if err != nil {
+		return nil, err
+	}
+	set := map[string]bool{}
+	for _, rr := range resp.Answer {
+		if n, ok := rr.(*dns.NS); ok {
+			set[dns.CanonicalName(n.Ns)] = true
+		}
+	}
+	return keys(set), nil
+}
+
+// show is a name as people write it, without the trailing dot.
+func show(name string) string {
+	if name == "." || !strings.HasSuffix(name, ".") {
+		return name
+	}
+	return strings.TrimSuffix(name, ".")
+}
+
+func showAll(names []string) string {
+	out := make([]string, len(names))
+	for i, n := range names {
+		out[i] = show(n)
+	}
+	return strings.Join(out, ", ")
+}
+
+func parentName(zone string) string {
+	labels := dns.SplitDomainName(zone)
+	if len(labels) <= 1 {
+		return "."
+	}
+	return dns.Fqdn(strings.Join(labels[1:], "."))
+}
+
+func describe(r Result) string {
+	switch r.Status {
+	case "refused":
+		return "refuses queries for the zone"
+	case "unreachable":
+		return "no answer (" + r.Error + ")"
+	case "no_address":
+		return r.Error
+	}
+	return nonEmpty(r.Error, r.Status)
+}
+
+func keys(m map[string]bool) []string {
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	sort.Strings(out)
+	return out
+}
+
+func canonAll(names []string) []string {
+	out := make([]string, 0, len(names))
+	for _, n := range names {
+		out = append(out, dns.CanonicalName(n))
+	}
+	return out
+}
+
+func canonAddrs(addrs []string) []string {
+	set := map[string]bool{}
+	for _, a := range addrs {
+		if ip, err := netip.ParseAddr(a); err == nil {
+			set[ip.String()] = true
+		}
+	}
+	return keys(set)
+}
+
+func errOr(err error, msg string) error {
+	if err != nil {
+		return err
+	}
+	return fmt.Errorf("%s", msg)
+}
+
+func nonEmpty(s, def string) string {
+	if s != "" {
+		return s
+	}
+	return def
+}

internal/nscheck/nscheck.go

@@ -0,0 +1,338 @@
+// Package nscheck asks other name servers about a zone over DNS:
+//
+//   - Serials: is every name server of a zone serving the serial this
+//     server serves? For a primary zone those are its secondaries (the NS
+//     records and also-notify targets); for a secondary zone, its
+//     primaries.
+//   - Delegation: does the parent zone (e.g. "lol." for adbraun.lol)
+//     delegate the zone to the name servers, with the glue addresses, that
+//     Jarl has? Only meaningful for zones on the internet, so it only runs
+//     when asked and reports a missing delegation as information, not as a
+//     problem.
+//
+// It only sends ordinary non-recursive SOA and NS queries.
+package nscheck
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net"
+	"net/netip"
+	"sort"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+
+	"github.com/miekg/dns"
+)
+
+// Key is a TSIG key to sign queries with, for servers that pick the view
+// (or allow the query) by key.
+type Key struct {
+	Name      string
+	Algorithm string // e.g. hmac-sha256
+	Secret    string // base64
+}
+
+// Target is a server to ask.
+type Target struct {
+	Name      string   // host name (FQDN), "" when only an address is known
+	Addresses []string // known addresses; nil: look the name up
+	Source    string   // why it is asked: "NS record", "also-notify", "primaries", ...
+	Port      string   // "" for the default
+	Key       *Key
+}
+
+// Result is one server's answer.
+type Result struct {
+	Name    string `json:"name,omitempty"`
+	Address string `json:"address,omitempty"`
+	Port    string `json:"port,omitempty"`
+	Source  string `json:"source"`
+	// Status is one of:
+	//   self               this server (not queried)
+	//   in_sync            serves the same serial
+	//   answered           serves the zone (no serial here to compare with)
+	//   behind, ahead      serves an older or newer serial (RFC 1982)
+	//   not_authoritative  answers, but doesn't serve the zone
+	//   refused            refuses the query (allow-query, or its view
+	//                      doesn't have the zone for this server)
+	//   unreachable        no answer in time
+	//   no_route           this host has no route to the address (e.g.
+	//                      IPv6 on a host without it); not a problem
+	//   no_address         the name has no address
+	//   error              anything else (see Error)
+	Status string  `json:"status"`
+	Serial *uint32 `json:"serial,omitempty"`
+	// Diff is Serial minus the reference serial (negative: behind).
+	Diff  int64  `json:"diff,omitempty"`
+	RTTMs int64  `json:"rtt_ms,omitempty"`
+	Error string `json:"error,omitempty"`
+}
+
+// OK reports whether the result is good news.
+func (r *Result) OK() bool {
+	switch r.Status {
+	case "in_sync", "answered", "self", "no_route":
+		return true
+	}
+	return false
+}
+
+// Checker sends the queries.
+type Checker struct {
+	// Timeout per query (default 2s; two tries per address).
+	Timeout time.Duration
+	// Port is the DNS port (default 53; tests use another).
+	Port string
+	// Resolver is the recursive resolver used to look up names
+	// (host:port); default: the first nameserver in /etc/resolv.conf.
+	Resolver string
+	// IsLocal reports whether an address belongs to this host (default:
+	// its interface addresses and loopback).
+	IsLocal func(netip.Addr) bool
+}
+
+func (c *Checker) timeout() time.Duration {
+	if c.Timeout > 0 {
+		return c.Timeout
+	}
+	return 2 * time.Second
+}
+
+func (c *Checker) port() string {
+	if c.Port != "" {
+		return c.Port
+	}
+	return "53"
+}
+
+func (c *Checker) isLocal(a netip.Addr) bool {
+	if c.IsLocal != nil {
+		return c.IsLocal(a)
+	}
+	return IsLocalAddr(a)
+}
+
+// IsLocalAddr reports whether a is a loopback address or one of this
+// host's interface addresses.
+func IsLocalAddr(a netip.Addr) bool {
+	if a.IsLoopback() {
+		return true
+	}
+	addrs, err := net.InterfaceAddrs()
+	if err != nil {
+		return false
+	}
+	for _, ia := range addrs {
+		if n, ok := ia.(*net.IPNet); ok {
+			if b, ok := netip.AddrFromSlice(n.IP); ok && b.Unmap() == a.Unmap() {
+				return true
+			}
+		}
+	}
+	return false
+}
+
+// SystemResolver returns the first nameserver in /etc/resolv.conf as
+// host:port.
+func SystemResolver() (string, error) {
+	cc, err := dns.ClientConfigFromFile("/etc/resolv.conf")
+	if err != nil {
+		return "", fmt.Errorf("reading /etc/resolv.conf: %w", err)
+	}
+	if len(cc.Servers) == 0 {
+		return "", errors.New("no nameserver in /etc/resolv.conf")
+	}
+	return net.JoinHostPort(cc.Servers[0], cc.Port), nil
+}
+
+func (c *Checker) resolver() (string, error) {
+	if c.Resolver != "" {
+		return c.Resolver, nil
+	}
+	return SystemResolver()
+}
+
+// exchange sends one query to addr (an IP address) with a retry, and
+// over TCP when the UDP answer is truncated.
+func (c *Checker) exchange(ctx context.Context, m *dns.Msg, addr string, key *Key) (*dns.Msg, time.Duration, error) {
+	return c.exchangeHostPort(ctx, m, net.JoinHostPort(addr, c.port()), key)
+}
+
+func (c *Checker) exchangeHostPort(ctx context.Context, m *dns.Msg, hostport string, key *Key) (*dns.Msg, time.Duration, error) {
+	cl := &dns.Client{Timeout: c.timeout()}
+	if key != nil {
+		name := dns.Fqdn(key.Name)
+		cl.TsigSecret = map[string]string{name: key.Secret}
+		m.SetTsig(name, dns.Fqdn(key.Algorithm), 300, time.Now().Unix())
+	}
+	var resp *dns.Msg
+	var rtt time.Duration
+	var err error
+	for try := 0; try < 2; try++ {
+		resp, rtt, err = cl.ExchangeContext(ctx, m, hostport)
+		if err == nil || ctx.Err() != nil {
+			break
+		}
+	}
+	if err == nil && resp.Truncated {
+		cl.Net = "tcp"
+		resp, rtt, err = cl.ExchangeContext(ctx, m, hostport)
+	}
+	return resp, rtt, err
+}
+
+// lookup returns a name's IPv4 and IPv6 addresses through the resolver.
+func (c *Checker) lookup(ctx context.Context, name string) ([]string, error) {
+	res, err := c.resolver()
+	if err != nil {
+		return nil, err
+	}
+	var out []string
+	var firstErr error
+	for _, t := range []uint16{dns.TypeA, dns.TypeAAAA} {
+		m := new(dns.Msg)
+		m.SetQuestion(dns.Fqdn(name), t)
+		resp, _, err := c.exchangeHostPort(ctx, m, res, nil)
+		if err != nil {
+			if firstErr == nil {
+				firstErr = err
+			}
+			continue
+		}
+		for _, rr := range resp.Answer {
+			switch v := rr.(type) {
+			case *dns.A:
+				out = append(out, v.A.String())
+			case *dns.AAAA:
+				out = append(out, v.AAAA.String())
+			}
+		}
+	}
+	if len(out) == 0 && firstErr != nil {
+		return nil, firstErr
+	}
+	return out, nil
+}
+
+// soa asks addr for zone's SOA and fills in r (whose Name, Address and
+// Source are set).
+func (c *Checker) soa(ctx context.Context, zone string, key *Key, ref *uint32, r *Result) {
+	m := new(dns.Msg)
+	m.SetQuestion(dns.Fqdn(zone), dns.TypeSOA)
+	m.RecursionDesired = false
+	port := r.Port
+	if port == "" {
+		port = c.port()
+	}
+	resp, rtt, err := c.exchangeHostPort(ctx, m, net.JoinHostPort(r.Address, port), key)
+	if err != nil {
+		var ne net.Error
+		switch {
+		case errors.Is(err, syscall.ENETUNREACH):
+			// Not the server's fault: typically IPv6 on a host without it.
+			r.Status, r.Error = "no_route", "this server has no route to that address"
+		case errors.As(err, &ne) && ne.Timeout(), errors.Is(err, context.DeadlineExceeded):
+			r.Status, r.Error = "unreachable", "no answer within "+c.timeout().String()
+		default:
+			r.Status, r.Error = "unreachable", err.Error()
+		}
+		return
+	}
+	r.RTTMs = rtt.Milliseconds()
+	switch {
+	case resp.Rcode == dns.RcodeRefused:
+		r.Status = "refused"
+		return
+	case resp.Rcode != dns.RcodeSuccess:
+		r.Status, r.Error = "error", "answered "+dns.RcodeToString[resp.Rcode]
+		return
+	case !resp.Authoritative:
+		r.Status = "not_authoritative"
+		return
+	}
+	for _, rr := range resp.Answer {
+		if s, ok := rr.(*dns.SOA); ok && strings.EqualFold(s.Hdr.Name, dns.Fqdn(zone)) {
+			serial := s.Serial
+			r.Serial = &serial
+			r.Status = "answered"
+			if ref != nil {
+				r.Status = "in_sync"
+				r.Diff = int64(int32(serial - *ref))
+				switch {
+				case r.Diff < 0:
+					r.Status = "behind"
+				case r.Diff > 0:
+					r.Status = "ahead"
+				}
+			}
+			return
+		}
+	}
+	r.Status, r.Error = "error", "the answer has no SOA record"
+}
+
+// Serials asks each target for zone's SOA serial and compares it with
+// ref, the serial this server serves (nil when unknown). Targets with
+// local addresses are this server and aren't asked. Results are sorted by
+// name and address.
+func (c *Checker) Serials(ctx context.Context, zone string, ref *uint32, targets []Target) []Result {
+	var mu sync.Mutex
+	var wg sync.WaitGroup
+	var out []Result
+	add := func(r Result) {
+		mu.Lock()
+		out = append(out, r)
+		mu.Unlock()
+	}
+	seen := map[string]bool{}
+	for _, t := range targets {
+		addrs := t.Addresses
+		if addrs == nil && t.Name != "" {
+			found, err := c.lookup(ctx, t.Name)
+			if err != nil || len(found) == 0 {
+				msg := "no A or AAAA record"
+				if err != nil {
+					msg = "looking up its address: " + err.Error()
+				}
+				add(Result{Name: t.Name, Source: t.Source, Status: "no_address", Error: msg})
+				continue
+			}
+			addrs = found
+		}
+		for _, a := range addrs {
+			ip, err := netip.ParseAddr(a)
+			if err != nil || seen[ip.String()+"#"+t.Port] {
+				continue
+			}
+			seen[ip.String()+"#"+t.Port] = true
+			r := Result{Name: t.Name, Address: ip.String(), Port: t.Port, Source: t.Source}
+			if c.isLocal(ip) {
+				r.Status = "self"
+				if ref != nil {
+					s := *ref
+					r.Serial = &s
+				}
+				add(r)
+				continue
+			}
+			wg.Add(1)
+			go func(r Result, key *Key) {
+				defer wg.Done()
+				c.soa(ctx, zone, key, ref, &r)
+				add(r)
+			}(r, t.Key)
+		}
+	}
+	wg.Wait()
+	sort.Slice(out, func(i, j int) bool {
+		if out[i].Name != out[j].Name {
+			return out[i].Name < out[j].Name
+		}
+		return out[i].Address < out[j].Address
+	})
+	return out
+}

internal/nscheck/nscheck_test.go

@@ -0,0 +1,245 @@
+package nscheck
+
+import (
+	"context"
+	"net"
+	"net/netip"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/miekg/dns"
+)
+
+// fake is a DNS server on one loopback address that answers with fn.
+type fake func(q dns.Question) *dns.Msg
+
+// serve starts fake servers on 127.0.0.x addresses, all on the same port,
+// and returns the port.
+func serve(t *testing.T, servers map[string]fake) string {
+	t.Helper()
+	port := ""
+	for addr, fn := range servers {
+		hp := net.JoinHostPort(addr, "0")
+		if port != "" {
+			hp = net.JoinHostPort(addr, port)
+		}
+		pc, err := net.ListenPacket("udp", hp)
+		if err != nil {
+			t.Skipf("can't listen on %s: %v", hp, err)
+		}
+		if port == "" {
+			_, port, _ = net.SplitHostPort(pc.LocalAddr().String())
+		}
+		fn := fn
+		srv := &dns.Server{PacketConn: pc, Handler: dns.HandlerFunc(func(w dns.ResponseWriter, req *dns.Msg) {
+			resp := fn(req.Question[0])
+			if resp == nil {
+				return // no answer: a timeout
+			}
+			rcode, aa := resp.Rcode, resp.Authoritative
+			resp.SetReply(req) // resets both
+			resp.Rcode, resp.Authoritative = rcode, aa
+			w.WriteMsg(resp)
+		})}
+		go srv.ActivateAndServe()
+		t.Cleanup(func() { srv.Shutdown() })
+	}
+	return port
+}
+
+func rr(s string) dns.RR {
+	r, err := dns.NewRR(s)
+	if err != nil {
+		panic(err)
+	}
+	return r
+}
+
+// authSOA answers SOA queries for zone authoritatively with serial.
+func authSOA(zone string, serial uint32) fake {
+	return func(q dns.Question) *dns.Msg {
+		m := new(dns.Msg)
+		m.Authoritative = true
+		if q.Qtype == dns.TypeSOA && strings.EqualFold(q.Name, zone) {
+			m.Answer = []dns.RR{rr(zone + " 3600 IN SOA ns1." + zone + " admin." + zone + " " + itoa(serial) + " 3600 900 1209600 300")}
+		}
+		return m
+	}
+}
+
+func itoa(n uint32) string { return strconv.FormatUint(uint64(n), 10) }
+
+func checker(port, resolver string) *Checker {
+	return &Checker{
+		Timeout:  300 * time.Millisecond,
+		Port:     port,
+		Resolver: resolver,
+		IsLocal:  func(a netip.Addr) bool { return a.String() == "127.0.0.9" },
+	}
+}
+
+func TestSerials(t *testing.T) {
+	zone := "example.com."
+	port := serve(t, map[string]fake{
+		"127.0.0.2": authSOA(zone, 2026100805),
+		"127.0.0.3": authSOA(zone, 2026100803),
+		"127.0.0.4": func(dns.Question) *dns.Msg { return new(dns.Msg) }, // answers, not authoritative
+		"127.0.0.5": func(dns.Question) *dns.Msg { m := new(dns.Msg); m.Rcode = dns.RcodeRefused; return m },
+		"127.0.0.6": func(dns.Question) *dns.Msg { return nil }, // never answers
+	})
+	ref := uint32(2026100805)
+	res := checker(port, "").Serials(context.Background(), zone, &ref, []Target{
+		{Name: "ns1.example.com.", Addresses: []string{"127.0.0.9"}, Source: "NS record"},
+		{Name: "ns2.example.com.", Addresses: []string{"127.0.0.2"}, Source: "NS record"},
+		{Name: "ns3.example.com.", Addresses: []string{"127.0.0.3"}, Source: "NS record"},
+		{Name: "ns4.example.com.", Addresses: []string{"127.0.0.4"}, Source: "NS record"},
+		{Addresses: []string{"127.0.0.5"}, Source: "also-notify"},
+		{Addresses: []string{"127.0.0.6", "127.0.0.2"}, Source: "also-notify"}, // 127.0.0.2 again: asked once
+	})
+	got := map[string]Result{}
+	for _, r := range res {
+		got[r.Address] = r
+	}
+	if len(res) != 6 {
+		t.Errorf("%d results: %+v", len(res), res)
+	}
+	check := func(addr, status string) {
+		t.Helper()
+		if got[addr].Status != status {
+			t.Errorf("%s: %+v, want %s", addr, got[addr], status)
+		}
+	}
+	check("127.0.0.9", "self")
+	check("127.0.0.2", "in_sync")
+	check("127.0.0.3", "behind")
+	check("127.0.0.4", "not_authoritative")
+	check("127.0.0.5", "refused")
+	check("127.0.0.6", "unreachable")
+	if got["127.0.0.3"].Diff != -2 || *got["127.0.0.3"].Serial != 2026100803 {
+		t.Errorf("behind: %+v", got["127.0.0.3"])
+	}
+	// Without a serial to compare with, an answer is just an answer.
+	res = checker(port, "").Serials(context.Background(), zone, nil, []Target{{Addresses: []string{"127.0.0.2"}}})
+	if res[0].Status != "answered" || !res[0].OK() {
+		t.Errorf("no reference: %+v", res[0])
+	}
+}
+
+func TestSerialWraparound(t *testing.T) {
+	// RFC 1982: 1 is ahead of 4294967295.
+	zone := "example.com."
+	port := serve(t, map[string]fake{"127.0.0.2": authSOA(zone, 1)})
+	ref := uint32(4294967295)
+	res := checker(port, "").Serials(context.Background(), zone, &ref, []Target{{Addresses: []string{"127.0.0.2"}}})
+	if res[0].Status != "ahead" || res[0].Diff != 2 {
+		t.Errorf("%+v", res[0])
+	}
+}
+
+// internet builds a small fake DNS tree: a resolver, a parent server for
+// "lol." and the child's servers.
+func internet(t *testing.T, parent fake) (port, resolver string) {
+	port = serve(t, map[string]fake{
+		// The recursive resolver.
+		"127.0.0.10": func(q dns.Question) *dns.Msg {
+			m := new(dns.Msg)
+			m.RecursionAvailable = true
+			switch {
+			case q.Qtype == dns.TypeSOA && q.Name == "lol.":
+				m.Answer = []dns.RR{rr("lol. 300 IN SOA ns.nic.lol. hostmaster.lol. 1 3600 900 1209600 300")}
+			case q.Qtype == dns.TypeNS && q.Name == "lol.":
+				m.Answer = []dns.RR{rr("lol. 300 IN NS ns.nic.lol.")}
+			case q.Qtype == dns.TypeA && q.Name == "ns.nic.lol.":
+				m.Answer = []dns.RR{rr("ns.nic.lol. 300 IN A 127.0.0.11")}
+			case q.Qtype == dns.TypeA && q.Name == "ns.elsewhere.net.":
+				m.Answer = []dns.RR{rr("ns.elsewhere.net. 300 IN A 127.0.0.13")}
+			}
+			return m
+		},
+		"127.0.0.11": parent,
+		// ns1.adbraun.lol: serves the zone.
+		"127.0.0.12": authSOA("adbraun.lol.", 7),
+		// ns.elsewhere.net: doesn't serve it (lame).
+		"127.0.0.13": func(dns.Question) *dns.Msg { m := new(dns.Msg); m.Rcode = dns.RcodeRefused; return m },
+	})
+	return port, net.JoinHostPort("127.0.0.10", port)
+}
+
+func levels(rep *DelegationReport) map[string][]string {
+	out := map[string][]string{}
+	for _, f := range rep.Findings {
+		out[f.Level] = append(out[f.Level], f.Message)
+	}
+	return out
+}
+
+func TestDelegationMismatch(t *testing.T) {
+	port, res := internet(t, func(q dns.Question) *dns.Msg {
+		m := new(dns.Msg)
+		if q.Qtype == dns.TypeNS && q.Name == "adbraun.lol." {
+			m.Ns = []dns.RR{rr("adbraun.lol. 3600 IN NS ns1.adbraun.lol."), rr("adbraun.lol. 3600 IN NS ns.elsewhere.net.")}
+			m.Extra = []dns.RR{rr("ns1.adbraun.lol. 3600 IN A 127.0.0.12")}
+		}
+		return m
+	})
+	ref := uint32(7)
+	rep := checker(port, res).Delegation(context.Background(), "adbraun.lol.",
+		[]string{"ns1.adbraun.lol.", "ns2.adbraun.lol."},
+		map[string][]string{"ns1.adbraun.lol.": {"127.0.0.99"}, "ns2.adbraun.lol.": {"127.0.0.14"}}, &ref)
+	if rep.Parent != "lol." || !rep.Delegated {
+		t.Fatalf("report: %+v", rep)
+	}
+	l := levels(rep)
+	all := strings.Join(append(append(l["error"], l["warning"]...), l["info"]...), "\n")
+	for _, want := range []string{
+		"delegates to ns.elsewhere.net, which the zone's NS records don't list",
+		"list ns2.adbraun.lol, which the parent doesn't delegate to",
+		"glue for ns1.adbraun.lol is 127.0.0.12, but the zone says 127.0.0.99",
+		"ns.elsewhere.net (127.0.0.13): refuses queries",
+	} {
+		if !strings.Contains(all, want) {
+			t.Errorf("missing finding %q in:\n%s", want, all)
+		}
+	}
+	if len(rep.Glue) != 1 || rep.Glue[0].Match {
+		t.Errorf("glue: %+v", rep.Glue)
+	}
+}
+
+func TestDelegationGood(t *testing.T) {
+	port, res := internet(t, func(q dns.Question) *dns.Msg {
+		m := new(dns.Msg)
+		if q.Qtype == dns.TypeNS && q.Name == "adbraun.lol." {
+			m.Ns = []dns.RR{rr("adbraun.lol. 3600 IN NS ns1.adbraun.lol.")}
+			m.Extra = []dns.RR{rr("ns1.adbraun.lol. 3600 IN A 127.0.0.12")}
+		}
+		return m
+	})
+	ref := uint32(7)
+	rep := checker(port, res).Delegation(context.Background(), "adbraun.lol.", []string{"ns1.adbraun.lol."}, map[string][]string{"ns1.adbraun.lol.": {"127.0.0.12"}}, &ref)
+	l := levels(rep)
+	if len(l["error"])+len(l["warning"])+len(l["info"]) != 0 || len(l["ok"]) != 2 {
+		t.Errorf("findings: %v", l)
+	}
+}
+
+func TestNotDelegated(t *testing.T) {
+	// An internal-only zone: one calm note, no warnings.
+	port, res := internet(t, func(q dns.Question) *dns.Msg {
+		m := new(dns.Msg)
+		m.Authoritative = true
+		m.Rcode = dns.RcodeNameError
+		m.Ns = []dns.RR{rr("lol. 300 IN SOA ns.nic.lol. hostmaster.lol. 1 3600 900 1209600 300")}
+		return m
+	})
+	rep := checker(port, res).Delegation(context.Background(), "adbraun.lol.", []string{"ns1.adbraun.lol."}, nil, nil)
+	l := levels(rep)
+	if rep.Delegated || len(l["info"]) != 1 || len(l["warning"])+len(l["error"]) != 0 {
+		t.Errorf("findings: %v", l)
+	}
+	if !strings.Contains(l["info"][0], "adbraun.lol isn't delegated from the parent zone lol, so the internet can't see it. That's expected for a zone used only on your own network") {
+		t.Errorf("note: %s", l["info"][0])
+	}
+}

internal/server/authn.go

@@ -6,10 +6,12 @@ import (
 	"errors"
 	"fmt"
 	"net/http"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
 
+	"jarl/internal/api"
 	"jarl/internal/auth"
 	"jarl/internal/model"
 	"jarl/internal/store"
@@ -79,11 +81,12 @@ func (l *rateLimiter) cleanup() {
 	}
 }
 
-// throttle allows an event once a minute per key (for log lines that
+// throttle allows an event once per interval (every, a minute if unset)
-// could otherwise repeat on every request).
+// per key, for log lines that could otherwise repeat on every request.
 type throttle struct {
-	mu   sync.Mutex
+	every time.Duration
-	last map[string]time.Time
+	mu    sync.Mutex
+	last  map[string]time.Time
 }
 
 func (t *throttle) allow(key string, now time.Time) bool {
@@ -92,11 +95,15 @@ func (t *throttle) allow(key string, now time.Time) bool {
 	if t.last == nil {
 		t.last = map[string]time.Time{}
 	}
-	if now.Sub(t.last[key]) < time.Minute {
+	every := t.every
+	if every == 0 {
+		every = time.Minute
+	}
+	if now.Sub(t.last[key]) < every {
 		return false
 	}
 	for k, v := range t.last {
-		if now.Sub(v) > time.Hour {
+		if now.Sub(v) > every+time.Hour {
 			delete(t.last, k)
 		}
 	}
@@ -206,12 +213,46 @@ func (s *Server) tokenPrincipal(ctx context.Context, tok, ip, agent string) (*au
 	if t.LastUsedAt == nil || now.Sub(*t.LastUsedAt) > time.Minute || t.LastUsedIP != ip {
 		s.store.TouchToken(ctx, t.ID, ip)
 	}
+	s.warnTokenExpiry(t, ip, now)
 	if previous && s.oldSecretWarn.allow(fmt.Sprintf("%d %s", t.ID, ip), now) {
 		s.log.Warn("token used with its old secret; update this client before the grace period ends", "token_id", t.ID, "token", t.Username+"/"+t.Name, "ip", ip, "valid_until", t.PrevExpiresAt.Format(time.RFC3339))
 	}
 	return &auth.Principal{User: u, Token: t, Method: "token", IP: ip, Agent: agent}, nil
 }
 
+// warnTokenExpiry logs, at most once a day per token, that a token in use
+// expires soon (or has expired). ip is where it was used ("" when found by
+// the daily check).
+func (s *Server) warnTokenExpiry(t *model.Token, ip string, now time.Time) {
+	if !t.ExpiresWithin(now, model.TokenExpiryWarning) || !s.expiryWarn.allow(strconv.FormatInt(t.ID, 10), now) {
+		return
+	}
+	args := []any{"token_id", t.ID, "token", t.Username + "/" + t.Name, "expires_at", t.ExpiresAt.Format(time.RFC3339)}
+	if ip != "" {
+		args = append(args, "ip", ip)
+	}
+	msg := "API token expires soon; give it a new lifetime with 'jarl token rotate'"
+	if now.After(*t.ExpiresAt) {
+		msg = "API token has expired; anything still using it is refused (rotate it with 'jarl token rotate')"
+	}
+	s.log.Warn(msg, args...)
+}
+
+// warnExpiringTokens runs warnTokenExpiry for every token (from
+// housekeeping), so tokens that are rarely used are reported too. Tokens
+// that expired more than TokenExpiryWarning ago are no longer reported.
+func (s *Server) warnExpiringTokens(ctx context.Context, now time.Time) {
+	ts, err := s.store.ListTokens(ctx, 0)
+	if err != nil {
+		return
+	}
+	for _, t := range ts {
+		if t.ExpiresAt != nil && now.Sub(*t.ExpiresAt) < model.TokenExpiryWarning {
+			s.warnTokenExpiry(t, "", now)
+		}
+	}
+}
+
 // authed wraps a handler with authentication and error handling.
 func (s *Server) authed(h handler) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
@@ -221,6 +262,10 @@ func (s *Server) authed(h handler) http.HandlerFunc {
 			s.writeErr(w, r, err, nil)
 			return
 		}
+		if p.Token != nil && p.Token.ExpiresWithin(time.Now(), model.TokenExpiryWarning) {
+			// The CLI warns its user when it sees this.
+			w.Header().Set(api.TokenExpiresHeader, p.Token.ExpiresAt.UTC().Format(time.RFC3339))
+		}
 		ctx := context.WithValue(r.Context(), keyPrincipal, p)
 		if se != nil {
 			ctx = context.WithValue(ctx, keySession, se)

internal/server/expiry_test.go

@@ -0,0 +1,117 @@
+package server
+
+import (
+	"context"
+	"net/http"
+	"testing"
+	"time"
+
+	"jarl/internal/api"
+	"jarl/internal/store"
+)
+
+// header sends a GET with a token and returns a response header.
+func (e *apiEnv) header(path, token, name string) string {
+	e.t.Helper()
+	r, _ := http.NewRequest("GET", e.ts.URL+api.Prefix+path, nil)
+	r.Header.Set("Authorization", "Bearer "+token)
+	resp, err := http.DefaultClient.Do(r)
+	if err != nil {
+		e.t.Fatal(err)
+	}
+	resp.Body.Close()
+	if resp.StatusCode != 200 {
+		e.t.Fatalf("GET %s = %d", path, resp.StatusCode)
+	}
+	return resp.Header.Get(name)
+}
+
+func TestTokenExpiryWarnings(t *testing.T) {
+	e := newAPIEnv(t)
+	e.do("POST", "/users", e.token, map[string]any{"username": "certbot", "password": "", "grant_strings": []string{"records:write"}}, 201)
+	var uid int64
+	for _, u := range e.list("/users", e.token) {
+		if u["username"] == "certbot" {
+			uid = int64(u["id"].(float64))
+		}
+	}
+	soon := e.do("POST", "/users/"+itoa(uid)+"/tokens", e.token, map[string]any{"name": "soon", "expires_in": "5d"}, 201)
+	later := e.do("POST", "/users/"+itoa(uid)+"/tokens", e.token, map[string]any{"name": "later", "expires_in": "90d"}, 201)
+	revoked := e.do("POST", "/users/"+itoa(uid)+"/tokens", e.token, map[string]any{"name": "gone", "expires_in": "1d"}, 201)
+	e.do("DELETE", "/tokens/"+itoa(int64(revoked["id"].(float64))), e.token, nil, 204)
+
+	// The dashboard lists the expiring token for the administrator...
+	names := func(token string) []string {
+		var out []string
+		for _, t := range e.do("GET", "/status", token, nil, 200)["expiring_tokens"].([]any) {
+			out = append(out, t.(map[string]any)["name"].(string))
+		}
+		return out
+	}
+	if got := names(e.token); len(got) != 1 || got[0] != "soon" {
+		t.Errorf("admin's expiring tokens = %v", got)
+	}
+	// ...and for its own user.
+	if got := names(soon["token"].(string)); len(got) != 1 || got[0] != "soon" {
+		t.Errorf("certbot's expiring tokens = %v", got)
+	}
+
+	// Requests with the expiring token say so (the CLI warns); others don't.
+	if h := e.header("/auth/whoami", soon["token"].(string), api.TokenExpiresHeader); h == "" {
+		t.Error("no expiry header for a token that expires in 5 days")
+	} else if exp, err := time.Parse(time.RFC3339, h); err != nil || time.Until(exp) < 4*24*time.Hour {
+		t.Errorf("expiry header %q", h)
+	}
+	if h := e.header("/auth/whoami", later["token"].(string), api.TokenExpiresHeader); h != "" {
+		t.Errorf("expiry header for a 90-day token: %q", h)
+	}
+	if h := e.header("/auth/whoami", e.token, api.TokenExpiresHeader); h != "" {
+		t.Errorf("expiry header for a token without expiry: %q", h)
+	}
+}
+
+func TestBackupStatus(t *testing.T) {
+	e := newAPIEnvConfig(t, Config{Version: "test", BackupDir: "/var/backups/jarl", BackupWarnAfter: 48 * time.Hour})
+	backup := func() map[string]any {
+		b, _ := e.do("GET", "/status", e.token, nil, 200)["backup"].(map[string]any)
+		return b
+	}
+	if b := backup(); b == nil || b["last_at"] != nil || b["stale"] != false || b["dir"] != "/var/backups/jarl" {
+		t.Errorf("before any backup: %v", b)
+	}
+	ctx := context.Background()
+	now := time.Now()
+	e.st.RecordBackup(ctx, store.BackupRecord{At: now.Add(-3 * time.Hour), File: "/var/backups/jarl/jarl-x.db", Size: 1234})
+	if b := backup(); b["last_file"] != "/var/backups/jarl/jarl-x.db" || b["stale"] != false || b["last_error"] != nil {
+		t.Errorf("fresh backup: %v", b)
+	}
+	// A failure newer than the last success is reported; an older one isn't.
+	e.st.RecordBackup(ctx, store.BackupRecord{At: now.Add(-time.Hour), Error: "disk full"})
+	if b := backup(); b["last_error"] != "disk full" {
+		t.Errorf("failed backup: %v", b)
+	}
+	e.st.RecordBackup(ctx, store.BackupRecord{At: now.Add(-72 * time.Hour), File: "/var/backups/jarl/jarl-y.db"})
+	e.st.RecordBackup(ctx, store.BackupRecord{At: now.Add(-80 * time.Hour), Error: "old"})
+	if b := backup(); b["stale"] != true || b["last_error"] != nil {
+		t.Errorf("old backup: %v", b)
+	}
+	// Only administrators see it.
+	e.do("POST", "/users", e.token, map[string]any{"username": "viewer", "password": "", "role": "readonly"}, 201)
+	var uid int64
+	for _, u := range e.list("/users", e.token) {
+		if u["username"] == "viewer" {
+			uid = int64(u["id"].(float64))
+		}
+	}
+	tok := e.do("POST", "/users/"+itoa(uid)+"/tokens", e.token, map[string]any{"name": "t"}, 201)["token"].(string)
+	if st := e.do("GET", "/status", tok, nil, 200); st["backup"] != nil {
+		t.Errorf("non-admin sees backups: %v", st["backup"])
+	}
+}
+
+func TestServerHeader(t *testing.T) {
+	e := newAPIEnv(t)
+	if h := e.header("/auth/whoami", e.token, "Server"); h != "Jarl/test" {
+		t.Errorf("Server: %q", h)
+	}
+}

internal/server/h_changes.go

@@ -1,6 +1,7 @@
 package server
 
 import (
+	"context"
 	"encoding/json"
 	"errors"
 	"net/http"
@@ -271,10 +272,55 @@ func (s *Server) status(w http.ResponseWriter, r *http.Request, p *auth.Principa
 			st.Named.Raw = ""
 		}
 	}
-	writeJSON(w, http.StatusOK, api.StatusResponse{Version: s.cfg.Version, Engine: raw(st), Time: time.Now()})
+	now := time.Now()
+	resp := api.StatusResponse{Version: s.cfg.Version, Engine: raw(st), Time: now, ExpiringTokens: s.expiringTokens(r.Context(), p, now)}
+	if p.IsAdmin() {
+		resp.Backup = s.backupStatus(r.Context(), now)
+	}
+	writeJSON(w, http.StatusOK, resp)
 	return nil
 }
 
+// backupStatus reports the newest backup, as 'jarld backup' recorded it,
+// and whether it is older than backup.warn_after. No backup at all is
+// reported but not flagged: backups may be made some other way.
+func (s *Server) backupStatus(ctx context.Context, now time.Time) *api.BackupStatus {
+	ok, failed := s.store.LastBackups(ctx)
+	bs := &api.BackupStatus{Dir: s.cfg.BackupDir, WarnAfter: int64(s.cfg.BackupWarnAfter / time.Second)}
+	if ok != nil {
+		bs.LastAt, bs.LastFile, bs.LastSize = &ok.At, ok.File, ok.Size
+		bs.Stale = s.cfg.BackupWarnAfter > 0 && now.Sub(ok.At) > s.cfg.BackupWarnAfter
+	}
+	if failed != nil && (ok == nil || failed.At.After(ok.At)) {
+		bs.LastError, bs.LastErrorAt = failed.Error, &failed.At
+	}
+	return bs
+}
+
+// expiringTokens lists the tokens the dashboard warns about: those that
+// expire within model.TokenExpiryWarning or expired less than that long
+// ago. Administrators see everyone's, others their own.
+func (s *Server) expiringTokens(ctx context.Context, p *auth.Principal, now time.Time) []api.Token {
+	out := []api.Token{}
+	uid := p.User.ID
+	if p.IsAdmin() {
+		uid = 0
+	} else if uid == 0 {
+		return out
+	}
+	ts, err := s.store.ListTokens(ctx, uid)
+	if err != nil {
+		s.log.Warn("listing tokens failed", "err", err)
+		return out
+	}
+	for _, t := range ts {
+		if t.ExpiresWithin(now, model.TokenExpiryWarning) && now.Sub(*t.ExpiresAt) < model.TokenExpiryWarning {
+			out = append(out, presentToken(t))
+		}
+	}
+	return out
+}
+
 func (s *Server) systemCheck(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
 	if err := require(p, auth.Action{Resource: auth.ResOptions}); err != nil {
 		return err

internal/server/h_nscheck.go

@@ -0,0 +1,269 @@
+package server
+
+// Name server checks (internal/nscheck): GET /zones/{zone}/serials and
+// GET /zones/{zone}/delegation. Both only run when asked for (the zone
+// page's Name servers tab, 'jarl zone serials' and 'jarl zone
+// delegation'); nothing checks in the background.
+
+import (
+	"context"
+	"fmt"
+	"net"
+	"net/http"
+	"net/netip"
+	"strings"
+	"time"
+
+	"jarl/internal/api"
+	"jarl/internal/auth"
+	"jarl/internal/dnsname"
+	"jarl/internal/model"
+	"jarl/internal/namedconf"
+	"jarl/internal/nscheck"
+)
+
+// checker is the nscheck.Checker the handlers use (tests replace it).
+func (s *Server) checker() *nscheck.Checker {
+	if s.nsChecker != nil {
+		c := *s.nsChecker
+		return &c
+	}
+	return &nscheck.Checker{}
+}
+
+func (s *Server) zoneSerials(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	st, err := s.state(r)
+	if err != nil {
+		return err
+	}
+	z, err := s.zoneByPath(r, st, p)
+	if err != nil {
+		return err
+	}
+	resp := api.SerialsResponse{Zone: z.Name, View: st.ViewName(z), Type: z.Type, Servers: []nscheck.Result{}, Notes: []string{}}
+	var targets []nscheck.Target
+	switch z.Type {
+	case model.ZonePrimary:
+		targets = append(targets, nsTargets(st, z)...)
+		an, notes := remoteTargets(st, inheritedOption(st, z, "also-notify"), "also-notify")
+		targets = append(targets, an...)
+		resp.Notes = append(resp.Notes, notes...)
+	case model.ZoneSecondary, model.ZoneMirror:
+		pr, notes := remoteTargets(st, st.FindOption(model.ScopeZone, z.ID, "primaries"), "primaries")
+		targets = append(targets, pr...)
+		resp.Notes = append(resp.Notes, notes...)
+	default:
+		return badRequest("only primary, secondary and mirror zones have serials to compare (this is a %s zone)", z.Type)
+	}
+	ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
+	defer cancel()
+	resp.Serial, resp.SerialSource, err = s.localSerial(ctx, st, z)
+	if err != nil {
+		resp.Notes = append(resp.Notes, "named on this server didn't report the zone's serial: "+err.Error())
+	}
+	if len(targets) == 0 {
+		if z.Type == model.ZonePrimary {
+			resp.Notes = append(resp.Notes, "The zone has no NS records or also-notify targets to check.")
+		} else {
+			resp.Notes = append(resp.Notes, "The zone has no primaries to check.")
+		}
+	}
+	if resp.View != "" {
+		resp.Notes = append(resp.Notes, fmt.Sprintf("The zone is in view %s. Other servers answer from the view that matches this server's address (or key), as they do for its zone transfers and NOTIFYs.", resp.View))
+	}
+	resp.Servers = s.checker().Serials(ctx, z.Name, resp.Serial, targets)
+	if len(resp.Servers) > 0 && !hasOther(resp.Servers) {
+		resp.Notes = append(resp.Notes, "Only this server is listed, so there is nothing to compare with.")
+	}
+	resp.CheckedAt = time.Now()
+	writeJSON(w, http.StatusOK, resp)
+	return nil
+}
+
+func (s *Server) zoneDelegation(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
+	st, err := s.state(r)
+	if err != nil {
+		return err
+	}
+	z, err := s.zoneByPath(r, st, p)
+	if err != nil {
+		return err
+	}
+	if z.Type != model.ZonePrimary {
+		return badRequest("the delegation check compares the parent's delegation with the zone's NS records, so it needs a primary zone (this is a %s zone; check it where its primary is)", z.Type)
+	}
+	c := s.checker()
+	if v := r.URL.Query().Get("resolver"); v != "" {
+		// An address only, always port 53: this is a DNS resolver.
+		a, err := netip.ParseAddr(v)
+		if err != nil {
+			return badRequest("resolver must be an IP address, e.g. 1.1.1.1 or 9.9.9.9")
+		}
+		c.Resolver = net.JoinHostPort(a.String(), "53")
+	}
+	var ns []string
+	for _, rec := range st.ZoneRecords(z.ID) {
+		if rec.Type == "NS" && rec.Name == z.Name {
+			ns = append(ns, rec.Data)
+		}
+	}
+	addrs := map[string][]string{}
+	for _, n := range ns {
+		if a := jarlAddresses(st, z.ViewID, dnsname.Fqdn(n)); len(a) > 0 {
+			addrs[dnsname.Fqdn(n)] = a
+		}
+	}
+	ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
+	defer cancel()
+	ref, _, _ := s.localSerial(ctx, st, z)
+	resp := api.DelegationResponse{DelegationReport: c.Delegation(ctx, z.Name, ns, addrs, ref), View: st.ViewName(z), Notes: []string{}, CheckedAt: time.Now()}
+	if resp.View != "" {
+		resp.Notes = append(resp.Notes, fmt.Sprintf("Compared with the zone in view %s. Resolvers on the internet get the view that matches their address; make sure that's this one.", resp.View))
+	}
+	writeJSON(w, http.StatusOK, resp)
+	return nil
+}
+
+// localSerial is the serial this server serves for z: named's, or for a
+// primary zone Jarl's when named isn't asked.
+func (s *Server) localSerial(ctx context.Context, st *model.State, z *model.Zone) (*uint32, string, error) {
+	serial, ok, err := s.eng.LoadedSerial(ctx, st, z)
+	if ok {
+		return &serial, "named", nil
+	}
+	if z.Type == model.ZonePrimary {
+		v := z.SOA.Serial
+		return &v, "jarl", err
+	}
+	return nil, "", err
+}
+
+// nsTargets are the name servers in the zone's apex NS records, with the
+// addresses Jarl has for them in the same view (or nil, to be looked up).
+func nsTargets(st *model.State, z *model.Zone) []nscheck.Target {
+	var out []nscheck.Target
+	for _, rec := range st.ZoneRecords(z.ID) {
+		if rec.Type != "NS" || rec.Name != z.Name {
+			continue
+		}
+		name := dnsname.Fqdn(rec.Data)
+		out = append(out, nscheck.Target{Name: name, Addresses: jarlAddresses(st, z.ViewID, name), Source: "NS record"})
+	}
+	return out
+}
+
+// jarlAddresses returns the A and AAAA records for name in the primary
+// zones of a view (nil when Jarl has none).
+func jarlAddresses(st *model.State, viewID int64, name string) []string {
+	var out []string
+	for _, z := range st.ZonesInView(viewID) {
+		if !z.HasRecords() || !dnsname.IsSubdomain(name, z.Name) {
+			continue
+		}
+		for _, rec := range st.ZoneRecords(z.ID) {
+			if rec.Name == name && (rec.Type == "A" || rec.Type == "AAAA") {
+				out = append(out, rec.Data)
+			}
+		}
+	}
+	return out
+}
+
+// inheritedOption finds an option for z as named does: the zone's own,
+// else its view's, else the server's.
+func inheritedOption(st *model.State, z *model.Zone, name string) *model.Option {
+	if o := st.FindOption(model.ScopeZone, z.ID, name); o != nil {
+		return o
+	}
+	if z.ViewID != 0 {
+		if o := st.FindOption(model.ScopeView, z.ViewID, name); o != nil {
+			return o
+		}
+	}
+	return st.FindOption(model.ScopeOptions, 0, name)
+}
+
+// remoteTargets turns an also-notify or primaries option into targets:
+// addresses with their port and key, following named remote-servers
+// lists.
+func remoteTargets(st *model.State, o *model.Option, source string) ([]nscheck.Target, []string) {
+	if o == nil {
+		return nil, nil
+	}
+	var out []nscheck.Target
+	var notes []string
+	var walk func(name, value string, port string, depth int)
+	walk = func(name, value, port string, depth int) {
+		stmt, err := namedconf.ParseOptionValue(name, value)
+		if err != nil {
+			notes = append(notes, fmt.Sprintf("couldn't read %s: %v", name, err))
+			return
+		}
+		vals := stmt.Values()
+		for i := 0; i+1 < len(vals); i++ {
+			if strings.EqualFold(vals[i], "port") {
+				port = vals[i+1]
+			}
+		}
+		blk := stmt.FirstBlock()
+		if blk == nil {
+			return
+		}
+		for _, el := range blk.Stmts {
+			if len(el.Items) == 0 || el.Items[0].Value == nil {
+				continue
+			}
+			first := el.Items[0].Value.Text
+			t := nscheck.Target{Source: source, Port: port}
+			words := el.Values()
+			for i := 0; i+1 < len(words); i++ {
+				switch strings.ToLower(words[i]) {
+				case "port":
+					t.Port = words[i+1]
+				case "key":
+					if k := st.KeyByName(words[i+1]); k != nil {
+						t.Key = &nscheck.Key{Name: k.Name, Algorithm: k.Algorithm, Secret: k.Secret}
+					}
+				}
+			}
+			if a, err := netip.ParseAddr(first); err == nil {
+				t.Addresses = []string{a.String()}
+				out = append(out, t)
+				continue
+			}
+			// A named list of servers.
+			if depth >= 5 {
+				continue
+			}
+			if lo := remoteList(st, first); lo != nil {
+				walk(lo.Name, lo.Value, port, depth+1)
+			} else {
+				notes = append(notes, fmt.Sprintf("%s refers to %q, which isn't an address or a server list Jarl knows", source, first))
+			}
+		}
+	}
+	walk(o.Name, o.Value, "", 0)
+	return out, notes
+}
+
+// remoteList finds a top-level remote-servers (or primaries) list.
+func remoteList(st *model.State, name string) *model.Option {
+	for _, o := range st.ScopedOptions(model.ScopeTopLevel, 0) {
+		switch o.Name {
+		case "remote-servers", "primaries", "masters":
+			if f := strings.Fields(o.Value); len(f) > 0 && strings.Trim(f[0], `"`) == name {
+				return o
+			}
+		}
+	}
+	return nil
+}
+
+func hasOther(rs []nscheck.Result) bool {
+	for _, r := range rs {
+		if r.Status != "self" {
+			return true
+		}
+	}
+	return false
+}

internal/server/nscheck_test.go

@@ -0,0 +1,71 @@
+package server
+
+import (
+	"net/netip"
+	"testing"
+	"time"
+
+	"jarl/internal/nscheck"
+)
+
+func TestZoneSerialsTargets(t *testing.T) {
+	e := newAPIEnv(t)
+	// Nothing listens on these addresses' DNS port: every server is
+	// "unreachable" quickly, and the test checks who is asked.
+	e.srv.nsChecker = &nscheck.Checker{Timeout: 200 * time.Millisecond, Port: "1", IsLocal: func(a netip.Addr) bool { return a.String() == "127.0.0.21" }}
+	e.do("POST", "/tsig-keys", e.token, map[string]any{"name": "xfer"}, 201)
+	e.do("POST", "/options", e.token, map[string]any{"scope": "toplevel", "name": "remote-servers", "value": "upstream port 5353 { 127.0.0.31; 127.0.0.32 key xfer; }"}, 201)
+	e.do("POST", "/zones", e.token, map[string]any{
+		"name":        "example.com",
+		"nameservers": []string{"ns1.example.com.", "ns2.example.com."},
+		"records":     []map[string]any{{"name": "ns1", "type": "A", "data": "127.0.0.21"}, {"name": "ns2", "type": "A", "data": "127.0.0.22"}},
+		"options":     []map[string]any{{"name": "also-notify", "value": "{ 127.0.0.23; 127.0.0.24 port 5300 key xfer; }"}},
+	}, 201)
+	e.do("POST", "/zones", e.token, map[string]any{"name": "copy.example", "type": "secondary", "options": []map[string]any{{"name": "primaries", "value": "{ upstream; 127.0.0.33; }"}}}, 201)
+
+	type row struct{ source, status, port string }
+	servers := func(zone string) map[string]row {
+		out := map[string]row{}
+		for _, s := range e.do("GET", "/zones/"+zone+"/serials", e.token, nil, 200)["servers"].([]any) {
+			m := s.(map[string]any)
+			port, _ := m["port"].(string)
+			out[m["address"].(string)] = row{m["source"].(string), m["status"].(string), port}
+		}
+		return out
+	}
+	got := servers("example.com")
+	want := map[string]row{
+		"127.0.0.21": {"NS record", "self", ""},
+		"127.0.0.22": {"NS record", "unreachable", ""},
+		"127.0.0.23": {"also-notify", "unreachable", ""},
+		"127.0.0.24": {"also-notify", "unreachable", "5300"},
+	}
+	if len(got) != len(want) {
+		t.Errorf("primary zone: %v", got)
+	}
+	for a, w := range want {
+		if got[a] != w {
+			t.Errorf("primary zone, %s: %+v, want %+v", a, got[a], w)
+		}
+	}
+	got = servers("copy.example")
+	want = map[string]row{
+		"127.0.0.31": {"primaries", "unreachable", "5353"},
+		"127.0.0.32": {"primaries", "unreachable", "5353"},
+		"127.0.0.33": {"primaries", "unreachable", ""},
+	}
+	if len(got) != len(want) {
+		t.Errorf("secondary zone: %v", got)
+	}
+	for a, w := range want {
+		if got[a] != w {
+			t.Errorf("secondary zone, %s: %+v, want %+v", a, got[a], w)
+		}
+	}
+
+	// Zone types without serials, and the delegation check's limits.
+	e.do("POST", "/zones", e.token, map[string]any{"name": "fwd.example", "type": "forward", "options": []map[string]any{{"name": "forwarders", "value": "{ 192.0.2.1; }"}}}, 201)
+	e.do("GET", "/zones/fwd.example/serials", e.token, nil, 400)
+	e.do("GET", "/zones/copy.example/delegation", e.token, nil, 400)
+	e.do("GET", "/zones/example.com/delegation?resolver=not-an-ip", e.token, nil, 400)
+}

internal/server/routes.go

@@ -65,6 +65,8 @@ func (s *Server) routes() {
 	m.HandleFunc("GET "+p+"/zones/{zone}", s.authed(s.getZone))
 	m.HandleFunc("PATCH "+p+"/zones/{zone}", s.authed(s.updateZone))
 	m.HandleFunc("DELETE "+p+"/zones/{zone}", s.authed(s.deleteZone))
+	m.HandleFunc("GET "+p+"/zones/{zone}/serials", s.authed(s.zoneSerials))
+	m.HandleFunc("GET "+p+"/zones/{zone}/delegation", s.authed(s.zoneDelegation))
 	m.HandleFunc("GET "+p+"/zones/{zone}/file", s.authed(s.zoneFile))
 	m.HandleFunc("POST "+p+"/zones/{zone}/import", s.authed(s.importZoneFile))
 	m.HandleFunc("GET "+p+"/zones/{zone}/records", s.authed(s.listRecords))

internal/server/server.go

@@ -19,6 +19,7 @@ import (
 	"jarl/internal/api"
 	"jarl/internal/auth"
 	"jarl/internal/engine"
+	"jarl/internal/nscheck"
 	"jarl/internal/stats"
 	"jarl/internal/store"
 )
@@ -42,6 +43,10 @@ type Config struct {
 	HTTPReq HTTPReq
 	// Socket is the local unix socket's path ("" for none); see socket.go.
 	Socket string
+	// BackupDir and BackupWarnAfter are the backup settings the dashboard
+	// reports on (backup.dir and backup.warn_after).
+	BackupDir       string
+	BackupWarnAfter time.Duration
 }
 
 // Server is the HTTP server.
@@ -57,6 +62,11 @@ type Server struct {
 	statsQ  *stats.Querier
 	// oldSecretWarn limits warnings about rotated tokens' old secrets.
 	oldSecretWarn throttle
+	// expiryWarn limits warnings about expiring tokens to one a day each.
+	expiryWarn throttle
+	// nsChecker, when set, replaces the default name server checker
+	// (tests point it at local DNS servers).
+	nsChecker *nscheck.Checker
 
 	mu   sync.Mutex
 	srvs []*http.Server
@@ -80,6 +90,7 @@ func New(cfg Config, eng *engine.Engine, log *slog.Logger, web fs.FS) *Server {
 		log = slog.Default()
 	}
 	s := &Server{cfg: cfg, eng: eng, store: eng.Store(), log: log, mux: http.NewServeMux(), limiter: newRateLimiter(), web: web}
+	s.expiryWarn.every = 24 * time.Hour
 	if cfg.Stats != nil {
 		s.statsQ = stats.NewQuerier(cfg.Stats)
 	}
@@ -236,6 +247,7 @@ func (s *Server) housekeeping(ctx context.Context) {
 				s.log.Warn("purging sessions failed", "err", err)
 			}
 			s.limiter.cleanup()
+			s.warnExpiringTokens(ctx, time.Now())
 			if s.certs != nil {
 				s.certs.reloadIfChanged()
 			}
@@ -330,6 +342,8 @@ func (s *Server) recoverer(next http.Handler) http.Handler {
 func (s *Server) securityHeaders(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		h := w.Header()
+		// The version is no secret: /healthz reports it too.
+		h.Set("Server", "Jarl/"+s.cfg.Version)
 		h.Set("X-Content-Type-Options", "nosniff")
 		h.Set("X-Frame-Options", "DENY")
 		h.Set("Referrer-Policy", "no-referrer")

internal/store/store.go

@@ -54,6 +54,24 @@ func Open(path string) (*Store, error) {
 	return s, nil
 }
 
+// OpenExisting opens an existing database without creating it or
+// applying migrations, for commands that work alongside a running jarld
+// that may be older than they are ('jarld backup').
+func OpenExisting(path string) (*Store, error) {
+	db, err := sql.Open("sqlite", "file:"+path+"?mode=rw&_pragma=busy_timeout(10000)&_pragma=foreign_keys(1)&_txlock=immediate")
+	if err != nil {
+		return nil, err
+	}
+	if err := db.Ping(); err != nil {
+		db.Close()
+		return nil, err
+	}
+	return &Store{db: db}, nil
+}
+
+// SchemaVersion is the database schema version this Jarl uses.
+func SchemaVersion() int { return len(migrations) }
+
 // Close closes the database.
 func (s *Store) Close() error { return s.db.Close() }
 
@@ -369,3 +387,48 @@ func boolInt(b bool) int {
 func placeholders(n int) string {
 	return strings.TrimSuffix(strings.Repeat("?,", n), ",")
 }
+
+// BackupRecord is a backup that 'jarld backup' made (or failed to make),
+// kept in the meta table for the dashboard.
+type BackupRecord struct {
+	At    time.Time `json:"at"`
+	File  string    `json:"file,omitempty"`
+	Size  int64     `json:"size,omitempty"`
+	Error string    `json:"error,omitempty"`
+}
+
+const (
+	metaBackupOK     = "backup.last_ok"
+	metaBackupFailed = "backup.last_failed"
+)
+
+// RecordBackup stores the outcome of a backup: r.Error empty for a
+// successful one.
+func (s *Store) RecordBackup(ctx context.Context, r BackupRecord) error {
+	key := metaBackupOK
+	if r.Error != "" {
+		key = metaBackupFailed
+	}
+	b, err := json.Marshal(r)
+	if err != nil {
+		return err
+	}
+	return s.SetMeta(ctx, key, string(b))
+}
+
+// LastBackups returns the newest successful and failed backups (nil for
+// none).
+func (s *Store) LastBackups(ctx context.Context) (ok, failed *BackupRecord) {
+	get := func(key string) *BackupRecord {
+		v, err := s.GetMeta(ctx, key)
+		if err != nil {
+			return nil
+		}
+		var r BackupRecord
+		if json.Unmarshal([]byte(v), &r) != nil {
+			return nil
+		}
+		return &r
+	}
+	return get(metaBackupOK), get(metaBackupFailed)
+}

internal/web/static/assets/pages/dashboard.js

@@ -1,4 +1,4 @@
-import { h, mount, time, problemsList, badge, plural } from "../ui.js";
+import { h, mount, time, relTime, problemsList, badge, plural } from "../ui.js";
 import { get, state } from "../api.js";
 
 export async function render(el) {
@@ -14,6 +14,7 @@ export async function render(el) {
       s.adopted ? "named loads Jarl's configuration" : "named does not load Jarl's configuration yet — run 'jarld adopt' on the server"),
     card("Zones", (s.counts || {}).zones ?? "–", plural((s.counts || {}).records || 0, "record") + " · " + plural((s.counts || {}).views || 0, "view")),
     card("Access", plural((s.counts || {}).acls || 0, "ACL"), plural((s.counts || {}).tsig_keys || 0, "TSIG key") + " · " + plural((s.counts || {}).options || 0, "option")),
+    st.backup ? backupCard(st.backup) : null,
   );
   const last = s.last_applied;
   const parts = [h("div", { class: "page-head" },
@@ -28,6 +29,26 @@ export async function render(el) {
       s.drift.map((d) => d.path + " (" + d.reason + ")").join(", "),
       h("div", { class: "small" }, "The next apply refuses to overwrite them unless forced. An administrator can rewrite them with 'jarld resync' (or System → Resync).")));
   }
+  const b = st.backup;
+  if (b && b.stale) {
+    parts.push(h("div", { class: "msg warn" },
+      h("strong", null, "The newest backup is " + relTime(b.last_at).replace(" ago", "") + " old. "),
+      "Backups should run daily: check 'systemctl status jarl-backup.timer' and 'journalctl -u jarl-backup' on the server."));
+  }
+  if (b && b.last_error) {
+    parts.push(h("div", { class: "msg err" },
+      h("strong", null, "The last backup failed "), time(b.last_error_at), ": " + b.last_error,
+      h("div", { class: "small" }, "See 'journalctl -u jarl-backup' on the server.")));
+  }
+  const exp = st.expiring_tokens || [];
+  if (exp.length) {
+    const now = new Date();
+    parts.push(h("div", { class: "msg warn" },
+      h("strong", null, exp.length === 1 ? "An API token expires soon: " : "API tokens expire soon: "),
+      exp.map((t, i) => [i ? ", " : "", h("a", { href: state.me.admin ? "#/users" : "#/account" }, t.username + "/" + t.name), " ",
+        new Date(t.expires_at) < now ? h("span", null, "expired ", time(t.expires_at)) : h("span", null, "expires ", time(t.expires_at))]),
+      h("div", { class: "small" }, "Rotate a token to give it a new lifetime and secret, keeping its permissions (API tokens → Rotate, or 'jarl token rotate'). Then update whatever uses it.")));
+  }
   parts.push(h("div", { class: "panel" },
     h("h2", null, "Last change"),
     last ? h("div", null,
@@ -50,6 +71,15 @@ export async function render(el) {
   mount(el, parts);
 }
 
+function backupCard(b) {
+  let value;
+  if (b.last_error) value = badge("failed", "err");
+  else if (!b.last_at) value = badge("none yet", "");
+  else value = badge(relTime(b.last_at), b.stale ? "warn" : "ok");
+  const sub = b.last_at ? h("span", { class: "mono small", title: b.last_file }, b.last_file.split("/").pop()) : "jarl-backup.timer writes to " + b.dir;
+  return card("Backup", value, sub);
+}
+
 function card(label, value, sub) {
   return h("div", { class: "card" }, h("div", { class: "label" }, label), h("div", { class: "value" }, value), sub ? h("div", { class: "sub" }, sub) : null);
 }

internal/web/static/assets/pages/users.js

@@ -104,7 +104,7 @@ export function tokenTable(tokens, redraw) {
     { label: "Name", render: (t) => [h("strong", null, t.name), rotationNote(t)] },
     { label: "ID", render: (t) => "jarl_" + t.public_id + "_…", tdClass: "mono small" },
     { label: "Restricted to", render: (t) => t.grant_strings && t.grant_strings.length ? h("div", { class: "grants" }, t.grant_strings.map((g) => h("div", null, g))) : h("span", { class: "muted" }, "all of the user's permissions"), tdClass: "small" },
-    { label: "Expires", render: (t) => t.expires_at ? time(t.expires_at) : h("span", { class: "muted" }, "never") },
+    { label: "Expires", render: (t) => t.expires_at ? [time(t.expires_at), expiryBadge(t)] : h("span", { class: "muted" }, "never") },
     { label: "Last used", render: (t) => t.last_used_at ? [time(t.last_used_at), " ", h("span", { class: "muted small" }, t.last_used_ip || "")] : h("span", { class: "muted" }, "never") },
     { label: "", render: (t) => [
       h("button", { class: "small", onclick: () => rotateDialog(t, redraw) }, "Rotate"), " ",
@@ -117,6 +117,16 @@ export function tokenTable(tokens, redraw) {
   ], tokens, { empty: "No tokens." });
 }
 
+// expiryBadge flags a token that expires within 14 days (the server's
+// TokenExpiryWarning) or has expired.
+function expiryBadge(t) {
+  if (t.revoked_at) return null;
+  const left = new Date(t.expires_at) - Date.now();
+  if (left < 0) return [" ", badge("expired", "err")];
+  if (left < 14 * 86400 * 1000) return [" ", badge("soon", "warn")];
+  return null;
+}
+
 // rotationNote says when a token last got a new secret, and whether its
 // old secret still works.
 function rotationNote(t) {

internal/web/static/assets/pages/zone.js

@@ -14,6 +14,7 @@ export async function render(el, params, query) {
     ["settings", "Settings"],
     ["options", "Options"],
     zone.type === "primary" ? ["file", "Zone file"] : null,
+    ["primary", "secondary", "mirror"].includes(zone.type) ? ["servers", "Name servers"] : null,
   ].filter(Boolean);
   const body = h("div");
   mount(el,
@@ -34,6 +35,106 @@ export async function render(el, params, query) {
   else if (tab === "settings") await settingsTab(body, zone);
   else if (tab === "options") await optionsEditor(body, { scope: "zone", scope_id: zone.id, catalogScope: "zone:" + zone.type, title: "Zone options" });
   else if (tab === "file") await fileTab(body, zone);
+  else if (tab === "servers") await serversTab(body, zone);
+}
+
+// ---- name servers: serials on the other servers, and the delegation
+
+const nsStatus = {
+  self: ["this server", ""], in_sync: ["in sync", "ok"], answered: ["answers", "ok"],
+  behind: [null, "warn"], ahead: [null, "warn"], not_authoritative: ["doesn't serve the zone", "err"],
+  refused: ["refused", "err"], unreachable: ["no answer", "err"], no_route: ["no route from here", ""],
+  no_address: ["no address", "err"], error: ["error", "err"],
+};
+
+function nsBadge(r) {
+  let [text, kind] = nsStatus[r.status] || [r.status, "err"];
+  if (r.status === "behind") text = "behind by " + -r.diff;
+  if (r.status === "ahead") text = "ahead by " + r.diff;
+  return [badge(text, kind), r.error && r.status !== "unreachable" ? h("div", { class: "muted small" }, r.error) : null];
+}
+
+function nsTable(servers, withSource) {
+  return table([
+    { label: "Server", render: (r) => r.name ? display(r.name) : h("span", { class: "muted" }, "–"), tdClass: "mono" },
+    { label: "Address", render: (r) => (r.address || "–") + (r.port ? " port " + r.port : ""), tdClass: "mono small" },
+    withSource ? { label: "From", render: (r) => r.source, tdClass: "small" } : null,
+    { label: "Status", render: nsBadge },
+    { label: "Serial", render: (r) => r.serial ?? "–", tdClass: "mono" },
+    { label: "Time", render: (r) => r.rtt_ms ? r.rtt_ms + " ms" : "", tdClass: "small muted" },
+  ].filter(Boolean), servers, { empty: "No other name servers to ask." });
+}
+
+async function serversTab(el, zone) {
+  const serials = h("div");
+  const loadSerials = async () => {
+    mount(serials, h("div", { class: "muted" }, "Asking the name servers…"));
+    try {
+      const r = await get("/zones/" + zone.id + "/serials");
+      const src = { named: "served by named here", jarl: "Jarl's; named wasn't asked" }[r.serial_source];
+      mount(serials,
+        h("p", null, "This server: serial ", h("strong", { class: "mono" }, r.serial ?? "unknown"), src ? h("span", { class: "muted" }, " (" + src + ")") : null),
+        nsTable(r.servers, true),
+        r.notes.map((n) => h("p", { class: "muted small" }, n)),
+        h("p", { class: "muted small" }, "Checked ", time(r.checked_at), "."));
+    } catch (err) {
+      mount(serials, errorBox(err));
+    }
+  };
+  const parts = [h("div", { class: "panel" },
+    h("div", { class: "page-head" },
+      h("h2", { class: "grow" }, zone.type === "primary" ? "Serials on the secondaries" : "Serials on the primaries"),
+      h("button", { class: "small", onclick: loadSerials }, "Check again")),
+    h("p", { class: "muted small" }, zone.type === "primary"
+      ? "Asks the servers in the zone's NS records and its also-notify targets which serial they serve. A secondary that is behind hasn't picked up the latest change yet."
+      : "Asks the zone's primaries which serial they serve, to see whether this server's copy is up to date."),
+    serials)];
+  if (zone.type === "primary") {
+    const out = h("div");
+    const resolver = input({ placeholder: "the server's own" });
+    const run = async (e) => {
+      e.preventDefault();
+      mount(out, h("div", { class: "muted" }, "Asking the parent zone's servers…"));
+      try {
+        const q = resolver.value.trim() ? { resolver: resolver.value.trim() } : {};
+        mount(out, delegationReport(await get("/zones/" + zone.id + "/delegation", q)));
+      } catch (err) {
+        mount(out, errorBox(err));
+      }
+    };
+    parts.push(h("div", { class: "panel" },
+      h("h2", null, "Delegation from the parent zone"),
+      h("p", { class: "muted small" }, "For zones on the internet: checks that the parent zone (for adbraun.lol, \"lol\") points to this zone's name servers with the right glue addresses, and that each of them serves the zone. A zone used only on your own network isn't delegated, and doesn't need this."),
+      h("form", { class: "inline-form", onsubmit: run },
+        h("div", { class: "f" }, h("label", null, "Resolver (optional)"), resolver),
+        h("div", { class: "f" }, h("label", null, " "), h("button", { type: "submit" }, "Check delegation"))),
+      out));
+  }
+  mount(el, parts);
+  await loadSerials();
+}
+
+function delegationReport(r) {
+  const lvl = { ok: "ok", info: "", warning: "warn", error: "err" };
+  const parents = {};
+  for (const p of r.parent_servers) {
+    const say = p.status === "delegated" ? "delegate to " + p.ns.map(display).join(", ") : p.status === "not_delegated" ? "no delegation" : p.status + (p.error ? ": " + p.error : "");
+    (parents[say] = parents[say] || []).push(display(p.name));
+  }
+  return [
+    r.parent ? h("p", null, "Parent zone ", h("strong", { class: "mono" }, display(r.parent)), h("span", { class: "muted" }, " (found through " + r.resolver + ")")) : null,
+    Object.keys(parents).length ? h("ul", { class: "small" }, Object.entries(parents).map(([say, who]) =>
+      h("li", null, who.length > 1 ? who[0] + " and " + (who.length - 1) + " more: " : who[0] + ": ", say))) : null,
+    h("div", null, r.findings.map((f) => h("div", { class: "msg " + lvl[f.level] }, f.message))),
+    r.glue.length ? table([
+      { label: "Glue for", render: (g) => display(g.name), tdClass: "mono" },
+      { label: "Parent", render: (g) => g.parent.join(", ") || "none", tdClass: "mono small" },
+      { label: "Zone", render: (g) => g.jarl.join(", ") || "none", tdClass: "mono small" },
+      { label: "", render: (g) => g.match ? badge("matches", "ok") : badge("differs", "warn") },
+    ], r.glue) : null,
+    r.servers.length ? [h("h3", null, "Delegated name servers"), nsTable(r.servers, false)] : null,
+    r.notes.map((n) => h("p", { class: "muted small" }, n)),
+  ];
 }
 
 async function deleteZone(zone) {

scripts/e2e.sh

@@ -104,6 +104,8 @@ bind:
   named_checkzone: $CHECKZONE
   rndc: $RNDC
   rndc_args: ["-k", "$E/legacy/rndc.key", "-s", "127.0.0.1", "-p", "$CTL"]
+backup:
+  dir: $E/backups
 EOF
 cd $E
 echo "secret-pass-123" | $JARLD -c jarl.yaml init --password-stdin --token --nameservers ns1.adbraun.test. --contact admin@adbraun.test > init.out 2>&1 || { cat init.out; exit 1; }
@@ -286,6 +288,49 @@ run "acme set --view external only" 0 $J acme unset adbraun.test --view internal
 expect_dig "acme TXT removed from internal" "" _acme-challenge.adbraun.test TXT
 run "zone file" 0 $J zone file adbraun.test --view external
 contains "_acme-challenge"
+# Name server checks: the external zone's NS (203.0.113.1) doesn't answer;
+# the internal one lists only this server.
+run "zone serials: internal, only this server" 0 $J zone serials adbraun.test --view internal
+contains "this server"
+run "zone serials: unreachable secondary (exit 2)" 2 $J zone serials adbraun.test --view external
+contains "NO ANSWER"
+
+# Server header and token expiry warnings
+if curl -sI http://127.0.0.1:$HTTP/healthz | grep -qi "^Server: Jarl/"; then ok "Server header"; else bad "no Server: Jarl header"; fi
+SOON=$($J token create soon --expires 3d -q)
+run "expiring token works" 0 $J --token "$SOON" whoami
+contains "warning: the API token expires"
+run "token list --expiring" 0 $J token list --expiring
+contains "soon"
+run "status lists the expiring token" 0 $J status
+contains "Token:.*soon"
+
+# Backups and restore
+run "jarld backup" 0 $JARLD -c jarl.yaml backup
+contains "Database backed up to $E/backups/jarl-"
+run "status shows the backup" 0 $J status
+contains "Backup:.*$E/backups/jarl-"
+run "restore refused while jarld runs" 1 $JARLD -c jarl.yaml restore latest --yes
+contains "jarld is running"
+run "a second jarld is refused" 1 $JARLD -c jarl.yaml serve
+contains "in use by a running jarld"
+run "record after the backup" 0 $J record add adbraun.test afterbackup A 192.168.1.99 --view internal
+expect_dig "afterbackup A" "192.168.1.99" afterbackup.adbraun.test A
+kill $JARLD_PID; wait $JARLD_PID 2>/dev/null
+run "jarld restore lists backups" 0 $JARLD -c jarl.yaml restore
+contains "jarl-.*\.db"
+run "jarld restore latest" 0 $JARLD -c jarl.yaml restore latest --yes
+contains "previous database is in"
+$JARLD -c jarl.yaml serve >> jarld.log 2>&1 &
+JARLD_PID=$!
+for i in $(seq 50); do curl -s -o /dev/null http://127.0.0.1:$HTTP/healthz && break; sleep 0.2; done
+run "restored database lacks the later record" 0 $J record list adbraun.test --view internal
+if echo "$OUT" | grep -q afterbackup; then bad "afterbackup still there after restore"; else ok "afterbackup gone after restore"; fi
+run "resync after restore" 0 $J resync --force
+expect_dig "afterbackup gone from named" "" afterbackup.adbraun.test A
+run "audit shows the restore" 0 $J audit --action system.restore
+contains "system.restore"
+
 run "status at the end" 0 $J status
 contains "Drift:.*none"
 

scripts/mkdeb.sh

@@ -6,12 +6,12 @@
 # The package installs, with Debian's paths:
 #   /usr/bin/jarld, /usr/bin/jarl
 #   /usr/share/bash-completion/completions/{jarl,jarld}
-#   /usr/lib/systemd/system/jarl.service
+#   /usr/lib/systemd/system/{jarl.service,jarl-backup.service,jarl-backup.timer}
 #   /usr/lib/sysusers.d/jarl.conf, /usr/lib/tmpfiles.d/jarl.conf
 #   /usr/share/doc/jarl/ (README, CHANGELOG, docs, contrib)
 # and its scripts (deploy/debian) create the jarl user and directories and
-# /etc/jarl/jarl.yaml on the first install, restart a running jarld on
+# /etc/jarl/jarl.yaml on the first install, turn on the daily backup timer,
-# upgrades, and stop it on removal. No file in /etc belongs to the package,
+# restart a running jarld on upgrades, and stop both on removal. No file in /etc belongs to the package,
 # so an upgrade never asks about configuration files.
 #
 # Variables: DEB_ARCH (Debian architecture, e.g. amd64, arm64, armhf),