jarl/docs/operations.md

# Running Jarl

## Day to day

| Task | How |
|------|-----|
| Is everything healthy? | `jarl status`, or the web UI dashboard: named reachable, adoption, drift, open changesets, configuration problems, the newest backup, tokens about to expire |
| Are the secondaries up to date? | `jarl zone serials ZONE`, or the zone's Name servers tab ([below](#checking-name-servers-and-the-delegation)) |
| Back up / restore | Daily by `jarl-backup.timer`; by hand `sudo jarld backup`; restore with `jarld restore` ([below](#backups)) |
| DNSSEC | `jarl zone dnssec status ZONE`, or the zone's DNSSEC tab ([dnssec.md](dnssec.md)) |
| What changed and who did it? | `jarl history`, `jarl changeset show ID` (with diff), `jarl audit` |
| Undo a change | `jarl changeset revert ID`, review it, then `jarl changeset apply NEW_ID` (or History → Revert in the UI) |
| Check the configuration | `jarl check`: full validation plus `named-checkconf -z` |
| Logs | `journalctl -u jarl` (jarld, including `msg=audit` lines), `journalctl -u named` |
| Restart / reload TLS | `systemctl restart jarl` / `systemctl reload jarl` |
| Work on the server itself | `sudo jarl ...`: root uses jarld's local socket and needs no token ([cli.md](cli.md#on-the-server-itself)) |
| Renew or replace a token | `jarl token rotate NAME --grace 1h` (same grants, new secret; [permissions.md](permissions.md#rotating-a-token)). `jarl token list --all --expiring` shows those that expire within 14 days |
| Upgrade Jarl | Debian packages: build the new ones and `sudo apt install ./dist/jarl*_amd64.deb` (server and CLI); otherwise `make && sudo make install && sudo systemctl restart jarl`. jarld upgrades its database when it starts |

Restarting jarld doesn't affect DNS: named keeps serving the files it
loaded. jarld is only needed to make changes.

## Backups

Everything Jarl knows is in its database (`/var/lib/jarl/jarl.db`): zones,
records, views, ACLs, TSIG keys, options, users, tokens, history and the
audit log. named's files are generated from it. The one exception is the
DNSSEC keys of signed zones, which named keeps in Jarl's key directory
(`/var/lib/bind/jarl/dnssec-keys`): every backup stores those too, inside the
backup file ([dnssec.md](dnssec.md#keys-backups-and-moving-servers)).

### The daily backup

The `jarl-backup.timer` systemd timer runs `jarld backup` once a day. The
Debian package turns it on; after `make install`, turn it on yourself:

```sh
systemctl enable --now jarl-backup.timer
systemctl list-timers jarl-backup.timer   # when it runs next
```

Each run:

1. writes a consistent copy of the database with SQLite's `VACUUM INTO`
   (safe while jarld runs and changes things) to a temporary file in the
   backup directory;
2. stores the DNSSEC keys in it (every file of the key directory, private
   keys included);
3. checks the copy with SQLite's `integrity_check`, and only then gives it
   its name, `jarl-<UTC time>.db`, e.g. `jarl-20261008T031512Z.db`. A file
   with that name is always a complete, intact backup;
4. deletes backups older than `retention`, but always keeps the newest
   `keep_at_least`, however old they are. If backups stop for a while (the
   machine was off, the disk was full), the old ones aren't all deleted by
   the next run. Ages come from the time in the name, not the file's
   modification time, which copying the directory would change. Only files
   named `jarl-<time>.db` (and `stats-<time>.db`) are ever deleted, so
   anything else you put in the directory is safe;
5. records the result in the database, so the dashboard and `jarl status`
   show when the newest backup was made, and whether the last attempt
   failed.

A failed run exits non-zero, so it shows in `systemctl --failed` and
`journalctl -u jarl-backup`.

What to keep and where is in `jarl.yaml`
([configuration.md](configuration.md#backups)); when it runs is the
timer's:

```yaml
backup:
  dir: /var/backups/jarl
  retention: 30d       # delete backups older than this (0 keeps them all)...
  keep_at_least: 7     # ...but always keep the newest 7
  stats: false         # also back up the statistics database
  warn_after: 2d       # dashboard warning when the newest backup is older
```

The settings are read by `jarld backup` itself each time it runs, from the
same file jarld uses, and checked the same way: a mistake there stops
`jarld` from starting rather than quietly breaking the backups.

To run at a different time, override the timer, and keep `warn_after`
above the new interval:

```sh
systemctl edit jarl-backup.timer
#   [Timer]
#   OnCalendar=
#   OnCalendar=*-*-* 02:30
```

The backup service runs as the `jarl` user in a sandbox that may only
write to `/var/lib/jarl` and `/var/backups/jarl` (created by
`tmpfiles.d/jarl.conf`, mode 0700). It has one capability,
`CAP_DAC_READ_SEARCH`, to read named's DNSSEC private keys, which only
named's user may read. To keep backups elsewhere, set
`backup.dir`, create the directory (owned by `jarl`, mode 0700) and allow
it:

```sh
systemctl edit jarl-backup.service
#   [Service]
#   ReadWritePaths=/srv/backups/jarl
```

Backups on the same disk don't survive the disk. Copy
`/var/backups/jarl` elsewhere (rsync, restic, borg, your usual backup) as
well.

### By hand

```sh
sudo jarld backup                       # into backup.dir, as the timer does
sudo jarld backup /root/before-upgrade.db  # one copy, never deleted automatically
sudo jarld restore                       # list the backups in backup.dir
sudo systemctl start jarl-backup         # or exactly what the timer runs
```

Query statistics live in a separate database (`/var/lib/jarl/stats.db`).
DNS doesn't need it, so it is backed up only with `backup.stats: true`, as
`stats-<UTC time>.db`.

Run it as root (files it creates are given to `jarl`): named's DNSSEC
private keys are readable only by named's user. As another user it fails
when there are keys; `--without-keys` backs up only the database.

The database holds password and token *hashes*, and TSIG secrets in the
clear, and backups hold the DNSSEC private keys too. Backups are written
with mode 0600 into a 0700 directory; keep any copies just as private.

### Restoring

`jarld restore` replaces the database with a backup. jarld must be
stopped: the restore refuses to run while jarld holds the database (and
jarld won't start while a restore runs).

```sh
sudo jarld restore                  # list the backups, newest first
sudo systemctl stop jarl
sudo jarld restore latest           # or: sudo jarld restore /var/backups/jarl/jarl-20261008T031512Z.db
```

It checks the backup's integrity and that this Jarl can read it (a backup
from a newer Jarl is refused), shows what the backup and the current
database hold (zones, records, users, last change), and asks before
changing anything (`--yes` skips the question). Then it:

* moves the current database aside as
  `jarl.db.before-restore-<UTC time>` (with its `-wal` and `-shm` files),
  so a restore can be undone the same way;
* copies the backup in place, upgrades it to the current schema if it is
  older, and records the restore in its audit log;
* puts the DNSSEC keys stored in the backup back into the key directory,
  owned by named's user (`bind.named_user`) with the modes they had. Key
  files that are there but not in the backup are left alone.
  `--without-keys` skips this; without it, a backup with keys needs root.

Then start jarld and bring named in line with the restored database:

```sh
sudo systemctl start jarl
jarl status                 # drift lists the files that differ
jarl resync --dry-run       # what would change
jarl resync --force         # write everything and reload named
```

Things to know:

* **Zone serials.** An older backup can carry lower zone serials than named
  last served. Secondaries ignore a serial that went backwards, and named
  refuses one for a signed zone. Jarl handles it: whenever it writes a zone
  file whose serial isn't above the one named has loaded, it raises the
  serial past named's (the resync result and the audit log show it). Check
  the secondaries afterwards with `jarl zone serials ZONE`.
* **DNSSEC keys** must be back before named loads a signed zone, or named
  makes new keys that the parent's DS record doesn't match. On a new
  machine, restore before adopting.
* **Tokens and passwords** go back to what they were when the backup was
  made: tokens created since are gone, and revoked ones work again. Rotate
  or revoke as needed.
* **Statistics**: to restore `stats-<time>.db`, stop jarld and copy it over
  `/var/lib/jarl/stats.db` (owner `jarl`, mode 0640).
* **On a new machine**: install Jarl, restore the backup, start jarld and
  run `jarl resync --force`; then adopt named as in
  [install.md](install.md) if this named doesn't include Jarl's
  configuration yet.

## Drift: files changed outside Jarl

Jarl keeps a checksum of every file it writes. If someone edits
`/etc/bind/jarl/named.conf` or a zone file by hand (or a restore puts old
files back), `jarl status` and the dashboard show **drift**, and changes
that would overwrite those files are refused with `409 drift`:

```
error: files managed by Jarl were changed outside Jarl: /var/lib/bind/jarl/internal/adbraun.lol.zone; review them and apply with force to overwrite
```

Decide which version is right:

* **Jarl's** (the usual case): `jarl resync --force` rewrites every file from
  the database and reloads named. Any single change with `--force` also
  overwrites the drifted files.
* **The hand edit**: put the same change into Jarl (for example
  `jarl zone import ZONE /var/lib/bind/jarl/internal/adbraun.lol.zone` for
  records), then `jarl resync --force`.

## When an apply fails

Jarl checks everything it can before it touches named, so failures are
rare. When one happens, nothing is left half-done:

| Stage | What you see | State afterwards |
|-------|--------------|------------------|
| validation | 422 with `problems` | Nothing written |
| named-checkconf / named-checkzone | 422 `validation_failed` with the checker's messages in `check`, mapped to the zone/option that caused them | Nothing written |
| `rndc reconfig` / `reload` fails, or a zone doesn't load within `reload_timeout` | 502 `bind_failed` | Previous files restored and named reloaded |

A failed change is never recorded as applied. A staged changeset stays open,
so you can fix it and apply again, and the failure is in the audit log
(`jarl audit --outcome failure`). named's own log usually explains a load
failure: `journalctl -u named`.

If named isn't running, changes fail at the reload step and are rolled back.
Start named, then apply again.

## Recovering access

Revoking a token never locks you out: web logins don't depend on tokens,
and root on the server can always use `sudo jarl` through the local
socket, without a token.

| Problem | Fix (on the server) |
|---------|---------------------|
| Revoked or lost the CLI's token | `sudo jarl token create cli --user admin`, or Account → API tokens in the web UI |
| Forgot the admin password | `sudo jarl user update admin --password`, or (also with jarld stopped) `sudo -u jarl jarld user passwd admin` |
| Locked out by the login rate limit | wait 15 minutes, or `systemctl restart jarl` (the limiter is in memory) |
| Lost every admin account | `sudo -u jarl jarld user add --admin rescue` |
| Lost the phone with the authenticator app, and the recovery codes | another administrator: Users → Reset 2FA, or `jarl user totp reset NAME`; on the server: `sudo -u jarl jarld user totp reset NAME` |
| Need a token without the web UI | `sudo jarl token create cli --user admin`, or `sudo -u jarl jarld user token admin cli` |

## A secondary server

A second name server runs its own named and jarld; its zones are of type
`secondary` and copy the primary's records by zone transfer, so you only
ever edit records on the primary. With both named in the CLI configuration
([cli.md](cli.md#several-servers-and-groups)):

```sh
# 1. One TSIG key with the same secret on both servers.
jarl -s ns1 tsig create xfer                     # prints the secret
jarl -s ns2 tsig create xfer --secret 'THE-SECRET'

# 2. On the primary: let the key transfer the zone, and list ns2 as a
#    name server (named sends NOTIFY to the zone's NS records).
jarl -s ns1 option set allow-transfer '{ key xfer; }' --zone adbraun.lol
jarl -s ns1 record add adbraun.lol @ NS ns2.adbraun.lol.

# 3. On the secondary: the zone, copied from the primary with the key.
jarl -s ns2 zone create adbraun.lol --type secondary --primary "192.0.2.53 key xfer"
```

If ns2 isn't one of the zone's NS records, add `also-notify { ADDRESS; };`
on the primary so it learns about changes quickly. With views, each view's
zones are transferred separately; match the secondary's requests to the
right view, for example with `match-clients { key xfer; ... };`.

Things that both servers need, such as ACLs, can be changed on both at once
with a group (`jarl -s dns acl update ...`): the change is checked on both
first and applied only if both accept it.

## Checking name servers and the delegation

Two checks ask other name servers about a zone over DNS. They only run when
you ask: on the zone's **Name servers** tab in the web UI, or with the CLI.
Nothing checks in the background.

### Are the other servers serving the current serial?

```
$ jarl zone serials adbraun.lol
adbraun.lol, primary, serial 2026100814 (served by named here)

SERVER           ADDRESS       FROM         STATUS       SERIAL      TIME
ns1.adbraun.lol  192.168.1.53  NS record    this server  2026100814
ns2.adbraun.lol  192.168.1.54  NS record    BEHIND by 2  2026100812  1 ms
```

It works for both sides of a zone transfer:

* **Primary zones**: asks the servers in the zone's NS records and its
  `also-notify` targets (the secondaries) which serial they serve, and
  compares it with what named on this server serves. A secondary that is
  behind hasn't picked up the latest change: check its logs, and that
  `allow-transfer` on this server lets it transfer the zone.
* **Secondary and mirror zones**: asks the zone's `primaries` (following
  named `remote-servers` lists, with their ports and TSIG keys), to see
  whether this server's copy is up to date.

Name server addresses come from Jarl's own A/AAAA records in the zone's
view where it has them, otherwise from a DNS lookup on the server. This
server itself is recognised by its addresses and isn't asked. Statuses:
*in sync*, *behind/ahead by N* (serials compare as in RFC 1982, so a
wrapped serial counts as ahead), *doesn't serve the zone* (the server
answers but isn't authoritative for it), *refused*, *no answer*, and *no
route from here* (typically an IPv6 address on a server without IPv6,
which isn't counted as a problem).

With views, other servers answer from the view that matches this server's
address (or TSIG key), exactly as they do for its zone transfers and
NOTIFYs; queries to primaries and also-notify targets that name a key are
signed with it.

`jarl zone serials` exits with 2 when a server isn't serving the current
serial, so it can run from cron or a monitoring system.

### Does the internet see the zone the way Jarl does?

For a zone published on the internet, the parent zone (`lol.` for
adbraun.lol, set through your registrar) must delegate it to the right name
servers, with glue addresses for name servers inside the zone.

```
$ jarl zone delegation adbraun.lol --view external
Zone:                              adbraun.lol
Parent zone:                       lol. (found through 127.0.0.53:53)
  a.nic.lol and 3 more:            delegate to ns1.adbraun.lol, ns2.adbraun.lol
NS at the parent:                  ns1.adbraun.lol, ns2.adbraun.lol
NS in the zone:                    ns1.adbraun.lol, ns2.adbraun.lol
Glue ns1.adbraun.lol:              203.0.113.53 (matches)
...
ok:      The parent delegates adbraun.lol to the same name servers as its NS records: ...
ok:      Every delegated name server answers for the zone with the current serial.
```

It finds the parent zone through the server's resolver (or `--resolver
9.9.9.9`), asks each of the parent's servers directly, and reports:

* name servers the parent delegates to that the zone's NS records don't
  list, and the other way round;
* glue that is missing, or differs from the zone's A/AAAA records;
* delegated servers that don't serve the zone (a *lame delegation*), don't
  answer, or serve another serial.

A zone that is only used on your own network isn't delegated at all, and
the check says so in a single note, not as a problem; it also exits with 0.
It exits with 2 only for errors (a lame delegation, missing glue). With
split DNS, check the zone in the view the internet sees (usually
`external`).

## Taking Jarl out of the picture

The generated files are plain BIND configuration. To stop using Jarl,
disable the service. named keeps working with `/etc/bind/jarl/` exactly as
it is. To go back to the configuration you had before adopting:

```sh
cp /etc/bind/named.conf.pre-jarl-<date> /etc/bind/named.conf
rndc reconfig
```

## Failed logins in the journal

A failed login to the web UI or the API (a wrong password, or a wrong
two-factor code), and a refused API token, are written to the journal as
well as the audit log, with the address first:

```
level=WARN msg="login failed" ip=203.0.113.9 user=admin reason="wrong username or password"
level=WARN msg="token refused" ip=203.0.113.9 reason="wrong token secret"
```

Guard's `jarl` preset, or a fail2ban filter, can ban such addresses at
the firewall (with Housecarl's ban lists). The address comes before the
user name, which a caller chooses, so a pattern anchored on
`msg="login failed" ip=` can't be fooled by a name. With `log.format:
json` they are JSON objects with the same fields in the same order.

## Monitoring

* `GET /healthz` (no authentication) reports that jarld is up.
* Prometheus: `/metrics` covers named up, configuration problems, drift,
  blocklist failures, backup age, DNSSEC steps waiting at the parent and
  expiring tokens, with example alerts ([metrics.md](metrics.md)).
* On the server, a check can use the local socket and needs no token:
  `sudo jarl status -o json`.
* `GET /api/v1/status` with a read-only token reports named reachability,
  drift and configuration problems. It is suitable for a monitoring check:
  ```sh
  jarl --token @/etc/jarl/monitor.token status -o json | jq -e '.engine.drift == [] and .engine.adopted'
  ```
* The audit log records failed logins as `auth.login_failed` /
  `auth.login_blocked`, and rejected tokens as `auth.token_failed`.
* Backups: `.backup.stale` and `.backup.last_error` in `jarl status -o
  json` (administrators), and `systemctl --failed` for the timer's runs:
  ```sh
  sudo jarl status -o json | jq -e '.backup.stale == false and (.backup.last_error // "") == ""'
  ```
* Secondaries: `jarl zone serials ZONE` exits with 2 when one isn't serving
  the current serial.
* Tokens: jarld logs a warning once a day for every token that expires
  within 14 days (`journalctl -u jarl | grep 'token expires'`), the
  dashboard lists them, and `jarl` prints a warning on stderr whenever it
  uses one, so a certbot hook's log shows it well before renewals fail.