jarl/docs/operations.md

# Running Jarl

## Day to day

| Task | How |
|------|-----|
| Is everything healthy? | `jarl status`, or the web UI dashboard: named reachable, adoption, drift, open changesets, configuration problems |
| What changed and who did it? | `jarl history`, `jarl changeset show ID` (with diff), `jarl audit` |
| Undo a change | `jarl changeset revert ID`, review it, then `jarl changeset apply NEW_ID` (or History → Revert in the UI) |
| Check the configuration | `jarl check`: full validation plus `named-checkconf -z` |
| Logs | `journalctl -u jarl` (jarld, including `msg=audit` lines), `journalctl -u named` |
| Restart / reload TLS | `systemctl restart jarl` / `systemctl reload jarl` |
| Work on the server itself | `sudo jarl ...`: root uses jarld's local socket and needs no token ([cli.md](cli.md#on-the-server-itself)) |
| Renew or replace a token | `jarl token rotate NAME --grace 1h` (same grants, new secret; [permissions.md](permissions.md#rotating-a-token)) |
| Upgrade Jarl | Debian package: build the new one and `sudo apt install ./dist/jarl_*.deb`; otherwise `make && sudo make install && sudo systemctl restart jarl`. jarld upgrades its database when it starts |

Restarting jarld doesn't affect DNS: named keeps serving the files it
loaded. jarld is only needed to make changes.

## Backups

Everything Jarl knows is in its database. The BIND files can be generated
from it again at any time.

```sh
sudo -u jarl jarld backup /var/backups/jarl/jarl-$(date +%F).db   # consistent copy, safe while jarld runs
```

For example, as a daily timer or cron job:

```sh
# /etc/cron.daily/jarl-backup
#!/bin/sh
set -e
install -d -o jarl -g jarl -m 0700 /var/backups/jarl
sudo -u jarl jarld backup /var/backups/jarl/jarl-$(date +%F).db
find /var/backups/jarl -name 'jarl-*.db' -mtime +30 -delete
```

Query statistics live in a separate database (`/var/lib/jarl/stats.db`).
It isn't needed to restore DNS, so back it up only if you want to keep the
history.

The database contains password and token *hashes* and the TSIG secrets in
the clear, so keep backups private.

### Restoring

```sh
systemctl stop jarl
install -o jarl -g jarl -m 0600 /var/backups/jarl/jarl-2026-10-01.db /var/lib/jarl/jarl.db
rm -f /var/lib/jarl/jarl.db-wal /var/lib/jarl/jarl.db-shm
systemctl start jarl
jarl resync --dry-run     # what differs between the restored database and the files
jarl resync --force       # write everything and reload named
```

An older backup can carry lower zone serials than named last served. Your
own named doesn't mind, but secondaries and caches ignore a serial that went
backwards. If you have secondaries, raise the serials after a restore:
`jarl zone update ZONE --serial <higher>`.

## Drift: files changed outside Jarl

Jarl keeps a checksum of every file it writes. If someone edits
`/etc/bind/jarl/named.conf` or a zone file by hand (or a restore puts old
files back), `jarl status` and the dashboard show **drift**, and changes
that would overwrite those files are refused with `409 drift`:

```
error: files managed by Jarl were changed outside Jarl: /var/lib/bind/jarl/internal/adbraun.lol.zone; review them and apply with force to overwrite
```

Decide which version is right:

* **Jarl's** (the usual case): `jarl resync --force` rewrites every file from
  the database and reloads named. Any single change with `--force` also
  overwrites the drifted files.
* **The hand edit**: put the same change into Jarl (for example
  `jarl zone import ZONE /var/lib/bind/jarl/internal/adbraun.lol.zone` for
  records), then `jarl resync --force`.

## When an apply fails

Jarl checks everything it can before it touches named, so failures are
rare. When one happens, nothing is left half-done:

| Stage | What you see | State afterwards |
|-------|--------------|------------------|
| validation | 422 with `problems` | Nothing written |
| named-checkconf / named-checkzone | 422 `validation_failed` with the checker's messages in `check`, mapped to the zone/option that caused them | Nothing written |
| `rndc reconfig` / `reload` fails, or a zone doesn't load within `reload_timeout` | 502 `bind_failed` | Previous files restored and named reloaded |

A failed change is never recorded as applied. A staged changeset stays open,
so you can fix it and apply again, and the failure is in the audit log
(`jarl audit --outcome failure`). named's own log usually explains a load
failure: `journalctl -u named`.

If named isn't running, changes fail at the reload step and are rolled back.
Start named, then apply again.

## Recovering access

Revoking a token never locks you out: web logins don't depend on tokens,
and root on the server can always use `sudo jarl` through the local
socket, without a token.

| Problem | Fix (on the server) |
|---------|---------------------|
| Revoked or lost the CLI's token | `sudo jarl token create cli --user admin`, or Account → API tokens in the web UI |
| Forgot the admin password | `sudo jarl user update admin --password`, or (also with jarld stopped) `sudo -u jarl jarld user passwd admin` |
| Locked out by the login rate limit | wait 15 minutes, or `systemctl restart jarl` (the limiter is in memory) |
| Lost every admin account | `sudo -u jarl jarld user add --admin rescue` |
| Need a token without the web UI | `sudo jarl token create cli --user admin`, or `sudo -u jarl jarld user token admin cli` |

## A secondary server

A second name server runs its own named and jarld; its zones are of type
`secondary` and copy the primary's records by zone transfer, so you only
ever edit records on the primary. With both named in the CLI configuration
([cli.md](cli.md#several-servers-and-groups)):

```sh
# 1. One TSIG key with the same secret on both servers.
jarl -s ns1 tsig create xfer                     # prints the secret
jarl -s ns2 tsig create xfer --secret 'THE-SECRET'

# 2. On the primary: let the key transfer the zone, and list ns2 as a
#    name server (named sends NOTIFY to the zone's NS records).
jarl -s ns1 option set allow-transfer '{ key xfer; }' --zone adbraun.lol
jarl -s ns1 record add adbraun.lol @ NS ns2.adbraun.lol.

# 3. On the secondary: the zone, copied from the primary with the key.
jarl -s ns2 zone create adbraun.lol --type secondary --primary "192.0.2.53 key xfer"
```

If ns2 isn't one of the zone's NS records, add `also-notify { ADDRESS; };`
on the primary so it learns about changes quickly. With views, each view's
zones are transferred separately; match the secondary's requests to the
right view, for example with `match-clients { key xfer; ... };`.

Things that both servers need, such as ACLs, can be changed on both at once
with a group (`jarl -s dns acl update ...`): the change is checked on both
first and applied only if both accept it.

## Taking Jarl out of the picture

The generated files are plain BIND configuration. To stop using Jarl,
disable the service. named keeps working with `/etc/bind/jarl/` exactly as
it is. To go back to the configuration you had before adopting:

```sh
cp /etc/bind/named.conf.pre-jarl-<date> /etc/bind/named.conf
rndc reconfig
```

## Monitoring

* `GET /healthz` (no authentication) reports that jarld is up.
* On the server, a check can use the local socket and needs no token:
  `sudo jarl status -o json`.
* `GET /api/v1/status` with a read-only token reports named reachability,
  drift and configuration problems. It is suitable for a monitoring check:
  ```sh
  jarl --token @/etc/jarl/monitor.token status -o json | jq -e '.engine.drift == [] and .engine.adopted'
  ```
* The audit log records failed logins as `auth.login_failed` /
  `auth.login_blocked`, and rejected tokens as `auth.token_failed`.