jarl/docs/operations.md

# Running Jarl

## Day to day

| Task | How |
|------|-----|
| Is everything healthy? | `jarl status`, or the web UI dashboard: named reachable, adoption, drift, open changesets, configuration problems |
| What changed and who did it? | `jarl history`, `jarl changeset show ID` (with diff), `jarl audit` |
| Undo a change | `jarl changeset revert ID`, review it, then `jarl changeset apply NEW_ID` (or History → Revert in the UI) |
| Check the configuration | `jarl check`: full validation plus `named-checkconf -z` |
| Logs | `journalctl -u jarl` (jarld, including `msg=audit` lines), `journalctl -u named` |
| Restart / reload TLS | `systemctl restart jarl` / `systemctl reload jarl` |

Restarting jarld doesn't affect DNS: named keeps serving the files it
loaded. jarld is only needed to make changes.

## Backups

Everything Jarl knows is in its database. The BIND files can be generated
from it again at any time.

```sh
sudo -u jarl jarld backup /var/backups/jarl/jarl-$(date +%F).db   # consistent copy, safe while jarld runs
```

For example, as a daily timer or cron job:

```sh
# /etc/cron.daily/jarl-backup
#!/bin/sh
set -e
install -d -o jarl -g jarl -m 0700 /var/backups/jarl
sudo -u jarl jarld backup /var/backups/jarl/jarl-$(date +%F).db
find /var/backups/jarl -name 'jarl-*.db' -mtime +30 -delete
```

Query statistics live in a separate database (`/var/lib/jarl/stats.db`).
It isn't needed to restore DNS, so back it up only if you want to keep the
history.

The database contains password and token *hashes* and the TSIG secrets in
the clear, so keep backups private.

### Restoring

```sh
systemctl stop jarl
install -o jarl -g jarl -m 0600 /var/backups/jarl/jarl-2026-10-01.db /var/lib/jarl/jarl.db
rm -f /var/lib/jarl/jarl.db-wal /var/lib/jarl/jarl.db-shm
systemctl start jarl
jarl resync --dry-run     # what differs between the restored database and the files
jarl resync --force       # write everything and reload named
```

An older backup can carry lower zone serials than named last served. Your
own named doesn't mind, but secondaries and caches ignore a serial that went
backwards. If you have secondaries, raise the serials after a restore:
`jarl zone update ZONE --serial <higher>`.

## Drift: files changed outside Jarl

Jarl keeps a checksum of every file it writes. If someone edits
`/etc/bind/jarl/named.conf` or a zone file by hand (or a restore puts old
files back), `jarl status` and the dashboard show **drift**, and changes
that would overwrite those files are refused with `409 drift`:

```
error: files managed by Jarl were changed outside Jarl: /var/lib/bind/jarl/internal/adbraun.lol.zone; review them and apply with force to overwrite
```

Decide which version is right:

* **Jarl's** (the usual case): `jarl resync --force` rewrites every file from
  the database and reloads named. Any single change with `--force` also
  overwrites the drifted files.
* **The hand edit**: put the same change into Jarl (for example
  `jarl zone import ZONE /var/lib/bind/jarl/internal/adbraun.lol.zone` for
  records), then `jarl resync --force`.

## When an apply fails

Jarl checks everything it can before it touches named, so failures are
rare. When one happens, nothing is left half-done:

| Stage | What you see | State afterwards |
|-------|--------------|------------------|
| validation | 422 with `problems` | Nothing written |
| named-checkconf / named-checkzone | 422 `validation_failed` with the checker's messages in `check`, mapped to the zone/option that caused them | Nothing written |
| `rndc reconfig` / `reload` fails, or a zone doesn't load within `reload_timeout` | 502 `bind_failed` | Previous files restored and named reloaded |

A failed change is never recorded as applied. A staged changeset stays open,
so you can fix it and apply again, and the failure is in the audit log
(`jarl audit --outcome failure`). named's own log usually explains a load
failure: `journalctl -u named`.

If named isn't running, changes fail at the reload step and are rolled back.
Start named, then apply again.

## Recovering access

| Problem | Fix (on the server) |
|---------|---------------------|
| Forgot the admin password | `sudo -u jarl jarld user passwd admin` |
| Locked out by the login rate limit | wait 15 minutes, or `systemctl restart jarl` (the limiter is in memory) |
| Lost every admin account | `sudo -u jarl jarld user add --admin rescue` |
| Need a token without the web UI | `sudo -u jarl jarld user token admin cli` |

## Taking Jarl out of the picture

The generated files are plain BIND configuration. To stop using Jarl,
disable the service. named keeps working with `/etc/bind/jarl/` exactly as
it is. To go back to the configuration you had before adopting:

```sh
cp /etc/bind/named.conf.pre-jarl-<date> /etc/bind/named.conf
rndc reconfig
```

## Monitoring

* `GET /healthz` (no authentication) reports that jarld is up.
* `GET /api/v1/status` with a read-only token reports named reachability,
  drift and configuration problems. It is suitable for a monitoring check:
  ```sh
  jarl --token @/etc/jarl/monitor.token status -o json | jq -e '.engine.drift == [] and .engine.adopted'
  ```
* The audit log records failed logins as `auth.login_failed` /
  `auth.login_blocked`, and rejected tokens as `auth.token_failed`.