jarl/docs/concepts.md

# How Jarl works

This page explains Jarl's design: where the data lives, how a change reaches
named, and why it works that way. Read it once before you adopt your server.

## The pieces

```
 browser ──(session + CSRF)──┐
                             ├──> jarld ──> SQLite DB (source of truth)
 jarl CLI ──(API token)──────┤      │
 scripts / certbot ──(token)─┘      ├──> /etc/bind/jarl/named.conf, keys.conf
                                    ├──> /var/lib/bind/jarl/*.zone
                                    └──> named-checkconf / named-checkzone / rndc ──> named
```

* **jarld** is one Go binary. It serves the REST API and the web UI (embedded
  in the binary; the UI uses only the public API), and it is the only thing
  that writes BIND's files. It runs as the unprivileged user `jarl` under
  systemd.
* **jarl** is the command line client. It talks to jarld's API with an API
  token, so it works from any machine that can reach jarld.
* **The database** (SQLite, `/var/lib/jarl/jarl.db`) holds the whole BIND
  configuration as structured data: views, ACLs, TSIG keys, zones, records,
  options, and the defaults for new zones. It also holds users, tokens,
  sessions, changesets and the audit log.

## The database is the source of truth

Jarl never edits BIND's files in place. Every time a change is applied it
renders the complete configuration from the database:

| File | Contents |
|------|----------|
| `/etc/bind/jarl/named.conf` | ACLs, options, views, zone statements, top-level statements such as `logging` and `controls` |
| `/etc/bind/jarl/keys.conf` | TSIG keys. They are kept apart so secrets never show up in diffs, history or the audit log |
| `/var/lib/bind/jarl/<view>/<zone>.zone` | One file per primary zone (`_default` when views aren't used) |
| `/var/cache/bind/jarl/...` | Written by named itself for secondary, stub and mirror zones |

`jarld adopt` turns `/etc/bind/named.conf` into a single
`include "/etc/bind/jarl/named.conf";`, so named loads only what Jarl
generates. Zone files go under `/var/lib/bind` because Debian's AppArmor
profile lets named write there: its journals, the signed copies of zones
it signs, and their keys (under `dnssec-keys/`, see [dnssec.md](dnssec.md)).

Because files are always regenerated, anything you edit by hand is
overwritten. Jarl records a checksum of every file it writes and reports
**drift** when a file changes behind its back. It then refuses to overwrite
the file until you say so (`--force`, or `jarl resync --force`).

Some of named.conf comes from Jarl itself rather than from your options:
the include of keys.conf, the statistics channel and the logging
statistics need, blocking's `response-policy` and policy zones, and the
view keys at the front of each view's `match-clients`. The Server options
page lists them under **Added by Jarl**, with why each is there and what
turns it on or off (`jarl option generated`). Where you have a statement
of your own (`logging`, `statistics-channels`), Jarl merges into it.

Comments are part of the data model: zones, records, ACLs, views, keys and
options can all carry a comment, and Jarl writes it into the generated file.
Comments in imported zone files are kept.

## Changesets: one change or a bundle, it's the same pipeline

Your brief asked whether each change should be checked and applied on its
own, or whether changes should be bundled and applied together. Jarl does
both, using a single mechanism:

* A **changeset** is an ordered list of operations (create a zone, add a
  record, rename an ACL, ...). Applying one is all-or-nothing.
* Every API call that changes something is applied at once **as a one-operation
  changeset** by default. Nothing extra is needed for small edits.
* To bundle changes, create a changeset and *stage* operations into it
  (API: `?changeset=ID` or the `X-Jarl-Changeset` header; CLI: `--changeset ID`
  or `JARL_CHANGESET`). Review the diff, then apply or discard it.
* `POST /api/v1/apply` (CLI: `jarl apply -f changes.yaml`) applies a list of
  operations as one changeset in a single request.
* `?dry_run=1` (CLI: `--dry-run`) runs the whole pipeline except installing
  files and reloading named, and shows the problems and the diff. Each
  part of a named.conf diff says which block it is in, after the `@@`
  line numbers: `@@ -50,6 +49,8 @@ view "internal" {`.

**Why bundling matters.** Some changes only make sense together. Turning
`www` from an A record into a CNAME means deleting the A record and adding
the CNAME, and named must never see the zone with neither, or with a
half-finished renumbering. A changeset reaches named in one step or not at
all.

Operations sent **in one request** (`jarl apply -f`, `POST /apply`, or one
staging request with several operations) are validated together, at their
end state, so their order doesn't matter. When operations are staged **one
at a time** (the web UI's review mode, `--changeset` on single commands),
each one is checked as it is added. You get immediate feedback, so delete
the A record before adding the CNAME. At apply time the whole changeset is
validated again against the configuration as it is then.

**Interfaces use different defaults**, because they are used differently:

| Interface | Default | Why |
|-----------|---------|-----|
| Web UI | *Review mode*: edits are collected in a draft changeset, the zone pages show pending rows (green = new, yellow = changed, struck out = deleted), and a banner leads to the review page, which has a diff and an Apply button. Users can switch to "Apply edits immediately". | People click through several related edits and want to see the result before it goes live. |
| CLI | *Immediate*: every command is applied (and validated) on its own. Bundle with `jarl apply -f` or `--changeset`. | Scripts want a clear success or failure per step, and ACME hooks need the record live when the command returns. |
| API | Immediate unless a changeset is named. | Same as the CLI. |

### What happens when a changeset is applied

1. **Replay.** The operations are replayed on a copy of the current
   configuration, checking permissions and (for staged operations) that the
   objects haven't changed since they were staged (optimistic locking via
   object versions). Concurrent edits produce a `409 conflict` instead of
   silently overwriting each other.
2. **Validate.** The resulting configuration is checked as a whole (see
   [validation.md](validation.md)): DNS rules such as CNAME exclusivity,
   glue, delegations and RRset TTLs; references to ACLs and keys; option
   syntax and scope against the BIND 9.20 grammar; and security lints such
   as open resolvers. Errors block the change, while warnings are shown and
   allowed. Problems that already existed are reported separately and don't
   block unrelated changes.
3. **Render and check.** The files are rendered into a staging directory and
   checked with the real `named-checkconf -z` and `named-checkzone`. Their
   messages are mapped back to the object that caused them.
4. **Install.** Changed files are written atomically (temp file + rename),
   keeping the previous contents in memory.
5. **Reload.** `rndc reconfig` if `named.conf` changed, then `rndc reload ZONE`
   for each changed zone.
6. **Verify.** named loads zones asynchronously and an `rndc reload` that
   fails only shows up in named's log. So Jarl polls `rndc zonestatus` until
   every changed zone reports its new serial. If one doesn't within
   `reload_timeout` (15s), Jarl restores the previous files, reloads again
   and reports the failure.
7. **Record.** The changeset is stored with its result, its diff and the
   *undo operations*, so it can be reverted later (`jarl changeset revert ID`,
   or the Revert button in History). A revert is itself a changeset you
   review before applying.

Serials are bumped automatically, and only when a zone's DNS data actually
changes (a comment-only edit doesn't bump it). Each zone follows its serial
policy: `date` (YYYYMMDDnn, the default), `increment` or `unixtime`, using
RFC 1982 arithmetic.

## Model: views, zones, records, options

* **Views** are ordered; named uses the first view whose `match-clients` and
  `match-destinations` match. Once one view exists, named requires every
  zone to be in a view, so creating the first view offers to move existing
  zones into it (`adopt_zones`). See [views.md](views.md) for a split-DNS
  walkthrough.
* **Zones** have a type: `primary`, `secondary`, `stub`, `static-stub`,
  `forward`, `mirror`, `hint`, `redirect`, or `in-view` (share a zone from
  another view). Only primary zones have records in Jarl. The same zone name
  can exist in several views (split DNS); copy one with `clone_from`.
  A CIDR such as `192.168.1.0/24` or `2001:db8::/48` creates the matching
  reverse zone (see [Zone types](#zone-types) and
  [Reverse zones](#reverse-zones) below).
* **Records** are stored canonically (owner FQDN, type, TTL or "zone
  default", data in presentation format). TXT data is entered as plain text;
  Jarl quotes it and splits it into 255-byte strings. Relative names are
  resolved against the zone, and `www.example.com` without a final dot in a
  zone `example.com` is caught as the classic `www.example.com.example.com.`
  mistake. Adding an A/AAAA record with `ptr` also maintains the PTR record in
  the matching reverse zone. In reverse zones a record can be named by its
  address (`10.10.0.20`), and a PTR target without a final dot is read as
  the full host name, because a PTR never points into the reverse zone.
* **Options** are named.conf statements attached to a scope: server
  `options { }`, a view, a zone, or the top level (`logging`, `controls`,
  `statistics-channels`, `include`, ...). The UI and CLI know every BIND 9.20
  statement (353 of them, from the ARM): where it may appear, its grammar,
  and whether it is deprecated. Values are written in named.conf syntax, and
  Jarl checks them before named does. Statements that Jarl manages itself
  (`zone`, `view`, `acl`, `key`, a zone's `type`/`file`, a view's
  `match-clients`, ...) can't be set as raw options.
  Options of a block are kept in one fixed order, the same in named.conf,
  the web UI and the CLI: grouped (files, listening, access control,
  recursion and forwarding, DNSSEC, transfers and notify, then the rest),
  alphabetical within a group. So the same option is in the same place in
  every view and zone. Top-level statements stay in the order they were
  added.
* **Address ranges.** Wherever named takes an address match list (ACLs,
  views' `match-clients` and `match-destinations`, options such as
  `allow-query` or `allow-recursion`) and in blocking's client groups, Jarl
  also takes a range of addresses: `10.0.10.101-10.0.10.199`,
  `2001:db8::100-2001:db8::1ff`, `!10.0.10.150-10.0.10.160`. BIND only
  matches addresses and prefixes, so Jarl keeps the range as you wrote it
  and writes the fewest prefixes that cover exactly that range to
  named.conf, with the range as a comment above them:
  ```
  acl "dhcp" {
  	// 10.0.10.101-10.0.10.199
  	10.0.10.101;
  	10.0.10.102/31;
  	10.0.10.104/29;
  	10.0.10.112/28;
  	10.0.10.128/26;
  	10.0.10.192/29;
  };
  ```
  Nothing outside the range is matched (writing such lists by hand, it is
  easy to round to `10.0.10.96/27` and let in more than you meant). Both
  ends are full addresses of the same family, and spaces around the dash
  are fine. A range takes at most 62 prefixes for IPv4 and 254 for IPv6.
  The web UI shows what a range becomes while you type it. A range in the
  IPv4-mapped space (`::ffff:0:0/96`) gets a warning: named can see IPv4
  clients as such addresses, so it may match IPv4 clients too.
* **Settings** are Jarl's defaults for new zones: name servers, SOA contact
  and timers, default TTL, serial policy.

### Zone types

Every type works for a domain as well as a reverse zone; a reverse zone is
just a zone with a special name.

| Type | What it does | Typical use |
|------|--------------|-------------|
| `primary` | This server holds the zone; Jarl manages its records | Almost every zone you create |
| `secondary` | Copies the zone from another server by zone transfer (AXFR/IXFR) and serves the copy | A backup name server ([operations.md](operations.md#a-secondary-server)) |
| `forward` | Sends queries for names in the zone to other resolvers (`forwarders`) | Names your router or Consul answers, e.g. forward `10.0.10.in-addr.arpa` to the router |
| `stub` | Keeps only the zone's NS records, fetched from its primaries, to find the right servers | Internal zones held elsewhere that aren't delegated publicly |
| `static-stub` | Like stub, but you list the server addresses (`server-addresses`) | The same, without asking the primaries |
| `mirror` | A copy that named validates with DNSSEC before using | The root zone (RFC 8806) |
| `hint` | The root hints file, so the resolver can find the root servers | Zone `.` only |
| `redirect` | Answers given instead of NXDOMAIN | Rarely; one per server, named `.` |
| `in-view` | Uses a zone defined in an earlier view instead of a copy | Split DNS where a zone is identical in two views |

Only primary zones have records in Jarl; the others are configuration.

### A new zone's name servers

Every primary zone needs at least one NS record, or named won't load it.
When creating a zone:

* **Name servers**: the host name(s) of the server(s) answering for the
  zone, normally this server, e.g. `ns1.adbraun.lol.`, plus a secondary if
  you have one. The first one is also the SOA's primary name server. Set
  your usual ones once with `jarl settings set --ns ns1.adbraun.lol.` (or
  Server options → Defaults for new zones) and they are filled in.
* **Name server addresses** are only needed when a name server's name is
  inside the zone being created: zone `adbraun.lol` with name server
  `ns1.adbraun.lol` needs an address record `ns1 10.0.10.2` in the zone
  (glue). For reverse zones, and zones whose name servers are named in
  another zone, leave it empty.

Other zone types don't take name servers.

### Reverse zones

Reverse names have one label per octet (`100.10.0.10.in-addr.arpa` for
10.0.10.100), or per nibble for IPv6, and a zone can only start at a label
boundary. So:

* A /8, /16 or /24 is one zone (any multiple of /4 for IPv6).
* A network in between is several zones: a /22 is four /24 zones. Jarl
  creates them together, up to 16 at a time.
* A /32 (or /128) is a zone for one address. Use it to answer for a few
  hosts while another server, such as your router, answers for the rest of
  the /24. When a /32 zone and its /24 zone are both on this server, the
  /32 zone answers for its address.
* A /25 to /31 has no zone of its own: its names live in the /24's zone.
  (RFC 2317 delegation, where an ISP hands you part of a /24, is set up by
  hand with CNAMEs.)

Jarl suggests the reverse zones your private views lack, creates them in
the right view filled with PTR records, and can keep PTR records in step
with a zone's A and AAAA records: [reverse.md](reverse.md). A zone created
inside another (a /24 inside a /8) takes the parent's records under it.

### Dynamic updates and TSIG keys

Jarl writes every zone file, so it can't let named change a zone through
dynamic updates (`allow-update`, `update-policy`, nsupdate): named's
changes would be overwritten. Automation goes through the API instead, with
an API token restricted to what it may change; `jarl acme` and the acme.sh
and certbot hooks do this for DNS-01 challenges ([acme.md](acme.md)).
TSIG keys remain for what servers do among themselves: zone transfers to
secondaries, NOTIFY, and choosing a view by key (`match-clients { key K; }`).

ACLs and TSIG keys are referenced by name. Jarl tracks every reference
(other ACLs, `match-clients`, `allow-*` options, `primaries ... key`, ...):
deleting something that is still in use is refused with the list of users,
and renaming updates every reference.

## Authentication and permissions

Your brief suggested HTTP basic auth for the web UI, with the web app
holding an API key. Jarl keeps the idea (each person has a username and
password, and every request is tied to a user and checked against that
user's permissions) but uses a login form and a session instead of basic
auth:

* Basic auth sends the password with every request and can't log out.
  Browsers cache it until they restart, and it can't be rate limited
  per user in a friendly way.
* A login form creates a session: an HttpOnly, SameSite=Strict cookie that
  expires after 12 hours (2 hours idle), plus a CSRF token the UI sends with
  every change. Failed logins are rate limited (8 per username and 20 per IP
  per 15 minutes). Passwords are hashed with argon2id.
* The web UI has no key of its own. It calls the same API as the CLI,
  authenticated as the logged-in user, so the audit log always names the
  person rather than "the web app".

The CLI and scripts use **API tokens** (`jarl_<id>_<secret>`). Only a hash
of each token is stored, and the full token is shown once.

* Each token belongs to a user. By default it has the user's permissions,
  or it can be **restricted** to a subset with its own grants and given an
  expiry. A token can never do more than its user.
* **Grants** are granular: resource (`zones`, `records`, `views`, `acls`,
  `tsig`, `options`, `audit`, `*`) × access (`read`/`write`), optionally
  narrowed by zone, view, record name pattern and record type. For example,
  `records:write zones=adbraun.lol names=_acme-challenge.* types=TXT` is
  what the ready-made `acme` role contains.
* Administrators can do everything, including managing users and server
  internals. Admin rights are never available through a restricted token.
* A token can be **rotated**: it gets a new secret but keeps its grants,
  optionally with a grace period for the old secret.

On the server itself, jarld also listens on a **local unix socket**. There
the kernel says which system user is connecting, and root and the `jarl`
user are administrators without a token, so `sudo jarl` needs no setup and
nobody with root on the server can be locked out. Everyone else needs a
token there too.

Details: [permissions.md](permissions.md).

## Audit log

Jarl records every login (successful, failed or blocked by the rate
limit), every failed token, every changeset (created, staged, applied or
failed, discarded, reverted), token creation and revocation, user change,
password change, TSIG secret reveal and adoption. Each event has the user,
how they authenticated (session, which token, or local), the client IP and
the outcome. Tokens also remember when and from where they were last used. It is in the database
(searchable in the UI and with `jarl audit`) and is also written to the
journal through jarld's log.

## What Jarl deliberately doesn't do (yet)

* **Dynamic updates** (`allow-update`, `update-policy`): named would write
  to zone files that Jarl regenerates, so the validator rejects them on Jarl
  zones. Use the API (for example `jarl acme`) instead of nsupdate.
* **Several servers from one jarld**: each jarld manages the named on its
  own host. A second server runs its own jarld with secondary zones, which
  copy the primary's zones by normal AXFR/IXFR (see "A secondary server" in
  [operations.md](operations.md)). The CLI can name both and run a command
  on both at once ([cli.md](cli.md#several-servers-and-groups)).