jarl/docs/certificates.md

# TLS certificates

Jarl can obtain TLS certificates from an ACME CA (Let's Encrypt by
default) and renew them itself, and put them where they are used:

- **web**: jarld's own HTTPS (the web UI, the API and the CLI's server).
- **dns**: files named can read, for DNS over TLS (DoT) and DNS over HTTPS
  (DoH) in named's `tls` blocks.

One certificate can have both uses. Jarl answers the ACME DNS-01
challenge itself: in a zone it serves, or through Porkbun's API while
Porkbun still serves the domain. So no other ACME client, hook script or
DNS provider plugin is needed, and the server doesn't have to be
reachable on port 80 or 443 from the internet.

A certificate you renew some other way (certbot, acme.sh, your own CA)
can be used too: Jarl reads it from its files and rereads it when they
change.

In the web UI: **Certificates** (administrators). In the CLI: `jarl cert`.
In the API: `/api/v1/certificates`. All of it is for administrators only:
a certificate's key can impersonate your server.

## Quick start

1. Set up the ACME account in `/etc/jarl/jarl.yaml` and restart jarld
   (`systemctl restart jarl`). Try Let's Encrypt's staging CA first: its
   certificates aren't trusted by browsers, but its rate limits are
   generous, so mistakes cost nothing.

   ```yaml
   acme:
     account:
       directory: letsencrypt-staging   # then: letsencrypt
       email: you@adbraun.lol           # optional
       agree_tos: true                  # after reading https://letsencrypt.org/repository/
   ```

2. See where the challenges would go, without asking the CA for anything:

   ```sh
   jarl cert check --domain ns1.adbraun.lol
   ```

   ```
   NAME             ROUTE    RECORD                           WHY
   ns1.adbraun.lol  porkbun  _acme-challenge.ns1.adbraun.lol  adbraun.lol is served by Porkbun (curitiba.ns.porkbun.com, ...), so the record goes in through Porkbun's API
   ```

3. Create the certificate. Jarl obtains it at once; the command waits for
   the outcome (usually under a minute, a few minutes through Porkbun):

   ```sh
   jarl cert create ns1 --domain ns1.adbraun.lol --use web --use dns
   ```

4. When it works, switch `directory` to `letsencrypt`, restart jarld and
   renew: `jarl cert issue ns1`.

From then on jarld renews it by itself.

## The ACME account

| Setting | Default | |
|--|--|--|
| `acme.account.directory` | `letsencrypt` | `letsencrypt`, `letsencrypt-staging`, or the directory URL of another ACME CA (ZeroSSL, Google, step-ca, ...). |
| `acme.account.email` | `""` | Contact address for the CA (optional; Let's Encrypt no longer sends expiry mails). |
| `acme.account.agree_tos` | `false` | You accept the CA's terms of service. Jarl registers no account without it, and says so. |
| `acme.account.eab_kid`, `eab_hmac` | `""` | External Account Binding, for CAs that need it (ZeroSSL, Google): from the CA's dashboard. |
| `acme.resolvers` | `1.1.1.1:53`, `9.9.9.9:53`, `8.8.8.8:53` | Recursive resolvers Jarl uses to see a domain as the CA does (below). |
| `acme.ca_bundle` | `""` | A PEM file of extra CA certificates to trust when talking to the ACME server (a private CA). |
| `acme.check_propagation` | `true` | Before telling the CA to look, wait until every name server of the zone serves the challenge (asked directly on port 53). |
| `certificates.dir` | `<bind.config_dir>/tls` | Where certificates with the dns use are written for named. |

The account is registered with the first certificate, and its key is
kept in Jarl's database. Changing `directory` registers a new account with
that CA; changing `email` updates the account.

## Where the challenge goes

For each name on a certificate, the CA looks up a TXT record at
`_acme-challenge.<name>` (a wildcard `*.example.com` uses
`_acme-challenge.example.com`). Jarl first looks the name up the way the
CA will, through `acme.resolvers` (public resolvers, not this server,
which may answer from an internal view), following a CNAME on
`_acme-challenge` if there is one. Then it finds the zone that holds the
record on the internet and that zone's public name servers:

| The zone's public name servers | Route | Jarl writes the record |
|--|--|--|
| Porkbun's (`*.ns.porkbun.com`), with `porkbun.api_key` set | `porkbun` | through Porkbun's DNS API (TTL 600, Porkbun's minimum), and deletes it afterwards. The key is the one the registrar features use ([configuration.md](configuration.md#porkbun-registrar-and-dns-host)); it needs DNS record write access, "API Access" switched on for the domain, and the domain in `porkbun.domains` if you set that |
| the name servers in Jarl's own zone of that name | `jarl` | in that zone, in every view that has it, as a change by `system` (in the history) |
| anything else | none | nowhere: the certificate fails with what to do |

The certificate's `challenge` setting chooses:

- **`auto`** (the default) decides again at **every** issue and renewal.
  While Porkbun serves `adbraun.lol`, the challenge goes to Porkbun. Once
  you point the domain's name servers at your own servers, the next
  renewal writes into Jarl's zone instead. Nothing needs changing.
- **`jarl`** or **`porkbun`** pins the route. Jarl still checks that the
  CA will see the record there and refuses to try otherwise (no wasted
  attempt against the CA's rate limits).

```sh
jarl cert update ns1 --challenge porkbun     # or jarl, or auto
```

`jarl cert check NAME` (or **Check the names** in the web UI) shows the
route for each name, and why, at any time.

### Domains Jarl can't write to: a CNAME

For a domain whose DNS is elsewhere (Cloudflare, say), point the
challenge name at a zone this server serves on the internet, once, by
hand:

```
_acme-challenge.www.example.org.  CNAME  www-example-org.acme.adbraun.lol.
```

Jarl follows the CNAME and writes the record at its target, in its zone
`acme.adbraun.lol` (which must be delegated to this server). Renewals
never touch the other provider again. That is the same idea as
[acme.md section 7](acme.md#7-a-dedicated-acme-zone-cname-delegation).

## Renewal

jarld looks at every certificate every 10 minutes:

- A certificate Jarl obtains is renewed at the time the CA suggests
  (ACME Renewal Information, RFC 9773; asked twice a day, so a CA that
  needs to replace certificates early, e.g. after revoking them, gets its
  way), or, without one, two thirds into its lifetime: day 60 of a 90-day
  certificate. Each renewal makes a new key.
- When obtaining fails, jarld tries again after 1, 2, 4, 8, 16 hours, then
  daily. The dashboard and the Certificates page show the error; so do
  `jarl cert list` and Prometheus (`jarl_certificate_failing`).
- New names or a new key type on a certificate make it due at once.
- `jarl cert issue NAME` (**Renew** in the web UI) obtains it now. It runs
  in the background: the command waits and reports, the web page updates
  itself.

A certificate counts as **expiring** (a warning) when less than a sixth of
its lifetime is left, which for a renewing certificate means renewal has
been failing for a while; and **expired** after its end.

## Use: web (jarld's HTTPS)

Only one certificate can have the web use. As soon as it has been
obtained, jarld serves it on all its `listen` addresses, **without a
restart**: each port speaks HTTPS and plain HTTP side by side.

- HTTPS connections get the certificate (HTTP/2 included).
- Plain HTTP from this server itself (`127.0.0.1`, `::1`) is served as
  before, so local scripts and `--server http://127.0.0.1:8053` keep
  working. So is plain HTTP from `trusted_proxies`.
- Plain HTTP from anywhere else: browsers are redirected to
  `https://<the same host and port>`; API requests get `400
  https_required` (and, if they carried a token, a note that it crossed
  the network unencrypted).

Point browsers and CLIs at the certificate's name:
`jarl config set server https://ns1.adbraun.lol:8053`. A CLI that uses the
server's IP address fails the name check.

A managed certificate takes over from `tls.cert_file`. Taking the web use
off (or deleting the certificate) goes back to `tls.cert_file`, or to
plain HTTP.

## Use: dns (for named)

Jarl writes the certificate to `<certificates.dir>/<name>/`:

```
/etc/bind/jarl/tls/ns1/fullchain.pem   (0644)
/etc/bind/jarl/tls/ns1/privkey.pem     (0640, group bind)
```

Debian's AppArmor profile lets named read `/etc/bind`, and group `bind`
lets it read the key. When a renewal changes the files and named's
configuration names them, Jarl runs `rndc reconfig`, and named serves the
new certificate at once (checked with BIND 9.20: DoT keeps working, with
the new certificate).

`jarl encrypted-dns setup` sets up DNS over TLS and HTTPS with it
([encrypted-dns.md](encrypted-dns.md)). By hand, a `tls` statement names
the files:

```sh
jarl option set tls 'dot { cert-file "/etc/bind/jarl/tls/ns1/fullchain.pem"; key-file "/etc/bind/jarl/tls/ns1/privkey.pem"; }' --toplevel
jarl option set listen-on 'port 853 tls dot { 10.0.10.5; }' --add
```

The Certificates page (and `jarl cert show`) says what named uses each
for: `DNS over TLS on 10.0.10.5, fd00::5 port 853`, `DNS over HTTPS on
… port 443`, or a hand-written `tls` statement no listener uses.

A certificate named uses can't be deleted or lose its `dns` use until
named no longer names its files. If you change `certificates.dir`, jarld
writes the certificates to the new place when it starts and points the
`tls` statements encrypted DNS wrote there (a hand-written `tls`
statement you change yourself).

## A certificate you renew yourself

```sh
jarl cert create own --cert-file /etc/ssl/ns1/fullchain.pem --key-file /etc/ssl/ns1/privkey.pem --use web
```

Jarl checks that the key belongs to the certificate, warns when the file
holds only the certificate without its intermediates (use the full
chain), and rereads both files when they change (within 10 minutes, or at
once with `jarl cert issue own`). A pair that doesn't match is reported
and the last good one stays in use. jarld runs as user `jarl`, which must
be able to read both files (e.g. `chgrp jarl` and `chmod 640`, or have
your renewal hook copy them somewhere it can read).

## Using a certificate outside Jarl

A certificate needn't be used by Jarl at all ("not used by Jarl" on the
Certificates page): Jarl obtains and renews it, and you export it for
another program, such as a mail server or a reverse proxy. **Export…** on
the Certificates page downloads each file or copies it to the clipboard
(copying needs the web UI over HTTPS, or from localhost). Or:

```sh
jarl cert export mail --dir /etc/ssl/mail             # fullchain.pem, and privkey.pem (mode 600)
jarl cert export mail --part bundle > /etc/haproxy/mail.pem
```

The parts are `fullchain` (the certificate and its chain), `cert` (the
certificate alone), `key` and `bundle` (the key, then the chain, in one
file, as HAProxy wants). Administrators can export any certificate, and
the owner of a certificate for another host (below) their own. Every
export of the key is in the audit log (`certificate.export_key`), and a key
only goes over HTTPS, the local socket, or to this server itself.

Jarl renews the certificate, not your copy: export it again after each
renewal (`jarl cert show` says when that is). For another host, `jarl
cert get` (below) does that for you. A program on this server can instead
read the files Jarl writes for named (use `dns`), which Jarl keeps current.

## Certificates for other hosts

`jarl cert get` gets a certificate onto another host (a mail server, a
reverse proxy) with one command: jarld obtains it from the CA and renews
it, answering the challenges in its own zones as for its own certificates;
the host asks for it, waits until it is there, downloads it and runs a
hook. The host needs only a token, not an ACME client or access to DNS.

**On Jarl**, as an administrator, give the hosts a user and a token each:

```sh
jarl user create hosts --no-password --grant "certificates:write names=*.adbraun.lol"
jarl token create mx --user hosts           # one token per host
```

The `certificates` grant says which names the user may have certificates
for. It is never part of `*`, so `operator` and `readonly` users can't
get certificates (or download keys) unless you add one. A certificate
belongs to the user that asked for it: its tokens can download it, other
users can't take its name or see it, and the certificates Jarl uses
itself are the administrators'. Hosts that should share certificates use
the same user (each with its own token, so you can revoke one host
alone); hosts that shouldn't, use different users. A user may own
`certificates.max_per_user` certificates (10 by default), or what `jarl
user update NAME --cert-limit N` (or the user's form) sets.

**On the host:**

```sh
export JARL_SERVER=https://jarl.adbraun.lol:8053 JARL_TOKEN=@/root/jarl.token
jarl cert get mail -d mail.adbraun.lol -d smtp.adbraun.lol --hook 'systemctl reload postfix dovecot'
sudo jarl cert schedule                    # a daily systemd timer runs 'jarl cert get'
jarl cert saved                            # what this host has, and when it expires
jarl cert saved mail                       # all of mail's settings
```

* **Asking again** with the same names returns the certificate jarld has:
  nothing is obtained, nothing is downloaded until jarld renews it.
  **Adding or removing a name** gets a new certificate. A different
  `--key-type` alone doesn't; delete the certificate for that.
* **Files**, in `--dir` (default `/etc/jarl/certs/NAME` as root, `./NAME`
  otherwise): `cert.pem`, `chain.pem` (the intermediates),
  `fullchain.pem`, `privkey.pem` (mode 600) and `jarl-cert.json` (serial,
  expiry, when it was fetched). Each file is replaced at once, never
  half written. The key is checked against the certificate before
  anything is written.
* **A combined file** (`--combined PATH`): the key, then the full chain,
  in one file (mode 600), as HAProxy wants. See [HAProxy](#haproxy).
* **Saved settings**: the names, `--dir`, `--combined`, `--hook`, the
  server and the token are saved in `/etc/jarl/cert-get/NAME.json` (as
  root; `~/.config/jarl/cert-get` otherwise), readable only by their
  owner. `jarl cert get` without a name does every saved certificate. A
  token given as `@FILE` is saved as that reference, not copied. `jarl
  cert saved NAME` shows them all (the token never). `jarl cert forget
  NAME` stops getting one (`--delete` deletes it on jarld too).
* **Changing a setting**: run `jarl cert get NAME` with just that option
  (`--hook '...'`, `--combined PATH`, `--dir DIR`); the others stay.
  `--hook ''` removes the hook, `--combined ''` stops writing the
  combined file and deletes it. The run still asks jarld for the
  certificate, but downloads nothing while this host has the current one.
  A new or moved combined file is written from the files this host
  already has, and the hook runs.
* **`--local`** doesn't ask jarld at all: it writes the combined file and
  runs the hook from the files this host has. Use it to apply a changed
  setting while jarld is unreachable, or to test a hook. New names still
  need jarld.
* **Several Jarl servers on one host**: each server has its own
  certificate names, so two servers may both have a `proxy`. The host
  keeps one `proxy` per settings directory, though: asking another server
  for a name saved here for the first one is refused, as is a `--dir`
  holding another certificate's files. Choose another name or `--dir`, or
  pass `--replace` to take the name (or directory) over. A `--dir` that
  another saved certificate uses is always refused.
* **The hook** runs with `sh` after new files are written or the
  combined file changes (and with `--local`), with `JARL_CERT_NAME`,
  `JARL_CERT_DIR`, `JARL_CERT_CERT`, `JARL_CERT_CHAIN`,
  `JARL_CERT_FULLCHAIN`, `JARL_CERT_KEY`, `JARL_CERT_COMBINED` (empty
  without `--combined`), `JARL_CERT_DOMAINS`, `JARL_CERT_SERIAL`,
  `JARL_CERT_NOT_AFTER` and `JARL_CERT_RENEWED` (1 when it replaced an
  older certificate). If it fails, `jarl cert get` exits non-zero and runs
  it again next time. It has 5 minutes.
* **`jarl cert schedule`** (Linux with systemd, as root) writes
  `/etc/systemd/system/jarl-cert-get.service` and `.timer`, which run
  `jarl cert get` daily at a random time within the hour after midnight,
  and catch up after the host was off. `--remove` takes them away. On
  other systems, run `jarl cert get` daily with their scheduler.

**On the Certificates page**, the administrators see each certificate's
owner and who downloaded it: the token, from where, when, and whether
that host has the current certificate or hasn't fetched the renewed one
yet. `jarl cert show NAME` says the same. The owner's own tokens see
their own downloads there too. In `jarl cert list`, such a certificate's
USES is `get`, and PULLED is the last download, with `(old)` while a host
hasn't fetched the renewed one.

### HAProxy

HAProxy wants the key and the chain in one file. Give `jarl cert get` a
`--combined` path outside `--dir`:

```sh
jarl cert get proxy -d adbraun.lol -d '*.adbraun.lol' \
    --combined /etc/haproxy/certs/adbraun.lol.pem --hook 'systemctl reload haproxy'
jarl cert get webmail -d webmail.adbraun.lol \
    --combined /etc/haproxy/certs/webmail.pem --hook 'systemctl reload haproxy'
```

```
frontend https
    bind :443 ssl crt /etc/haproxy/certs/
```

HAProxy loads every file in a `crt` directory and picks one by SNI, so
`/etc/haproxy/certs` should hold only combined files. Don't point `crt`
at `--dir` (`/etc/jarl/certs/NAME`): HAProxy would try to load
`cert.pem`, `privkey.pem` and `jarl-cert.json` there too, and refuse
to start. A `--combined` path that would overwrite a file in a saved
certificate's directory, or another certificate's combined file, is
refused.

The combined file is mode 600, owned by the user running `jarl cert get`
(root under the timer). HAProxy reads its certificates as root before it
drops privileges, so that's fine.

To add a combined file to a certificate this host already has, run
`jarl cert get proxy --combined /etc/haproxy/certs/adbraun.lol.pem`: it's
written from the files on disk and the hook runs, with nothing
downloaded.

## Security notes

- Certificates Jarl obtains, their keys and the ACME account key are kept
  in Jarl's database, so they are in its backups (like TSIG secrets). The
  API returns a key only to an administrator who exports it, and records
  that in the audit log.
- Every publicly trusted certificate is published in Certificate
  Transparency logs, with all its names. A name like `ns1.adbraun.lol`
  used for both the web UI and DNS reveals less than separate names. Avoid
  wildcards where one name will do: a wildcard's key is good for every
  name in the domain.
- Using one certificate for jarld and named puts its key in two places.
  On one server, where jarld already writes named's configuration, that
  changes little.

## API

| Method | Path | |
|--|--|--|
| GET | `/certificates` | `certificates` (each with `status` {`state`: ok, pending, failing, expiring, expired; `message`}, `next_attempt`, `issuing_since` while being obtained, `serving`, `files`) and `account` |
| POST | `/certificates` | `{"name": "ns1", "domains": [...], "uses": ["web", "dns"], "challenge": "auto", "key_type": "ec256"}`, or `{"name", "source": "file", "cert_file", "key_file", "uses"}`. One Jarl obtains starts being obtained at once (`"issue": false` leaves it to the loop) |
| POST | `/certificates/check` | `{"domains": [...], "challenge": "auto"}`: where each challenge would go (`routes`: `route`, `target`, `zone`, `public_ns`, `why` or `error`) |
| GET, PATCH, DELETE | `/certificates/{name}` | `domains` and `uses` replace the whole list |
| POST | `/certificates/{name}/issue` | obtain now: 202, then poll `GET /certificates/{name}` until `issuing_since` is gone; a file certificate is reread (200) |
| POST | `/certificates/{name}/check` | the routes for an existing certificate |
| GET | `/certificates/{name}/pem?part=fullchain` | the certificate as PEM, to save: `fullchain` (default), `cert`, `key` or `bundle` (key and chain). Exports of the key are audited |

## Testing with Pebble

[Pebble](https://github.com/letsencrypt/pebble) is Let's Encrypt's test
CA. Point it at a test named, and Jarl at it:

```sh
docker run -d --name pebble --network host -e PEBBLE_VA_NOSLEEP=1 \
  ghcr.io/letsencrypt/pebble -config test/config/pebble-config.json -dnsserver 127.0.0.1:53
docker cp pebble:/test/certs/pebble.minica.pem /etc/jarl/pebble.pem
```

```yaml
acme:
  account: {directory: "https://localhost:14000/dir", agree_tos: true}
  resolvers: ["127.0.0.1:53"]
  ca_bundle: /etc/jarl/pebble.pem
```