jarl/docs/certificates.md

TLS certificates

Jarl can obtain TLS certificates from an ACME CA (Let's Encrypt by default) and renew them itself, and put them where they are used:

• web: jarld's own HTTPS (the web UI, the API and the CLI's server).
• dns: files named can read, for DNS over TLS (DoT) and DNS over HTTPS (DoH) in named's tls blocks.

One certificate can have both uses. Jarl answers the ACME DNS-01 challenge itself: in a zone it serves, or through Porkbun's API while Porkbun still serves the domain. So no other ACME client, hook script or DNS provider plugin is needed, and the server doesn't have to be reachable on port 80 or 443 from the internet.

A certificate you renew some other way (certbot, acme.sh, your own CA) can be used too: Jarl reads it from its files and rereads it when they change.

In the web UI: Certificates (administrators). In the CLI: jarl cert. In the API: /api/v1/certificates. All of it is for administrators only: a certificate's key can impersonate your server.

Quick start

1. Set up the ACME account in /etc/jarl/jarl.yaml and restart jarld (systemctl restart jarl). Try Let's Encrypt's staging CA first: its certificates aren't trusted by browsers, but its rate limits are generous, so mistakes cost nothing.

   acme:
     account:
       directory: letsencrypt-staging   # then: letsencrypt
       email: you@adbraun.lol           # optional
       agree_tos: true                  # after reading https://letsencrypt.org/repository/
   

2. See where the challenges would go, without asking the CA for anything:

   jarl cert check --domain ns1.adbraun.lol
   

   NAME             ROUTE    RECORD                           WHY
   ns1.adbraun.lol  porkbun  _acme-challenge.ns1.adbraun.lol  adbraun.lol is served by Porkbun (curitiba.ns.porkbun.com, ...), so the record goes in through Porkbun's API
   

3. Create the certificate. Jarl obtains it at once; the command waits for the outcome (usually under a minute, a few minutes through Porkbun):

   jarl cert create ns1 --domain ns1.adbraun.lol --use web --use dns
   

4. When it works, switch directory to letsencrypt, restart jarld and renew: jarl cert issue ns1.

From then on jarld renews it by itself.

The ACME account

Setting	Default
acme.account.directory	letsencrypt	letsencrypt, letsencrypt-staging, or the directory URL of another ACME CA (ZeroSSL, Google, step-ca, ...).
acme.account.email	""	Contact address for the CA (optional; Let's Encrypt no longer sends expiry mails).
acme.account.agree_tos	false	You accept the CA's terms of service. Jarl registers no account without it, and says so.
acme.account.eab_kid, eab_hmac	""	External Account Binding, for CAs that need it (ZeroSSL, Google): from the CA's dashboard.
acme.resolvers	1.1.1.1:53, 9.9.9.9:53, 8.8.8.8:53	Recursive resolvers Jarl uses to see a domain as the CA does (below).
acme.ca_bundle	""	A PEM file of extra CA certificates to trust when talking to the ACME server (a private CA).
acme.check_propagation	true	Before telling the CA to look, wait until every name server of the zone serves the challenge (asked directly on port 53).
certificates.dir	<bind.config_dir>/tls	Where certificates with the dns use are written for named.

The account is registered with the first certificate, and its key is kept in Jarl's database. Changing directory registers a new account with that CA; changing email updates the account.

Where the challenge goes

For each name on a certificate, the CA looks up a TXT record at _acme-challenge.<name> (a wildcard *.example.com uses _acme-challenge.example.com). Jarl first looks the name up the way the CA will, through acme.resolvers (public resolvers, not this server, which may answer from an internal view), following a CNAME on _acme-challenge if there is one. Then it finds the zone that holds the record on the internet and that zone's public name servers:

The zone's public name servers	Route	Jarl writes the record
Porkbun's (*.ns.porkbun.com), with porkbun.api_key set	porkbun	through Porkbun's DNS API (TTL 600, Porkbun's minimum), and deletes it afterwards. The key is the one the registrar features use (configuration.md); it needs DNS record write access, "API Access" switched on for the domain, and the domain in porkbun.domains if you set that
the name servers in Jarl's own zone of that name	jarl	in that zone, in every view that has it, as a change by system (in the history)
anything else	none	nowhere: the certificate fails with what to do

The certificate's challenge setting chooses:

• auto (the default) decides again at every issue and renewal. While Porkbun serves adbraun.lol, the challenge goes to Porkbun. Once you point the domain's name servers at your own servers, the next renewal writes into Jarl's zone instead. Nothing needs changing.
• jarl or porkbun pins the route. Jarl still checks that the CA will see the record there and refuses to try otherwise (no wasted attempt against the CA's rate limits).

jarl cert update ns1 --challenge porkbun     # or jarl, or auto

jarl cert check NAME (or Check the names in the web UI) shows the route for each name, and why, at any time.

Domains Jarl can't write to: a CNAME

For a domain whose DNS is elsewhere (Cloudflare, say), point the challenge name at a zone this server serves on the internet, once, by hand:

_acme-challenge.www.example.org.  CNAME  www-example-org.acme.adbraun.lol.

Jarl follows the CNAME and writes the record at its target, in its zone acme.adbraun.lol (which must be delegated to this server). Renewals never touch the other provider again. That is the same idea as acme.md section 7.

Renewal

jarld looks at every certificate every 10 minutes:

• A certificate Jarl obtains is renewed at the time the CA suggests (ACME Renewal Information, RFC 9773; asked twice a day, so a CA that needs to replace certificates early, e.g. after revoking them, gets its way), or, without one, two thirds into its lifetime: day 60 of a 90-day certificate. Each renewal makes a new key.
• When obtaining fails, jarld tries again after 1, 2, 4, 8, 16 hours, then daily. The dashboard and the Certificates page show the error; so do jarl cert list and Prometheus (jarl_certificate_failing).
• New names or a new key type on a certificate make it due at once.
• jarl cert issue NAME (Renew in the web UI) obtains it now. It runs in the background: the command waits and reports, the web page updates itself.

A certificate counts as expiring (a warning) when less than a sixth of its lifetime is left, which for a renewing certificate means renewal has been failing for a while; and expired after its end.

Use: web (jarld's HTTPS)

Only one certificate can have the web use. As soon as it has been obtained, jarld serves it on all its listen addresses, without a restart: each port speaks HTTPS and plain HTTP side by side.

• HTTPS connections get the certificate (HTTP/2 included).
• Plain HTTP from this server itself (127.0.0.1, ::1) is served as before, so local scripts and --server http://127.0.0.1:8053 keep working. So is plain HTTP from trusted_proxies.
• Plain HTTP from anywhere else: browsers are redirected to https://<the same host and port>; API requests get 400 https_required (and, if they carried a token, a note that it crossed the network unencrypted).

Point browsers and CLIs at the certificate's name: jarl config set server https://ns1.adbraun.lol:8053. A CLI that uses the server's IP address fails the name check.

A managed certificate takes over from tls.cert_file. Taking the web use off (or deleting the certificate) goes back to tls.cert_file, or to plain HTTP.

Use: dns (for named)

Jarl writes the certificate to <certificates.dir>/<name>/:

/etc/bind/jarl/tls/ns1/fullchain.pem   (0644)
/etc/bind/jarl/tls/ns1/privkey.pem     (0640, group bind)

Debian's AppArmor profile lets named read /etc/bind, and group bind lets it read the key. When a renewal changes the files and named's configuration names them, Jarl runs rndc reconfig, and named serves the new certificate at once (checked with BIND 9.20: DoT keeps working, with the new certificate).

jarl encrypted-dns setup sets up DNS over TLS and HTTPS with it (encrypted-dns.md). By hand, a tls statement names the files:

jarl option set tls 'dot { cert-file "/etc/bind/jarl/tls/ns1/fullchain.pem"; key-file "/etc/bind/jarl/tls/ns1/privkey.pem"; }' --toplevel
jarl option set listen-on 'port 853 tls dot { 10.0.10.5; }' --add

The Certificates page (and jarl cert show) says what named uses each for: DNS over TLS on 10.0.10.5, fd00::5 port 853, DNS over HTTPS on … port 443, or a hand-written tls statement no listener uses.

A certificate named uses can't be deleted or lose its dns use until named no longer names its files. If you change certificates.dir, jarld writes the certificates to the new place when it starts and points the tls statements encrypted DNS wrote there (a hand-written tls statement you change yourself).

A certificate you renew yourself

jarl cert create own --cert-file /etc/ssl/ns1/fullchain.pem --key-file /etc/ssl/ns1/privkey.pem --use web

Jarl checks that the key belongs to the certificate, warns when the file holds only the certificate without its intermediates (use the full chain), and rereads both files when they change (within 10 minutes, or at once with jarl cert issue own). A pair that doesn't match is reported and the last good one stays in use. jarld runs as user jarl, which must be able to read both files (e.g. chgrp jarl and chmod 640, or have your renewal hook copy them somewhere it can read).

Using a certificate outside Jarl

A certificate needn't be used by Jarl at all ("not used by Jarl" on the Certificates page): Jarl obtains and renews it, and you export it for another program, such as a mail server or a reverse proxy. Export… on the Certificates page downloads each file or copies it to the clipboard (copying needs the web UI over HTTPS, or from localhost). Or:

jarl cert export mail --dir /etc/ssl/mail             # fullchain.pem, and privkey.pem (mode 600)
jarl cert export mail --part bundle > /etc/haproxy/mail.pem

The parts are fullchain (the certificate and its chain), cert (the certificate alone), key and bundle (the key, then the chain, in one file, as HAProxy wants). Administrators can export any certificate, and the owner of a certificate for another host (below) their own. Every export of the key is in the audit log (certificate.export_key), and a key only goes over HTTPS, the local socket, or to this server itself.

Jarl renews the certificate, not your copy: export it again after each renewal (jarl cert show says when that is). For another host, jarl cert get (below) does that for you. A program on this server can instead read the files Jarl writes for named (use dns), which Jarl keeps current.

Certificates for other hosts

jarl cert get gets a certificate onto another host (a mail server, a reverse proxy) with one command: jarld obtains it from the CA and renews it, answering the challenges in its own zones as for its own certificates; the host asks for it, waits until it is there, downloads it and runs a hook. The host needs only a token, not an ACME client or access to DNS.

On Jarl, as an administrator, give the hosts a user and a token each:

jarl user create hosts --no-password --grant "certificates:write names=*.adbraun.lol"
jarl token create mx --user hosts           # one token per host

The certificates grant says which names the user may have certificates for. It is never part of *, so operator and readonly users can't get certificates (or download keys) unless you add one. A certificate belongs to the user that asked for it: its tokens can download it, other users can't take its name or see it, and the certificates Jarl uses itself are the administrators'. Hosts that should share certificates use the same user (each with its own token, so you can revoke one host alone); hosts that shouldn't, use different users. A user may own certificates.max_per_user certificates (10 by default), or what jarl user update NAME --cert-limit N (or the user's form) sets.

On the host:

export JARL_SERVER=https://jarl.adbraun.lol:8053 JARL_TOKEN=@/root/jarl.token
jarl cert get mail -d mail.adbraun.lol -d smtp.adbraun.lol --hook 'systemctl reload postfix dovecot'
sudo jarl cert schedule                    # a daily systemd timer runs 'jarl cert get'
jarl cert saved                            # what this host has, and when it expires
jarl cert saved mail                       # all of mail's settings

• Asking again with the same names returns the certificate jarld has: nothing is obtained, nothing is downloaded until jarld renews it. Adding or removing a name gets a new certificate. A different --key-type alone doesn't; delete the certificate for that.
• Files, in --dir (default /etc/jarl/certs/NAME as root, ./NAME otherwise): cert.pem, chain.pem (the intermediates), fullchain.pem, privkey.pem (mode 600) and jarl-cert.json (serial, expiry, when it was fetched). Each file is replaced at once, never half written. The key is checked against the certificate before anything is written.
• A combined file (--combined PATH): the key, then the full chain, in one file (mode 600), as HAProxy wants. See HAProxy.
• Saved settings: the names, --dir, --combined, --hook, the server and the token are saved in /etc/jarl/cert-get/NAME.json (as root; ~/.config/jarl/cert-get otherwise), readable only by their owner. jarl cert get without a name does every saved certificate. A token given as @FILE is saved as that reference, not copied. jarl cert saved NAME shows them all (the token never). jarl cert forget NAME stops getting one (--delete deletes it on jarld too).
• Changing a setting: run jarl cert get NAME with just that option (--hook '...', --combined PATH, --dir DIR); the others stay. --hook '' removes the hook, --combined '' stops writing the combined file and deletes it. The run still asks jarld for the certificate, but downloads nothing while this host has the current one. A new or moved combined file is written from the files this host already has, and the hook runs.
• --local doesn't ask jarld at all: it writes the combined file and runs the hook from the files this host has. Use it to apply a changed setting while jarld is unreachable, or to test a hook. New names still need jarld.
• Several Jarl servers on one host: each server has its own certificate names, so two servers may both have a proxy. The host keeps one proxy per settings directory, though: asking another server for a name saved here for the first one is refused, as is a --dir holding another certificate's files. Choose another name or --dir, or pass --replace to take the name (or directory) over. A --dir that another saved certificate uses is always refused.
• The hook runs with sh after new files are written or the combined file changes (and with --local), with JARL_CERT_NAME, JARL_CERT_DIR, JARL_CERT_CERT, JARL_CERT_CHAIN, JARL_CERT_FULLCHAIN, JARL_CERT_KEY, JARL_CERT_COMBINED (empty without --combined), JARL_CERT_DOMAINS, JARL_CERT_SERIAL, JARL_CERT_NOT_AFTER and JARL_CERT_RENEWED (1 when it replaced an older certificate). If it fails, jarl cert get exits non-zero and runs it again next time. It has 5 minutes.
• jarl cert schedule (Linux with systemd, as root) writes /etc/systemd/system/jarl-cert-get.service and .timer, which run jarl cert get daily at a random time within the hour after midnight, and catch up after the host was off. --remove takes them away. On other systems, run jarl cert get daily with their scheduler.

On the Certificates page, the administrators see each certificate's owner and who downloaded it: the token, from where, when, and whether that host has the current certificate or hasn't fetched the renewed one yet. jarl cert show NAME says the same. The owner's own tokens see their own downloads there too. In jarl cert list, such a certificate's USES is get, and PULLED is the last download, with (old) while a host hasn't fetched the renewed one.

HAProxy

HAProxy wants the key and the chain in one file. Give jarl cert get a --combined path outside --dir:

jarl cert get proxy -d adbraun.lol -d '*.adbraun.lol' \
    --combined /etc/haproxy/certs/adbraun.lol.pem --hook 'systemctl reload haproxy'
jarl cert get webmail -d webmail.adbraun.lol \
    --combined /etc/haproxy/certs/webmail.pem --hook 'systemctl reload haproxy'

frontend https
    bind :443 ssl crt /etc/haproxy/certs/

HAProxy loads every file in a crt directory and picks one by SNI, so /etc/haproxy/certs should hold only combined files. Don't point crt at --dir (/etc/jarl/certs/NAME): HAProxy would try to load cert.pem, privkey.pem and jarl-cert.json there too, and refuse to start. A --combined path that would overwrite a file in a saved certificate's directory, or another certificate's combined file, is refused.

The combined file is mode 600, owned by the user running jarl cert get (root under the timer). HAProxy reads its certificates as root before it drops privileges, so that's fine.

To add a combined file to a certificate this host already has, run jarl cert get proxy --combined /etc/haproxy/certs/adbraun.lol.pem: it's written from the files on disk and the hook runs, with nothing downloaded.

Security notes

• Certificates Jarl obtains, their keys and the ACME account key are kept in Jarl's database, so they are in its backups (like TSIG secrets). The API returns a key only to an administrator who exports it, and records that in the audit log.
• Every publicly trusted certificate is published in Certificate Transparency logs, with all its names. A name like ns1.adbraun.lol used for both the web UI and DNS reveals less than separate names. Avoid wildcards where one name will do: a wildcard's key is good for every name in the domain.
• Using one certificate for jarld and named puts its key in two places. On one server, where jarld already writes named's configuration, that changes little.

API

Method	Path
GET	/certificates	certificates (each with status {state: ok, pending, failing, expiring, expired; message}, next_attempt, issuing_since while being obtained, serving, files) and account
POST	/certificates	{"name": "ns1", "domains": [...], "uses": ["web", "dns"], "challenge": "auto", "key_type": "ec256"}, or {"name", "source": "file", "cert_file", "key_file", "uses"}. One Jarl obtains starts being obtained at once ("issue": false leaves it to the loop)
POST	/certificates/check	{"domains": [...], "challenge": "auto"}: where each challenge would go (routes: route, target, zone, public_ns, why or error)
GET, PATCH, DELETE	/certificates/{name}	domains and uses replace the whole list
POST	/certificates/{name}/issue	obtain now: 202, then poll GET /certificates/{name} until issuing_since is gone; a file certificate is reread (200)
POST	/certificates/{name}/check	the routes for an existing certificate
GET	/certificates/{name}/pem?part=fullchain	the certificate as PEM, to save: fullchain (default), cert, key or bundle (key and chain). Exports of the key are audited

Testing with Pebble

Pebble is Let's Encrypt's test CA. Point it at a test named, and Jarl at it:

docker run -d --name pebble --network host -e PEBBLE_VA_NOSLEEP=1 \
  ghcr.io/letsencrypt/pebble -config test/config/pebble-config.json -dnsserver 127.0.0.1:53
docker cp pebble:/test/certs/pebble.minica.pem /etc/jarl/pebble.pem

acme:
  account: {directory: "https://localhost:14000/dir", agree_tos: true}
  resolvers: ["127.0.0.1:53"]
  ca_bundle: /etc/jarl/pebble.pem