jarl/docs/acme.md

# Let's Encrypt (ACME DNS-01) with Jarl

This page is about **other** ACME clients (certbot, acme.sh, lego,
Traefik) publishing their DNS-01 challenges through Jarl. For
certificates for the DNS server itself (Jarl's web UI, named's DNS over
TLS and HTTPS), Jarl can be the ACME client: see
[certificates.md](certificates.md).

The DNS-01 challenge proves that you control a domain by publishing a TXT
record at `_acme-challenge.<domain>`. Jarl does this through its API, so no
dynamic updates (`allow-update`, nsupdate) are needed. That matters because
Jarl regenerates zone files and doesn't allow dynamic zones.

**When a `jarl acme` command returns, named is already serving the record.**
Jarl reloads the zone and checks the new serial with `rndc zonestatus`
before it reports success, so hooks need no "sleep and hope" delay. If you
also run secondary servers, they pick the change up through NOTIFY within
seconds. Add a short `--dns-propagation-seconds`/`--dnssleep` only if your
CA checks them too.

## 1. A token that can only publish challenges

```sh
jarl user create certbot --no-password --role acme --comment "Let's Encrypt"
jarl token create certbot --user certbot --role acme -q > certbot.token
sudo install -m 0600 certbot.token /etc/jarl/certbot.token && rm certbot.token
```

The `acme` role is `records:write names=_acme-challenge.* types=TXT`. The
token can add and remove challenge TXT records in any zone and nothing
else. To restrict it to your zones, use an explicit grant instead:

```sh
jarl user create certbot --no-password --grant "records:write zones=adbraun.lol names=_acme-challenge.* types=TXT"
```

## 2. What `jarl acme` does

```sh
jarl acme set   DOMAIN VALUE...     # publish
jarl acme unset DOMAIN [VALUE...]   # remove these values, or all of them
```

* `DOMAIN` is the name on the certificate. `_acme-challenge.` is added for
  you, and a leading `*.` is dropped, since a wildcard and its apex share one
  challenge name. A name that already starts with `_acme-challenge.` is used
  as it is.
* The zone is found automatically: the longest matching primary zone.
* If the zone exists in several **views** (split DNS), the record is written
  into every view, so the CA (on the internet, in your external view) and
  your internal resolvers see the same thing. `--view NAME` limits it to one,
  and so does a token for one view (`views=external`): in a linked zone its
  records are then local to that view.
* All values and views go into one changeset. Setting a value that is
  already there succeeds, so a retried hook doesn't fail, and removing a
  value that isn't there succeeds as well.
* The record TTL is 60 seconds (`--ttl`).

## 3. certbot

```sh
sudo certbot certonly --manual --preferred-challenges dns \
  --manual-auth-hook    'jarl --token @/etc/jarl/certbot.token acme certbot-auth' \
  --manual-cleanup-hook 'jarl --token @/etc/jarl/certbot.token acme certbot-cleanup' \
  -d adbraun.lol -d '*.adbraun.lol'
```

The hooks read `CERTBOT_DOMAIN` and `CERTBOT_VALIDATION` from certbot. Add
`--server https://ns1.adbraun.lol:8053` when certbot runs on another host,
or `--server http://127.0.0.1:8053` on the DNS server itself (root's jarl
CLI has no configuration file). certbot stores the hooks in the renewal
configuration, so `certbot renew` works unattended.

## 4. acme.sh

Install the DNS API hook that comes with Jarl:

```sh
cp contrib/acme.sh/dns_jarl.sh ~/.acme.sh/dnsapi/    # or from /usr/share/doc/jarl-cli/contrib/ (jarl-cli package)
export JARL_SERVER=https://ns1.adbraun.lol:8053
export JARL_TOKEN=$(cat /etc/jarl/certbot.token)
acme.sh --issue --dns dns_jarl -d adbraun.lol -d '*.adbraun.lol'
```

The hook calls the `jarl` CLI, which must be installed on the same host. It
remembers `JARL_SERVER`, `JARL_TOKEN`, `JARL_VIEW` (optional) and `JARL_BIN`
(path to jarl, optional) in acme.sh's account configuration for renewals.
When they are unset, jarl's own configuration file is used.

## 5. lego

lego's `exec` provider runs a program to publish and remove the record.
Jarl ships one, `contrib/lego/jarl-lego.sh`, which calls the `jarl` CLI
(so jarl must be installed on the same host):

```sh
sudo install -m 0755 contrib/lego/jarl-lego.sh /usr/local/bin/jarl-lego   # or from /usr/share/doc/jarl-cli/contrib/ (jarl-cli package)
export EXEC_PATH=/usr/local/bin/jarl-lego
export JARL_SERVER=https://ns1.adbraun.lol:8053
export JARL_TOKEN=@/etc/jarl/certbot.token
lego run --accept-tos --email you@adbraun.lol --dns exec -d adbraun.lol -d '*.adbraun.lol'
```

That is lego 5. `lego run` also renews: run the same command daily (a
cron job or systemd timer), and it renews once a third of the
certificate's lifetime is left. **lego 4** takes its options before the
command, and renews with `renew`:

```sh
lego --accept-tos --email you@adbraun.lol --dns exec -d adbraun.lol -d '*.adbraun.lol' run     # first time
lego --email you@adbraun.lol --dns exec -d adbraun.lol -d '*.adbraun.lol' renew                # then daily
```

Upgrading from lego 4 to 5, run `lego migrate` once (with `--path` if you
keep lego's data somewhere other than `./.lego`) before anything else:
lego 5 stores accounts and certificates differently. The exec and httpreq
providers, and so Jarl's side, are the same in both.

* lego runs `jarl-lego present FQDN VALUE` and `jarl-lego cleanup FQDN VALUE`,
  and the script passes them to `jarl acme set|unset --exact`. FQDN is the
  full record name (`_acme-challenge.adbraun.lol.`), so `--exact` stops
  jarl from adding `_acme-challenge.` a second time.
* `JARL_TOKEN` may be the token itself or `@FILE`. `JARL_VIEW` limits the
  record to one view, and `JARL_BIN` points at jarl if it isn't in `PATH`.
  When `JARL_SERVER` and `JARL_TOKEN` are unset, jarl's configuration file
  is used.
* Leave `EXEC_MODE` unset: the script needs lego's default mode, not `RAW`.
* lego waits until the record is visible on the zone's name servers
  (`EXEC_PROPAGATION_TIMEOUT`, 60 seconds by default). Jarl answers only
  once named serves the record, so this check passes on its first try,
  as long as the view that answers lego's host has the record: with
  `JARL_VIEW` on a host inside your network it doesn't. See
  [Split DNS and lego's propagation check](#split-dns-and-legos-propagation-check).

### Split DNS and lego's propagation check

Before lego asks the CA to look, it checks the record itself, asking the
zone's name servers (`ns1.adbraun.lol`) directly. With views, they answer
lego's host from the view that matches *it*. On a host inside your
network that is the internal view. If the challenge was written into
another view only (`JARL_VIEW=external`, `--view external`, a token
limited to `views=external`, or the httpreq endpoint
`.../httpreq/view/external`), lego never sees it, and gives up after
its propagation timeout without the CA having looked:

```
propagation: time limit exceeded: last error: authoritative nameservers:
NS ns1.adbraun.lol.:53 returned NXDOMAIN for _acme-challenge.stats.adbraun.lol.
```

A name whose authorization is still valid from an earlier run ("authorization
already valid; skipping challenge") isn't checked at all, so one name can
work and the next fail with the same settings.

The ways round it, best first:

1. **Don't limit the view.** Leave `JARL_VIEW` unset (and give the token
   no `views=` filter, the httpreq endpoint no `/view/...`): the record
   goes into every view that has the zone, the internal one too, and
   lego's check passes at once. The challenge is a random value that is
   useless once the CA has checked it, and it is removed straight after.
2. **Skip lego's check of the name servers.** The record is already
   served when Jarl answers, so lego's check adds nothing:

   ```sh
   lego run --dns exec --dns.propagation.disable-ans ...    # lego 5
   lego --dns exec --dns.propagation-disable-ans ... run    # lego 4.20 to 4.x
   lego --dns exec --dns.disable-cp ... run                 # older lego 4
   ```

   or replace the checks with a fixed wait, which also gives secondary
   name servers time to catch up: `--dns.propagation.wait 30s` (lego 5;
   `--dns.propagation-wait 30s` in lego 4.20 and later). In lego 5 these
   can also be set in the environment: `LEGO_DNS_PROPAGATION_DISABLE_ANS=true`,
   `LEGO_DNS_PROPAGATION_WAIT=30s`.
3. **In Traefik**, the same under `dnsChallenge` (Traefik 3.2 and later;
   older versions have `disablePropagationCheck: true`):

   ```yaml
   dnsChallenge:
     provider: httpreq
     propagation:
       disableANSChecks: true     # or delayBeforeChecks: 30s
   ```
4. **Run lego where the external view answers it**, such as a host outside
   your network. Rarely the easy option.

certbot (section 3) doesn't check by itself, and acme.sh (section 4)
checks through public DNS-over-HTTPS resolvers, so neither runs into this.

## 6. lego and Traefik over HTTP (httpreq), without the jarl CLI

lego's `httpreq` provider publishes the challenge with an HTTP request
instead of running a program. jarld can answer those requests itself, so
the machine running lego needs nothing from Jarl: no `jarl` binary, no
script, only a few environment variables. That suits Traefik (which
uses lego inside) and lego in a container.

|  | `exec` + `jarl-lego` (section 5) | `httpreq` (this section) |
|--|--|--|
| Needs on lego's host | the `jarl` CLI and the script | nothing |
| Turned on in jarld | always available | `acme.httpreq.enabled: true` |
| How the token travels | Bearer header, from the jarl CLI | HTTP Basic password, so HTTPS is required |
| Works in Traefik | no | yes |

Both end up doing exactly the same thing in Jarl (the shared code is in
`internal/acme`): `jarl acme set|unset --exact FQDN VALUE`.

### How it works

lego is given a base URL, a username and a password. For every name on
the certificate it sends

```http
POST /api/v1/acme/httpreq/present HTTP/1.1
Authorization: Basic base64(lego:jarl_...)
Content-Type: application/json

{"fqdn": "_acme-challenge.adbraun.lol.", "value": "LHDhK3oGRvkiefQnx7OOczTY5Tic_xZ6HcMOc_gmtoM"}
```

and, once the CA has checked the record, the same body to
`/api/v1/acme/httpreq/cleanup`. lego treats any 2xx answer as success.
`fqdn` is the full record name, after lego has followed any CNAME (so CNAME
delegation to an ACME zone, section 7, just works).

For each request jarld:

1. refuses it unless the endpoint is enabled (`404 disabled`);
2. refuses it unless it came over HTTPS (`403 https_required`, see
   [HTTPS and reverse proxies](#https-and-reverse-proxies));
3. checks the credentials: the password must be a valid API token and the
   username the user that owns it (`401`). Failures count towards the
   same per-IP lockout as other token failures and appear in the audit log
   as `auth.token_failed`;
4. checks the body: `fqdn` must be a domain name and `value` a DNS-01
   challenge value (letters, digits, `-` and `_`) (`400`);
5. finds the longest primary zone that contains `fqdn` among the zones the
   token can see, in every view that has it (`404` if there is none);
6. sets or removes the TXT record (TTL 60) in one changeset, checked by the
   token's grants like any other change (`403` if they don't allow it),
   and answers `200` once named serves the change.

```json
{"ok": true, "fqdn": "_acme-challenge.adbraun.lol.", "changeset_id": 412, "zones": ["adbraun.lol (view external)", "adbraun.lol (view internal)"]}
```

Setting a value that is already there, and removing one that isn't,
both succeed, so lego can retry safely. The changeset description ends in
"(lego httpreq)", so `jarl history` and the audit log show where a change
came from.

HTTP Basic is accepted on these routes only. Everywhere else the API
still takes only Bearer tokens and web sessions.

### 1. Enable the endpoint

In `/etc/jarl/jarl.yaml`:

```yaml
acme:
  httpreq:
    enabled: true
    require_https: true          # the default; see below
    allow_loopback_http: false   # the default
```

Then `sudo systemctl restart jarl`. jarld's log says at startup whether
the endpoint can work, for example `acme.httpreq is enabled but jarld has
no tls and no trusted_proxies, so every request will be refused`.

### HTTPS and reverse proxies

The token is in every request, so by default jarld only accepts requests
that reached it over HTTPS. Everything else in Jarl (the web UI, the CLI,
the rest of the API) works over plain HTTP exactly as before; only this
endpoint checks. jarld decides like this:

| The request came... | Counts as HTTPS when |
|--|--|
| straight to jarld with `tls` configured | always |
| from an address in `trusted_proxies` | that proxy says the client used HTTPS: `X-Forwarded-Proto: https`, or if that header is missing, `Forwarded: proto=https` (RFC 7239) |
| from any other address without TLS | never. `X-Forwarded-Proto` and `Forwarded` are ignored, since anyone can send them |

Proxies append to these headers, so when a header holds a list
(`X-Forwarded-Proto: https, http`) jarld uses the **last** value: the one
the nearest proxy, the one it trusts, added. A value a client made up comes
earlier in the list and is ignored. The same rule decides whether the web
UI's session cookie gets the Secure flag (`session.cookie_secure: auto`).

**Setup A: jarld does TLS itself.** Set `tls.cert_file` and
`tls.key_file` ([configuration.md](configuration.md)). Nothing else is
needed.

**Setup B: behind a reverse proxy that terminates TLS.** Add the proxy's
address (as jarld sees it) to `trusted_proxies`, and make sure the proxy
sets `X-Forwarded-Proto`:

```yaml
listen: ["127.0.0.1:8053"]
trusted_proxies: [127.0.0.1]     # the proxy runs on this host
```

* **nginx** doesn't set it by default:
  ```nginx
  location / {
      proxy_pass http://127.0.0.1:8053;
      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
  }
  ```
* **Caddy** (`reverse_proxy 127.0.0.1:8053`) and **Traefik** set
  `X-Forwarded-Proto` themselves, and replace the value a client sent.
* **HAProxy**: `http-request set-header X-Forwarded-Proto https if { ssl_fc }`.

lego then uses the proxy's HTTPS URL as its endpoint.

**Setup C: lego runs on the DNS server itself** and talks to
`http://127.0.0.1:8053`. Traffic over loopback never leaves the machine,
so it can be allowed:

```yaml
acme:
  httpreq:
    enabled: true
    allow_loopback_http: true
```

This accepts plain HTTP from `127.0.0.1` and `::1` only, and only when
that address is **not** in `trusted_proxies`. A trusted proxy on the same
host is judged by its headers instead, otherwise everything it forwarded
from the internet would look local. Don't use `allow_loopback_http` if a
reverse proxy runs on the DNS server but isn't in `trusted_proxies`.

**Turning the check off** (`require_https: false`) accepts plain HTTP
from anywhere. jarld logs a warning at startup. Only do this on a network
where you'd also send passwords in the clear.

When a request is refused for not being HTTPS, the answer explains why,
for example: `this request came over plain HTTP from 192.0.2.9, which sent
X-Forwarded-Proto or Forwarded but isn't in trusted_proxies, so the header
was ignored`. jarld also logs a warning and an `acme.httpreq_refused` audit
event. The token in such a request may have crossed the network
unencrypted; if it did, revoke it (`jarl token revoke ID`) and make a new
one.

**Check a setup** without a token. The status endpoint answers without
authentication and says how jarld sees the request:

```sh
curl https://ns1.adbraun.lol/api/v1/acme/httpreq
{"accepted": true, "enabled": true, "https": true, "transport": "HTTPS to trusted proxy 127.0.0.1 (X-Forwarded-Proto: https)"}
```

Run it from where lego runs, against the URL lego will use.

### 2. A user and token for lego

```sh
jarl user create lego --no-password --role acme --comment "lego / Traefik"
jarl token create lego --user lego --role acme -q > lego.token
```

The username lego sends must be the token's user, `lego` here. For CNAME
delegation into an ACME zone, use the zone grant from section 7 instead
of the `acme` role.

### 3. lego

```sh
export HTTPREQ_ENDPOINT=https://ns1.adbraun.lol:8053/api/v1/acme/httpreq
export HTTPREQ_USERNAME=lego
export HTTPREQ_PASSWORD_FILE=/etc/jarl/lego.token     # or HTTPREQ_PASSWORD=jarl_...
lego run --accept-tos --email you@adbraun.lol --dns httpreq -d adbraun.lol -d '*.adbraun.lol'
```

That is lego 5; in lego 4 the options go before the command (`lego
--email ... --dns httpreq -d ... run`, and `renew` to renew). See
[section 5](#5-lego).

Leave `HTTPREQ_MODE` unset (its `RAW` mode sends a different body, which
jarld refuses with a message saying so). If jarld's certificate doesn't come from a public
CA, lego has to trust yours: `SSL_CERT_FILE=/path/to/your-ca.pem` (lego's
`LEGO_CA_CERTIFICATES` covers only the ACME server). To write into one view only, put
it in the endpoint: `.../api/v1/acme/httpreq/view/external`; on a host inside
your network, read [Split DNS and lego's propagation check](#split-dns-and-legos-propagation-check)
first.

### 4. Traefik

Traefik passes the same variables to lego. Static configuration:

```yaml
certificatesResolvers:
  letsencrypt:
    acme:
      email: you@adbraun.lol
      storage: /letsencrypt/acme.json
      dnsChallenge:
        provider: httpreq
```

and in the Traefik container's environment:

```yaml
environment:
  HTTPREQ_ENDPOINT: https://ns1.adbraun.lol:8053/api/v1/acme/httpreq
  HTTPREQ_USERNAME: lego
  HTTPREQ_PASSWORD_FILE: /run/secrets/jarl_token
```

### Trying it by hand

```sh
TOKEN=$(cat lego.token)
curl -u "lego:$TOKEN" -H 'Content-Type: application/json' \
  -d '{"fqdn":"_acme-challenge.adbraun.lol.","value":"test-value"}' \
  https://ns1.adbraun.lol:8053/api/v1/acme/httpreq/present
dig @ns1.adbraun.lol _acme-challenge.adbraun.lol TXT +short
curl -u "lego:$TOKEN" -H 'Content-Type: application/json' \
  -d '{"fqdn":"_acme-challenge.adbraun.lol.","value":"test-value"}' \
  https://ns1.adbraun.lol:8053/api/v1/acme/httpreq/cleanup
```

### Answers and what they mean

| Status and code | Meaning |
|--|--|
| `200` | Done; named serves the change. |
| `400 bad_request` | The body isn't `{"fqdn", "value"}` (or is lego's `RAW` mode), or `fqdn` / `value` is invalid. |
| `401 unauthenticated` | No credentials, an invalid, expired or revoked token, or a username that isn't the token's user. |
| `403 https_required` | Not HTTPS, see above. If the token crossed a network unencrypted, revoke it. |
| `403 forbidden` | The token's grants don't allow this TXT record. |
| `404 disabled` | `acme.httpreq.enabled` is off. |
| `404 not_found` | No primary zone that the token can see contains `fqdn` (in that view). |
| `422` and others | The change failed Jarl's validation or named's check, as with any change. |
| `429 rate_limited` | Too many failed logins from this address; wait a few minutes. |

## 7. A dedicated ACME zone (CNAME delegation)

Instead of letting a token write into your main zone, point each challenge
name at a small zone that exists only for ACME. The token may then write
only there:

```sh
jarl zone create acme.adbraun.lol --ns ns1.adbraun.lol.
jarl record add adbraun.lol acme NS ns1.adbraun.lol.          # delegate it
jarl record add adbraun.lol _acme-challenge     CNAME apex.acme.adbraun.lol.
jarl record add adbraun.lol _acme-challenge.www CNAME www.acme.adbraun.lol.

jarl user create acme --no-password --grant "records:write zones=acme.adbraun.lol types=TXT"
jarl token create acme --user acme -q > acme.token
```

The CA follows the CNAME, so the TXT record has to be written at its
target:

* **certbot**: add `--follow-cname` to both hooks, e.g.
  `jarl --token @/etc/jarl/acme.token acme certbot-auth --follow-cname`.
  jarl looks up `_acme-challenge.<domain>` and writes the TXT record at the
  target (`www.acme.adbraun.lol`). The lookup uses the host's resolvers.
  Add `--resolver 127.0.0.1` to ask the Jarl server directly, for example
  when the host's resolver sees a different view.
* **acme.sh** does the lookup itself:
  * `--challenge-alias acme.adbraun.lol` puts every TXT record at
    `_acme-challenge.acme.adbraun.lol`. Point all the CNAMEs there
    (`_acme-challenge.www CNAME _acme-challenge.acme.adbraun.lol.`).
  * `--domain-alias www.acme.adbraun.lol` writes at exactly that name.

  The hook passes acme.sh's full record name to `jarl acme --exact`, so both
  modes work.
* **lego** (`exec` or `httpreq`) follows the CNAME itself (unless `LEGO_DISABLE_CNAME_SUPPORT`
  is set) and passes the target to `jarl-lego` or jarld, which write there as
  given. The `acme` role allows only names that start with
  `_acme-challenge`, so the token needs the grant above, not the role. lego
  uses the host's resolvers for the lookup; `--dns.resolvers
  ns1.adbraun.lol:53` asks the Jarl server instead.

A leaked ACME token can then only touch TXT records in the ACME zone, and
the main zone is never modified by automation.

## Troubleshooting

* `no primary zone that you can see contains ...`: the token's grants
  don't cover the zone, or the zone isn't a primary zone in Jarl.
* `permission denied: this needs records:write ...` (exit code 3): the
  record name or type isn't allowed for the token. Compare with
  `jarl --token @FILE whoami`.
* The CA says the TXT record is missing: check from outside with
  `dig @ns1.adbraun.lol _acme-challenge.adbraun.lol TXT`. With views, make
  sure the record is in the view that matches the CA (it is by default)
  and that `allow-query` lets the internet query the zone.
* lego says `propagation: time limit exceeded ... returned NXDOMAIN for
  _acme-challenge...`: the record is in a view that doesn't answer lego's
  host, usually `JARL_VIEW=external` on a host inside your network. See
  [Split DNS and lego's propagation check](#split-dns-and-legos-propagation-check).
* `jarl audit --user certbot` shows every attempt.