jarl/docs/acme.md

Let's Encrypt (ACME DNS-01) with Jarl

The DNS-01 challenge proves that you control a domain by publishing a TXT record at _acme-challenge.<domain>. Jarl does this through its API, so no dynamic updates (allow-update, nsupdate) are needed. That matters because Jarl regenerates zone files and doesn't allow dynamic zones.

When a jarl acme command returns, named is already serving the record. Jarl reloads the zone and checks the new serial with rndc zonestatus before it reports success, so hooks need no "sleep and hope" delay. If you also run secondary servers, they pick the change up through NOTIFY within seconds. Add a short --dns-propagation-seconds/--dnssleep only if your CA checks them too.

1. A token that can only publish challenges

jarl user create certbot --no-password --role acme --comment "Let's Encrypt"
jarl token create certbot --user certbot --role acme -q > certbot.token
sudo install -m 0600 certbot.token /etc/jarl/certbot.token && rm certbot.token

The acme role is records:write names=_acme-challenge.* types=TXT. The token can add and remove challenge TXT records in any zone and nothing else. To restrict it to your zones, use an explicit grant instead:

jarl user create certbot --no-password --grant "records:write zones=adbraun.lol names=_acme-challenge.* types=TXT"

2. What jarl acme does

jarl acme set   DOMAIN VALUE...     # publish
jarl acme unset DOMAIN [VALUE...]   # remove these values, or all of them

• DOMAIN is the name on the certificate. _acme-challenge. is added for you, and a leading *. is dropped, since a wildcard and its apex share one challenge name. A name that already starts with _acme-challenge. is used as it is.
• The zone is found automatically: the longest matching primary zone.
• If the zone exists in several views (split DNS), the record is written into every view, so the CA (on the internet, in your external view) and your internal resolvers see the same thing. --view NAME limits it to one.
• All values and views go into one changeset. Setting a value that is already there succeeds, so a retried hook doesn't fail, and removing a value that isn't there succeeds as well.
• The record TTL is 60 seconds (--ttl).

3. certbot

sudo certbot certonly --manual --preferred-challenges dns \
  --manual-auth-hook    'jarl --token @/etc/jarl/certbot.token acme certbot-auth' \
  --manual-cleanup-hook 'jarl --token @/etc/jarl/certbot.token acme certbot-cleanup' \
  -d adbraun.lol -d '*.adbraun.lol'

The hooks read CERTBOT_DOMAIN and CERTBOT_VALIDATION from certbot. Add --server https://ns1.adbraun.lol:8053 when certbot runs on another host, or --server http://127.0.0.1:8053 on the DNS server itself (root's jarl CLI has no configuration file). certbot stores the hooks in the renewal configuration, so certbot renew works unattended.

4. acme.sh

Install the DNS API hook that comes with Jarl:

cp contrib/acme.sh/dns_jarl.sh ~/.acme.sh/dnsapi/    # or from /usr/share/doc/jarl-cli/contrib/ (jarl-cli package)
export JARL_SERVER=https://ns1.adbraun.lol:8053
export JARL_TOKEN=$(cat /etc/jarl/certbot.token)
acme.sh --issue --dns dns_jarl -d adbraun.lol -d '*.adbraun.lol'

The hook calls the jarl CLI, which must be installed on the same host. It remembers JARL_SERVER, JARL_TOKEN, JARL_VIEW (optional) and JARL_BIN (path to jarl, optional) in acme.sh's account configuration for renewals. When they are unset, jarl's own configuration file is used.

5. lego

lego's exec provider runs a program to publish and remove the record. Jarl ships one, contrib/lego/jarl-lego.sh, which calls the jarl CLI (so jarl must be installed on the same host):

sudo install -m 0755 contrib/lego/jarl-lego.sh /usr/local/bin/jarl-lego   # or from /usr/share/doc/jarl-cli/contrib/ (jarl-cli package)
export EXEC_PATH=/usr/local/bin/jarl-lego
export JARL_SERVER=https://ns1.adbraun.lol:8053
export JARL_TOKEN=@/etc/jarl/certbot.token
lego --email you@adbraun.lol --dns exec -d adbraun.lol -d '*.adbraun.lol' run

Renew with the same variables and lego ... renew in place of run.

• lego runs jarl-lego present FQDN VALUE and jarl-lego cleanup FQDN VALUE, and the script passes them to jarl acme set|unset --exact. FQDN is the full record name (_acme-challenge.adbraun.lol.), so --exact stops jarl from adding _acme-challenge. a second time.
• JARL_TOKEN may be the token itself or @FILE. JARL_VIEW limits the record to one view, and JARL_BIN points at jarl if it isn't in PATH. When JARL_SERVER and JARL_TOKEN are unset, jarl's configuration file is used.
• Leave EXEC_MODE unset: the script needs lego's default mode, not RAW.
• lego waits until the record is visible on the zone's name servers (EXEC_PROPAGATION_TIMEOUT, 60 seconds by default). Jarl answers only once named serves the record, so this check passes on its first try.

6. lego and Traefik over HTTP (httpreq), without the jarl CLI

lego's httpreq provider publishes the challenge with an HTTP request instead of running a program. jarld can answer those requests itself, so the machine running lego needs nothing from Jarl: no jarl binary, no script, only a few environment variables. That suits Traefik (which uses lego inside) and lego in a container.

	exec + jarl-lego (section 5)	httpreq (this section)
Needs on lego's host	the jarl CLI and the script	nothing
Turned on in jarld	always available	acme.httpreq.enabled: true
How the token travels	Bearer header, from the jarl CLI	HTTP Basic password, so HTTPS is required
Works in Traefik	no	yes

Both end up doing exactly the same thing in Jarl (the shared code is in internal/acme): jarl acme set|unset --exact FQDN VALUE.

How it works

lego is given a base URL, a username and a password. For every name on the certificate it sends

POST /api/v1/acme/httpreq/present HTTP/1.1
Authorization: Basic base64(lego:jarl_...)
Content-Type: application/json

{"fqdn": "_acme-challenge.adbraun.lol.", "value": "LHDhK3oGRvkiefQnx7OOczTY5Tic_xZ6HcMOc_gmtoM"}

and, once the CA has checked the record, the same body to /api/v1/acme/httpreq/cleanup. lego treats any 2xx answer as success. fqdn is the full record name, after lego has followed any CNAME (so CNAME delegation to an ACME zone, section 7, just works).

For each request jarld:

1. refuses it unless the endpoint is enabled (404 disabled);
2. refuses it unless it came over HTTPS (403 https_required, see HTTPS and reverse proxies);
3. checks the credentials: the password must be a valid API token and the username the user that owns it (401). Failures count towards the same per-IP lockout as other token failures and appear in the audit log as auth.token_failed;
4. checks the body: fqdn must be a domain name and value a DNS-01 challenge value (letters, digits, - and _) (400);
5. finds the longest primary zone that contains fqdn among the zones the token can see, in every view that has it (404 if there is none);
6. sets or removes the TXT record (TTL 60) in one changeset, checked by the token's grants like any other change (403 if they don't allow it), and answers 200 once named serves the change.

{"ok": true, "fqdn": "_acme-challenge.adbraun.lol.", "changeset_id": 412, "zones": ["adbraun.lol (view external)", "adbraun.lol (view internal)"]}

Setting a value that is already there, and removing one that isn't, both succeed, so lego can retry safely. The changeset description ends in "(lego httpreq)", so jarl history and the audit log show where a change came from.

HTTP Basic is accepted on these routes only. Everywhere else the API still takes only Bearer tokens and web sessions.

1. Enable the endpoint

In /etc/jarl/jarl.yaml:

acme:
  httpreq:
    enabled: true
    require_https: true          # the default; see below
    allow_loopback_http: false   # the default

Then sudo systemctl restart jarl. jarld's log says at startup whether the endpoint can work, for example acme.httpreq is enabled but jarld has no tls and no trusted_proxies, so every request will be refused.

HTTPS and reverse proxies

The token is in every request, so by default jarld only accepts requests that reached it over HTTPS. Everything else in Jarl (the web UI, the CLI, the rest of the API) works over plain HTTP exactly as before; only this endpoint checks. jarld decides like this:

The request came...	Counts as HTTPS when
straight to jarld with tls configured	always
from an address in trusted_proxies	that proxy says the client used HTTPS: X-Forwarded-Proto: https, or if that header is missing, Forwarded: proto=https (RFC 7239)
from any other address without TLS	never. X-Forwarded-Proto and Forwarded are ignored, since anyone can send them

Proxies append to these headers, so when a header holds a list (X-Forwarded-Proto: https, http) jarld uses the last value: the one the nearest proxy, the one it trusts, added. A value a client made up comes earlier in the list and is ignored. The same rule decides whether the web UI's session cookie gets the Secure flag (session.cookie_secure: auto).

Setup A: jarld does TLS itself. Set tls.cert_file and tls.key_file (configuration.md). Nothing else is needed.

Setup B: behind a reverse proxy that terminates TLS. Add the proxy's address (as jarld sees it) to trusted_proxies, and make sure the proxy sets X-Forwarded-Proto:

listen: ["127.0.0.1:8053"]
trusted_proxies: [127.0.0.1]     # the proxy runs on this host

• nginx doesn't set it by default:

  location / {
      proxy_pass http://127.0.0.1:8053;
      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
  }
  

• Caddy (reverse_proxy 127.0.0.1:8053) and Traefik set X-Forwarded-Proto themselves, and replace the value a client sent.
• HAProxy: http-request set-header X-Forwarded-Proto https if { ssl_fc }.

lego then uses the proxy's HTTPS URL as its endpoint.

Setup C: lego runs on the DNS server itself and talks to http://127.0.0.1:8053. Traffic over loopback never leaves the machine, so it can be allowed:

acme:
  httpreq:
    enabled: true
    allow_loopback_http: true

This accepts plain HTTP from 127.0.0.1 and ::1 only, and only when that address is not in trusted_proxies. A trusted proxy on the same host is judged by its headers instead, otherwise everything it forwarded from the internet would look local. Don't use allow_loopback_http if a reverse proxy runs on the DNS server but isn't in trusted_proxies.

Turning the check off (require_https: false) accepts plain HTTP from anywhere. jarld logs a warning at startup. Only do this on a network where you'd also send passwords in the clear.

When a request is refused for not being HTTPS, the answer explains why, for example: this request came over plain HTTP from 192.0.2.9, which sent X-Forwarded-Proto or Forwarded but isn't in trusted_proxies, so the header was ignored. jarld also logs a warning and an acme.httpreq_refused audit event. The token in such a request may have crossed the network unencrypted; if it did, revoke it (jarl token revoke ID) and make a new one.

Check a setup without a token. The status endpoint answers without authentication and says how jarld sees the request:

curl https://ns1.adbraun.lol/api/v1/acme/httpreq
{"accepted": true, "enabled": true, "https": true, "transport": "HTTPS to trusted proxy 127.0.0.1 (X-Forwarded-Proto: https)"}

Run it from where lego runs, against the URL lego will use.

2. A user and token for lego

jarl user create lego --no-password --role acme --comment "lego / Traefik"
jarl token create lego --user lego --role acme -q > lego.token

The username lego sends must be the token's user, lego here. For CNAME delegation into an ACME zone, use the zone grant from section 7 instead of the acme role.

3. lego

export HTTPREQ_ENDPOINT=https://ns1.adbraun.lol:8053/api/v1/acme/httpreq
export HTTPREQ_USERNAME=lego
export HTTPREQ_PASSWORD_FILE=/etc/jarl/lego.token     # or HTTPREQ_PASSWORD=jarl_...
lego --email you@adbraun.lol --dns httpreq -d adbraun.lol -d '*.adbraun.lol' run

Leave HTTPREQ_MODE unset (its RAW mode sends a different body, which jarld refuses with a message saying so). If jarld's certificate doesn't come from a public CA, lego has to trust yours: SSL_CERT_FILE=/path/to/your-ca.pem (lego's LEGO_CA_CERTIFICATES covers only the ACME server). To write into one view only, put it in the endpoint: .../api/v1/acme/httpreq/view/external.

4. Traefik

Traefik passes the same variables to lego. Static configuration:

certificatesResolvers:
  letsencrypt:
    acme:
      email: you@adbraun.lol
      storage: /letsencrypt/acme.json
      dnsChallenge:
        provider: httpreq

and in the Traefik container's environment:

environment:
  HTTPREQ_ENDPOINT: https://ns1.adbraun.lol:8053/api/v1/acme/httpreq
  HTTPREQ_USERNAME: lego
  HTTPREQ_PASSWORD_FILE: /run/secrets/jarl_token

Trying it by hand

TOKEN=$(cat lego.token)
curl -u "lego:$TOKEN" -H 'Content-Type: application/json' \
  -d '{"fqdn":"_acme-challenge.adbraun.lol.","value":"test-value"}' \
  https://ns1.adbraun.lol:8053/api/v1/acme/httpreq/present
dig @ns1.adbraun.lol _acme-challenge.adbraun.lol TXT +short
curl -u "lego:$TOKEN" -H 'Content-Type: application/json' \
  -d '{"fqdn":"_acme-challenge.adbraun.lol.","value":"test-value"}' \
  https://ns1.adbraun.lol:8053/api/v1/acme/httpreq/cleanup

Answers and what they mean

Status and code	Meaning
200	Done; named serves the change.
400 bad_request	The body isn't {"fqdn", "value"} (or is lego's RAW mode), or fqdn / value is invalid.
401 unauthenticated	No credentials, an invalid, expired or revoked token, or a username that isn't the token's user.
403 https_required	Not HTTPS, see above. If the token crossed a network unencrypted, revoke it.
403 forbidden	The token's grants don't allow this TXT record.
404 disabled	acme.httpreq.enabled is off.
404 not_found	No primary zone that the token can see contains fqdn (in that view).
422 and others	The change failed Jarl's validation or named's check, as with any change.
429 rate_limited	Too many failed logins from this address; wait a few minutes.

7. A dedicated ACME zone (CNAME delegation)

Instead of letting a token write into your main zone, point each challenge name at a small zone that exists only for ACME. The token may then write only there:

jarl zone create acme.adbraun.lol --ns ns1.adbraun.lol.
jarl record add adbraun.lol acme NS ns1.adbraun.lol.          # delegate it
jarl record add adbraun.lol _acme-challenge     CNAME apex.acme.adbraun.lol.
jarl record add adbraun.lol _acme-challenge.www CNAME www.acme.adbraun.lol.

jarl user create acme --no-password --grant "records:write zones=acme.adbraun.lol types=TXT"
jarl token create acme --user acme -q > acme.token

The CA follows the CNAME, so the TXT record has to be written at its target:

• certbot: add --follow-cname to both hooks, e.g. jarl --token @/etc/jarl/acme.token acme certbot-auth --follow-cname. jarl looks up _acme-challenge.<domain> and writes the TXT record at the target (www.acme.adbraun.lol). The lookup uses the host's resolvers. Add --resolver 127.0.0.1 to ask the Jarl server directly, for example when the host's resolver sees a different view.

• acme.sh does the lookup itself:

  • --challenge-alias acme.adbraun.lol puts every TXT record at _acme-challenge.acme.adbraun.lol. Point all the CNAMEs there (_acme-challenge.www CNAME _acme-challenge.acme.adbraun.lol.).
  • --domain-alias www.acme.adbraun.lol writes at exactly that name.

  The hook passes acme.sh's full record name to jarl acme --exact, so both modes work.

• lego (exec or httpreq) follows the CNAME itself (unless LEGO_DISABLE_CNAME_SUPPORT is set) and passes the target to jarl-lego or jarld, which write there as given. The acme role allows only names that start with _acme-challenge, so the token needs the grant above, not the role. lego uses the host's resolvers for the lookup; --dns.resolvers ns1.adbraun.lol:53 asks the Jarl server instead.

A leaked ACME token can then only touch TXT records in the ACME zone, and the main zone is never modified by automation.

Troubleshooting

• no primary zone that you can see contains ...: the token's grants don't cover the zone, or the zone isn't a primary zone in Jarl.
• permission denied: this needs records:write ... (exit code 3): the record name or type isn't allowed for the token. Compare with jarl --token @FILE whoami.
• The CA says the TXT record is missing: check from outside with dig @ns1.adbraun.lol _acme-challenge.adbraun.lol TXT. With views, make sure the record is in the view that matches the CA (it is by default) and that allow-query lets the internet query the zone.
• jarl audit --user certbot shows every attempt.