adam/jarl
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases 34 Wiki Activity

Add an httpreq endpoint for lego and Traefik

Browse Source 
jarld can now answer lego's httpreq DNS provider
(POST /api/v1/acme/httpreq/present and /cleanup), so lego and Traefik
publish DNS-01 challenges with only environment variables, without the
jarl CLI. lego's username and API token arrive as HTTP Basic auth,
accepted on these routes only.

The endpoint is off by default (acme.httpreq.enabled) and HTTPS-only by
default: jarld's own TLS, or a trusted proxy that says so with
X-Forwarded-Proto or Forwarded (the last value counts). Plain HTTP can be
allowed from loopback only or everywhere. GET /api/v1/acme/httpreq shows
how jarld sees a request, to check a proxy setup.

jarl acme and the endpoint share their code in internal/acme. Documented
in docs/acme.md, configuration.md, api.md and permissions.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
abrown38andClaude Opus 5.5 committed 2026-10-08 13:48:47 +00:00
1 parent 0221c78c8a
commit 6b44b92975
18 files changed
+1106 -73
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CHANGELOG.md
+14
View File
@@ -24,6 +24,20 @@ The first version, to be released as 0.1.0.
the web UI. Networks smaller than a /24 explain that they belong in the the web UI. Networks smaller than a /24 explain that they belong in the
enclosing /24's zone. enclosing /24's zone.
- docs/operations.md: setting up a secondary server with its own jarld. - docs/operations.md: setting up a secondary server with its own jarld.
- An endpoint for lego's `httpreq` DNS provider
(`/api/v1/acme/httpreq/present` and `/cleanup`), so lego and Traefik can
publish DNS-01 challenges with only environment variables, without the
jarl CLI. lego's username and API token arrive as HTTP Basic auth, which
is accepted on these routes only. Off by default
(`acme.httpreq.enabled`), and HTTPS-only by default: jarld's own TLS, or a
trusted proxy that says so with `X-Forwarded-Proto` or `Forwarded`.
Plain HTTP can be allowed from loopback only (`allow_loopback_http`) or
everywhere (`require_https: false`). `GET /api/v1/acme/httpreq` shows how
jarld sees a request, to check a proxy setup. `jarl acme` and the endpoint
share their code (`internal/acme`).
- Trusted proxies may also send the standard `Forwarded: proto=https`
header. When a proxy header lists several values, the last one (added by
the nearest proxy) counts.
- lego support: `contrib/lego/jarl-lego.sh` is a program for lego's `exec` - lego support: `contrib/lego/jarl-lego.sh` is a program for lego's `exec`
DNS provider that publishes DNS-01 challenges through `jarl acme`, DNS provider that publishes DNS-01 challenges through `jarl acme`,
including challenge names reached through a CNAME (docs/acme.md). including challenge names reached through a CNAME (docs/acme.md).
README.md
+3 -2
View File
@@ -47,7 +47,8 @@ error: www.adbraun.lol has a CNAME, so it can't have any other records (found A)
and exportable as JSON. Stored as aggregates in a separate database with and exportable as JSON. Stored as aggregates in a separate database with
retention you set. retention you set.
* **Let's Encrypt DNS-01** built in: `jarl acme` plus certbot, acme.sh and lego * **Let's Encrypt DNS-01** built in: `jarl acme` plus certbot, acme.sh and lego
hooks, including CNAME delegation to a dedicated ACME zone. hooks, an HTTPS-only endpoint for lego's and Traefik's `httpreq` provider,
and CNAME delegation to a dedicated ACME zone.
* **Adopts an existing server**: imports named.conf and zone files * **Adopts an existing server**: imports named.conf and zone files
(comments included), then switches named over with an automatic (comments included), then switches named over with an automatic
fallback. fallback.
@@ -88,7 +89,7 @@ in `/etc/jarl/jarl.yaml` to reach it from your management network.
| [docs/api.md](docs/api.md) | REST API reference | | [docs/api.md](docs/api.md) | REST API reference |
| [docs/permissions.md](docs/permissions.md) | Users, tokens, grants and roles | | [docs/permissions.md](docs/permissions.md) | Users, tokens, grants and roles |
| [docs/views.md](docs/views.md) | Split DNS: adding views to a server that has none | | [docs/views.md](docs/views.md) | Split DNS: adding views to a server that has none |
| [docs/acme.md](docs/acme.md) | Let's Encrypt DNS-01 with certbot, acme.sh and lego | | [docs/acme.md](docs/acme.md) | Let's Encrypt DNS-01 with certbot, acme.sh, lego and Traefik |
| [docs/stats.md](docs/stats.md) | Query statistics: enabling, what is collected, retention | | [docs/stats.md](docs/stats.md) | Query statistics: enabling, what is collected, retention |
| [docs/validation.md](docs/validation.md) | Every check Jarl makes | | [docs/validation.md](docs/validation.md) | Every check Jarl makes |
| [docs/operations.md](docs/operations.md) | Backups, drift, failures, recovery, monitoring | | [docs/operations.md](docs/operations.md) | Backups, drift, failures, recovery, monitoring |
cmd/jarl/acme.go
+9 -53
View File
@@ -10,6 +10,7 @@ import (
"github.com/spf13/cobra" "github.com/spf13/cobra"
"jarl/internal/acme"
"jarl/internal/api" "jarl/internal/api"
"jarl/internal/client" "jarl/internal/client"
) )
@@ -75,28 +76,16 @@ func (o *acmeOptions) challengeName(domain string) (string, error) {
} }
// acmeZones finds the primary zones (one per view) that contain name. // acmeZones finds the primary zones (one per view) that contain name.
func acmeZones(c *client.Client, name, view string) ([]api.Zone, error) { func acmeZones(c *client.Client, name, view string) ([]acme.Zone, error) {
var zones []api.Zone var zones []api.Zone
if err := c.Get(ctx(), "/zones", q("type", "primary"), &zones); err != nil { if err := c.Get(ctx(), "/zones", q("type", "primary"), &zones); err != nil {
return nil, err return nil, err
} }
best := "" refs := make([]acme.Zone, len(zones))
var out []api.Zone for i, z := range zones {
for _, z := range zones { refs[i] = acme.Zone{Name: z.Name, View: z.View}
if view != "" && !strings.EqualFold(z.View, view) {
continue
}
zn := strings.ToLower(z.Name)
if !(name == zn || zn == "." || strings.HasSuffix(name, "."+zn)) {
continue
}
switch {
case len(zn) > len(best):
best, out = zn, []api.Zone{z}
case zn == best:
out = append(out, z)
}
} }
out := acme.Zones(name, refs, view)
if len(out) == 0 { if len(out) == 0 {
where := "" where := ""
if view != "" { if view != "" {
@@ -121,40 +110,7 @@ func acmeChange(o *acmeOptions, domain string, values []string, set bool) error
if err != nil { if err != nil {
return err return err
} }
var reqs []api.ChangeRequest return submitBatch(c, acme.Description(name, set), acme.Requests(name, zones, values, set, o.ttl))
for _, z := range zones {
base := map[string]any{"zone": z.Name, "name": name, "type": "TXT"}
if z.View != "" {
base["view"] = z.View
}
with := func(extra map[string]any) map[string]any {
m := map[string]any{}
for k, v := range base {
m[k] = v
}
for k, v := range extra {
m[k] = v
}
return m
}
if !set && len(values) == 0 {
reqs = append(reqs, api.ChangeRequest{Op: "record.delete", Data: rawJSON(with(map[string]any{"missing_ok": true}))})
continue
}
for _, v := range values {
// Deleting first makes a repeated 'set' (a retried hook)
// succeed instead of failing on a duplicate record.
reqs = append(reqs, api.ChangeRequest{Op: "record.delete", Data: rawJSON(with(map[string]any{"data": v, "missing_ok": true}))})
if set {
reqs = append(reqs, api.ChangeRequest{Op: "record.create", Data: rawJSON(with(map[string]any{"data": v, "ttl": o.ttl, "comment": "ACME DNS-01 challenge"}))})
}
}
}
verb := "remove"
if set {
verb = "set"
}
return submitBatch(c, fmt.Sprintf("ACME: %s challenge for %s", verb, dot(name)), reqs)
} }
// envOrError returns an environment variable that a hook needs. // envOrError returns an environment variable that a hook needs.
@@ -206,7 +162,7 @@ acme.sh: copy contrib/acme.sh/dns_jarl.sh into ~/.acme.sh/dnsapi/ and use
}, },
} }
o.flags(set) o.flags(set)
set.Flags().Uint32Var(&o.ttl, "ttl", 60, "TTL of the challenge record") set.Flags().Uint32Var(&o.ttl, "ttl", acme.DefaultTTL, "TTL of the challenge record")
unset := &cobra.Command{ unset := &cobra.Command{
Use: "unset DOMAIN [VALUE...]", Use: "unset DOMAIN [VALUE...]",
Short: "Remove challenge values for DOMAIN (all of them when no VALUE is given)", Short: "Remove challenge values for DOMAIN (all of them when no VALUE is given)",
@@ -233,7 +189,7 @@ acme.sh: copy contrib/acme.sh/dns_jarl.sh into ~/.acme.sh/dnsapi/ and use
}, },
} }
o.flags(auth) o.flags(auth)
auth.Flags().Uint32Var(&o.ttl, "ttl", 60, "TTL of the challenge record") auth.Flags().Uint32Var(&o.ttl, "ttl", acme.DefaultTTL, "TTL of the challenge record")
cleanup := &cobra.Command{ cleanup := &cobra.Command{
Use: "certbot-cleanup", Use: "certbot-cleanup",
Short: "certbot --manual-cleanup-hook: remove $CERTBOT_VALIDATION for $CERTBOT_DOMAIN", Short: "certbot --manual-cleanup-hook: remove $CERTBOT_VALIDATION for $CERTBOT_DOMAIN",
cmd/jarld/main.go
+5
View File
@@ -287,6 +287,11 @@ func serveCmd() *cobra.Command {
TrustedProxies: a.cfg.Proxies(), SessionLifetime: time.Duration(a.cfg.Session.Lifetime), TrustedProxies: a.cfg.Proxies(), SessionLifetime: time.Duration(a.cfg.Session.Lifetime),
SessionIdle: time.Duration(a.cfg.Session.IdleTimeout), CookieSecure: a.cfg.Session.CookieSecure, SessionIdle: time.Duration(a.cfg.Session.IdleTimeout), CookieSecure: a.cfg.Session.CookieSecure,
Version: version.Version, Stats: collector, Version: version.Version, Stats: collector,
HTTPReq: server.HTTPReq{
Enabled: a.cfg.ACME.HTTPReq.Enabled,
AllowPlainHTTP: !a.cfg.ACME.HTTPReq.RequireHTTPS,
AllowLoopbackHTTP: a.cfg.ACME.HTTPReq.AllowLoopbackHTTP,
},
}, a.eng, a.log, web.FS()) }, a.eng, a.log, web.FS())
hup := make(chan os.Signal, 1) hup := make(chan os.Signal, 1)
signal.Notify(hup, syscall.SIGHUP) signal.Notify(hup, syscall.SIGHUP)
docs/acme.md
+246 -4
View File
@@ -108,7 +108,249 @@ Renew with the same variables and `lego ... renew` in place of `run`.
(`EXEC_PROPAGATION_TIMEOUT`, 60 seconds by default). Jarl answers only (`EXEC_PROPAGATION_TIMEOUT`, 60 seconds by default). Jarl answers only
once named serves the record, so this check passes on its first try. once named serves the record, so this check passes on its first try.
## 6. A dedicated ACME zone (CNAME delegation) ## 6. lego and Traefik over HTTP (httpreq), without the jarl CLI
lego's `httpreq` provider publishes the challenge with an HTTP request
instead of running a program. jarld can answer those requests itself, so
the machine running lego needs nothing from Jarl: no `jarl` binary, no
script, only a few environment variables. That suits Traefik (which
uses lego inside) and lego in a container.
| | `exec` + `jarl-lego` (section 5) | `httpreq` (this section) |
|--|--|--|
| Needs on lego's host | the `jarl` CLI and the script | nothing |
| Turned on in jarld | always available | `acme.httpreq.enabled: true` |
| How the token travels | Bearer header, from the jarl CLI | HTTP Basic password, so HTTPS is required |
| Works in Traefik | no | yes |
Both end up doing exactly the same thing in Jarl (the shared code is in
`internal/acme`): `jarl acme set|unset --exact FQDN VALUE`.
### How it works
lego is given a base URL, a username and a password. For every name on
the certificate it sends
```http
POST /api/v1/acme/httpreq/present HTTP/1.1
Authorization: Basic base64(lego:jarl_...)
Content-Type: application/json
{"fqdn": "_acme-challenge.adbraun.lol.", "value": "LHDhK3oGRvkiefQnx7OOczTY5Tic_xZ6HcMOc_gmtoM"}
```
and, once the CA has checked the record, the same body to
`/api/v1/acme/httpreq/cleanup`. lego treats any 2xx answer as success.
`fqdn` is the full record name, after lego has followed any CNAME (so CNAME
delegation to an ACME zone, section 7, just works).
For each request jarld:
1. refuses it unless the endpoint is enabled (`404 disabled`);
2. refuses it unless it came over HTTPS (`403 https_required`, see
[HTTPS and reverse proxies](#https-and-reverse-proxies));
3. checks the credentials: the password must be a valid API token and the
username the user that owns it (`401`). Failures count towards the
same per-IP lockout as other token failures and appear in the audit log
as `auth.token_failed`;
4. checks the body: `fqdn` must be a domain name and `value` a DNS-01
challenge value (letters, digits, `-` and `_`) (`400`);
5. finds the longest primary zone that contains `fqdn` among the zones the
token can see, in every view that has it (`404` if there is none);
6. sets or removes the TXT record (TTL 60) in one changeset, checked by the
token's grants like any other change (`403` if they don't allow it),
and answers `200` once named serves the change.
```json
{"ok": true, "fqdn": "_acme-challenge.adbraun.lol.", "changeset_id": 412, "zones": ["adbraun.lol (view external)", "adbraun.lol (view internal)"]}
```
Setting a value that is already there, and removing one that isn't,
both succeed, so lego can retry safely. The changeset description ends in
"(lego httpreq)", so `jarl history` and the audit log show where a change
came from.
HTTP Basic is accepted on these routes only. Everywhere else the API
still takes only Bearer tokens and web sessions.
### 1. Enable the endpoint
In `/etc/jarl/jarl.yaml`:
```yaml
acme:
httpreq:
enabled: true
require_https: true # the default; see below
allow_loopback_http: false # the default
```
Then `sudo systemctl restart jarl`. jarld's log says at startup whether
the endpoint can work, for example `acme.httpreq is enabled but jarld has
no tls and no trusted_proxies, so every request will be refused`.
### HTTPS and reverse proxies
The token is in every request, so by default jarld only accepts requests
that reached it over HTTPS. Everything else in Jarl (the web UI, the CLI,
the rest of the API) works over plain HTTP exactly as before; only this
endpoint checks. jarld decides like this:
| The request came... | Counts as HTTPS when |
|--|--|
| straight to jarld with `tls` configured | always |
| from an address in `trusted_proxies` | that proxy says the client used HTTPS: `X-Forwarded-Proto: https`, or if that header is missing, `Forwarded: proto=https` (RFC 7239) |
| from any other address without TLS | never. `X-Forwarded-Proto` and `Forwarded` are ignored, since anyone can send them |
Proxies append to these headers, so when a header holds a list
(`X-Forwarded-Proto: https, http`) jarld uses the **last** value: the one
the nearest proxy, the one it trusts, added. A value a client made up comes
earlier in the list and is ignored. The same rule decides whether the web
UI's session cookie gets the Secure flag (`session.cookie_secure: auto`).
**Setup A: jarld does TLS itself.** Set `tls.cert_file` and
`tls.key_file` ([configuration.md](configuration.md)). Nothing else is
needed.
**Setup B: behind a reverse proxy that terminates TLS.** Add the proxy's
address (as jarld sees it) to `trusted_proxies`, and make sure the proxy
sets `X-Forwarded-Proto`:
```yaml
listen: ["127.0.0.1:8053"]
trusted_proxies: [127.0.0.1] # the proxy runs on this host
```
* **nginx** doesn't set it by default:
```nginx
location / {
proxy_pass http://127.0.0.1:8053;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
```
* **Caddy** (`reverse_proxy 127.0.0.1:8053`) and **Traefik** set
`X-Forwarded-Proto` themselves, and replace the value a client sent.
* **HAProxy**: `http-request set-header X-Forwarded-Proto https if { ssl_fc }`.
lego then uses the proxy's HTTPS URL as its endpoint.
**Setup C: lego runs on the DNS server itself** and talks to
`http://127.0.0.1:8053`. Traffic over loopback never leaves the machine,
so it can be allowed:
```yaml
acme:
httpreq:
enabled: true
allow_loopback_http: true
```
This accepts plain HTTP from `127.0.0.1` and `::1` only, and only when
that address is **not** in `trusted_proxies`. A trusted proxy on the same
host is judged by its headers instead, otherwise everything it forwarded
from the internet would look local. Don't use `allow_loopback_http` if a
reverse proxy runs on the DNS server but isn't in `trusted_proxies`.
**Turning the check off** (`require_https: false`) accepts plain HTTP
from anywhere. jarld logs a warning at startup. Only do this on a network
where you'd also send passwords in the clear.
When a request is refused for not being HTTPS, the answer explains why,
for example: `this request came over plain HTTP from 192.0.2.9, which sent
X-Forwarded-Proto or Forwarded but isn't in trusted_proxies, so the header
was ignored`. jarld also logs a warning and an `acme.httpreq_refused` audit
event. The token in such a request may have crossed the network
unencrypted; if it did, revoke it (`jarl token revoke ID`) and make a new
one.
**Check a setup** without a token. The status endpoint answers without
authentication and says how jarld sees the request:
```sh
curl https://ns1.adbraun.lol/api/v1/acme/httpreq
{"accepted": true, "enabled": true, "https": true, "transport": "HTTPS to trusted proxy 127.0.0.1 (X-Forwarded-Proto: https)"}
```
Run it from where lego runs, against the URL lego will use.
### 2. A user and token for lego
```sh
jarl user create lego --no-password --role acme --comment "lego / Traefik"
jarl token create lego --user lego --role acme -q > lego.token
```
The username lego sends must be the token's user, `lego` here. For CNAME
delegation into an ACME zone, use the zone grant from section 7 instead
of the `acme` role.
### 3. lego
```sh
export HTTPREQ_ENDPOINT=https://ns1.adbraun.lol:8053/api/v1/acme/httpreq
export HTTPREQ_USERNAME=lego
export HTTPREQ_PASSWORD_FILE=/etc/jarl/lego.token # or HTTPREQ_PASSWORD=jarl_...
lego --email you@adbraun.lol --dns httpreq -d adbraun.lol -d '*.adbraun.lol' run
```
Leave `HTTPREQ_MODE` unset (its `RAW` mode sends a different body, which
jarld refuses with a message saying so). If jarld's certificate doesn't come from a public
CA, lego has to trust yours: `SSL_CERT_FILE=/path/to/your-ca.pem` (lego's
`LEGO_CA_CERTIFICATES` covers only the ACME server). To write into one view only, put
it in the endpoint: `.../api/v1/acme/httpreq/view/external`.
### 4. Traefik
Traefik passes the same variables to lego. Static configuration:
```yaml
certificatesResolvers:
letsencrypt:
acme:
email: you@adbraun.lol
storage: /letsencrypt/acme.json
dnsChallenge:
provider: httpreq
```
and in the Traefik container's environment:
```yaml
environment:
HTTPREQ_ENDPOINT: https://ns1.adbraun.lol:8053/api/v1/acme/httpreq
HTTPREQ_USERNAME: lego
HTTPREQ_PASSWORD_FILE: /run/secrets/jarl_token
```
### Trying it by hand
```sh
TOKEN=$(cat lego.token)
curl -u "lego:$TOKEN" -H 'Content-Type: application/json' \
-d '{"fqdn":"_acme-challenge.adbraun.lol.","value":"test-value"}' \
https://ns1.adbraun.lol:8053/api/v1/acme/httpreq/present
dig @ns1.adbraun.lol _acme-challenge.adbraun.lol TXT +short
curl -u "lego:$TOKEN" -H 'Content-Type: application/json' \
-d '{"fqdn":"_acme-challenge.adbraun.lol.","value":"test-value"}' \
https://ns1.adbraun.lol:8053/api/v1/acme/httpreq/cleanup
```
### Answers and what they mean
| Status and code | Meaning |
|--|--|
| `200` | Done; named serves the change. |
| `400 bad_request` | The body isn't `{"fqdn", "value"}` (or is lego's `RAW` mode), or `fqdn` / `value` is invalid. |
| `401 unauthenticated` | No credentials, an invalid, expired or revoked token, or a username that isn't the token's user. |
| `403 https_required` | Not HTTPS, see above. If the token crossed a network unencrypted, revoke it. |
| `403 forbidden` | The token's grants don't allow this TXT record. |
| `404 disabled` | `acme.httpreq.enabled` is off. |
| `404 not_found` | No primary zone that the token can see contains `fqdn` (in that view). |
| `422` and others | The change failed Jarl's validation or named's check, as with any change. |
| `429 rate_limited` | Too many failed logins from this address; wait a few minutes. |
## 7. A dedicated ACME zone (CNAME delegation)
Instead of letting a token write into your main zone, point each challenge Instead of letting a token write into your main zone, point each challenge
name at a small zone that exists only for ACME. The token may then write name at a small zone that exists only for ACME. The token may then write
@@ -141,9 +383,9 @@ target:
The hook passes acme.sh's full record name to `jarl acme --exact`, so both The hook passes acme.sh's full record name to `jarl acme --exact`, so both
modes work. modes work.
* **lego** follows the CNAME itself (unless `LEGO_DISABLE_CNAME_SUPPORT` * **lego** (`exec` or `httpreq`) follows the CNAME itself (unless `LEGO_DISABLE_CNAME_SUPPORT`
is set) and passes the target to `jarl-lego`, which writes there with is set) and passes the target to `jarl-lego` or jarld, which write there as
`--exact`. The `acme` role allows only names that start with given. The `acme` role allows only names that start with
`_acme-challenge`, so the token needs the grant above, not the role. lego `_acme-challenge`, so the token needs the grant above, not the role. lego
uses the host's resolvers for the lookup; `--dns.resolvers uses the host's resolvers for the lookup; `--dns.resolvers
ns1.adbraun.lol:53` asks the Jarl server instead. ns1.adbraun.lol:53` asks the Jarl server instead.
docs/api.md
+18
View File
@@ -28,6 +28,9 @@ change something (anything but GET/HEAD) must send it as `X-CSRF-Token`.
`GET /auth/whoami` returns the user, the token (if any), the effective `GET /auth/whoami` returns the user, the token (if any), the effective
permissions and whether the caller is an administrator. permissions and whether the caller is an administrator.
**HTTP Basic**, only on the lego `httpreq` routes (below): the username is
the token's user and the password the API token.
## Errors ## Errors
```http ```http
@@ -286,6 +289,21 @@ Needs `stats:read`. When statistics are disabled, every endpoint but
curl -sf -H "$AUTH" "$API/stats/top?kind=nxdomain&since=7d&limit=10" | jq -r '.[] | "\(.count)\t\(.key)"' curl -sf -H "$AUTH" "$API/stats/top?kind=nxdomain&since=7d&limit=10" | jq -r '.[] | "\(.count)\t\(.key)"'
``` ```
### ACME: lego's httpreq provider
Off unless `acme.httpreq.enabled` is set, and by default HTTPS only. The
full description is in [acme.md](acme.md#6-lego-and-traefik-over-http-httpreq-without-the-jarl-cli).
| Method | Path | Body |
|--------|------|------|
| GET | `/acme/httpreq` | No authentication. `{"enabled", "https", "accepted", "transport"}`: whether the endpoint is on and how jarld sees this request (HTTPS or not, and why) |
| POST | `/acme/httpreq/present` | `{"fqdn": "_acme-challenge.adbraun.lol.", "value": "..."}`. Sets the TXT value in the longest matching primary zone, in every view that has it |
| POST | `/acme/httpreq/cleanup` | Same body. Removes that value (succeeds if it's already gone) |
| POST | `/acme/httpreq/view/{view}/present`, `.../cleanup` | The same, in one view only |
Authenticate with HTTP Basic (user and API token) or a Bearer token.
Success is `200 {"ok": true, "fqdn": ..., "changeset_id": ..., "zones": [...]}`.
### Users and tokens ### Users and tokens
| Method | Path | Body | | Method | Path | Body |
docs/configuration.md
+21 -1
View File
@@ -20,7 +20,7 @@ The CLI has its own small configuration file, described in [cli.md](cli.md).
|-----|---------|---------| |-----|---------|---------|
| `listen` | `["127.0.0.1:8053"]` | Addresses to listen on (`host:port`, several allowed). Keep these on localhost and management networks. | | `listen` | `["127.0.0.1:8053"]` | Addresses to listen on (`host:port`, several allowed). Keep these on localhost and management networks. |
| `tls.cert_file`, `tls.key_file` | (empty: plain HTTP) | PEM certificate chain and key. Both or neither. The files are rechecked every 10 minutes and on SIGHUP (`systemctl reload jarl`), so renewed certificates are used without a restart. | | `tls.cert_file`, `tls.key_file` | (empty: plain HTTP) | PEM certificate chain and key. Both or neither. The files are rechecked every 10 minutes and on SIGHUP (`systemctl reload jarl`), so renewed certificates are used without a restart. |
| `trusted_proxies` | `[]` | Addresses or prefixes of reverse proxies whose `X-Forwarded-For` and `X-Forwarded-Proto` headers are believed. Needed for correct client IPs in the audit log and for rate limiting behind a proxy. | | `trusted_proxies` | `[]` | Addresses or prefixes of reverse proxies whose `X-Forwarded-For` and `X-Forwarded-Proto` (or `Forwarded: proto=`) headers are believed. Needed for correct client IPs in the audit log and for rate limiting behind a proxy, and for knowing that a proxied request came over HTTPS (the session cookie's Secure flag, `acme.httpreq`). From other addresses these headers are ignored. |
| `session.lifetime` | `12h` | A web session ends this long after login... | | `session.lifetime` | `12h` | A web session ends this long after login... |
| `session.idle_timeout` | `2h` | ...or after this long without requests. | | `session.idle_timeout` | `2h` | ...or after this long without requests. |
| `session.cookie_secure` | `auto` | `auto` sets the cookie's Secure flag when the request came over HTTPS (directly or via a trusted proxy); `always` / `never` force it. | | `session.cookie_secure` | `auto` | `auto` sets the cookie's Secure flag when the request came over HTTPS (directly or via a trusted proxy); `always` / `never` force it. |
@@ -51,6 +51,20 @@ The `stats:` section turns on query statistics (off by default). See
[stats.md](stats.md) for every setting and what it changes in named's [stats.md](stats.md) for every setting and what it changes in named's
configuration. Changing it takes effect when jarld restarts. configuration. Changing it takes effect when jarld restarts.
### ACME (lego httpreq endpoint)
An endpoint for lego's `httpreq` DNS provider, so lego and Traefik can
publish DNS-01 challenges without the jarl CLI. See
[acme.md](acme.md#6-lego-and-traefik-over-http-httpreq-without-the-jarl-cli)
for how it works and how to set up HTTPS for it. Changes take effect when
jarld restarts.
| Key | Default | Meaning |
|-----|---------|---------|
| `acme.httpreq.enabled` | `false` | Answer `POST /api/v1/acme/httpreq/present` and `/cleanup`. Off, they answer `404 disabled`. |
| `acme.httpreq.require_https` | `true` | Refuse requests that didn't come over HTTPS: jarld's own `tls`, or a proxy in `trusted_proxies` that sends `X-Forwarded-Proto: https`. The rest of Jarl is unaffected by this setting. `false` accepts plain HTTP from anywhere (logged as a warning at startup). |
| `acme.httpreq.allow_loopback_http` | `false` | Also accept plain HTTP from `127.0.0.1` and `::1` (lego on the DNS server itself), unless that address is in `trusted_proxies`. |
### BIND ### BIND
| Key | Default | Meaning | | Key | Default | Meaning |
@@ -107,3 +121,9 @@ location / {
proxy_read_timeout 300s; # applying a large changeset can take a while proxy_read_timeout 300s; # applying a large changeset can take a while
} }
``` ```
`X-Forwarded-Proto` tells jarld that the client used HTTPS. It sets the
session cookie's Secure flag, and the lego `httpreq` endpoint needs it
(Caddy and Traefik send it without being told). Check with
`curl https://YOUR-PROXY/api/v1/acme/httpreq`, whose `transport` field says
how jarld saw the request.
docs/development.md
+1
View File
@@ -23,6 +23,7 @@ internal/
stats/ query statistics: statistics-channel poller, response-log tailer, collector, separate SQLite store stats/ query statistics: statistics-channel poller, response-log tailer, collector, separate SQLite store
web/static/ the web UI: plain ES modules, no build step, embedded with go:embed web/static/ the web UI: plain ES modules, no build step, embedded with go:embed
api/, client/ JSON types and the Go client shared by server and CLI api/, client/ JSON types and the Go client shared by server and CLI
acme/ ACME DNS-01 challenge records, shared by jarl acme and the httpreq endpoint
config/ jarld's YAML configuration config/ jarld's YAML configuration
deploy/ systemd unit, sysusers.d and tmpfiles.d files deploy/ systemd unit, sysusers.d and tmpfiles.d files
contrib/acme.sh/ acme.sh DNS API hook contrib/acme.sh/ acme.sh DNS API hook
docs/permissions.md
+1
View File
@@ -6,6 +6,7 @@
|-----|-----|-------| |-----|-----|-------|
| People in the web UI | Username + password → session cookie | argon2id password hashes. The session cookie is HttpOnly and SameSite=Strict and lasts 12h (2h idle). Every change also needs the session's CSRF token, which the UI sends in `X-CSRF-Token`. Changing a password logs out the user's other sessions. | | People in the web UI | Username + password → session cookie | argon2id password hashes. The session cookie is HttpOnly and SameSite=Strict and lasts 12h (2h idle). Every change also needs the session's CSRF token, which the UI sends in `X-CSRF-Token`. Changing a password logs out the user's other sessions. |
| CLI, scripts, certbot | API token: `Authorization: Bearer jarl_...` | Shown once at creation, stored hashed. Optional expiry. Revocable. Last use time and IP are recorded. | | CLI, scripts, certbot | API token: `Authorization: Bearer jarl_...` | Shown once at creation, stored hashed. Optional expiry. Revocable. Last use time and IP are recorded. |
| lego / Traefik (`httpreq`) | HTTP Basic: the token's user and the API token | Only on `/api/v1/acme/httpreq/...`, only when `acme.httpreq.enabled` is set, and by default only over HTTPS ([acme.md](acme.md)). The token's grants apply as usual. |
| Root on the server | `jarld` subcommands (`init`, `user`, `import`, `adopt`, `check`, `resync`, `backup`) | Work directly on the database. They are for bootstrapping and recovery, and are audited as "local". | | Root on the server | `jarld` subcommands (`init`, `user`, `import`, `adopt`, `check`, `resync`, `backup`) | Work directly on the database. They are for bootstrapping and recovery, and are audited as "local". |
API tokens are how you give a script, a tool or a person limited access: API tokens are how you give a script, a tool or a person limited access:
internal/acme/acme.go
+93
View File
@@ -0,0 +1,93 @@
// Package acme turns "publish (or remove) this DNS-01 challenge value"
// into record changes. The jarl CLI (jarl acme) and jarld's httpreq
// endpoint for lego both use it, so they behave the same way.
package acme
import (
"encoding/json"
"strings"
"jarl/internal/api"
"jarl/internal/dnsname"
)
// DefaultTTL is the TTL of challenge records.
const DefaultTTL = 60
// Comment is stored with every challenge record.
const Comment = "ACME DNS-01 challenge"
// Zone is a primary zone that may receive challenge records.
type Zone struct {
Name string // as Jarl stores it, e.g. "adbraun.lol."
View string // "" when views are not used
}
// Zones returns the zones that hold name: the longest matching zone, in
// every view that has it, or only in view when view is set. The record is
// written into every view so that the CA (usually in an external view) and
// internal resolvers see the same thing.
func Zones(name string, zones []Zone, view string) []Zone {
name = dnsname.Fqdn(name)
best := ""
var out []Zone
for _, z := range zones {
if view != "" && !strings.EqualFold(z.View, view) {
continue
}
zn := dnsname.Fqdn(z.Name)
if !dnsname.IsSubdomain(name, zn) {
continue
}
switch {
case len(zn) > len(best):
best, out = zn, []Zone{z}
case zn == best:
out = append(out, z)
}
}
return out
}
// Requests returns the changes that set (or, with set false, remove) the
// TXT values at name in each zone. Setting first deletes the same value, so
// a retried hook succeeds instead of failing on a duplicate record, and
// removing a value that isn't there succeeds too. Removing with no values
// removes every TXT record at name.
func Requests(name string, zones []Zone, values []string, set bool, ttl uint32) []api.ChangeRequest {
name = dnsname.Fqdn(name)
var reqs []api.ChangeRequest
add := func(op string, z Zone, extra map[string]any) {
m := map[string]any{"zone": z.Name, "name": name, "type": "TXT"}
if z.View != "" {
m["view"] = z.View
}
for k, v := range extra {
m[k] = v
}
b, _ := json.Marshal(m)
reqs = append(reqs, api.ChangeRequest{Op: op, Data: b})
}
for _, z := range zones {
if !set && len(values) == 0 {
add("record.delete", z, map[string]any{"missing_ok": true})
continue
}
for _, v := range values {
add("record.delete", z, map[string]any{"data": v, "missing_ok": true})
if set {
add("record.create", z, map[string]any{"data": v, "ttl": ttl, "comment": Comment})
}
}
}
return reqs
}
// Description is the changeset description for a change at name.
func Description(name string, set bool) string {
verb := "remove"
if set {
verb = "set"
}
return "ACME: " + verb + " challenge for " + dnsname.Display(dnsname.Fqdn(name))
}
internal/acme/acme_test.go
+71
View File
@@ -0,0 +1,71 @@
package acme
import (
"encoding/json"
"testing"
)
func TestZones(t *testing.T) {
zones := []Zone{
{Name: "example.com.", View: "internal"},
{Name: "example.com.", View: "external"},
{Name: "acme.example.com.", View: "external"},
{Name: "example.net."},
}
cases := []struct {
name, view string
want []Zone
}{
{"_acme-challenge.example.com.", "", zones[:2]},
{"_acme-challenge.example.com", "External", zones[1:2]},
{"www.acme.example.com.", "", zones[2:3]},
{"_acme-challenge.WWW.example.net.", "", zones[3:4]},
{"_acme-challenge.example.org.", "", nil},
{"_acme-challenge.notexample.com.", "", nil},
}
for _, c := range cases {
got := Zones(c.name, zones, c.view)
if len(got) != len(c.want) {
t.Errorf("Zones(%s, %q) = %v, want %v", c.name, c.view, got, c.want)
continue
}
for i := range got {
if got[i] != c.want[i] {
t.Errorf("Zones(%s, %q) = %v, want %v", c.name, c.view, got, c.want)
}
}
}
}
func TestRequests(t *testing.T) {
zones := []Zone{{Name: "example.com.", View: "internal"}, {Name: "example.com.", View: "external"}}
ops := func(set bool, values ...string) []string {
var out []string
for _, r := range Requests("_acme-challenge.Example.com", zones, values, set, 60) {
var d map[string]any
json.Unmarshal(r.Data, &d)
if d["name"] != "_acme-challenge.example.com." || d["type"] != "TXT" {
t.Errorf("data = %v", d)
}
s := r.Op + " " + d["view"].(string)
if v, ok := d["data"]; ok {
s += " " + v.(string)
}
out = append(out, s)
}
return out
}
want := []string{"record.delete internal v1", "record.create internal v1", "record.delete external v1", "record.create external v1"}
if got := ops(true, "v1"); len(got) != 4 || got[0] != want[0] || got[1] != want[1] || got[3] != want[3] {
t.Errorf("set = %v", got)
}
if got := ops(false, "v1"); len(got) != 2 || got[0] != "record.delete internal v1" {
t.Errorf("unset = %v", got)
}
if got := ops(false); len(got) != 2 || got[1] != "record.delete external" {
t.Errorf("unset all = %v", got)
}
if d := Description("_acme-challenge.example.com.", true); d != "ACME: set challenge for _acme-challenge.example.com" {
t.Errorf("description = %q", d)
}
}
internal/config/config.go
+37
View File
@@ -29,6 +29,27 @@ type Config struct {
BIND BIND `yaml:"bind"` BIND BIND `yaml:"bind"`
Audit Audit `yaml:"audit"` Audit Audit `yaml:"audit"`
Stats Stats `yaml:"stats"` Stats Stats `yaml:"stats"`
ACME ACME `yaml:"acme"`
}
// ACME configures the ACME helpers.
type ACME struct {
HTTPReq HTTPReq `yaml:"httpreq"`
}
// HTTPReq configures the endpoint for lego's httpreq DNS provider
// (docs/acme.md). lego sends an API token as an HTTP Basic password, so
// by default the endpoint only answers requests that came over HTTPS.
type HTTPReq struct {
Enabled bool `yaml:"enabled"`
// RequireHTTPS refuses requests that didn't arrive over HTTPS: jarld's
// own TLS, or a trusted proxy that says so in X-Forwarded-Proto or
// Forwarded.
RequireHTTPS bool `yaml:"require_https"`
// AllowLoopbackHTTP also accepts plain HTTP from 127.0.0.1 and ::1
// (lego on the DNS server itself), unless that address is a trusted
// proxy.
AllowLoopbackHTTP bool `yaml:"allow_loopback_http"`
} }
// TLS configures HTTPS. // TLS configures HTTPS.
@@ -172,6 +193,7 @@ func Default() *Config {
Retention: StatsRetention{Minute: Duration(7 * 24 * time.Hour), Hour: Duration(90 * 24 * time.Hour), Day: Duration(730 * 24 * time.Hour)}, Retention: StatsRetention{Minute: Duration(7 * 24 * time.Hour), Hour: Duration(90 * 24 * time.Hour), Day: Duration(730 * 24 * time.Hour)},
TopN: 100, TopN: 100,
}, },
ACME: ACME{HTTPReq: HTTPReq{RequireHTTPS: true}},
} }
} }
@@ -374,4 +396,19 @@ stats:
top_n: 100 # names and clients kept per hour and per day top_n: 100 # names and clients kept per hour and per day
anonymize_clients: false # store 10.10.0.0/24 instead of 10.10.0.37 anonymize_clients: false # store 10.10.0.0/24 instead of 10.10.0.37
exclude_clients: [] # e.g. [127.0.0.1] to ignore local monitoring exclude_clients: [] # e.g. [127.0.0.1] to ignore local monitoring
# An endpoint for lego's "httpreq" DNS provider (docs/acme.md), so lego and
# Traefik can publish DNS-01 challenges without the jarl CLI. Off by
# default. lego sends an API token as an HTTP Basic password, so by default
# requests are only accepted over HTTPS: jarld's own tls (above), or a
# reverse proxy in trusted_proxies that sends "X-Forwarded-Proto: https"
# (or "Forwarded: proto=https"). Nothing else in Jarl depends on this.
acme:
httpreq:
enabled: false
require_https: true
# Also accept plain HTTP from 127.0.0.1 and ::1, for lego running on
# this server. Leave false if a reverse proxy runs on this server but
# isn't in trusted_proxies: its requests would look local.
allow_loopback_http: false
` `
internal/config/config_test.go
+16
View File
@@ -18,6 +18,22 @@ func TestExampleParses(t *testing.T) {
if c.BIND.ZoneDir != d.BIND.ZoneDir || time.Duration(c.Session.IdleTimeout) != 2*time.Hour || c.Listen[0] != "127.0.0.1:8053" { if c.BIND.ZoneDir != d.BIND.ZoneDir || time.Duration(c.Session.IdleTimeout) != 2*time.Hour || c.Listen[0] != "127.0.0.1:8053" {
t.Errorf("example differs from defaults: %+v", c) t.Errorf("example differs from defaults: %+v", c)
} }
if c.ACME != d.ACME || c.ACME.HTTPReq.Enabled || !c.ACME.HTTPReq.RequireHTTPS {
t.Errorf("acme = %+v, want the defaults %+v", c.ACME, d.ACME)
}
}
func TestHTTPReqDefaults(t *testing.T) {
path := filepath.Join(t.TempDir(), "jarl.yaml")
// Enabling the endpoint keeps require_https on unless it is turned off.
os.WriteFile(path, []byte("acme: {httpreq: {enabled: true}}\n"), 0o644)
c, err := Load(path, false)
if err != nil {
t.Fatal(err)
}
if h := c.ACME.HTTPReq; !h.Enabled || !h.RequireHTTPS || h.AllowLoopbackHTTP {
t.Errorf("httpreq = %+v", h)
}
} }
func TestLoad(t *testing.T) { func TestLoad(t *testing.T) {
internal/server/acme_test.go
+248
View File
@@ -0,0 +1,248 @@
package server
import (
"bytes"
"crypto/tls"
"encoding/json"
"net/http"
"net/http/httptest"
"net/netip"
"strings"
"testing"
"jarl/internal/api"
)
// hreq is one request to the httpreq endpoint, as it would arrive from
// remote (with TLS when tlsOn is set), with extra headers.
type hreq struct {
path string // after /acme/httpreq
remote string
tlsOn bool
header map[string]string
user string
token string
body any
}
func (e *apiEnv) httpreq(q hreq) *httptest.ResponseRecorder {
e.t.Helper()
var rd *bytes.Reader
if q.body != nil {
b, _ := json.Marshal(q.body)
rd = bytes.NewReader(b)
} else {
rd = bytes.NewReader(nil)
}
method := "POST"
if q.path == "" {
method = "GET"
}
r := httptest.NewRequest(method, api.Prefix+"/acme/httpreq"+q.path, rd)
r.RemoteAddr = q.remote
if r.RemoteAddr == "" {
r.RemoteAddr = "192.0.2.9:40000"
}
if q.tlsOn {
r.TLS = &tls.ConnectionState{}
}
r.Header.Set("Content-Type", "application/json")
for k, v := range q.header {
r.Header.Set(k, v)
}
if q.token != "" {
r.SetBasicAuth(q.user, q.token)
}
w := httptest.NewRecorder()
e.srv.Handler().ServeHTTP(w, r)
return w
}
func errCode(w *httptest.ResponseRecorder) string {
var b api.ErrorBody
json.Unmarshal(w.Body.Bytes(), &b)
return b.Error.Code
}
func challenge(value string) map[string]any {
return map[string]any{"fqdn": "_acme-challenge.example.com.", "value": value}
}
func (e *apiEnv) txtValues(zone, name string) []string {
e.t.Helper()
var out []string
for _, r := range e.list("/zones/"+zone+"/records", e.token) {
if r["type"] == "TXT" && r["name"] == name {
out = append(out, r["data"].(string))
}
}
return out
}
func TestHTTPReqDisabledByDefault(t *testing.T) {
e := newAPIEnv(t)
w := e.httpreq(hreq{path: "/present", tlsOn: true, user: "admin", token: e.token, body: challenge("abc")})
if w.Code != 404 || errCode(w) != "disabled" {
t.Errorf("disabled endpoint = %d %s", w.Code, w.Body)
}
w = e.httpreq(hreq{})
var st map[string]any
json.Unmarshal(w.Body.Bytes(), &st)
if w.Code != 200 || st["enabled"] != false {
t.Errorf("status = %d %s", w.Code, w.Body)
}
}
func TestHTTPReqTransport(t *testing.T) {
proxy := netip.MustParsePrefix("10.0.0.2/32")
e := newAPIEnvConfig(t, Config{Version: "test", TrustedProxies: []netip.Prefix{proxy}, HTTPReq: HTTPReq{Enabled: true}})
e.exampleZone()
cases := []struct {
name string
remote string
tlsOn bool
header map[string]string
ok bool
}{
{"plain HTTP", "192.0.2.9:1", false, nil, false},
{"jarld's own TLS", "192.0.2.9:1", true, nil, true},
{"header from an untrusted client", "192.0.2.9:1", false, map[string]string{"X-Forwarded-Proto": "https"}, false},
{"trusted proxy, https", "10.0.0.2:1", false, map[string]string{"X-Forwarded-Proto": "https"}, true},
{"trusted proxy, http", "10.0.0.2:1", false, map[string]string{"X-Forwarded-Proto": "http"}, false},
{"trusted proxy, no header", "10.0.0.2:1", false, nil, false},
{"client-sent https, proxy appended http", "10.0.0.2:1", false, map[string]string{"X-Forwarded-Proto": "https, http"}, false},
{"client-sent http, proxy appended https", "10.0.0.2:1", false, map[string]string{"X-Forwarded-Proto": "http,https"}, true},
{"trusted proxy, Forwarded", "10.0.0.2:1", false, map[string]string{"Forwarded": `for=192.0.2.9;proto=https`}, true},
{"trusted proxy, Forwarded http", "10.0.0.2:1", false, map[string]string{"Forwarded": `for=192.0.2.9;Proto="http"`}, false},
{"loopback, not allowed", "127.0.0.1:1", false, nil, false},
}
for _, c := range cases {
w := e.httpreq(hreq{path: "/present", remote: c.remote, tlsOn: c.tlsOn, header: c.header, user: "admin", token: e.token, body: challenge("v1")})
if c.ok && w.Code != 200 || !c.ok && (w.Code != 403 || errCode(w) != "https_required") {
t.Errorf("%s: %d %s", c.name, w.Code, w.Body)
}
if !c.ok && !strings.Contains(w.Body.String(), "unencrypted") {
t.Errorf("%s: no warning about the exposed token: %s", c.name, w.Body)
}
// The status check agrees.
s := e.httpreq(hreq{remote: c.remote, tlsOn: c.tlsOn, header: c.header})
var st map[string]any
json.Unmarshal(s.Body.Bytes(), &st)
if st["accepted"] != c.ok || st["transport"] == "" {
t.Errorf("%s: status = %s", c.name, s.Body)
}
}
// Cookies follow the same rules (cookie_secure: auto).
r := httptest.NewRequest("GET", "/", nil)
r.RemoteAddr = "10.0.0.2:1"
r.Header.Set("Forwarded", "proto=https")
if !e.srv.isHTTPS(r) {
t.Error("Forwarded: proto=https from a trusted proxy isn't HTTPS")
}
}
func TestHTTPReqLoopbackAndPlain(t *testing.T) {
e := newAPIEnvConfig(t, Config{Version: "test", TrustedProxies: []netip.Prefix{netip.MustParsePrefix("::1/128")}, HTTPReq: HTTPReq{Enabled: true, AllowLoopbackHTTP: true}})
e.exampleZone()
for remote, ok := range map[string]bool{
"127.0.0.1:1": true, // lego on this host
"192.0.2.9:1": false, // still HTTPS-only from the network
"[::1]:1": false, // a trusted proxy: its headers decide, and it sent none
"10.20.30.40:1": false,
} {
w := e.httpreq(hreq{path: "/present", remote: remote, user: "admin", token: e.token, body: challenge("v1")})
if ok && w.Code != 200 || !ok && w.Code != 403 {
t.Errorf("%s: %d %s", remote, w.Code, w.Body)
}
}
e = newAPIEnvConfig(t, Config{Version: "test", HTTPReq: HTTPReq{Enabled: true, AllowPlainHTTP: true}})
e.exampleZone()
if w := e.httpreq(hreq{path: "/present", user: "admin", token: e.token, body: challenge("v1")}); w.Code != 200 {
t.Errorf("require_https off: %d %s", w.Code, w.Body)
}
}
func TestHTTPReqChallenges(t *testing.T) {
e := newAPIEnvConfig(t, Config{Version: "test", HTTPReq: HTTPReq{Enabled: true}})
e.exampleZone()
e.do("POST", "/users", e.token, map[string]any{"username": "lego", "password": "lego-password-123", "grant_strings": []string{"records:write names=_acme-challenge.* types=TXT"}}, 201)
var legoID int64
for _, u := range e.list("/users", e.token) {
if u["username"] == "lego" {
legoID = int64(u["id"].(float64))
}
}
tok := e.do("POST", "/users/"+itoa(legoID)+"/tokens", e.token, map[string]any{"name": "lego"}, 201)["token"].(string)
send := func(path, user, token string, body any) *httptest.ResponseRecorder {
return e.httpreq(hreq{path: path, tlsOn: true, user: user, token: token, body: body})
}
const name = "_acme-challenge.example.com."
// present, a retry of it, and a second value (apex and wildcard).
for _, v := range []string{"v1", "v1", "v2"} {
if w := send("/present", "lego", tok, challenge(v)); w.Code != 200 {
t.Fatalf("present %s: %d %s", v, w.Code, w.Body)
}
}
if got := e.txtValues("example.com", name); len(got) != 2 {
t.Errorf("after present: %v", got)
}
// cleanup removes one value; repeating it still succeeds.
for i := 0; i < 2; i++ {
if w := send("/cleanup", "lego", tok, challenge("v1")); w.Code != 200 {
t.Fatalf("cleanup: %d %s", w.Code, w.Body)
}
}
if got := e.txtValues("example.com", name); len(got) != 1 || !strings.Contains(got[0], "v2") {
t.Errorf("after cleanup: %v", got)
}
// Credentials.
for _, c := range []struct {
name, user, token string
}{
{"no credentials", "", ""},
{"wrong user", "admin", tok},
{"bad token", "lego", "jarl_aaaaaaaaaaaaaaaa_bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"},
} {
w := send("/present", c.user, c.token, challenge("v3"))
if w.Code != 401 || w.Header().Get("WWW-Authenticate") == "" {
t.Errorf("%s: %d %s", c.name, w.Code, w.Body)
}
}
// A Bearer token works too.
w := e.httpreq(hreq{path: "/cleanup", tlsOn: true, header: map[string]string{"Authorization": "Bearer " + tok}, body: challenge("v2")})
if w.Code != 200 {
t.Errorf("bearer: %d %s", w.Code, w.Body)
}
// Bodies and names.
for _, c := range []struct {
name string
body any
code int
}{
{"RAW mode", map[string]any{"domain": "example.com", "token": "t", "keyAuth": "k"}, 400},
{"no value", map[string]any{"fqdn": name}, 400},
{"value with a quote", map[string]any{"fqdn": name, "value": `a"b`}, 400},
{"wildcard", map[string]any{"fqdn": "*.example.com.", "value": "v"}, 400},
{"outside the token's grants", map[string]any{"fqdn": "www.example.com.", "value": "v"}, 403},
{"no such zone", map[string]any{"fqdn": "_acme-challenge.example.net.", "value": "v"}, 404},
} {
if w := send("/present", "lego", tok, c.body); w.Code != c.code {
t.Errorf("%s: %d, want %d: %s", c.name, w.Code, c.code, w.Body)
}
}
if w := send("/view/nosuch/present", "lego", tok, challenge("v4")); w.Code != 404 {
t.Errorf("unknown view: %d %s", w.Code, w.Body)
}
if w := e.httpreq(hreq{path: "/present", tlsOn: true, user: "lego", token: tok, body: challenge("v5")}); w.Code != http.StatusOK {
t.Errorf("final present: %d", w.Code)
}
}
func (e *apiEnv) exampleZone() {
e.t.Helper()
e.do("POST", "/zones", e.token, map[string]any{"name": "example.com", "records": []map[string]any{{"name": "ns1", "type": "A", "data": "192.0.2.1"}}}, 201)
}
internal/server/h_acme.go
+230
View File
@@ -0,0 +1,230 @@
package server
// An endpoint for lego's "httpreq" DNS provider, so lego (and Traefik,
// which uses lego) can publish DNS-01 challenges without the jarl CLI.
// docs/acme.md explains the setup; this is how it works.
//
// lego is configured with HTTPREQ_ENDPOINT, HTTPREQ_USERNAME and
// HTTPREQ_PASSWORD. For each challenge it sends
//
// POST <endpoint>/present {"fqdn": "_acme-challenge.example.com.", "value": "..."}
// POST <endpoint>/cleanup (the same body, once the CA has checked it)
//
// with the username and password as HTTP Basic auth, and treats any 2xx
// answer as success. fqdn is the full record name, after lego has followed
// any CNAME (into a dedicated ACME zone, for example).
//
// Jarl's endpoint is /api/v1/acme/httpreq, or /api/v1/acme/httpreq/view/NAME
// to write into one view only. The password is a Jarl API token and the
// username is the user that owns it. The request then does exactly what
// "jarl acme set|unset --exact FQDN VALUE" does (package acme): it writes
// the TXT record in the longest matching primary zone, in every view that
// has the zone, as one changeset, and answers once named serves it. The
// token's grants apply as usual, and the audit log records the change.
//
// Basic auth is accepted on these routes only; the rest of the API takes
// Bearer tokens and sessions. The endpoint is off unless
// acme.httpreq.enabled is set, and because the token travels in every
// request it refuses requests that didn't come over HTTPS (see transport)
// unless acme.httpreq.require_https is turned off.
import (
"fmt"
"net"
"net/http"
"net/netip"
"regexp"
"strings"
"jarl/internal/acme"
"jarl/internal/auth"
"jarl/internal/dnsname"
"jarl/internal/engine"
"jarl/internal/model"
)
// HTTPReq configures the httpreq endpoint. The zero value is off, and
// when on, HTTPS-only.
type HTTPReq struct {
Enabled bool
// AllowPlainHTTP accepts requests over plain HTTP from anywhere
// (require_https: false).
AllowPlainHTTP bool
// AllowLoopbackHTTP accepts plain HTTP from 127.0.0.1 and ::1, unless
// that address is a trusted proxy.
AllowLoopbackHTTP bool
}
// httpreqBody is what lego sends. In its RAW mode (HTTPREQ_MODE=RAW) it
// sends domain, token and keyAuth instead, which Jarl doesn't support.
type httpreqBody struct {
FQDN string `json:"fqdn"`
Value string `json:"value"`
Domain string `json:"domain"`
Token string `json:"token"`
KeyAuth string `json:"keyAuth"`
}
// A DNS-01 value is the base64url SHA-256 digest of the key authorization
// (RFC 8555, section 8.4): 43 characters from this alphabet.
var acmeValueRe = regexp.MustCompile(`^[A-Za-z0-9_-]{1,255}$`)
const httpreqRealm = `Basic realm="jarl httpreq", charset="UTF-8"`
// httpreq handles POST .../present (set) and .../cleanup.
func (s *Server) httpreq(set bool) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, 64<<10)
if err := s.handleHTTPReq(w, r, set); err != nil {
s.writeErr(w, r, err, nil)
}
}
}
func (s *Server) handleHTTPReq(w http.ResponseWriter, r *http.Request, set bool) error {
if !s.cfg.HTTPReq.Enabled {
return &httpError{http.StatusNotFound, "disabled", "the httpreq endpoint is off; set acme.httpreq.enabled: true in jarl.yaml and restart jarld (docs/acme.md)"}
}
if ok, why := s.httpreqTransport(r); !ok {
ip := s.clientIP(r)
s.log.Warn("httpreq request refused: not HTTPS", "ip", ip, "why", why)
s.eng.Audit(r.Context(), &auth.Principal{IP: ip, Agent: r.UserAgent(), Method: "token"}, "acme.httpreq_refused", "", 0, "", "denied", "not HTTPS: "+why)
msg := "the httpreq endpoint only accepts HTTPS (acme.httpreq.require_https), and this request came over " + why + "."
if r.Header.Get("Authorization") != "" {
msg += " Its token may have crossed the network unencrypted; if so, revoke it and create a new one."
}
return &httpError{http.StatusForbidden, "https_required", msg}
}
p, err := s.httpreqPrincipal(w, r)
if err != nil {
return err
}
var in httpreqBody
if err := decodeJSON(r, &in); err != nil {
return err
}
if in.FQDN == "" && (in.Domain != "" || in.KeyAuth != "") {
return badRequest("this looks like lego's RAW mode (domain, token, keyAuth); Jarl needs the default mode: unset HTTPREQ_MODE")
}
if in.FQDN == "" || in.Value == "" {
return badRequest(`expected {"fqdn": "_acme-challenge.example.com.", "value": "..."}`)
}
name := dnsname.Fqdn(in.FQDN)
if strings.HasPrefix(name, "*.") {
return badRequest("fqdn %q is a wildcard; send the challenge name (_acme-challenge.<domain>)", in.FQDN)
}
if err := dnsname.CheckOwner(name); err != nil {
return badRequest("fqdn %q: %v", in.FQDN, err)
}
if !acmeValueRe.MatchString(in.Value) {
return badRequest("value must be a DNS-01 challenge value (letters, digits, - and _)")
}
view := r.PathValue("view")
st, err := s.store.LoadState(r.Context())
if err != nil {
return err
}
zones := acme.Zones(name, primaryZonesFor(st, p), view)
if len(zones) == 0 {
where := ""
if view != "" {
where = " in view " + view
}
return notFound("no primary zone%s that this token can see contains %s", where, dnsname.Display(name))
}
reqs := toRequests(acme.Requests(name, zones, []string{in.Value}, set, acme.DefaultTTL))
res, err := s.eng.Submit(r.Context(), reqs, p, "api", acme.Description(name, set)+" (lego httpreq)", engine.ApplyOptions{})
if err != nil {
s.writeErr(w, r, err, res)
return nil
}
out := map[string]any{"ok": true, "fqdn": name, "changeset_id": res.ChangesetID}
var where []string
for _, z := range zones {
if z.View != "" {
where = append(where, dnsname.Display(z.Name)+" (view "+z.View+")")
} else {
where = append(where, dnsname.Display(z.Name))
}
}
out["zones"] = where
writeJSON(w, http.StatusOK, out)
return nil
}
// primaryZonesFor lists the primary zones p can see.
func primaryZonesFor(st *model.State, p *auth.Principal) []acme.Zone {
var out []acme.Zone
for _, z := range st.SortedZones() {
vn := st.ViewName(z)
if z.Type == model.ZonePrimary && p.CanSeeZone(z.Name, vn) {
out = append(out, acme.Zone{Name: z.Name, View: vn})
}
}
return out
}
// httpreqTransport reports whether the endpoint accepts r's transport.
func (s *Server) httpreqTransport(r *http.Request) (bool, string) {
https, why := s.transport(r)
if https || s.cfg.HTTPReq.AllowPlainHTTP {
return true, why
}
if s.cfg.HTTPReq.AllowLoopbackHTTP {
host, _, _ := net.SplitHostPort(r.RemoteAddr)
if a, err := netip.ParseAddr(host); err == nil && a.Unmap().IsLoopback() && !s.trusted(a) {
return true, "plain HTTP from " + host + " (allowed by allow_loopback_http)"
}
}
return false, why
}
// httpreqPrincipal authenticates an httpreq request: HTTP Basic with the
// token's user as the username and the API token as the password. A
// Bearer token works too (handy with curl).
func (s *Server) httpreqPrincipal(w http.ResponseWriter, r *http.Request) (*auth.Principal, error) {
user, tok, ok := r.BasicAuth()
if !ok {
if scheme, _, _ := strings.Cut(r.Header.Get("Authorization"), " "); strings.EqualFold(scheme, "Bearer") {
p, _, err := s.authenticate(r)
return p, err
}
w.Header().Set("WWW-Authenticate", httpreqRealm)
return nil, &httpError{http.StatusUnauthorized, "unauthenticated", "send the Jarl user as the HTTP Basic username (HTTPREQ_USERNAME) and an API token as the password (HTTPREQ_PASSWORD)"}
}
ip := s.clientIP(r)
if s.limiter.blocked("ip:"+ip, failsPerIP) {
return nil, &httpError{http.StatusTooManyRequests, "rate_limited", "too many failed attempts; try again later"}
}
fail := func(reason string) error {
s.limiter.fail("ip:" + ip)
s.eng.Audit(r.Context(), &auth.Principal{User: &model.User{Username: user}, IP: ip, Agent: r.UserAgent(), Method: "token"}, "auth.token_failed", "", 0, "", "failure", "httpreq: "+reason)
w.Header().Set("WWW-Authenticate", httpreqRealm)
return &httpError{http.StatusUnauthorized, "unauthenticated", "invalid credentials: " + reason}
}
p, err := s.tokenPrincipal(r.Context(), strings.TrimSpace(tok), ip, r.UserAgent())
if err != nil {
return nil, fail(err.Error() + " (HTTPREQ_PASSWORD must be a Jarl API token)")
}
if !strings.EqualFold(strings.TrimSpace(user), p.User.Username) {
return nil, fail(fmt.Sprintf("the token doesn't belong to user %q (HTTPREQ_USERNAME must be the token's user)", user))
}
return p, nil
}
// httpreqStatus answers GET /acme/httpreq without authentication, so a
// setup can be checked (e.g. with curl through the reverse proxy): is the
// endpoint on, and would this request count as HTTPS?
func (s *Server) httpreqStatus(w http.ResponseWriter, r *http.Request) error {
https, why := s.transport(r)
accepted, _ := s.httpreqTransport(r)
out := map[string]any{"enabled": s.cfg.HTTPReq.Enabled, "https": https, "transport": why}
if s.cfg.HTTPReq.Enabled {
out["accepted"] = accepted
if !accepted {
out["hint"] = "requests like this one are refused; see 'HTTPS and reverse proxies' in docs/acme.md"
}
}
writeJSON(w, http.StatusOK, out)
return nil
}
internal/server/routes.go
+6
View File
@@ -100,6 +100,12 @@ func (s *Server) routes() {
m.HandleFunc("POST "+p+"/system/check", s.authed(s.systemCheck)) m.HandleFunc("POST "+p+"/system/check", s.authed(s.systemCheck))
m.HandleFunc("POST "+p+"/system/resync", s.authed(s.systemResync)) m.HandleFunc("POST "+p+"/system/resync", s.authed(s.systemResync))
m.HandleFunc("GET "+p+"/audit", s.authed(s.listAudit)) m.HandleFunc("GET "+p+"/audit", s.authed(s.listAudit))
// lego's httpreq DNS provider (h_acme.go)
m.HandleFunc("GET "+p+"/acme/httpreq", s.public(s.httpreqStatus))
m.HandleFunc("POST "+p+"/acme/httpreq/present", s.httpreq(true))
m.HandleFunc("POST "+p+"/acme/httpreq/cleanup", s.httpreq(false))
m.HandleFunc("POST "+p+"/acme/httpreq/view/{view}/present", s.httpreq(true))
m.HandleFunc("POST "+p+"/acme/httpreq/view/{view}/cleanup", s.httpreq(false))
// Query statistics // Query statistics
m.HandleFunc("GET "+p+"/stats/status", s.authed(s.statsStatus)) m.HandleFunc("GET "+p+"/stats/status", s.authed(s.statsStatus))
m.HandleFunc("GET "+p+"/stats/summary", s.authed(s.statsSummary)) m.HandleFunc("GET "+p+"/stats/summary", s.authed(s.statsSummary))
internal/server/server.go
+75 -8
View File
@@ -16,6 +16,7 @@ import (
"sync" "sync"
"time" "time"
"jarl/internal/api"
"jarl/internal/auth" "jarl/internal/auth"
"jarl/internal/engine" "jarl/internal/engine"
"jarl/internal/stats" "jarl/internal/stats"
@@ -37,6 +38,8 @@ type Config struct {
Version string Version string
// Stats is the query statistics collector (nil when disabled). // Stats is the query statistics collector (nil when disabled).
Stats *stats.Collector Stats *stats.Collector
// HTTPReq configures the endpoint for lego's httpreq DNS provider.
HTTPReq HTTPReq
} }
// Server is the HTTP server. // Server is the HTTP server.
@@ -150,6 +153,7 @@ func (s *Server) Serve(ctx context.Context, ready func()) error {
} }
} }
} }
s.logHTTPReq(tlsCfg != nil)
if ready != nil { if ready != nil {
ready() ready()
} }
@@ -374,13 +378,76 @@ func (s *Server) trusted(a netip.Addr) bool {
return false return false
} }
func (s *Server) isHTTPS(r *http.Request) bool { // logHTTPReq says at startup whether the httpreq endpoint can work.
if r.TLS != nil { func (s *Server) logHTTPReq(tls bool) {
return true h := s.cfg.HTTPReq
switch {
case !h.Enabled:
case h.AllowPlainHTTP:
s.log.Warn("acme.httpreq accepts plain HTTP (require_https: false): API tokens may travel unencrypted", "path", api.Prefix+"/acme/httpreq")
case !tls && len(s.cfg.TrustedProxies) == 0 && !h.AllowLoopbackHTTP:
s.log.Warn("acme.httpreq is enabled but jarld has no tls and no trusted_proxies, so every request will be refused as not HTTPS", "path", api.Prefix+"/acme/httpreq")
default:
s.log.Info("acme.httpreq endpoint enabled", "path", api.Prefix+"/acme/httpreq", "tls", tls, "trusted_proxies", len(s.cfg.TrustedProxies), "allow_loopback_http", h.AllowLoopbackHTTP)
} }
host, _, _ := net.SplitHostPort(r.RemoteAddr) }
if a, err := netip.ParseAddr(host); err == nil && s.trusted(a) {
return strings.EqualFold(r.Header.Get("X-Forwarded-Proto"), "https") func (s *Server) isHTTPS(r *http.Request) bool {
} https, _ := s.transport(r)
return false return https
}
// transport reports whether r reached Jarl over HTTPS, and explains how
// that was decided (for error messages and the httpreq status check).
//
// A request is HTTPS when jarld terminated TLS itself (tls in jarl.yaml),
// or when it came straight from a trusted proxy (trusted_proxies) that
// says the client used HTTPS: X-Forwarded-Proto, or, if that is absent,
// the proto parameter of Forwarded (RFC 7239). Proxies append to these
// headers, so the last value is the one the nearest proxy set; anything a
// client sent itself comes earlier and is ignored. From any other address
// the headers are ignored, since anyone could send them.
func (s *Server) transport(r *http.Request) (https bool, why string) {
if r.TLS != nil {
return true, "HTTPS to jarld (tls in jarl.yaml)"
}
host, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
host = r.RemoteAddr
}
peer, err := netip.ParseAddr(host)
if err != nil || !s.trusted(peer) {
if r.Header.Get("X-Forwarded-Proto") != "" || r.Header.Get("Forwarded") != "" {
return false, fmt.Sprintf("plain HTTP from %s, which sent X-Forwarded-Proto or Forwarded but isn't in trusted_proxies, so the header was ignored", host)
}
return false, fmt.Sprintf("plain HTTP from %s (jarld has no tls configured, and %s isn't a trusted proxy)", host, host)
}
proto, header := forwardedProto(r)
switch {
case proto == "":
return false, fmt.Sprintf("trusted proxy %s, which sent neither X-Forwarded-Proto nor Forwarded: proto=, so the client's protocol is unknown", host)
case proto == "https":
return true, fmt.Sprintf("HTTPS to trusted proxy %s (%s: https)", host, header)
default:
return false, fmt.Sprintf("trusted proxy %s, which says the client used %s (%s: %s)", host, proto, header, proto)
}
}
// forwardedProto returns the client's protocol as the nearest proxy
// reported it, and the header it came from.
func forwardedProto(r *http.Request) (proto, header string) {
if vals := r.Header.Values("X-Forwarded-Proto"); len(vals) > 0 {
parts := strings.Split(strings.Join(vals, ","), ",")
return strings.ToLower(strings.TrimSpace(parts[len(parts)-1])), "X-Forwarded-Proto"
}
if vals := r.Header.Values("Forwarded"); len(vals) > 0 {
elems := strings.Split(strings.Join(vals, ","), ",")
for _, pair := range strings.Split(elems[len(elems)-1], ";") {
k, v, _ := strings.Cut(strings.TrimSpace(pair), "=")
if strings.EqualFold(k, "proto") {
return strings.ToLower(strings.Trim(strings.TrimSpace(v), `"`)), "Forwarded"
}
}
}
return "", ""
} }
internal/server/server_test.go
+12 -5
View File
@@ -26,12 +26,23 @@ import (
type apiEnv struct { type apiEnv struct {
t *testing.T t *testing.T
ts *httptest.Server ts *httptest.Server
srv *Server
st *store.Store st *store.Store
token string // admin token token string // admin token
} }
// newAPIEnv starts an API server; a statistics collector may be given. // newAPIEnv starts an API server; a statistics collector may be given.
func newAPIEnv(t *testing.T, collector ...*stats.Collector) *apiEnv { func newAPIEnv(t *testing.T, collector ...*stats.Collector) *apiEnv {
t.Helper()
cfg := Config{Version: "test"}
if len(collector) > 0 {
cfg.Stats = collector[0]
}
return newAPIEnvConfig(t, cfg)
}
// newAPIEnvConfig starts an API server with the given configuration.
func newAPIEnvConfig(t *testing.T, cfg Config) *apiEnv {
t.Helper() t.Helper()
dir := t.TempDir() dir := t.TempDir()
st, err := store.Open(filepath.Join(dir, "jarl.db")) st, err := store.Open(filepath.Join(dir, "jarl.db"))
@@ -44,10 +55,6 @@ func newAPIEnv(t *testing.T, collector ...*stats.Collector) *apiEnv {
Paths: render.Paths{ConfigDir: filepath.Join(dir, "etc"), ZoneDir: filepath.Join(dir, "zones"), CacheDir: filepath.Join(dir, "cache")}, Paths: render.Paths{ConfigDir: filepath.Join(dir, "etc"), ZoneDir: filepath.Join(dir, "zones"), CacheDir: filepath.Join(dir, "cache")},
StagingDir: filepath.Join(dir, "staging"), NoReload: true, SkipCheck: true, StagingDir: filepath.Join(dir, "staging"), NoReload: true, SkipCheck: true,
}, quiet) }, quiet)
cfg := Config{Version: "test"}
if len(collector) > 0 {
cfg.Stats = collector[0]
}
srv := New(cfg, eng, quiet, nil) srv := New(cfg, eng, quiet, nil)
ts := httptest.NewServer(srv.Handler()) ts := httptest.NewServer(srv.Handler())
t.Cleanup(ts.Close) t.Cleanup(ts.Close)
@@ -56,7 +63,7 @@ func newAPIEnv(t *testing.T, collector ...*stats.Collector) *apiEnv {
st.CreateUser(context.Background(), admin) st.CreateUser(context.Background(), admin)
tok, pid, sh := auth.NewToken() tok, pid, sh := auth.NewToken()
st.CreateToken(context.Background(), &model.Token{UserID: admin.ID, Name: "test", PublicID: pid, SecretHash: sh}) st.CreateToken(context.Background(), &model.Token{UserID: admin.ID, Name: "test", PublicID: pid, SecretHash: sh})
env := &apiEnv{t: t, ts: ts, st: st, token: tok} env := &apiEnv{t: t, ts: ts, srv: srv, st: st, token: tok}
env.do("PATCH", "/settings", env.token, map[string]any{"nameservers": []string{"ns1.example.com."}}, 200) env.do("PATCH", "/settings", env.token, map[string]any{"nameservers": []string{"ns1.example.com."}}, 200)
return env return env
} }