adam/jarl
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases 34 Wiki Activity

Add blocking: blocklists and block/allow rules with BIND response policy zones

Browse Source 
jarld downloads blocklists (Adblock, hosts or plain domains), checks them
daily and reloads named only when a list changed; your own block and
allow rules take precedence, and the most specific rule wins. Lists and
rules apply per view (default: every view that offers recursion). Web UI
Blocking page with name checks, jarl blocklist and jarl block, the
/blocking API, blocked counts in the statistics. Database schema 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
adamandClaude Opus 5.5 committed 2026-10-10 09:24:19 +00:00
1 parent 436cd79ec9
commit 9fc23310bd
54 files changed
+4269 -85
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/api.md
+26 -4
View File
@@ -411,6 +411,26 @@ send the request with that changeset (`?changeset=` or
| GET | `/tsig-keys/{id}` | `?secret=1` includes the secret (needs `tsig:write`, audited) |
| PATCH / DELETE | `/tsig-keys/{id}` | PATCH `{"secret": "..."}` rotates |
### Blocking
Needs `blocking:read` (`blocking:write` to change). See
[blocking.md](blocking.md).
| Method | Path | Body / query |
|--------|------|--------------|
| GET | `/blocking/lists` | each with `views` (where it applies), `auto_views` (not chosen: every view that offers recursion), `state` (ok, waiting, error, disabled, idle), `status` (`entries`, `records`, `skipped`, `checked_at`, `updated_at`, `error`) and `memory_mb` (named's memory for it, estimated) |
| POST | `/blocking/lists` | `{"name": "hagezi-multi", "url": "https://...", "views": ["internal"], "comment": ""}` (`views` empty or absent: automatic); jarld downloads it right after |
| GET / PATCH / DELETE | `/blocking/lists/{id or name}` | PATCH any of `name`, `url`, `views`, `disabled`, `comment` |
| POST | `/blocking/lists/{id or name}/refresh` | download now and wait; `?force=1` even when the server says nothing changed. Returns the list; a failed update is in `state` and `status.error` |
| GET | `/blocking/rules` | each with `views` and `auto_views` |
| POST | `/blocking/rules` | `{"name": "doubleclick.net", "action": "block", "exact": false, "views": [], "comment": ""}` (`action` block or allow; `exact`: only the name, not the names below it) |
| PATCH / DELETE | `/blocking/rules/{id}` | |
| DELETE | `/blocking/rules` | `?name=DOMAIN[&view=V][&missing_ok=1]`: the rules for a domain |
| GET | `/blocking/check` | `?name=DOMAIN`: per view with blocking, `{view, result: blocked\|allowed\|none, entry, rule_id, list}` |
Lists and rules are configuration like the rest: they can be staged, and
listing with `?changeset=` marks them `pending`.
### Options and settings
| Method | Path | Body / query |
@@ -462,6 +482,8 @@ An **Operation** is `{"op": "...", "id": 0, "base_version": 0, "data": {...}}`:
| `tsig.create` / `tsig.update` / `tsig.delete` | key | key fields |
| `option.create` / `option.update` / `option.delete` | option | option fields |
| `settings.update` | | settings fields |
| `blocklist.create` / `blocklist.update` / `blocklist.delete` | blocklist (update/delete) | list fields; views as `views` (names) or `view_ids` |
| `blockrule.create` / `blockrule.update` / `blockrule.delete` | rule (update/delete) | rule fields, likewise |
Operations later in a batch can refer to objects created earlier in it by
name (zone and view names). One request can therefore create a view, a zone
@@ -471,7 +493,7 @@ in it and its records.
| Method | Path | Notes |
|--------|------|-------|
| GET | `/status` | named status, adoption, drift, open changesets, current problems, object counts; `expiring_tokens` (tokens that expire within 14 days or expired in the last 14: all for administrators, otherwise the caller's); `backup` (administrators: `last_at`, `last_file`, `last_size`, `last_error`, `last_error_at`, `dir`, `warn_after` in seconds, `stale`) |
| GET | `/status` | named status, adoption, drift, open changesets, current problems, object counts; `expiring_tokens` (tokens that expire within 14 days or expired in the last 14: all for administrators, otherwise the caller's); `backup` (administrators: `last_at`, `last_file`, `last_size`, `last_error`, `last_error_at`, `dir`, `warn_after` in seconds, `stale`); `blocklist_errors` (with `blocking:read`: lists whose last update failed, `{name, error, updated_at}`) |
| POST | `/system/check` | validation + named-checkconf of the current configuration (`options:read`) |
| POST | `/system/resync` | `{"dry_run": false, "force": false}`: rewrite every file from the database and reload (admin) |
| GET | `/audit` | `?user=&action=&outcome=&object_type=&changeset=&q=&since=&until=&limit=200&before=ID` |
@@ -484,11 +506,11 @@ Needs `stats:read`. When statistics are disabled, every endpoint but
| Method | Path | Query | Returns |
|--------|------|-------|---------|
| GET | `/stats/status` | | `{enabled, query_log, statistics_channel, log_file, last_poll, last_log_line, log_lines_read, first_data, retention, warnings}` |
| GET | `/stats/summary` | `since` (90m, 24h, 7d, 2w; default 24h), `top` (default 10) | totals, `avg_qps`, `peak`, `rcodes`, `qtypes`, `views`, `zones`, `answers`, `transport`, `cache_hit_ratio`, `nxdomain_ratio`, `tops` |
| GET | `/stats/summary` | `since` (90m, 24h, 7d, 2w; default 24h), `top` (default 10) | totals, `avg_qps`, `peak`, `rcodes`, `qtypes`, `views`, `zones`, `answers`, `transport`, `cache_hit_ratio`, `nxdomain_ratio` (without blocked answers), `blocked`, `blocked_ratio`, `blocklists` (blocked per list; `rules` for your rules), `tops` |
| GET | `/stats/timeline` | `since`, `step` (auto, 1m, 5m, 15m, 1h, 6h, 1d), `key` (repeatable or comma separated; `rcode:*` = every counter of a group; default `queries`) | `{step, times: [unix...], series: [{key, values}]}` |
| GET | `/stats/top` | `kind` (name, client, nxdomain, servfail, refused), `since`, `limit` (default 20) | `[{key, count}]`; client and refused entries add `name`, the host name of Jarl's record for the address (the summary's client list too) |
| GET | `/stats/top` | `kind` (name, client, nxdomain, servfail, refused, blocked), `since`, `limit` (default 20) | `[{key, count}]`; client and refused entries add `name`, the host name of Jarl's record for the address (the summary's client list too) |
| GET | `/stats/heatmap` | `since` (default 28d), `key` (default queries) | `{key, from, avg[7][24], samples[7][24]}`, Monday first, local time, `-1` = no data |
| GET | `/stats/breakdown` | `group` (qtype, rcode, view, zone, answer, transport, rtt, dnssec, cache, ratelimit), `since` | `[{key, count}]` |
| GET | `/stats/breakdown` | `group` (qtype, rcode, view, zone, answer, transport, rtt, dnssec, cache, ratelimit, blocklist), `since` | `[{key, count}]` |
```sh
curl -sf -H "$AUTH" "$API/stats/top?kind=nxdomain&since=7d&limit=10" | jq -r '.[] | "\(.count)\t\(.key)"'
docs/blocking.md
+245
View File
@@ -0,0 +1,245 @@
# Blocking
Jarl can make named answer "this name doesn't exist" (NXDOMAIN) for ads,
trackers and malware, like Pi-hole does, for every device that uses your
server to resolve names. It has two parts:
- **Blocklists**: lists of domains published by others, which jarld
downloads and keeps up to date.
- **Your rules**: domains you block or allow yourself. They take precedence
over every list.
It uses BIND's own mechanism for this, response policy zones (RPZ), so named
does the blocking itself; jarld only prepares the lists.
- [Quick start](#quick-start)
- [Where blocking applies](#where-blocking-applies)
- [Your rules](#your-rules)
- [Blocklists](#blocklists)
- [Checking a name](#checking-a-name)
- [Statistics](#statistics)
- [Memory and performance](#memory-and-performance)
- [How it works](#how-it-works)
- [Troubleshooting](#troubleshooting)
## Quick start
In the web UI, open **Blocking**, pick one of the well-known lists, and
click **Add list**. Or with the CLI:
```sh
jarl blocklist add hagezi-multi https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/multi.txt
jarl blocklist list # "ok" once downloaded, usually within seconds
jarl block check ads.doubleclick.net # blocked? by what?
jarl block allow cdn.example.com # something you need was blocked
```
Like every change in Jarl, adding a list or rule can be staged and reviewed
first (the web UI does this by default: apply it under **Changes**).
## Where blocking applies
Blocking changes the answers named gives when it resolves names for a
client (recursion). It doesn't change what the internet sees of your own
zones: other resolvers ask your server without recursion.
**Views.** A list or rule applies to the views you choose, or by default
to every view that offers recursion. With the usual split (an `internal`
view that resolves for your network, an `external` view that only answers
for your zones) that is just `internal`. A view that doesn't offer
recursion gets nothing, and naming one gives a warning. Without views,
blocking applies to the whole server.
**What it can't do:**
- Devices that don't use your server for DNS aren't affected. That includes
browsers using their own DNS over HTTPS, and devices with a hard-coded
resolver (8.8.8.8). A firewall rule that sends port 53 to your server, and
your network's DNS settings, are the usual answers.
- A device that validates DNSSEC itself sees a blocked name as a
validation failure rather than "doesn't exist". It's still blocked.
- A list can contain names in your own zones. They are blocked like any
other; allow them with a rule if that's not what you want.
## Your rules
A rule **blocks** or **allows** a domain and every name below it, or with
`--exact` (web UI: "only this exact name") just the domain itself.
```sh
jarl block add doubleclick.net # doubleclick.net and everything below it
jarl block add telemetry.example.com --exact # just that name
jarl block allow cdn.example.com --view internal # an exception, in one view
jarl block list
jarl block delete doubleclick.net # or: --id ID
```
**Precedence:** your rules come before every blocklist, and among your
rules the most specific one wins. With these rules:
| Rule | |
|------|-|
| block `example.com` | and below |
| allow `cdn.example.com` | exact |
`example.com` and `www.example.com` are blocked, `cdn.example.com` is
allowed, and `img.cdn.example.com` is blocked again (the allow rule was for
the exact name). An allow rule beats a blocklist entry whatever the lists
say.
There is one rule per domain and view; a second one is refused. A rule
takes effect as soon as the change is applied.
## Blocklists
**Formats.** Lists are read line by line; one list may mix formats:
| Line | Blocks |
|------|--------|
| `\|\|ads.example^` (Adblock) | the name and every name below it |
| `*.ads.example` | the name and every name below it |
| `0.0.0.0 ads.example` (hosts file, any address) | just the name |
| `ads.example` (plain domain) | just the name |
Comments (`#`, `!`), headers and blank lines are ignored. Lines that DNS
can't act on are skipped and counted: Adblock exceptions (`@@`), cosmetic
and URL rules, addresses, invalid names, and single labels (a broken list
can't block `com`). Prefer lists in Adblock format: they say which entries
cover the names below them.
**Well-known lists** (sizes in October 2026; they change):
| List | URL | Domains | Memory per view |
|------|-----|--------:|----------------:|
| Hagezi Light | `https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/light.txt` | 55,000 | ~65 MB |
| Hagezi Multi Normal | `https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/multi.txt` | 159,000 | ~190 MB |
| Hagezi Pro | `https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/pro.txt` | 195,000 | ~235 MB |
| OISD Small | `https://small.oisd.nl/` | 58,000 | ~70 MB |
| OISD Big | `https://big.oisd.nl/` | 241,000 | ~290 MB |
| StevenBlack hosts | `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` | 73,000 | ~45 MB |
One good list is usually enough; lists overlap a lot. Hagezi Multi Normal
is a balanced start; Light and OISD Small block less but almost never
something you need.
**Updates.** jarld checks each list every `blocking.refresh` (a day by
default) and downloads it only when the server says it changed. A new
version is checked with `named-checkzone` and replaces the old one only
when its content differs; named then reloads it. If a download or check
fails, named keeps using the previous version, jarld tries again within
the hour, and the dashboard shows the error. To update now:
```sh
jarl blocklist refresh # every list; or: refresh hagezi-multi [--force]
```
**Managing lists:**
```sh
jarl blocklist update hagezi-multi --view internal # --view auto: every view with recursion
jarl blocklist update hagezi-multi --url URL --name NEW-NAME
jarl blocklist disable hagezi-multi # stop using it, keep it (enable: back without a download)
jarl blocklist delete hagezi-multi
```
A list's state is **ok** (in use), **waiting** (not downloaded yet),
**error** (the last update failed; the previous version stays in use),
**disabled**, or **idle** (it applies to no view, so it does nothing).
## Checking a name
`jarl block check NAME`, or the box at the top of the Blocking page, says
for each view whether a name is blocked or allowed, and by which rule or
list entry:
```
$ jarl block check img.ads.example.com
view internal: blocked by blocklist hagezi-multi (entry *.ads.example.com)
view guest: allowed by your rule 12 (entry *.ads.example.com)
```
Views without blocking aren't listed.
In the web UI a blocked name gets an **Allow** button, and the statistics'
most blocked names link to this check.
## Statistics
With [query statistics](stats.md) on (including `query_log`), Jarl counts
what blocking does: blocked queries over time and as a share of all
queries, which rules and lists blocked them, and the most blocked names
(`jarl stats summary`, `jarl stats top blocked`, `jarl stats breakdown
blocklist`). Blocked answers are left out of the NXDOMAIN figures, which
are about names that really don't exist.
Without statistics named doesn't log what it blocks at all.
## Memory and performance
**Memory** is the cost to watch. named keeps each list in memory, once per
view it applies to: about 0.6 KB per policy record (an Adblock entry is
two records: the name, and the names below it). The Blocking page and
`jarl blocklist list` show the estimate per list. Hagezi Pro in one view
took named from 25 MB to 260 MB in a test with BIND 9.20; while a list
reloads, the old and new versions briefly coexist.
**Answers aren't slowed**: named looks blocked names up in memory, and
answers them without asking the internet.
**Updating a list** (Hagezi Pro: download, parse, check) takes jarld about
2 seconds. While named loads the new version, the queries that the loading
thread would have answered wait, up to half a second; this only happens
when the list changed, at most once per `blocking.refresh`.
**Other changes are unaffected**: the lists' contents never go through
Jarl's apply, so adding a record takes as long with a list as without.
Downloads are limited by `blocking.max_download_mb` and
`blocking.max_entries` ([configuration.md](configuration.md#blocking)).
## How it works
For each view (or the server, without views) Jarl writes a
`response-policy` statement and the policy zones it lists, in this order:
```
response-policy {
zone "rules.jarl.invalid" min-update-interval 0; # your rules
zone "hagezi-multi.blocklist.jarl.invalid" min-update-interval 0;
} qname-wait-recurse no break-dnssec yes;
```
- Policy zones are named under `.invalid`, so they can't clash with real
zones (zone names under `jarl.invalid` are refused). They allow no
queries or transfers.
- The rules zone of each view is a file next to your zone files
(`/var/lib/bind/jarl/VIEW/rules.jarl.invalid.rpz`) and shows up in the
changeset diff like any other.
- A list's file is `/var/cache/bind/jarl/blocklists/list-ID.rpz`, shared by
the views that use it. Until the first download it is an empty policy
zone, so named always starts. Files of deleted lists are removed.
- `qname-wait-recurse no` blocks without first resolving the real answer;
`break-dnssec yes` also blocks for clients that ask for DNSSEC data;
`min-update-interval 0` applies a new version at once.
- named looks names up in a policy zone like in any zone, so a wildcard
doesn't apply below a name the zone also contains. Jarl writes the zones
so that the most specific rule still wins.
- A `response-policy` option of your own can't be combined with Jarl's
blocking in the same view: Jarl refuses the change and says so.
- Rewrites are logged to Jarl's response log when statistics read it, and
not at all otherwise.
Lists and rules need the `blocking` permission ([permissions.md](permissions.md)).
## Troubleshooting
- **"no domains found in N lines"**: the URL didn't return a list (often an
HTML error page). Check it in a browser.
- **A list has no effect / is "idle"**: no view it applies to offers
recursion. Pick the view your devices use (`--view internal`).
- **Something is still answered**: `jarl block check NAME` shows whether
Jarl blocks it. If it does, the device probably doesn't use your server
(DNS over HTTPS, another resolver), or has the answer cached for a
while.
- **Something you need is blocked**: `jarl block check NAME` shows which
list; `jarl block allow NAME` overrides it.
docs/cli.md
+20 -2
View File
@@ -419,15 +419,33 @@ working.
```sh
jarl stats status
jarl stats summary --since 7d
jarl stats top names | clients | nxdomain | servfail | refused [--since 24h] [--limit 20]
jarl stats top names | clients | nxdomain | servfail | refused | blocked [--since 24h] [--limit 20]
jarl stats timeline --since 6h [--key rcode:NXDOMAIN] [--table]
jarl stats heatmap --since 28d
jarl stats breakdown qtype | rcode | view | zone | answer | rtt
jarl stats breakdown qtype | rcode | view | zone | answer | rtt | blocklist
```
Statistics must be enabled in `jarl.yaml`, and `-o json` exports the data.
See [stats.md](stats.md).
## Blocking
```sh
jarl blocklist list
jarl blocklist add NAME URL [--view V]... # default: every view that offers recursion
jarl blocklist update NAME [--url U] [--name N] [--view V | --view auto] [--comment C]
jarl blocklist disable | enable | delete NAME
jarl blocklist refresh [NAME...] [--force] # download now (all lists without NAME)
jarl block add DOMAIN [--exact] [--view V]... # block it (and the names below it)
jarl block allow DOMAIN [--exact] [--view V]... # an exception that beats every list
jarl block list
jarl block delete DOMAIN [--view V] | --id ID
jarl block check DOMAIN # blocked? by which rule or list?
```
See [blocking.md](blocking.md).
## ACME DNS-01
```sh
docs/configuration.md
+11
View File
@@ -134,6 +134,17 @@ jarl.yaml; chmod 640 jarl.yaml`, as the Debian package installs it); jarld
warns at start when everyone can read it. At porkbun.com, switch on "API
Access" for the domain, and limit the key to the domain if you can.
### Blocking
How jarld downloads blocklists ([blocking.md](blocking.md)). The lists and
rules themselves are configuration in the database.
| Key | Default | Meaning |
|-----|---------|---------|
| `blocking.refresh` | `24h` | How often each blocklist is checked for changes (at least `1h`); it is downloaded only when it changed ([blocking.md](blocking.md)). |
| `blocking.max_download_mb` | `64` | Larger downloads are refused (the previous version stays in use). |
| `blocking.max_entries` | `1000000` | Lists with more domains are refused. |
## Example: management network with TLS
```yaml
docs/development.md
+2
View File
@@ -28,6 +28,8 @@ internal/
nscheck/ DNS queries to other name servers: zone serials, the parent's delegation and DS records
dnssec/ reads named's DNSSEC key and state files; DS records; the signing phases (docs/dnssec.md)
porkbun/ Porkbun API client (name servers, glue, DS records) and an in-memory fake for tests
blocking/ blocklist parser, response policy zone writers, and matching a name the way named does
(the downloads are in engine/blocklists.go)
config/ jarld's YAML configuration
deploy/ systemd units (jarld, the daily backup service and timer), sysusers.d and tmpfiles.d files
deploy/debian/ the Debian package's control template and postinst/prerm/postrm
docs/permissions.md
+2 -1
View File
@@ -135,6 +135,7 @@ RESOURCE:ACCESS [zones=PATTERNS] [views=PATTERNS] [names=PATTERNS] [types=TYPES]
| `options` | server options, top-level statements, the defaults for new zones, `jarl check` |
| `audit` | everyone's audit events and changesets (without it, users see only their own) |
| `stats` | query statistics, including client addresses and the names they look up (read only) |
| `blocking` | blocklists and block/allow rules ([blocking.md](blocking.md)); `write` also downloads lists now |
| `*` | all of the above |
`ACCESS` is `read` or `write` (write includes read). Filters narrow a grant.
@@ -168,7 +169,7 @@ you can then edit.
| Role | Grants |
|------|--------|
| `readonly` | `*:read` (includes statistics) |
| `dns-editor` | `zones:write`, `records:write`, `views:read`, `acls:read`, `options:read` |
| `dns-editor` | `zones:write`, `records:write`, `views:read`, `acls:read`, `options:read`, `blocking:read` |
| `operator` | `*:write` (everything except user management) |
| `acme` | `records:write names=_acme-challenge.* types=TXT` |
docs/stats.md
+5 -1
View File
@@ -102,10 +102,14 @@ lists, views, zones). named's own counters can't be filtered.
| `view:NAME` | response log | answers per view (`_default` without views) |
| `zone:NAME` | response log | answers per local zone; `zone:-` for names in none of your zones |
| `responses` | response log | lines read (a health check) |
| `blocked`, `blocklist:NAME` | response log | queries [blocking](blocking.md) answered, and by which list (`blocklist:rules` for your own rules) |
**Top lists** (per hour and per day, the `top_n` largest): `name` (most
queried names), `client`, `nxdomain` (names answered NXDOMAIN), `servfail`
(names answered SERVFAIL), `refused` (clients that were refused).
(names answered SERVFAIL), `refused` (clients that were refused), `blocked`
(names blocking answered). Blocked answers are NXDOMAIN too, but they
aren't counted as names that don't exist: not in the `nxdomain` list, and
not in the summary's NXDOMAIN ratio.
Nothing else is stored. In particular there's no per-query record, so you
can't look up "who asked for X at 14:02". That is deliberate, both for size
docs/validation.md
+14
View File
@@ -30,6 +30,7 @@ showing them.
| `record_out_of_zone` | error | The name must be inside the zone (catches `www.example.com` without the final dot) |
| `record_duplicate` | error | The same record can't exist twice |
| `cname_conflict` | error | A CNAME must be the only record at its name (RFC 1034), so no CNAME next to A, MX, TXT... |
| `disabled_conflict` | warning | A disabled record that couldn't be enabled again as the zone is now (a CNAME now shares its name) |
| `cname_multiple` | error | A name can have only one CNAME |
| `cname_apex` | error | No CNAME at the zone apex (it needs SOA and NS) |
| `cname_loop` | error | CNAMEs that point back to themselves |
@@ -100,6 +101,19 @@ applied.
| `dnssec_old_keys` | info | Signing a zone whose key directory still has old keys |
| `serial_not_higher` | error | An explicit serial on a signed zone must be above the one named has loaded (automatic serials are raised by themselves) |
## Blocking
From [blocking.md](blocking.md).
| Code | Severity | Rule |
|------|----------|------|
| `blocking_rule_duplicate` | error | One rule per domain and view |
| `blocking_response_policy` | error | No `response-policy` option where Jarl's blocking writes that statement (the view, or the server) |
| `zone_name_reserved` | error | Zone names under `jarl.invalid` are reserved for blocking's policy zones |
| `blocking_view_missing` | error | A list or rule names a view that doesn't exist |
| `blocking_no_view` | warning | A list or rule applies to no view: none offers recursion, and blocking only changes recursive answers |
| `blocking_no_recursion` | warning | A list or rule names a view that doesn't offer recursion, where it has no effect |
## Security lints
| Code | Severity | What it catches |