Add optional query statistics
Off by default (stats in jarl.yaml). When enabled, Jarl adds a loopback statistics channel and a response log to named's configuration (merged into existing logging/statistics-channels statements) and switches response logging with rndc responselog, because the responselog option only counts when named starts. jarld polls named's counters every minute and tails the response log (resuming after restarts and across rotation). It stores aggregates, never individual queries, in a separate SQLite database: per-minute, hourly and daily counters (queries, result codes, record types, views, zones, cache hits, upstream latency, DNSSEC) and hourly/daily top lists (names, clients, NXDOMAIN and SERVFAIL names, refused clients), each with its own retention. Clients can be anonymized or excluded. - Web UI: Statistics page with KPI tiles, queries and failed answers over time, a weekday x hour heatmap, top lists and breakdowns; SVG charts with hover/keyboard readouts, a table view and light/dark palettes validated for colour-vision deficiency. - CLI: jarl stats status|summary|top|timeline|heatmap|breakdown. - API: /api/v1/stats/*, behind a new "stats" permission resource. - jarld brings named's configuration up to date with jarl.yaml at startup (engine.SyncConfig), never overwriting hand-edited files. - resync now refuses to overwrite drifted files without force, as documented. - tmpfiles.d creates /var/log/named (allowed by named's AppArmor profile on Debian 13 and Ubuntu). - The e2e harness runs named with -f like the Debian unit. - docs/stats.md, plus API, CLI, configuration, permissions and README updates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
abrown38andClaude Opus 5.5 committed
1 parent
0795a1001e
commit
3b7bee67f8
44 files changed
+4163
-42
No files matched your search
+18
@@ -268,6 +268,24 @@ in it and its records.
|
||||
| POST | `/system/resync` | `{"dry_run": false, "force": false}`: rewrite every file from the database and reload (admin) |
|
||||
| GET | `/audit` | `?user=&action=&outcome=&object_type=&changeset=&q=&since=&until=&limit=200&before=ID` |
|
||||
|
||||
### Query statistics
|
||||
|
||||
Needs `stats:read`. When statistics are disabled, every endpoint but
|
||||
`status` answers 404 `stats_disabled`. See [stats.md](stats.md).
|
||||
|
||||
| Method | Path | Query | Returns |
|
||||
|--------|------|-------|---------|
|
||||
| GET | `/stats/status` | | `{enabled, query_log, statistics_channel, log_file, last_poll, last_log_line, log_lines_read, first_data, retention, warnings}` |
|
||||
| GET | `/stats/summary` | `since` (90m, 24h, 7d, 2w; default 24h), `top` (default 10) | totals, `avg_qps`, `peak`, `rcodes`, `qtypes`, `views`, `zones`, `answers`, `transport`, `cache_hit_ratio`, `nxdomain_ratio`, `tops` |
|
||||
| GET | `/stats/timeline` | `since`, `step` (auto, 1m, 5m, 15m, 1h, 6h, 1d), `key` (repeatable or comma separated; `rcode:*` = every counter of a group; default `queries`) | `{step, times: [unix...], series: [{key, values}]}` |
|
||||
| GET | `/stats/top` | `kind` (name, client, nxdomain, servfail, refused), `since`, `limit` (default 20) | `[{key, count}]` |
|
||||
| GET | `/stats/heatmap` | `since` (default 28d), `key` (default queries) | `{key, from, avg[7][24], samples[7][24]}`, Monday first, local time, `-1` = no data |
|
||||
| GET | `/stats/breakdown` | `group` (qtype, rcode, view, zone, answer, transport, rtt, dnssec, cache, ratelimit), `since` | `[{key, count}]` |
|
||||
|
||||
```sh
|
||||
curl -sf -H "$AUTH" "$API/stats/top?kind=nxdomain&since=7d&limit=10" | jq -r '.[] | "\(.count)\t\(.key)"'
|
||||
```
|
||||
|
||||
### Users and tokens
|
||||
|
||||
| Method | Path | Body |
|
||||
|
||||
+14
@@ -218,6 +218,20 @@ jarl resync [--force] [--dry-run] # rewrite every file from the database
|
||||
|
||||
Grant syntax and roles are in [permissions.md](permissions.md).
|
||||
|
||||
## Query statistics
|
||||
|
||||
```sh
|
||||
jarl stats status
|
||||
jarl stats summary --since 7d
|
||||
jarl stats top names | clients | nxdomain | servfail | refused [--since 24h] [--limit 20]
|
||||
jarl stats timeline --since 6h [--key rcode:NXDOMAIN] [--table]
|
||||
jarl stats heatmap --since 28d
|
||||
jarl stats breakdown qtype | rcode | view | zone | answer | rtt
|
||||
```
|
||||
|
||||
Statistics must be enabled in `jarl.yaml`, and `-o json` exports the data.
|
||||
See [stats.md](stats.md).
|
||||
|
||||
## ACME DNS-01
|
||||
|
||||
```sh
|
||||
|
||||
@@ -45,6 +45,12 @@ Durations accept Go syntax (`90s`, `15m`, `12h`) and days (`30d`).
|
||||
Audit events are logged as `msg=audit` lines as well as stored in the
|
||||
database.
|
||||
|
||||
### Query statistics
|
||||
|
||||
The `stats:` section turns on query statistics (off by default). See
|
||||
[stats.md](stats.md) for every setting and what it changes in named's
|
||||
configuration. Changing it takes effect when jarld restarts.
|
||||
|
||||
### BIND
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|
||||
@@ -20,6 +20,7 @@ internal/
|
||||
store/ SQLite (modernc.org/sqlite, no cgo): state, changesets, users, tokens, sessions, audit
|
||||
auth/ argon2id passwords, tokens, sessions, grants and roles
|
||||
server/ HTTP API, authentication, the web UI
|
||||
stats/ query statistics: statistics-channel poller, response-log tailer, collector, separate SQLite store
|
||||
web/static/ the web UI: plain ES modules, no build step, embedded with go:embed
|
||||
api/, client/ JSON types and the Go client shared by server and CLI
|
||||
config/ jarld's YAML configuration
|
||||
|
||||
@@ -201,6 +201,12 @@ The CLI's settings are stored in `~/.config/jarl/config.yaml` (mode 0600).
|
||||
Create more tokens in the web UI (Account → API tokens) or with
|
||||
`jarl token create NAME`. See [cli.md](cli.md).
|
||||
|
||||
## Optional: query statistics
|
||||
|
||||
To see query volume, top names and clients in the UI and CLI, set
|
||||
`stats: {enabled: true}` in `/etc/jarl/jarl.yaml` and restart jarl. See
|
||||
[stats.md](stats.md).
|
||||
|
||||
## TLS certificate from Let's Encrypt
|
||||
|
||||
Jarl can publish the DNS-01 challenge for its own certificate. On the DNS
|
||||
|
||||
@@ -34,6 +34,10 @@ sudo -u jarl jarld backup /var/backups/jarl/jarl-$(date +%F).db
|
||||
find /var/backups/jarl -name 'jarl-*.db' -mtime +30 -delete
|
||||
```
|
||||
|
||||
Query statistics live in a separate database (`/var/lib/jarl/stats.db`).
|
||||
It isn't needed to restore DNS, so back it up only if you want to keep the
|
||||
history.
|
||||
|
||||
The database contains password and token *hashes* and the TSIG secrets in
|
||||
the clear, so keep backups private.
|
||||
|
||||
|
||||
+2
-1
@@ -37,6 +37,7 @@ RESOURCE:ACCESS [zones=PATTERNS] [views=PATTERNS] [names=PATTERNS] [types=TYPES]
|
||||
| `tsig` | TSIG keys (`write` is needed to see a secret) |
|
||||
| `options` | server options, top-level statements, the defaults for new zones, `jarl check` |
|
||||
| `audit` | everyone's audit events and changesets (without it, users see only their own) |
|
||||
| `stats` | query statistics, including client addresses and the names they look up (read only) |
|
||||
| `*` | all of the above |
|
||||
|
||||
`ACCESS` is `read` or `write` (write includes read). Filters narrow a grant.
|
||||
@@ -69,7 +70,7 @@ you can then edit.
|
||||
|
||||
| Role | Grants |
|
||||
|------|--------|
|
||||
| `readonly` | `*:read` |
|
||||
| `readonly` | `*:read` (includes statistics) |
|
||||
| `dns-editor` | `zones:write`, `records:write`, `views:read`, `acls:read`, `options:read` |
|
||||
| `operator` | `*:write` (everything except user management) |
|
||||
| `acme` | `records:write names=_acme-challenge.* types=TXT` |
|
||||
|
||||
+182
@@ -0,0 +1,182 @@
|
||||
# Query statistics
|
||||
|
||||
Jarl can show what your DNS server is doing:
|
||||
- how many queries it answers over time, and the busiest hours of the week,
|
||||
- the most queried names, and the clients asking the most,
|
||||
- names that don't exist (NXDOMAIN), names that fail to resolve (SERVFAIL), and clients being refused,
|
||||
- record types, result codes, views and zones,
|
||||
- cache hit rate, and how fast upstream servers answer recursive queries.
|
||||
|
||||
All of it is in the web UI (**Statistics**), the CLI (`jarl stats`) and the
|
||||
API (`/api/v1/stats/...`).
|
||||
|
||||
Statistics are **off by default**. They are kept apart from your
|
||||
configuration in a database of their own, as aggregates only (never
|
||||
individual queries), and old data is removed automatically.
|
||||
|
||||
## Turning it on
|
||||
|
||||
Add to `/etc/jarl/jarl.yaml`, then restart jarl:
|
||||
|
||||
```yaml
|
||||
stats:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
```sh
|
||||
sudo systemctl restart jarl
|
||||
jarl stats status
|
||||
```
|
||||
|
||||
When jarld starts with statistics enabled, it:
|
||||
|
||||
1. adds a **statistics channel** to named's configuration, listening on
|
||||
`127.0.0.1:8058` only, so named publishes its counters as JSON,
|
||||
2. adds a **response log**: a logging channel writing one line per answered
|
||||
query to `/var/log/named/jarl-responses.log` (rotated by named itself at
|
||||
20 MB, 2 old files kept), plus `responselog yes;`,
|
||||
3. applies this through the usual checked pipeline (named-checkconf,
|
||||
reconfig, verification) and turns response logging on with
|
||||
`rndc responselog on` (the option alone only counts when named starts),
|
||||
4. reads both every minute and writes aggregates to `/var/lib/jarl/stats.db`.
|
||||
|
||||
If you already have a `logging` or `statistics-channels` statement, Jarl
|
||||
adds its channel to yours instead of replacing it. A `responselog` you set
|
||||
yourself is respected.
|
||||
|
||||
The tmpfiles.d entry creates `/var/log/named` (owned by bind, readable by
|
||||
group bind, which the `jarl` user belongs to). Debian's and Ubuntu's AppArmor
|
||||
profile for named allows writing there. If you installed Jarl before this
|
||||
version, run `sudo systemd-tmpfiles --create jarl.conf` once.
|
||||
|
||||
**Turning it off**: set `enabled: false` and restart jarl. Jarl removes
|
||||
the channel and log from named's configuration and turns response logging
|
||||
off again. The statistics database stays until you delete it
|
||||
(`/var/lib/jarl/stats.db`). Deleting it is always safe.
|
||||
|
||||
## Settings
|
||||
|
||||
```yaml
|
||||
stats:
|
||||
enabled: false
|
||||
query_log: true # per-query data; false = named's counters only
|
||||
database: /var/lib/jarl/stats.db
|
||||
statistics_channel: 127.0.0.1:8058 # must be a loopback address
|
||||
log_file: /var/log/named/jarl-responses.log
|
||||
log_size: 20m
|
||||
log_versions: 2
|
||||
retention:
|
||||
minute: 7d # per-minute counters
|
||||
hour: 90d # hourly counters and top lists
|
||||
day: 730d # daily counters and top lists
|
||||
top_n: 100 # names and clients kept per hour and per day
|
||||
anonymize_clients: false # store 10.10.0.0/24 instead of 10.10.0.37 (IPv6: /48)
|
||||
exclude_clients: [] # e.g. [10.10.0.250] to ignore a monitoring probe
|
||||
```
|
||||
|
||||
`query_log: false` keeps only what named counts itself: totals, record
|
||||
types, result codes, cache and recursion figures. It costs nothing on
|
||||
named's side. The per-name, per-client, per-view and per-zone figures need
|
||||
the response log. Logging every answer costs named some CPU and disk I/O,
|
||||
which is negligible at homelab volumes. At tens of thousands of queries per
|
||||
second, consider counters only.
|
||||
|
||||
`exclude_clients` drops those clients from the log-derived figures (top
|
||||
lists, views, zones). named's own counters can't be filtered.
|
||||
|
||||
## What is collected
|
||||
|
||||
**Counters** (every minute, summed into hours and days):
|
||||
|
||||
| Counter | Source | Meaning |
|
||||
|---------|--------|---------|
|
||||
| `queries` | named | queries received |
|
||||
| `rcode:NOERROR`, `rcode:NXDOMAIN`, ... | named | answers by result code |
|
||||
| `qtype:A`, `qtype:AAAA`, ... | named | queries by record type |
|
||||
| `answer:authoritative`, `answer:recursive` | named | answered from your zones, or by recursion/cache |
|
||||
| `recursion` | named | queries that needed recursion (cache misses) |
|
||||
| `cache:hits`, `cache:misses` | named | client queries answered from the cache or not |
|
||||
| `rtt:<10ms` ... `rtt:>1600ms` | named | how long other servers took to answer named |
|
||||
| `dnssec:valid`, `dnssec:failed` | named | DNSSEC validation results |
|
||||
| `transport:udp`, `transport:tcp`, `dropped`, `ratelimit:*` | named | |
|
||||
| `view:NAME` | response log | answers per view (`_default` without views) |
|
||||
| `zone:NAME` | response log | answers per local zone; `zone:-` for names in none of your zones |
|
||||
| `responses` | response log | lines read (a health check) |
|
||||
|
||||
**Top lists** (per hour and per day, the `top_n` largest): `name` (most
|
||||
queried names), `client`, `nxdomain` (names answered NXDOMAIN), `servfail`
|
||||
(names answered SERVFAIL), `refused` (clients that were refused).
|
||||
|
||||
Nothing else is stored. In particular there's no per-query record, so you
|
||||
can't look up "who asked for X at 14:02". That is deliberate, both for size
|
||||
and privacy.
|
||||
|
||||
## Using it
|
||||
|
||||
**Web UI**: Statistics. Pick a range (1 hour to 90 days); everything on the
|
||||
page follows it. Charts have a hover readout (or focus them and use the
|
||||
arrow keys) and a **Table** button showing the same data as a table. The
|
||||
heatmap shows the average queries per hour for each weekday and hour over at
|
||||
least the last 7 days. The page refreshes every minute.
|
||||
|
||||
**CLI**:
|
||||
|
||||
```sh
|
||||
jarl stats status # collecting? any problems?
|
||||
jarl stats summary --since 7d # totals, ratios, top lists
|
||||
jarl stats top names --since 24h --limit 50
|
||||
jarl stats top clients
|
||||
jarl stats top nxdomain # misconfigured hosts, typos, malware
|
||||
jarl stats top refused # who asks for what they may not have
|
||||
jarl stats timeline --since 6h # sparkline; --table for every bucket
|
||||
jarl stats timeline --since 7d --key rcode:NXDOMAIN --key rcode:SERVFAIL
|
||||
jarl stats heatmap --since 28d
|
||||
jarl stats breakdown qtype | rcode | view | zone | answer | rtt | dnssec | cache | transport | ratelimit
|
||||
jarl stats summary --since 30d -o json # export (also -o yaml)
|
||||
```
|
||||
|
||||
**API**: see [api.md](api.md#query-statistics).
|
||||
|
||||
**Permissions**: the `stats` resource (`stats:read`). The `readonly` and
|
||||
`operator` roles include it, `dns-editor` doesn't. Statistics show client
|
||||
addresses and what they look up, so grant it with care.
|
||||
|
||||
## How data is kept
|
||||
|
||||
- Minute counters are kept for `retention.minute`, hourly counters and top
|
||||
lists for `retention.hour`, daily ones for `retention.day`. Older data is
|
||||
deleted every hour.
|
||||
- Charts use the finest resolution that covers the range: minutes up to 48
|
||||
hours, hours up to 90 days, days beyond.
|
||||
- Top lists are exact within an hour (up to 200,000 distinct names or
|
||||
clients per hour, beyond which the rarest are dropped). Daily lists are
|
||||
built from the hourly top lists, so for very long tails they are
|
||||
approximate.
|
||||
- After a restart, jarld continues the log where it stopped, including a
|
||||
rotation that happened in the meantime. named's counters are read again
|
||||
from scratch, so the minutes jarld was down count no queries.
|
||||
- Days follow the server's local time zone. Hours are UTC hours, which is
|
||||
the same as local hours except in time zones with a half-hour offset.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
`jarl stats status` (and the top of the Statistics page) lists problems:
|
||||
|
||||
- **named's statistics channel is not reachable**: named isn't running, or
|
||||
doesn't have Jarl's configuration with statistics yet. Check
|
||||
`journalctl -u jarl` for the line about updating named's configuration
|
||||
at startup. If the files were changed by hand (drift), jarld refuses to
|
||||
rewrite them; run `jarl resync --force` after checking them.
|
||||
- **named answered N queries but Jarl read no response log lines**:
|
||||
`/var/log/named` is missing or not writable by named
|
||||
(`sudo systemd-tmpfiles --create jarl.conf`), or response logging was
|
||||
turned off. Jarl tries `rndc responselog on` again by itself every 5
|
||||
minutes. Check with `sudo rndc status | grep logging`.
|
||||
- **No top lists, only totals**: `query_log` is false.
|
||||
|
||||
## Debian 13 and Ubuntu
|
||||
|
||||
This works with the `bind9` packages of Debian 13 (BIND 9.20.26) and Ubuntu
|
||||
26.04 (9.20.24). Both are built with JSON statistics support (`json-c`), and
|
||||
their AppArmor profile allows `/var/log/named`. Debian also builds in dnstap,
|
||||
but Jarl doesn't need or use it, so the same setup works on both.
|
||||
Reference in new issue
Block a user