jarl/docs/stats.md

# Query statistics

Jarl can show what your DNS server is doing:
- how many queries it answers over time, and the busiest hours of the week,
- the most queried names, and the clients asking the most,
- names that don't exist (NXDOMAIN), names that fail to resolve (SERVFAIL), and clients being refused,
- record types, result codes, views and zones,
- cache hit rate, and how fast upstream servers answer recursive queries.

All of it is in the web UI (**Statistics**), the CLI (`jarl stats`) and the
API (`/api/v1/stats/...`).

Statistics are **off by default**. They are kept apart from your
configuration in a database of their own, as aggregates only (never
individual queries), and old data is removed automatically.

## Turning it on

Add to `/etc/jarl/jarl.yaml`, then restart jarl:

```yaml
stats:
  enabled: true
```

```sh
sudo systemctl restart jarl
jarl stats status
```

When jarld starts with statistics enabled, it:

1. adds a **statistics channel** to named's configuration, listening on
   `127.0.0.1:8058` only, so named publishes its counters as JSON,
2. adds a **response log**: a logging channel writing one line per answered
   query to `/var/log/named/jarl-responses.log` (rotated by named itself at
   20 MB, 2 old files kept), plus `responselog yes;`,
3. applies this through the usual checked pipeline (named-checkconf,
   reconfig, verification) and turns response logging on with
   `rndc responselog on` (the option alone only counts when named starts),
4. reads both every minute and writes aggregates to `/var/lib/jarl/stats.db`.

If you already have a `logging` or `statistics-channels` statement, Jarl
adds its channel to yours instead of replacing it. A `responselog` you set
yourself is respected.

The tmpfiles.d entry creates `/var/log/named` (owned by bind, readable by
group bind, which the `jarl` user belongs to). Debian's and Ubuntu's AppArmor
profile for named allows writing there. If you installed Jarl before this
version, run `sudo systemd-tmpfiles --create jarl.conf` once.

**Turning it off**: set `enabled: false` and restart jarl. Jarl removes
the channel and log from named's configuration and turns response logging
off again. The statistics database stays until you delete it
(`/var/lib/jarl/stats.db`). Deleting it is always safe.

## Settings

```yaml
stats:
  enabled: false
  query_log: true          # per-query data; false = named's counters only
  database: /var/lib/jarl/stats.db
  statistics_channel: 127.0.0.1:8058   # must be a loopback address
  log_file: /var/log/named/jarl-responses.log
  log_size: 20m
  log_versions: 2
  retention:
    minute: 7d             # per-minute counters
    hour: 90d              # hourly counters and top lists
    day: 730d              # daily counters and top lists
  top_n: 100               # names and clients kept per hour and per day
  anonymize_clients: false # store 10.10.0.0/24 instead of 10.10.0.37 (IPv6: /48)
  exclude_clients: []      # e.g. [10.10.0.250] to ignore a monitoring probe
```

`query_log: false` keeps only what named counts itself: totals, record
types, result codes, cache and recursion figures. It costs nothing on
named's side. The per-name, per-client, per-view and per-zone figures need
the response log. Logging every answer costs named some CPU and disk I/O,
which is negligible at homelab volumes. At tens of thousands of queries per
second, consider counters only.

`exclude_clients` drops those clients from the log-derived figures (top
lists, views, zones). named's own counters can't be filtered.

## What is collected

**Counters** (every minute, summed into hours and days):

| Counter | Source | Meaning |
|---------|--------|---------|
| `queries` | named | queries received |
| `rcode:NOERROR`, `rcode:NXDOMAIN`, ... | named | answers by result code |
| `qtype:A`, `qtype:AAAA`, ... | named | queries by record type |
| `answer:authoritative`, `answer:recursive` | named | answered from your zones, or by recursion/cache |
| `recursion` | named | queries that needed recursion (cache misses) |
| `cache:hits`, `cache:misses` | named | client queries answered from the cache or not |
| `rtt:<10ms` ... `rtt:>1600ms` | named | how long other servers took to answer named |
| `dnssec:valid`, `dnssec:failed` | named | DNSSEC validation results |
| `transport:udp`, `transport:tcp`, `dropped`, `ratelimit:*` | named | |
| `view:NAME` | response log | answers per view (`_default` without views) |
| `zone:NAME` | response log | answers per local zone; `zone:-` for names in none of your zones |
| `responses` | response log | lines read (a health check) |

**Top lists** (per hour and per day, the `top_n` largest): `name` (most
queried names), `client`, `nxdomain` (names answered NXDOMAIN), `servfail`
(names answered SERVFAIL), `refused` (clients that were refused).

Nothing else is stored. In particular there's no per-query record, so you
can't look up "who asked for X at 14:02". That is deliberate, both for size
and privacy.

## Using it

**Web UI**: Statistics. Pick a range (1 hour to 90 days); everything on the
page follows it. Charts have a hover readout (or focus them and use the
arrow keys) and a **Table** button showing the same data as a table. The
heatmap shows the average queries per hour for each weekday and hour over at
least the last 7 days. The page refreshes every minute.

**CLI**:

```sh
jarl stats status                          # collecting? any problems?
jarl stats summary --since 7d              # totals, ratios, top lists
jarl stats top names --since 24h --limit 50
jarl stats top clients
jarl stats top nxdomain                    # misconfigured hosts, typos, malware
jarl stats top refused                     # who asks for what they may not have
jarl stats timeline --since 6h             # sparkline; --table for every bucket
jarl stats timeline --since 7d --key rcode:NXDOMAIN --key rcode:SERVFAIL
jarl stats heatmap --since 28d
jarl stats breakdown qtype | rcode | view | zone | answer | rtt | dnssec | cache | transport | ratelimit
jarl stats summary --since 30d -o json     # export (also -o yaml)
```

**API**: see [api.md](api.md#query-statistics).

**Permissions**: the `stats` resource (`stats:read`). The `readonly` and
`operator` roles include it, `dns-editor` doesn't. Statistics show client
addresses and what they look up, so grant it with care.

## How data is kept

- Minute counters are kept for `retention.minute`, hourly counters and top
  lists for `retention.hour`, daily ones for `retention.day`. Older data is
  deleted every hour.
- Charts use the finest resolution that covers the range: minutes up to 48
  hours, hours up to 90 days, days beyond.
- Top lists are exact within an hour (up to 200,000 distinct names or
  clients per hour, beyond which the rarest are dropped). Daily lists are
  built from the hourly top lists, so for very long tails they are
  approximate.
- After a restart, jarld continues the log where it stopped, including a
  rotation that happened in the meantime. named's counters are read again
  from scratch, so the minutes jarld was down count no queries.
- Days follow the server's local time zone. Hours are UTC hours, which is
  the same as local hours except in time zones with a half-hour offset.

## Troubleshooting

`jarl stats status` (and the top of the Statistics page) lists problems:

- **named's statistics channel is not reachable**: named isn't running, or
  doesn't have Jarl's configuration with statistics yet. Check
  `journalctl -u jarl` for the line about updating named's configuration
  at startup. If the files were changed by hand (drift), jarld refuses to
  rewrite them; run `jarl resync --force` after checking them.
- **named answered N queries but Jarl read no response log lines**:
  `/var/log/named` is missing or not writable by named
  (`sudo systemd-tmpfiles --create jarl.conf`), or response logging was
  turned off. Jarl tries `rndc responselog on` again by itself every 5
  minutes. Check with `sudo rndc status | grep logging`.
- **No top lists, only totals**: `query_log` is false.

## Debian 13 and Ubuntu

This works with the `bind9` packages of Debian 13 (BIND 9.20.26) and Ubuntu
26.04 (9.20.24). Both are built with JSON statistics support (`json-c`), and
their AppArmor profile allows `/var/log/named`. Debian also builds in dnstap,
but Jarl doesn't need or use it, so the same setup works on both.