jarl/docs/stats.md

# Query statistics

Jarl can show what your DNS server is doing:
- how many queries it answers over time, and the busiest hours of the week,
- the most queried names, and the clients asking the most,
- names that don't exist (NXDOMAIN), names that fail to resolve (SERVFAIL), and clients being refused,
- record types, result codes, views and zones,
- cache hit rate, and how fast upstream servers answer recursive queries.

All of it is in the web UI (**Statistics**), the CLI (`jarl stats`) and the
API (`/api/v1/stats/...`).

Statistics are **off by default**. They are kept apart from your
configuration in a database of their own, as aggregates only (never
individual queries), and old data is removed automatically.

## Turning it on

Add to `/etc/jarl/jarl.yaml`, then restart jarl:

```yaml
stats:
  enabled: true
```

```sh
sudo systemctl restart jarl
jarl stats status
```

When jarld starts with statistics enabled, it:

1. adds a **statistics channel** to named's configuration, listening on
   `127.0.0.1:8058` only, so named publishes its counters as JSON,
2. adds a **response log**: a logging channel writing one line per answered
   query to `/var/log/named/jarl-responses.log` (rotated by named itself at
   20 MB, 2 old files kept), plus `responselog yes;`,
3. applies this through the usual checked pipeline (named-checkconf,
   reconfig, verification) and turns response logging on with
   `rndc responselog on` (the option alone only counts when named starts),
4. reads both every minute and writes aggregates to `/var/lib/jarl/stats.db`.

If you already have a `logging` or `statistics-channels` statement, Jarl
adds its channel to yours instead of replacing it. A `responselog` you set
yourself is respected.

The tmpfiles.d entry creates `/var/log/named` (owned by bind, readable by
group bind, which the `jarl` user belongs to). Debian's and Ubuntu's AppArmor
profile for named allows writing there. If you installed Jarl before this
version, run `sudo systemd-tmpfiles --create jarl.conf` once.

**Turning it off**: set `enabled: false` and restart jarl. Jarl removes
the channel and log from named's configuration and turns response logging
off again. The statistics database stays until you delete it
(`/var/lib/jarl/stats.db`). Deleting it is always safe.

## Settings

```yaml
stats:
  enabled: false
  query_log: true          # per-query data; false = named's counters only
  database: /var/lib/jarl/stats.db
  statistics_channel: 127.0.0.1:8058   # must be a loopback address
  log_file: /var/log/named/jarl-responses.log
  log_size: 20m
  log_versions: 2
  retention:
    minute: 7d             # per-minute counters
    hour: 90d              # hourly counters and top lists
    day: 730d              # daily counters and top lists
  top_n: 100               # names and clients kept per hour and per day
  anonymize_clients: false # store 10.10.0.0/24 instead of 10.10.0.37 (IPv6: /48)
  exclude_clients: []      # e.g. [10.10.0.250] to ignore a monitoring probe
```

`query_log: false` keeps only what named counts itself: totals, record
types, result codes, cache and recursion figures. It costs nothing on
named's side. The per-name, per-client, per-view and per-zone figures need
the response log. Logging every answer costs named some CPU and disk I/O,
which is negligible at homelab volumes. At tens of thousands of queries per
second, consider counters only.

`exclude_clients` drops those clients from the log-derived figures (top
lists, views, zones). named's own counters can't be filtered.

## What is collected

**Counters** (every minute, summed into hours and days):

| Counter | Source | Meaning |
|---------|--------|---------|
| `queries` | named | queries received |
| `rcode:NOERROR`, `rcode:NXDOMAIN`, ... | named | answers by result code |
| `qtype:A`, `qtype:AAAA`, ... | named | queries by record type |
| `answer:authoritative`, `answer:recursive` | named | answered from your zones, or by recursion/cache |
| `recursion` | named | queries that needed recursion (cache misses) |
| `cache:hits`, `cache:misses` | named | client queries answered from the cache or not |
| `rtt:<10ms` ... `rtt:>1600ms` | named | how long other servers took to answer named |
| `dnssec:valid`, `dnssec:failed` | named | DNSSEC validation results |
| `transport:udp`, `transport:tcp`, `dropped`, `ratelimit:*` | named | |
| `view:NAME` | response log | answers per view (`_default` without views) |
| `zone:NAME` | response log | answers per local zone; `zone:-` for names in none of your zones |
| `responses` | response log | lines read (a health check) |
| `blocked`, `blocklist:NAME` | response log | queries [blocking](blocking.md) answered, and by which list (`blocklist:rules` for your own rules) |
| `rrl:drop`, `rrl:slip`, `rrl:would-drop`, `rrl:would-slip` | response log | responses [rate limiting](exposure.md#response-rate-limiting) limited, or would have in log-only mode (named's own `ratelimit:*` counters don't move then) |

**Server** (sampled every minute; stored as gauges, `g:KEY` the sum of
the samples and `gn:KEY` their number, so a chart shows the average):

| Gauge | Source | Meaning |
|-------|--------|---------|
| `g:named:rss` | /proc | named's resident memory (what the system sees), bytes |
| `g:named:mem` | named | the memory named counts itself as using, bytes |
| `g:named:cpu` | /proc | named's CPU use over the minute, in tenths of a percent of one CPU |
| `g:cache:mem`, `g:cache:rrsets` | named | the cache's memory and RRsets, all views |
| `g:host:mem-used`, `g:host:load` | /proc | the host's memory in use, and its 1-minute load average times 100 |

Jarl finds named's process by name (the one whose command line mentions
named's main configuration file); reading its memory and CPU time needs no
privileges. With `/proc` mounted `hidepid`, it isn't visible and those
figures are left out. The current values also include named's sockets, its
recursive and TCP clients against their limits (`rndc status`), the most
at once since it started, and the space left on the file systems holding
the zone files, Jarl's cache, named.conf, the response log and this
database.

**Top lists** (per hour and per day, the `top_n` largest): `name` (most
queried names), `client`, `nxdomain` (names answered NXDOMAIN), `servfail`
(names answered SERVFAIL), `refused` (clients that were refused), `blocked`
(names blocking answered), `blocked_client` (clients by blocked queries). Blocked answers are NXDOMAIN too, but they
aren't counted as names that don't exist: not in the `nxdomain` list, and
not in the summary's NXDOMAIN ratio.

Nothing else is stored. In particular there's no per-query record, so you
can't look up "who asked for X at 14:02". That is deliberate, both for size
and privacy.

**Client names.** Where the client lists (busiest and refused clients) show
an address Jarl has a record for, they show its host name too: the PTR
record's target, or else the name of an A/AAAA record with that address
(disabled records don't count, and only records you may read). This is
looked up in Jarl's own data when the list is shown, not stored and not
asked of DNS, so it is instant and adds no queries of its own. The name is
what the records say now: with DHCP, an address seen last week may have
been another machine. Anonymized clients (`anonymize_clients`) are
prefixes and get no name.

## Using it

**Web UI**: Statistics. Pick a range (1 hour to 90 days); everything on the
page follows it. Charts have a hover readout (or focus them and use the
arrow keys) and a **Table** button showing the same data as a table. The
heatmap shows the average queries per hour for each weekday and hour over at
least the last 7 days. The page refreshes every minute. **Server**, at the
bottom, shows named's memory, CPU, cache and clients, the host and the
disks now, and named's memory and CPU over the range.

**CLI**:

```sh
jarl stats status                          # collecting? any problems?
jarl stats summary --since 7d              # totals, ratios, top lists
jarl stats top names --since 24h --limit 50
jarl stats top clients
jarl stats top nxdomain                    # misconfigured hosts, typos, malware
jarl stats top refused                     # who asks for what they may not have
jarl stats timeline --since 6h             # sparkline; --table for every bucket
jarl stats timeline --since 7d --key rcode:NXDOMAIN --key rcode:SERVFAIL
jarl stats heatmap --since 28d
jarl stats breakdown qtype | rcode | view | zone | answer | rtt | dnssec | cache | transport | ratelimit | rrl
jarl stats top ratelimited                 # client networks rate limiting limits (or would)
jarl stats server                          # memory, CPU, cache, clients, host, disks now
jarl stats timeline --since 7d --key g:named:rss --table   # named's memory over a week
jarl stats summary --since 30d -o json     # export (also -o yaml)
```

**API**: see [api.md](api.md#query-statistics).

**Prometheus**: with `metrics.enabled`, the counters are also served at
`/metrics` ([metrics.md](metrics.md)).

**Permissions**: the `stats` resource (`stats:read`). The `readonly` and
`operator` roles include it, `dns-editor` doesn't. Statistics show client
addresses and what they look up, so grant it with care.

## How data is kept

- Minute counters are kept for `retention.minute`, hourly counters and top
  lists for `retention.hour`, daily ones for `retention.day`. Older data is
  deleted every hour.
- Charts use the finest resolution that covers the range: minutes up to 48
  hours, hours up to 90 days, days beyond.
- Top lists are exact within an hour (up to 200,000 distinct names or
  clients per hour, beyond which the rarest are dropped). Daily lists are
  built from the hourly top lists, so for very long tails they are
  approximate.
- After a restart, jarld continues the log where it stopped, including a
  rotation that happened in the meantime. named's counters are read again
  from scratch, so the minutes jarld was down count no queries.
- Days follow the server's local time zone. Hours are UTC hours, which is
  the same as local hours except in time zones with a half-hour offset.

## Troubleshooting

`jarl stats status` (and the top of the Statistics page) lists problems:

- **named's statistics channel is not reachable**: named isn't running, or
  doesn't have Jarl's configuration with statistics yet. Check
  `journalctl -u jarl` for the line about updating named's configuration
  at startup. If the files were changed by hand (drift), jarld refuses to
  rewrite them; run `jarl resync --force` after checking them.
- **named answered N queries but Jarl read no response log lines**:
  `/var/log/named` is missing or not writable by named
  (`sudo systemd-tmpfiles --create jarl.conf`), or response logging was
  turned off. Jarl tries `rndc responselog on` again by itself every 5
  minutes. Check with `sudo rndc status | grep logging`.
- **No top lists, only totals**: `query_log` is false.

## Debian 13 and Ubuntu

This works with the `bind9` packages of Debian 13 (BIND 9.20.26) and Ubuntu
26.04 (9.20.24). Both are built with JSON statistics support (`json-c`), and
their AppArmor profile allows `/var/log/named`. Debian also builds in dnstap,
but Jarl doesn't need or use it, so the same setup works on both.