jarl/docs/stats.md

# Query statistics

Jarl can show what your DNS server is doing:
- how many queries it answers over time, and the busiest hours of the week,
- the most queried names, and the clients asking the most,
- names that don't exist (NXDOMAIN), names that fail to resolve (SERVFAIL), and clients being refused,
- record types, result codes, views and zones,
- cache hit rate, and how fast upstream servers answer recursive queries.

All of it is in the web UI (**Statistics**), the CLI (`jarl stats`) and the
API (`/api/v1/stats/...`).

Statistics are **off by default**. They are kept apart from your
configuration in a database of their own, as aggregates only (never
individual queries), and old data is removed automatically.

## Turning it on

Add to `/etc/jarl/jarl.yaml`, then restart jarl:

```yaml
stats:
  enabled: true
```

```sh
sudo systemctl restart jarl
jarl stats status
```

When jarld starts with statistics enabled, it:

1. adds a **statistics channel** to named's configuration, listening on
   `127.0.0.1:8058` only, so named publishes its counters as JSON,
2. adds a **response log**: a logging channel writing one line per answered
   query to `/var/log/named/jarl-responses.log` (rotated by named itself at
   20 MB, 2 old files kept), plus `responselog yes;`,
3. applies this through the usual checked pipeline (named-checkconf,
   reconfig, verification) and turns response logging on with
   `rndc responselog on` (the option alone only counts when named starts),
4. reads both every minute and writes aggregates to `/var/lib/jarl/stats.db`.

If you already have a `logging` or `statistics-channels` statement, Jarl
adds its channel to yours instead of replacing it. A `responselog` you set
yourself is respected.

The tmpfiles.d entry creates `/var/log/named` (owned by bind, readable by
group bind, which the `jarl` user belongs to). Debian's and Ubuntu's AppArmor
profile for named allows writing there. If you installed Jarl before this
version, run `sudo systemd-tmpfiles --create jarl.conf` once.

**Turning it off**: set `enabled: false` and restart jarl. Jarl removes
the channel and log from named's configuration and turns response logging
off again. The statistics database stays until you delete it
(`/var/lib/jarl/stats.db`). Deleting it is always safe.

## Settings

```yaml
stats:
  enabled: false
  query_log: true          # per-query data; false = named's counters only
  rate_limit_log: true     # with a rate-limit: also log what it limits (query-errors)
  database: /var/lib/jarl/stats.db
  statistics_channel: 127.0.0.1:8058   # must be a loopback address
  log_file: /var/log/named/jarl-responses.log
  log_size: 20m
  log_versions: 2
  retention:
    minute: 7d             # per-minute counters
    hour: 90d              # hourly counters and top lists
    day: 730d              # daily counters and top lists
    queries: 7d            # every answer, for exploring (0: none)
  top_n: 100               # names and clients kept per hour and per day
  anonymize_clients: false # store 10.10.0.0/24 instead of 10.10.0.37 (IPv6: /48)
  exclude_clients: []      # e.g. [10.10.0.250] to ignore a monitoring probe
```

`query_log: false` keeps only what named counts itself: totals, record
types, result codes, cache and recursion figures. It costs nothing on
named's side. The per-name, per-client, per-view and per-zone figures need
the response log. Logging every answer costs named some CPU and disk I/O,
which is negligible at homelab volumes. At tens of thousands of queries per
second, consider counters only.

`rate_limit_log` matters only with a `rate-limit` set: Jarl then sends
named's `query-errors` category (where rate limiting logs what it limits,
or would in log-only mode) to the response log too, and keeps it in
syslog. That other query errors come along is the price; `false` stops it,
and the Exposure page can't show rate limiting's activity then
([exposure.md](exposure.md#response-rate-limiting)).

`exclude_clients` drops those clients from the log-derived figures (top
lists, views, zones). named's own counters can't be filtered.

## What is collected

**Counters** (every minute, summed into hours and days):

| Counter | Source | Meaning |
|---------|--------|---------|
| `queries` | named | queries received |
| `rcode:NOERROR`, `rcode:NXDOMAIN`, ... | named | answers by result code |
| `qtype:A`, `qtype:AAAA`, ... | named | queries by record type |
| `answer:authoritative`, `answer:recursive` | named | answered from your zones, or by recursion/cache |
| `recursion` | named | queries that needed recursion (cache misses) |
| `cache:hits`, `cache:misses` | named | client queries answered from the cache or not |
| `rtt:<10ms` ... `rtt:>1600ms` | named | how long other servers took to answer named |
| `dnssec:valid`, `dnssec:failed` | named | DNSSEC validation results |
| `transport:udp`, `transport:tcp`, `dropped`, `ratelimit:*` | named | |
| `view:NAME` | response log | answers per view (`_default` without views) |
| `zone:NAME` | response log | answers per local zone; `zone:-` for names in none of your zones |
| `responses` | response log | lines read (a health check) |
| `blocked`, `blocklist:NAME` | response log | queries [blocking](blocking.md) answered, and by which list (`blocklist:rules` for your own rules) |
| `rrl:drop`, `rrl:slip`, `rrl:would-drop`, `rrl:would-slip` | response log | responses [rate limiting](exposure.md#response-rate-limiting) limited, or would have in log-only mode (named's own `ratelimit:*` counters don't move then) |

**Server** (sampled every minute; stored as gauges, `g:KEY` the sum of
the samples and `gn:KEY` their number, so a chart shows the average):

| Gauge | Source | Meaning |
|-------|--------|---------|
| `g:named:rss` | /proc | named's resident memory (what the system sees), bytes |
| `g:named:mem` | named | the memory named counts itself as using, bytes |
| `g:named:cpu` | /proc | named's CPU use over the minute, in tenths of a percent of one CPU |
| `g:cache:mem`, `g:cache:rrsets` | named | the cache's memory and RRsets, all views |
| `g:host:mem-used`, `g:host:load` | /proc | the host's memory in use, and its 1-minute load average times 100 |

Jarl finds named's process by name (the one whose command line mentions
named's main configuration file); reading its memory and CPU time needs no
privileges. With `/proc` mounted `hidepid`, it isn't visible and those
figures are left out. The current values also include named's sockets, its
recursive and TCP clients against their limits (`rndc status`), the most
at once since it started, and the space left on the file systems holding
the zone files, Jarl's cache, named.conf, the response log and this
database.

**Top lists** (per hour and per day, the `top_n` largest): `name` (most
queried names), `client`, `nxdomain` (names answered NXDOMAIN), `servfail`
(names answered SERVFAIL), `refused` (clients that were refused), `blocked`
(names blocking answered), `blocked_client` (clients by blocked queries). Blocked answers are NXDOMAIN too, but they
aren't counted as names that don't exist: not in the `nxdomain` list, and
not in the summary's NXDOMAIN ratio.

Nothing else is stored. In particular there's no per-query record, so you
can't look up "who asked for X at 14:02". That is deliberate, both for size
and privacy.

**Client names.** Where the client lists (busiest and refused clients) show
an address Jarl has a record for, they show its host name too: the PTR
record's target, or else the name of an A/AAAA record with that address
(disabled records don't count, and only records you may read). This is
looked up in Jarl's own data when the list is shown, not stored and not
asked of DNS, so it is instant and adds no queries of its own. The name is
what the records say now: with DHCP, an address seen last week may have
been another machine. Anonymized clients (`anonymize_clients`) are
prefixes and get no name.

## Using it

**Web UI**: Statistics. Pick a range (1 hour to 90 days); everything on the
page follows it. Charts have a hover readout (or focus them and use the
arrow keys) and a **Table** button showing the same data as a table. The
heatmap shows the average queries per hour for each weekday and hour over at
least the last 7 days. The page refreshes every minute. **Server**, at the
bottom, shows named's memory, CPU, cache and clients, the host and the
disks now, and named's memory and CPU over the range.

**CLI**:

```sh
jarl stats status                          # collecting? any problems?
jarl stats summary --since 7d              # totals, ratios, top lists
jarl stats top names --since 24h --limit 50
jarl stats top clients
jarl stats top nxdomain                    # misconfigured hosts, typos, malware
jarl stats top refused                     # who asks for what they may not have
jarl stats timeline --since 6h             # sparkline; --table for every bucket
jarl stats timeline --since 7d --key rcode:NXDOMAIN --key rcode:SERVFAIL
jarl stats heatmap --since 28d
jarl stats breakdown qtype | rcode | view | zone | answer | rtt | dnssec | cache | transport | ratelimit | rrl
jarl stats top ratelimited                 # client networks rate limiting limits (or would)
jarl stats server                          # memory, CPU, cache, clients, host, disks now
jarl stats timeline --since 7d --key g:named:rss --table   # named's memory over a week
jarl stats summary --since 30d -o json     # export (also -o yaml)
```

**API**: see [api.md](api.md#query-statistics).

**Prometheus**: with `metrics.enabled`, the counters are also served at
`/metrics` ([metrics.md](metrics.md)).

**Permissions**: the `stats` resource (`stats:read`). The `readonly` and
`operator` roles include it, `dns-editor` doesn't. Statistics show client
addresses and what they look up, so grant it with care.

## Exploring

The counters and top lists say how much, not who asked for what: "100
NXDOMAIN answers in the external view" can't tell you which clients asked.
For that, jarld also keeps every answer for a while, the **query log**: time,
client, view, name, record type, result, and the blocklist (or your rules)
that blocked it. It is in the statistics database, from the response log
named writes anyway, and answers older than `retention.queries` (default
7 days) are deleted every hour; `0` keeps none.

With it, every client, name, view, record type, result and blocklist on the
Statistics page is a link to the **explore** page (`#/stats/explore`),
which shows, for what you clicked:

* the answers over time (and how many were blocked);
* the top clients, names, views, record types, results and blocklists
  among them, each a link that narrows the filter further;
* the latest 100 answers, every cell a link.

So "external, NXDOMAIN" leads to the clients asking, a client to everything
it asks, one of its names to who else asks for it. The page also opens
from **Explore the answers** at the top of the Statistics page, and has a
filter bar to type filters in: a client, a name (or it and the names below
it), a view, a record type, a result, blocked or not; it suggests values
from what is shown. Filters combine (this
client, in this view, blocked only) and each has a ✕ to take it out. A name
can be widened to the names below it (`domain=`). The Statistics page's
view switches apply here too.

`anonymize_clients` and `exclude_clients` apply to the query log as to
everything else. Its size: each answer takes about 150 to 200 bytes with
its indexes, so 50,000 answers a day for 7 days is about 60 MB, 200,000 a
day about 250 MB. The Statistics page's server section shows the
statistics database's size and how many answers the query log holds, next
to the disks (with a bar of how full each is); the dashboard's Storage
panel has all of Jarl's databases.

## Views

With views, the Statistics page has a toggle per view (and **All** and
**None**): switch some off to see only the others, for example your
internal and VPS views without the internet's noise on the external one,
or several linked views together. The choice is remembered in your
browser. The CLI takes `--view` on every `jarl stats` command (repeat it,
or separate views with commas), and the API `?views=internal,vps`.

Only what named's response log says can be split by view: answers (shown
as *answers* instead of queries), result codes, record types, zones,
blocking, rate limiting and the top lists. named's own counters (answer
sources, transport, upstream response times, DNSSEC, the cache and the
server) are for the whole server, and the page says so; asking for them
per view (`jarl stats breakdown answer --view internal`) is refused.

Per-view figures are kept from the version that added them (Jarl 0.4.1);
the page says since when. They take some more room in the statistics
database, roughly in proportion to the number of views.

## How data is kept

- Minute counters are kept for `retention.minute`, hourly counters and top
  lists for `retention.hour`, daily ones for `retention.day`. Older data is
  deleted every hour.
- Charts use the finest resolution that covers the range: minutes up to 48
  hours, hours up to 90 days, days beyond.
- Top lists are exact within an hour (up to 200,000 distinct names or
  clients per hour, beyond which the rarest are dropped). Daily lists are
  built from the hourly top lists, so for very long tails they are
  approximate.
- After a restart, jarld continues the log where it stopped, including a
  rotation that happened in the meantime. named's counters are read again
  from scratch, so the minutes jarld was down count no queries.
- Days follow the server's local time zone. Hours are UTC hours, which is
  the same as local hours except in time zones with a half-hour offset.

## Troubleshooting

`jarl stats status` (and the top of the Statistics page) lists problems:

- **named's statistics channel is not reachable**: named isn't running, or
  doesn't have Jarl's configuration with statistics yet. Check
  `journalctl -u jarl` for the line about updating named's configuration
  at startup. If the files were changed by hand (drift), jarld refuses to
  rewrite them; run `jarl resync --force` after checking them.
- **named answered N queries but Jarl read no response log lines**:
  `/var/log/named` is missing or not writable by named
  (`sudo systemd-tmpfiles --create jarl.conf`), or response logging was
  turned off. Jarl tries `rndc responselog on` again by itself every 5
  minutes. Check with `sudo rndc status | grep logging`.
- **No top lists, only totals**: `query_log` is false.

## Debian 13 and Ubuntu

This works with the `bind9` packages of Debian 13 (BIND 9.20.26) and Ubuntu
26.04 (9.20.24). Both are built with JSON statistics support (`json-c`), and
their AppArmor profile allows `/var/log/named`. Debian also builds in dnstap,
but Jarl doesn't need or use it, so the same setup works on both.