jarl/docs/stats.md

Query statistics

Jarl can show what your DNS server is doing:

• how many queries it answers over time, and the busiest hours of the week,
• the most queried names, and the clients asking the most,
• names that don't exist (NXDOMAIN), names that fail to resolve (SERVFAIL), and clients being refused,
• record types, result codes, views and zones,
• cache hit rate, and how fast upstream servers answer recursive queries.

All of it is in the web UI (Statistics), the CLI (jarl stats) and the API (/api/v1/stats/...).

Statistics are off by default. They are kept apart from your configuration in a database of their own, as aggregates only (never individual queries), and old data is removed automatically.

Turning it on

Add to /etc/jarl/jarl.yaml, then restart jarl:

stats:
  enabled: true

sudo systemctl restart jarl
jarl stats status

When jarld starts with statistics enabled, it:

1. adds a statistics channel to named's configuration, listening on 127.0.0.1:8058 only, so named publishes its counters as JSON,
2. adds a response log: a logging channel writing one line per answered query to /var/log/named/jarl-responses.log (rotated by named itself at 20 MB, 2 old files kept), plus responselog yes;,
3. applies this through the usual checked pipeline (named-checkconf, reconfig, verification) and turns response logging on with rndc responselog on (the option alone only counts when named starts),
4. reads both every minute and writes aggregates to /var/lib/jarl/stats.db.

If you already have a logging or statistics-channels statement, Jarl adds its channel to yours instead of replacing it. A responselog you set yourself is respected.

The tmpfiles.d entry creates /var/log/named (owned by bind, readable by group bind, which the jarl user belongs to). Debian's and Ubuntu's AppArmor profile for named allows writing there. If you installed Jarl before this version, run sudo systemd-tmpfiles --create jarl.conf once.

Turning it off: set enabled: false and restart jarl. Jarl removes the channel and log from named's configuration and turns response logging off again. The statistics database stays until you delete it (/var/lib/jarl/stats.db). Deleting it is always safe.

Settings

stats:
  enabled: false
  query_log: true          # per-query data; false = named's counters only
  rate_limit_log: true     # with a rate-limit: also log what it limits (query-errors)
  database: /var/lib/jarl/stats.db
  statistics_channel: 127.0.0.1:8058   # must be a loopback address
  log_file: /var/log/named/jarl-responses.log
  log_size: 20m
  log_versions: 2
  retention:
    minute: 7d             # per-minute counters
    hour: 90d              # hourly counters and top lists
    day: 730d              # daily counters and top lists
    queries: 7d            # every answer, for exploring (0: none)
  top_n: 100               # names and clients kept per hour and per day
  anonymize_clients: false # store 10.10.0.0/24 instead of 10.10.0.37 (IPv6: /48)
  exclude_clients: []      # e.g. [10.10.0.250] to ignore a monitoring probe

query_log: false keeps only what named counts itself: totals, record types, result codes, cache and recursion figures. It costs nothing on named's side. The per-name, per-client, per-view and per-zone figures need the response log. Logging every answer costs named some CPU and disk I/O, which is negligible at homelab volumes. At tens of thousands of queries per second, consider counters only.

rate_limit_log matters only with a rate-limit set: Jarl then sends named's query-errors category (where rate limiting logs what it limits, or would in log-only mode) to the response log too, and keeps it in syslog. That other query errors come along is the price; false stops it, and the Exposure page can't show rate limiting's activity then (exposure.md).

exclude_clients drops those clients from the log-derived figures (top lists, views, zones). named's own counters can't be filtered.

What is collected

Counters (every minute, summed into hours and days):

Counter	Source	Meaning
queries	named	queries received
rcode:NOERROR, rcode:NXDOMAIN, ...	named	answers by result code
qtype:A, qtype:AAAA, ...	named	queries by record type
answer:authoritative, answer:recursive	named	answered from your zones, or by recursion/cache
recursion	named	queries that needed recursion (cache misses)
cache:hits, cache:misses	named	client queries answered from the cache or not
rtt:<10ms ... rtt:>1600ms	named	how long other servers took to answer named
dnssec:valid, dnssec:failed	named	DNSSEC validation results
transport:udp, transport:tcp, dropped, ratelimit:*	named
view:NAME	response log	answers per view (_default without views)
zone:NAME	response log	answers per local zone; zone:- for names in none of your zones
responses	response log	lines read (a health check)
blocked, blocklist:NAME	response log	queries blocking answered, and by which list (blocklist:rules for your own rules)
rrl:drop, rrl:slip, rrl:would-drop, rrl:would-slip	response log	responses rate limiting limited, or would have in log-only mode (named's own ratelimit:* counters don't move then)

Server (sampled every minute; stored as gauges, g:KEY the sum of the samples and gn:KEY their number, so a chart shows the average):

Gauge	Source	Meaning
g:named:rss	/proc	named's resident memory (what the system sees), bytes
g:named:mem	named	the memory named counts itself as using, bytes
g:named:cpu	/proc	named's CPU use over the minute, in tenths of a percent of one CPU
g:cache:mem, g:cache:rrsets	named	the cache's memory and RRsets, all views
g:host:mem-used, g:host:load	/proc	the host's memory in use, and its 1-minute load average times 100

Jarl finds named's process by name (the one whose command line mentions named's main configuration file); reading its memory and CPU time needs no privileges. With /proc mounted hidepid, it isn't visible and those figures are left out. The current values also include named's sockets, its recursive and TCP clients against their limits (rndc status), the most at once since it started, and the space left on the file systems holding the zone files, Jarl's cache, named.conf, the response log and this database.

Top lists (per hour and per day, the top_n largest): name (most queried names), client, nxdomain (names answered NXDOMAIN), servfail (names answered SERVFAIL), refused (clients that were refused), blocked (names blocking answered), blocked_client (clients by blocked queries). Blocked answers are NXDOMAIN too, but they aren't counted as names that don't exist: not in the nxdomain list, and not in the summary's NXDOMAIN ratio.

Nothing else is stored. In particular there's no per-query record, so you can't look up "who asked for X at 14:02". That is deliberate, both for size and privacy.

Client names. Where the client lists (busiest and refused clients) show an address Jarl has a record for, they show its host name too: the PTR record's target, or else the name of an A/AAAA record with that address (disabled records don't count, and only records you may read). This is looked up in Jarl's own data when the list is shown, not stored and not asked of DNS, so it is instant and adds no queries of its own. The name is what the records say now: with DHCP, an address seen last week may have been another machine. Anonymized clients (anonymize_clients) are prefixes and get no name.

Using it

Web UI: Statistics. Pick a range (1 hour to 90 days); everything on the page follows it. Charts have a hover readout (or focus them and use the arrow keys) and a Table button showing the same data as a table. The heatmap shows the average queries per hour for each weekday and hour over at least the last 7 days. The page refreshes every minute. Server, at the bottom, shows named's memory, CPU, cache and clients, the host and the disks now, and named's memory and CPU over the range.

CLI:

jarl stats status                          # collecting? any problems?
jarl stats summary --since 7d              # totals, ratios, top lists
jarl stats top names --since 24h --limit 50
jarl stats top clients
jarl stats top nxdomain                    # misconfigured hosts, typos, malware
jarl stats top refused                     # who asks for what they may not have
jarl stats timeline --since 6h             # sparkline; --table for every bucket
jarl stats timeline --since 7d --key rcode:NXDOMAIN --key rcode:SERVFAIL
jarl stats heatmap --since 28d
jarl stats breakdown qtype | rcode | view | zone | answer | rtt | dnssec | cache | transport | ratelimit | rrl
jarl stats top ratelimited                 # client networks rate limiting limits (or would)
jarl stats server                          # memory, CPU, cache, clients, host, disks now
jarl stats timeline --since 7d --key g:named:rss --table   # named's memory over a week
jarl stats summary --since 30d -o json     # export (also -o yaml)

API: see api.md.

Prometheus: with metrics.enabled, the counters are also served at /metrics (metrics.md).

Permissions: the stats resource (stats:read). The readonly and operator roles include it, dns-editor doesn't. Statistics show client addresses and what they look up, so grant it with care.

Exploring

The counters and top lists say how much, not who asked for what: "100 NXDOMAIN answers in the external view" can't tell you which clients asked. For that, jarld also keeps every answer for a while, the query log: time, client, view, name, record type, result, and the blocklist (or your rules) that blocked it. It is in the statistics database, from the response log named writes anyway, and answers older than retention.queries (default 7 days) are deleted every hour; 0 keeps none.

With it, every client, name, view, record type, result and blocklist on the Statistics page is a link to the explore page (#/stats/explore), which shows, for what you clicked:

• the answers over time (and how many were blocked);
• the top clients, names, views, record types, results and blocklists among them, each a link that narrows the filter further;
• the latest 100 answers, every cell a link.

So "external, NXDOMAIN" leads to the clients asking, a client to everything it asks, one of its names to who else asks for it. The page also opens from Explore the answers at the top of the Statistics page, and has a filter bar to type filters in: a client, a name (or it and the names below it), a view, a record type, a result, blocked or not; it suggests values from what is shown. Filters combine (this client, in this view, blocked only) and each has a ✕ to take it out. A name can be widened to the names below it (domain=). The Statistics page's view switches apply here too.

anonymize_clients and exclude_clients apply to the query log as to everything else. Its size: each answer takes about 150 to 200 bytes with its indexes, so 50,000 answers a day for 7 days is about 60 MB, 200,000 a day about 250 MB. The Statistics page's server section shows the statistics database's size and how many answers the query log holds, next to the disks (with a bar of how full each is); the dashboard's Storage panel has all of Jarl's databases.

Views

With views, the Statistics page has a toggle per view (and All and None): switch some off to see only the others, for example your internal and VPS views without the internet's noise on the external one, or several linked views together. The choice is remembered in your browser. The CLI takes --view on every jarl stats command (repeat it, or separate views with commas), and the API ?views=internal,vps.

Only what named's response log says can be split by view: answers (shown as answers instead of queries), result codes, record types, zones, blocking, rate limiting and the top lists. named's own counters (answer sources, transport, upstream response times, DNSSEC, the cache and the server) are for the whole server, and the page says so; asking for them per view (jarl stats breakdown answer --view internal) is refused.

Per-view figures are kept from the version that added them (Jarl 0.4.1); the page says since when. They take some more room in the statistics database, roughly in proportion to the number of views.

How data is kept

• Minute counters are kept for retention.minute, hourly counters and top lists for retention.hour, daily ones for retention.day. Older data is deleted every hour.
• Charts use the finest resolution that covers the range: minutes up to 48 hours, hours up to 90 days, days beyond.
• Top lists are exact within an hour (up to 200,000 distinct names or clients per hour, beyond which the rarest are dropped). Daily lists are built from the hourly top lists, so for very long tails they are approximate.
• After a restart, jarld continues the log where it stopped, including a rotation that happened in the meantime. named's counters are read again from scratch, so the minutes jarld was down count no queries.
• Days follow the server's local time zone. Hours are UTC hours, which is the same as local hours except in time zones with a half-hour offset.

Troubleshooting

jarl stats status (and the top of the Statistics page) lists problems:

• named's statistics channel is not reachable: named isn't running, or doesn't have Jarl's configuration with statistics yet. Check journalctl -u jarl for the line about updating named's configuration at startup. If the files were changed by hand (drift), jarld refuses to rewrite them; run jarl resync --force after checking them.
• named answered N queries but Jarl read no response log lines: /var/log/named is missing or not writable by named (sudo systemd-tmpfiles --create jarl.conf), or response logging was turned off. Jarl tries rndc responselog on again by itself every 5 minutes. Check with sudo rndc status | grep logging.
• No top lists, only totals: query_log is false.

Debian 13 and Ubuntu

This works with the bind9 packages of Debian 13 (BIND 9.20.26) and Ubuntu 26.04 (9.20.24). Both are built with JSON statistics support (json-c), and their AppArmor profile allows /var/log/named. Debian also builds in dnstap, but Jarl doesn't need or use it, so the same setup works on both.