jarl/docs/metrics.md

Prometheus metrics

jarld can serve metrics for Prometheus at /metrics:

• health you can alert on: named up, validation problems, files changed outside Jarl, blocklists that fail to download, backup age, DNSSEC zones waiting for you at the parent, and API tokens about to expire,
• queries, once query statistics are on: named's counters (result codes, record types, cache, upstream times, DNSSEC validation, rate limiting) and Jarl's own (responses per view and zone, blocked queries per blocklist), plus named's memory, CPU, cache and sockets.

Metrics are off by default. Everything is set in /etc/jarl/jarl.yaml; nothing is stored in the database.

Turning it on

Prometheus on the same server needs only:

metrics:
  enabled: true

sudo systemctl restart jarl
curl -s http://127.0.0.1:8053/metrics | head

/metrics is served on the addresses in listen (and their TLS, if set) and on the local socket. Query counters also need stats.enabled: true (stats.md); without it you get only the health metrics.

Prometheus on another machine

Give /metrics an address of its own and say who may scrape it:

metrics:
  enabled: true
  listen: ["0.0.0.0:9450"]       # serves only /metrics, over plain HTTP
  allow: ["192.168.1.20"]        # the Prometheus server

# prometheus.yml
scrape_configs:
  - job_name: jarl
    static_configs:
      - targets: ["dns.lan:9450"]

Or use a token instead of (or as well as) the address list. Any address that sends it may scrape:

metrics:
  enabled: true
  listen: ["0.0.0.0:9450"]
  token: "…"                     # e.g. from: openssl rand -hex 24

# prometheus.yml
scrape_configs:
  - job_name: jarl
    authorization:
      credentials: "…"           # or credentials_file: /etc/prometheus/jarl.token
    static_configs:
      - targets: ["dns.lan:9450"]

With a token in jarl.yaml, keep the file readable by root and jarld only (chown root:jarl /etc/jarl/jarl.yaml; chmod 640 /etc/jarl/jarl.yaml); jarld warns at start when everyone can read it. On metrics.listen the token travels unencrypted, so use it on a network you trust, or scrape the main listener over HTTPS instead (scheme: https and its port in Prometheus).

Settings

Key	Default	Meaning
metrics.enabled	false	Serve /metrics.
metrics.listen	[]	Extra ADDRESS:PORTs that serve only /metrics, over plain HTTP.
metrics.allow	["127.0.0.1", "::1"]	Addresses or prefixes that may scrape without a token. Behind a reverse proxy in trusted_proxies, this is the client's address from X-Forwarded-For. [] with a token: the token is required everywhere.
metrics.token	""	A bearer token (at least 16 characters) that lets any address scrape.

A scrape is allowed when it comes from an address in allow, or carries the token, or comes over the local socket from root or jarld's own user. A wrong token is refused (and counts towards the per-address limit on failed logins). Jarl's API tokens don't work for /metrics.

The firewall must let Prometheus reach the port. allow is checked by jarld, not the kernel.

What is exported

Everything is prefixed jarl_. The text format is checked with promtool check metrics.

Health (always)

These are read at most every 15 seconds; a scrape in between gets the same values.

Metric	Meaning
jarl_build_info{version}	Always 1.
jarl_start_time_seconds	When jarld started.
jarl_metrics_state_ok	0 when Jarl's state couldn't be read (see jarld's log).
jarl_adopted	1 when named loads Jarl's configuration.
jarl_named_up	1 when named answers rndc status (missing with bind.no_reload).
jarl_named_recursive_clients, …_soft_limit, …_limit	Recursive clients now, and named's limits (rndc status).
jarl_named_tcp_clients, …_limit	TCP clients now, and the limit.
jarl_open_changesets	Changesets with staged changes not applied yet.
jarl_last_apply_timestamp_seconds	When the last changeset was applied.
jarl_problems{severity}	Validation problems (error, warning, info) as on the dashboard, including named's own warnings.
jarl_drifted_files	Managed files changed or removed outside Jarl.
jarl_objects{kind}	Views, zones, records, ACLs, TSIG keys and options.
jarl_backup_last_success_timestamp_seconds, jarl_backup_last_size_bytes	The newest backup by jarld backup.
jarl_backup_last_failure_timestamp_seconds	A failed backup newer than the last good one.
jarl_backup_stale	1 when the newest backup is older than backup.warn_after.
jarl_blocking_paused	1 while blocking is paused (only with blocklists or rules).
jarl_blocking_paused_until_timestamp_seconds	When a timed pause ends.
jarl_blocklist_enabled{list}	1 for lists that are switched on.
jarl_blocklist_entries{list}	Domains in the last good download.
jarl_blocklist_failing{list}	1 when the last download or check of an enabled list failed.
jarl_blocklist_last_check_timestamp_seconds{list}, jarl_blocklist_last_change_timestamp_seconds{list}	When the list was last checked, and last changed.
jarl_dnssec_phase{zone,view,phase}	Always 1, one per signed zone; phase is one of the DNSSEC phases. ready means: add the DS at the parent. removing_ds: remove it.
jarl_token_expiry_timestamp_seconds{id,user,token}	When each API token with an expiry date expires.
jarl_certificate_expiry_timestamp_seconds{certificate,source}	When each certificate Jarl manages expires (certificates.md).
jarl_certificate_failing{certificate,source}	1 when obtaining (or reading) the certificate failed the last time, or it has expired.
jarl_certificate_renewal_failures{certificate}	Failed attempts to obtain a certificate since its last success.

Queries (with stats.enabled)

The counters start at zero when jarld starts (Prometheus handles that). named's counters move once a minute, when Jarl polls named's statistics channel. Counters from the response log (responses, views, zones, blocking) move as lines are read and need stats.query_log. Clients in stats.exclude_clients are left out of those. Names and clients are not exported, because there would be too many of them. Use the Statistics page for them.

Metric	Meaning
jarl_queries_total	Queries named received.
jarl_responses_by_rcode_total{rcode}	Responses by result code (NOERROR, NXDOMAIN, SERVFAIL, REFUSED, …).
jarl_queries_by_type_total{type}	Queries by record type.
jarl_queries_by_transport_total{transport}	udp and tcp.
jarl_answers_total{source}	authoritative or recursive.
jarl_recursion_queries_total, jarl_dropped_queries_total, jarl_duplicate_queries_total	Queries that caused recursion, that named dropped, and duplicates.
jarl_ratelimit_responses_total{action}	Responses rate limiting dropped or slipped (named's counters; they don't move in log-only mode).
jarl_ratelimit_logged_total{action}	Rate limited responses from named's log: drop, slip, and in log-only mode would-drop, would-slip (exposure.md).
jarl_upstream_responses_by_rtt_total{rtt}	Upstream answers during recursion by round-trip time (<10ms … >1600ms).
jarl_dnssec_validations_total{result}	valid, valid-negative and failed.
jarl_cache_lookups_total{result}	hits and misses.
jarl_logged_responses_total	Responses read from the response log.
jarl_responses_by_view_total{view}	Logged responses per view (_default without views).
jarl_responses_by_zone_total{zone}	Logged responses per local zone; - for names in no local zone.
jarl_blocked_queries_total	Queries blocking answered.
jarl_blocked_by_list_total{list}	Blocked queries per blocklist; rules for your own rules.
jarl_stats_poll_ok, jarl_stats_last_poll_timestamp_seconds	Whether the last poll of named's statistics channel worked, and when it was.
jarl_stats_log_lines_total, jarl_stats_last_log_line_timestamp_seconds	Response log lines read, and the time of the last one.
jarl_stats_warnings	Warnings on the Statistics page (e.g. no log lines while named answers queries).
jarl_named_start_time_seconds, jarl_named_config_time_seconds	When named started, and last loaded its configuration.
jarl_named_resident_memory_bytes, jarl_named_threads, jarl_named_cpu_seconds_total	named's process (/proc).
jarl_named_memory_in_use_bytes	Memory named counts as in use.
jarl_named_cache_rrsets, jarl_named_cache_memory_bytes	named's caches, all views together.
jarl_named_sockets{proto}	Open UDP and TCP sockets.
jarl_named_recursive_clients_highwater, jarl_named_tcp_clients_highwater	The most clients at once since named started.

Host metrics (load, memory, disks) are left to node_exporter.

Example queries

# Queries per second, and the share that failed
rate(jarl_queries_total[5m])
rate(jarl_responses_by_rcode_total{rcode="SERVFAIL"}[5m]) / ignoring(rcode) rate(jarl_queries_total[5m])

# Blocked share of logged responses, per list
rate(jarl_blocked_by_list_total[1h]) / ignoring(list) group_left rate(jarl_logged_responses_total[1h])

# Busiest zones
topk(5, rate(jarl_responses_by_zone_total{zone!="-"}[1h]))

# Cache hit rate
rate(jarl_cache_lookups_total{result="hits"}[5m]) / ignoring(result) sum without(result) (rate(jarl_cache_lookups_total[5m]))

Example alerts

groups:
  - name: jarl
    rules:
      - alert: JarlDown
        expr: up{job="jarl"} == 0
        for: 5m
      - alert: NamedDown
        expr: jarl_named_up == 0
        for: 2m
      - alert: JarlConfigErrors
        expr: jarl_problems{severity="error"} > 0
        for: 15m
      - alert: JarlDrift
        expr: jarl_drifted_files > 0
        for: 15m
      - alert: BlocklistFailing
        expr: jarl_blocklist_failing == 1
        for: 6h
      - alert: BackupStale
        expr: jarl_backup_stale == 1
      - alert: DNSSECWaitingForYou
        expr: jarl_dnssec_phase{phase=~"ready|removing_ds|unsigned"} == 1
        for: 1h
        annotations:
          summary: "{{ $labels.zone }}: DNSSEC is waiting for a step at the parent ({{ $labels.phase }})"
      - alert: TokenExpiresSoon
        expr: jarl_token_expiry_timestamp_seconds - time() < 7 * 86400
        annotations:
          summary: "API token {{ $labels.token }} of {{ $labels.user }} expires within a week"
      - alert: CertificateFailing
        expr: jarl_certificate_failing == 1
        for: 12h
        annotations:
          summary: "Certificate {{ $labels.certificate }} can't be renewed (see the Certificates page)"
      - alert: CertificateExpiresSoon
        expr: jarl_certificate_expiry_timestamp_seconds - time() < 10 * 86400
        annotations:
          summary: "Certificate {{ $labels.certificate }} expires within 10 days"
      - alert: BlockingPausedForLong
        expr: jarl_blocking_paused == 1
        for: 24h
      - alert: NamedRecursiveClientsHigh
        expr: jarl_named_recursive_clients / jarl_named_recursive_clients_soft_limit > 0.8
        for: 5m
      - alert: StatsNotCollecting
        expr: jarl_stats_poll_ok == 0 or jarl_stats_warnings > 0
        for: 15m