jarl/docs/metrics.md

# Prometheus metrics

jarld can serve metrics for [Prometheus](https://prometheus.io/) at
`/metrics`:
- **health** you can alert on: named up, validation problems, files changed
  outside Jarl, blocklists that fail to download, backup age, DNSSEC zones
  waiting for you at the parent, and API tokens about to expire,
- **queries**, once [query statistics](stats.md) are on: named's counters
  (result codes, record types, cache, upstream times, DNSSEC validation, rate
  limiting) and Jarl's own (responses per view and zone, blocked queries per
  blocklist), plus named's memory, CPU, cache and sockets.

Metrics are **off by default**. Everything is set in `/etc/jarl/jarl.yaml`;
nothing is stored in the database.

## Turning it on

Prometheus on the same server needs only:

```yaml
metrics:
  enabled: true
```

```sh
sudo systemctl restart jarl
curl -s http://127.0.0.1:8053/metrics | head
```

`/metrics` is served on the addresses in `listen` (and their TLS, if set)
and on the local socket. Query counters also need `stats.enabled: true`
([stats.md](stats.md)); without it you get only the health metrics.

### Prometheus on another machine

Give `/metrics` an address of its own and say who may scrape it:

```yaml
metrics:
  enabled: true
  listen: ["0.0.0.0:9450"]       # serves only /metrics, over plain HTTP
  allow: ["192.168.1.20"]        # the Prometheus server
```

```yaml
# prometheus.yml
scrape_configs:
  - job_name: jarl
    static_configs:
      - targets: ["dns.lan:9450"]
```

Or use a token instead of (or as well as) the address list. Any address
that sends it may scrape:

```yaml
metrics:
  enabled: true
  listen: ["0.0.0.0:9450"]
  token: "…"                     # e.g. from: openssl rand -hex 24
```

```yaml
# prometheus.yml
scrape_configs:
  - job_name: jarl
    authorization:
      credentials: "…"           # or credentials_file: /etc/prometheus/jarl.token
    static_configs:
      - targets: ["dns.lan:9450"]
```

With a token in jarl.yaml, keep the file readable by root and jarld only
(`chown root:jarl /etc/jarl/jarl.yaml; chmod 640 /etc/jarl/jarl.yaml`);
jarld warns at start when everyone can read it. On `metrics.listen` the
token travels unencrypted, so use it on a network you trust, or scrape the
main listener over HTTPS instead (`scheme: https` and its port in
Prometheus).

### Settings

| Key | Default | Meaning |
|-----|---------|---------|
| `metrics.enabled` | `false` | Serve `/metrics`. |
| `metrics.listen` | `[]` | Extra `ADDRESS:PORT`s that serve only `/metrics`, over plain HTTP. |
| `metrics.allow` | `["127.0.0.1", "::1"]` | Addresses or prefixes that may scrape without a token. Behind a reverse proxy in `trusted_proxies`, this is the client's address from `X-Forwarded-For`. `[]` with a token: the token is required everywhere. |
| `metrics.token` | `""` | A bearer token (at least 16 characters) that lets any address scrape. |

A scrape is allowed when it comes from an address in `allow`, **or**
carries the token, **or** comes over the local socket from root or jarld's
own user. A wrong token is refused (and counts towards the per-address
limit on failed logins). Jarl's API tokens don't work for `/metrics`.

The firewall must let Prometheus reach the port. `allow` is checked by
jarld, not the kernel.

## What is exported

Everything is prefixed `jarl_`. The text format is checked with `promtool
check metrics`.

### Health (always)

These are read at most every 15 seconds; a scrape in between gets the
same values.

| Metric | Meaning |
|--------|---------|
| `jarl_build_info{version}` | Always 1. |
| `jarl_start_time_seconds` | When jarld started. |
| `jarl_metrics_state_ok` | 0 when Jarl's state couldn't be read (see jarld's log). |
| `jarl_adopted` | 1 when named loads Jarl's configuration. |
| `jarl_named_up` | 1 when named answers `rndc status` (missing with `bind.no_reload`). |
| `jarl_named_recursive_clients`, `…_soft_limit`, `…_limit` | Recursive clients now, and named's limits (`rndc status`). |
| `jarl_named_tcp_clients`, `…_limit` | TCP clients now, and the limit. |
| `jarl_open_changesets` | Changesets with staged changes not applied yet. |
| `jarl_last_apply_timestamp_seconds` | When the last changeset was applied. |
| `jarl_problems{severity}` | Validation problems (`error`, `warning`, `info`) as on the dashboard, including named's own warnings. |
| `jarl_drifted_files` | Managed files changed or removed outside Jarl. |
| `jarl_objects{kind}` | Views, zones, records, ACLs, TSIG keys and options. |
| `jarl_backup_last_success_timestamp_seconds`, `jarl_backup_last_size_bytes` | The newest backup by `jarld backup`. |
| `jarl_backup_last_failure_timestamp_seconds` | A failed backup newer than the last good one. |
| `jarl_backup_stale` | 1 when the newest backup is older than `backup.warn_after`. |
| `jarl_blocking_paused` | 1 while blocking is paused (only with blocklists or rules). |
| `jarl_blocking_paused_until_timestamp_seconds` | When a timed pause ends. |
| `jarl_blocklist_enabled{list}` | 1 for lists that are switched on. |
| `jarl_blocklist_entries{list}` | Domains in the last good download. |
| `jarl_blocklist_failing{list}` | 1 when the last download or check of an enabled list failed. |
| `jarl_blocklist_last_check_timestamp_seconds{list}`, `jarl_blocklist_last_change_timestamp_seconds{list}` | When the list was last checked, and last changed. |
| `jarl_dnssec_phase{zone,view,phase}` | Always 1, one per signed zone; `phase` is one of the [DNSSEC phases](dnssec.md). `ready` means: add the DS at the parent. `removing_ds`: remove it. |
| `jarl_token_expiry_timestamp_seconds{id,user,token}` | When each API token with an expiry date expires. |
| `jarl_certificate_expiry_timestamp_seconds{certificate,source}` | When each certificate Jarl manages expires ([certificates.md](certificates.md)). |
| `jarl_certificate_failing{certificate,source}` | 1 when obtaining (or reading) the certificate failed the last time, or it has expired. |
| `jarl_certificate_renewal_failures{certificate}` | Failed attempts to obtain a certificate since its last success. |
| `jarl_encrypted_dns_up{kind,address,port}` | 1 when the DoT or DoH listener answered with the right, current certificate at the last check ([encrypted-dns.md](encrypted-dns.md#health)). |

### Queries (with `stats.enabled`)

The counters start at zero when jarld starts (Prometheus handles that).
named's counters move once a minute, when Jarl polls named's statistics
channel. Counters from the response log (`responses`, views, zones,
blocking) move as lines are read and need `stats.query_log`. Clients in
`stats.exclude_clients` are left out of those. Names and clients are not
exported, because there would be too many of them. Use the Statistics page
for them.

| Metric | Meaning |
|--------|---------|
| `jarl_queries_total` | Queries named received. |
| `jarl_responses_by_rcode_total{rcode}` | Responses by result code (`NOERROR`, `NXDOMAIN`, `SERVFAIL`, `REFUSED`, …). |
| `jarl_queries_by_type_total{type}` | Queries by record type. |
| `jarl_queries_by_transport_total{transport}` | `udp` and `tcp`. |
| `jarl_answers_total{source}` | `authoritative` or `recursive`. |
| `jarl_recursion_queries_total`, `jarl_dropped_queries_total`, `jarl_duplicate_queries_total` | Queries that caused recursion, that named dropped, and duplicates. |
| `jarl_ratelimit_responses_total{action}` | Responses rate limiting `dropped` or `slipped` (named's counters; they don't move in log-only mode). |
| `jarl_ratelimit_logged_total{action}` | Rate limited responses from named's log: `drop`, `slip`, and in log-only mode `would-drop`, `would-slip` ([exposure.md](exposure.md#response-rate-limiting)). |
| `jarl_upstream_responses_by_rtt_total{rtt}` | Upstream answers during recursion by round-trip time (`<10ms` … `>1600ms`). |
| `jarl_dnssec_validations_total{result}` | `valid`, `valid-negative` and `failed`. |
| `jarl_cache_lookups_total{result}` | `hits` and `misses`. |
| `jarl_logged_responses_total` | Responses read from the response log. |
| `jarl_responses_by_view_total{view}` | Logged responses per view (`_default` without views). |
| `jarl_responses_by_zone_total{zone}` | Logged responses per local zone; `-` for names in no local zone. |
| `jarl_blocked_queries_total` | Queries blocking answered. |
| `jarl_blocked_by_list_total{list}` | Blocked queries per blocklist; `rules` for your own rules. |
| `jarl_stats_poll_ok`, `jarl_stats_last_poll_timestamp_seconds` | Whether the last poll of named's statistics channel worked, and when it was. |
| `jarl_stats_log_lines_total`, `jarl_stats_last_log_line_timestamp_seconds` | Response log lines read, and the time of the last one. |
| `jarl_stats_warnings` | Warnings on the Statistics page (e.g. no log lines while named answers queries). |
| `jarl_named_start_time_seconds`, `jarl_named_config_time_seconds` | When named started, and last loaded its configuration. |
| `jarl_named_resident_memory_bytes`, `jarl_named_threads`, `jarl_named_cpu_seconds_total` | named's process (`/proc`). |
| `jarl_named_memory_in_use_bytes` | Memory named counts as in use. |
| `jarl_named_cache_rrsets`, `jarl_named_cache_memory_bytes` | named's caches, all views together. |
| `jarl_named_sockets{proto}` | Open UDP and TCP sockets. |
| `jarl_named_recursive_clients_highwater`, `jarl_named_tcp_clients_highwater` | The most clients at once since named started. |

Host metrics (load, memory, disks) are left to
[node_exporter](https://github.com/prometheus/node_exporter).

## Example queries

```promql
# Queries per second, and the share that failed
rate(jarl_queries_total[5m])
rate(jarl_responses_by_rcode_total{rcode="SERVFAIL"}[5m]) / ignoring(rcode) rate(jarl_queries_total[5m])

# Blocked share of logged responses, per list
rate(jarl_blocked_by_list_total[1h]) / ignoring(list) group_left rate(jarl_logged_responses_total[1h])

# Busiest zones
topk(5, rate(jarl_responses_by_zone_total{zone!="-"}[1h]))

# Cache hit rate
rate(jarl_cache_lookups_total{result="hits"}[5m]) / ignoring(result) sum without(result) (rate(jarl_cache_lookups_total[5m]))
```

## Example alerts

```yaml
groups:
  - name: jarl
    rules:
      - alert: JarlDown
        expr: up{job="jarl"} == 0
        for: 5m
      - alert: NamedDown
        expr: jarl_named_up == 0
        for: 2m
      - alert: JarlConfigErrors
        expr: jarl_problems{severity="error"} > 0
        for: 15m
      - alert: JarlDrift
        expr: jarl_drifted_files > 0
        for: 15m
      - alert: BlocklistFailing
        expr: jarl_blocklist_failing == 1
        for: 6h
      - alert: BackupStale
        expr: jarl_backup_stale == 1
      - alert: DNSSECWaitingForYou
        expr: jarl_dnssec_phase{phase=~"ready|removing_ds|unsigned"} == 1
        for: 1h
        annotations:
          summary: "{{ $labels.zone }}: DNSSEC is waiting for a step at the parent ({{ $labels.phase }})"
      - alert: TokenExpiresSoon
        expr: jarl_token_expiry_timestamp_seconds - time() < 7 * 86400
        annotations:
          summary: "API token {{ $labels.token }} of {{ $labels.user }} expires within a week"
      - alert: CertificateFailing
        expr: jarl_certificate_failing == 1
        for: 12h
        annotations:
          summary: "Certificate {{ $labels.certificate }} can't be renewed (see the Certificates page)"
      - alert: CertificateExpiresSoon
        expr: jarl_certificate_expiry_timestamp_seconds - time() < 10 * 86400
        annotations:
          summary: "Certificate {{ $labels.certificate }} expires within 10 days"
      - alert: EncryptedDNSDown
        expr: jarl_encrypted_dns_up == 0
        for: 20m
        annotations:
          summary: "DNS over {{ $labels.kind }} on {{ $labels.address }} port {{ $labels.port }} isn't working (see the Encrypted DNS page)"
      - alert: BlockingPausedForLong
        expr: jarl_blocking_paused == 1
        for: 24h
      - alert: NamedRecursiveClientsHigh
        expr: jarl_named_recursive_clients / jarl_named_recursive_clients_soft_limit > 0.8
        for: 5m
      - alert: StatsNotCollecting
        expr: jarl_stats_poll_ok == 0 or jarl_stats_warnings > 0
        for: 15m
```