Statistics: traffic, drops, devices and connections over time

internal/stats samples every minute what the kernel already counts: the
interfaces' counters (per zone too), Housecarl's table (hits per rule,
built-in and policy; drops by chain; quotas; per-device sets), the
connection tracking table, and housecarld's and the host's memory, CPU
and load. Logged drops feed top lists of sources and what they tried.
Aggregates go to a database of their own (stats.db; per minute 7 days,
per hour 90 days, per day 2 years), with the store and queries from
Jarl's.

Traffic per inside device is a setting (stats_per_device): counting sets
stats_up/down_v4/v6 and rules at the top of the forward chain, IPv6 per
/64, carried across reloads. Set names starting with meter_, conn_,
quota_ or stats_ are reserved (set_name_reserved).

GET /stats/{status,summary,timeline,top,hits,heatmap}, hc stats, a
Statistics page with Jarl's charts, two charts on the dashboard, and a
conntrack_full warning. e2e: 188 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/stats.md

@@ -0,0 +1,140 @@
+# Statistics
+
+Housecarl keeps what the firewall did over time:
+
+- traffic per interface and zone, received and sent;
+- packets dropped, by chain, and when drops happen (a weekday × hour
+  heatmap);
+- hits per rule, built-in and policy, and the rules that matched nothing;
+- the sources blocked most and what they tried (from the log);
+- traffic per inside device (a setting, below);
+- connections tracked, against the kernel's limit;
+- what each quota counted;
+- housecarld's and the machine's memory, CPU and load.
+
+All of it is in the web UI (**Traffic → Statistics**, and two charts on
+the dashboard), the CLI (`hc stats`) and the API (`/api/v1/stats/...`),
+for anyone with `log:read`.
+
+Statistics are **on by default**. They cost nothing on the packets'
+path: housecarld reads counters the kernel keeps anyway, once a minute.
+Only traffic per device adds to the ruleset, so it is a setting you turn
+on. The figures are kept as aggregates in a database of their own,
+`/var/lib/housecarl/stats.db`, separate from the configuration. Deleting
+it is always safe (stop housecarl first), and backups leave it out.
+
+## Settings
+
+In `/etc/housecarl/housecarl.yaml` (restart housecarl after a change):
+
+```yaml
+stats:
+  enabled: true
+  database: /var/lib/housecarl/stats.db
+  retention:
+    minute: 7d     # per-minute figures
+    hour: 90d      # hourly figures and top lists
+    day: 730d      # daily figures and top lists
+  top_n: 100       # entries kept per top list, per hour and day
+```
+
+Old data is removed once an hour. A range picks the finest data that
+covers it: per minute up to 2 days, hourly up to 90 days, then daily.
+
+## Traffic per device
+
+```sh
+hc settings set --stats-per-device on
+```
+
+(or **Settings → Statistics** in the web UI). This is a firewall setting,
+so it goes through the usual review and probation. Housecarl then adds
+four sets to its table, and four rules at the top of the forward chain
+(shown as the built-in `stats`):
+
+```
+iifname != "wan" oifname "wan" update @stats_up_v4 { ip saddr }
+iifname != "wan" oifname "wan" update @stats_up_v6 { ip6 saddr and ffff:ffff:ffff:ffff:: }
+iifname "wan" oifname != "wan" update @stats_down_v4 { ip daddr }
+iifname "wan" oifname != "wan" update @stats_down_v6 { ip6 daddr and ffff:ffff:ffff:ffff:: }
+```
+
+Each inside address talking to an external zone gets a counter: what it
+sent out, and what came back to it. IPv6 addresses are counted per /64,
+since one device uses many addresses. Counting is done in the kernel, a
+set lookup per forwarded packet. An address quiet for an hour drops out
+of the sets, and a reload keeps the counts.
+
+With the flowtable on, established connections skip the forward chain,
+so only their first packets are counted (warning `stats_flowtable`). Set
+names starting with `stats_` (and `meter_`, `conn_`, `quota_`) are
+Housecarl's own and can't be used for your sets.
+
+## What is collected
+
+Every minute housecarld samples its sources and stores what changed:
+
+| Figure | Source |
+|---|---|
+| `rx:IF`, `tx:IF` (bytes), `rxp:IF`, `txp:IF` (packets) | the interfaces' counters (`/sys/class/net`); the loopback is left out |
+| `zrx:ZONE`, `ztx:ZONE` | the same, summed per zone: received from and sent to the zone's interfaces |
+| `hits:MARK`, `hitb:MARK` | each rule's (`r12`), built-in's (`b-established`), port forward's (`f3`) and policy's (`p-input`) counter, in packets and bytes |
+| `drop:CHAIN`, `drops` | the packets counted by what drops them: rules that drop or reject, the drop policies, antispoofing (`early`) and invalid packets (`invalid`) |
+| `quota:r12` | bytes counted against a rule's quota |
+| `g:conntrack`, `g:conntrack_max` | connections tracked, and the limit (`nf_conntrack_max`) |
+| `g:rss`, `g:cpu` | housecarld's memory (bytes) and CPU (tenths of a percent of one CPU) |
+| `g:mem`, `g:load` | the host's memory in use, and its 1-minute load average × 100 |
+
+`g:` figures are gauges: the API averages them over a bucket.
+
+Top lists, per hour and per day, the `top_n` largest:
+
+| List | |
+|---|---|
+| `blocked_src` | sources of logged drops |
+| `blocked_port` | what they tried: `tcp/22`, `udp/53`, `icmp/echo-request` |
+| `device_up`, `device_down` | bytes per inside address, sent and received (traffic per device) |
+| `quota_r12` | a per-address quota's use, per address |
+
+The blocked lists only see what is **logged**: the input and forward
+policies' drops with `hc settings set --log-drops on`, and rules with
+logging. The drop counts above see everything.
+
+A counter that goes down (an interface that came back, a quota reset)
+counts again from zero. What happens while housecarld is stopped isn't
+counted.
+
+## The CLI
+
+```sh
+hc stats --since 7d              # summary: traffic per zone, drops, busiest rules, top lists
+hc stats status                  # collecting? since when, the database, figures now
+hc stats traffic --since 24h     # per zone over time (--interfaces: per interface)
+hc stats top devices --since 30d # devices by traffic (blocked, tried, quota RULE-ID)
+hc stats rules --unused          # rules that matched nothing in 30 days
+hc stats connections --since 24h
+```
+
+Ranges are written `90m`, `24h`, `7d` or `4w`. `-o json` gives the API's
+answer.
+
+## The API
+
+| Endpoint | |
+|---|---|
+| `GET /stats/status` | `enabled`, `database`, `size`, `first_data`, `last_sample`, `sample_error`, `per_device`, `retention`, `latest` (the newest gauges) |
+| `GET /stats/summary?since=24h&limit=10` | `zones` and `interfaces` (`name`, `received`, `sent`), `drops`, `drops_by_chain`, `rules` (busiest, named), `peak_connections`, `conntrack_max`, `blocked_sources`, `blocked_ports`, `devices_up`, `devices_down` |
+| `GET /stats/timeline?keys=zrx:*,drops&since=24h&step=auto` | `step`, `times` (bucket starts), `series` (`key`, `values`); `GROUP:*` means every key of a group; `step` is `auto`, `1m`, `5m`, `15m`, `1h`, `6h` or `1d`; a gauge without a sample is -1 |
+| `GET /stats/top?kind=device_down&since=7d&limit=20` | a top list (`key`, `count`) |
+| `GET /stats/hits?since=30d` | packets per rule, built-in and policy (`key`, `count`, `name`), with the enabled rules that counted none |
+| `GET /stats/heatmap?key=drops&since=28d` | `avg[weekday][hour]` (Monday first, local time; -1 where there is no data) |
+
+With statistics off, `/stats/status` says `enabled: false` and the others
+answer 503.
+
+## Warnings
+
+When the connection tracking table is over 80% full, the status shows
+the warning `conntrack_full`: once it is full, the kernel drops new
+connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens
+so many on the Connections page.