abrown38andClaude Opus 5.5
cf30715c5a
internal/stats samples every minute what the kernel already counts: the
interfaces' counters (per zone too), Housecarl's table (hits per rule,
built-in and policy; drops by chain; quotas; per-device sets), the
connection tracking table, and housecarld's and the host's memory, CPU
and load. Logged drops feed top lists of sources and what they tried.
Aggregates go to a database of their own (stats.db; per minute 7 days,
per hour 90 days, per day 2 years), with the store and queries from
Jarl's.
Traffic per inside device is a setting (stats_per_device): counting sets
stats_up/down_v4/v6 and rules at the top of the forward chain, IPv6 per
/64, carried across reloads. Set names starting with meter_, conn_,
quota_ or stats_ are reserved (set_name_reserved).
GET /stats/{status,summary,timeline,top,hits,heatmap}, hc stats, a
Statistics page with Jarl's charts, two charts on the dashboard, and a
conntrack_full warning. e2e: 188 checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
141 lines
6.4 KiB
Markdown
141 lines
6.4 KiB
Markdown
# Statistics
|
||
|
||
Housecarl keeps what the firewall did over time:
|
||
|
||
- traffic per interface and zone, received and sent;
|
||
- packets dropped, by chain, and when drops happen (a weekday × hour
|
||
heatmap);
|
||
- hits per rule, built-in and policy, and the rules that matched nothing;
|
||
- the sources blocked most and what they tried (from the log);
|
||
- traffic per inside device (a setting, below);
|
||
- connections tracked, against the kernel's limit;
|
||
- what each quota counted;
|
||
- housecarld's and the machine's memory, CPU and load.
|
||
|
||
All of it is in the web UI (**Traffic → Statistics**, and two charts on
|
||
the dashboard), the CLI (`hc stats`) and the API (`/api/v1/stats/...`),
|
||
for anyone with `log:read`.
|
||
|
||
Statistics are **on by default**. They cost nothing on the packets'
|
||
path: housecarld reads counters the kernel keeps anyway, once a minute.
|
||
Only traffic per device adds to the ruleset, so it is a setting you turn
|
||
on. The figures are kept as aggregates in a database of their own,
|
||
`/var/lib/housecarl/stats.db`, separate from the configuration. Deleting
|
||
it is always safe (stop housecarl first), and backups leave it out.
|
||
|
||
## Settings
|
||
|
||
In `/etc/housecarl/housecarl.yaml` (restart housecarl after a change):
|
||
|
||
```yaml
|
||
stats:
|
||
enabled: true
|
||
database: /var/lib/housecarl/stats.db
|
||
retention:
|
||
minute: 7d # per-minute figures
|
||
hour: 90d # hourly figures and top lists
|
||
day: 730d # daily figures and top lists
|
||
top_n: 100 # entries kept per top list, per hour and day
|
||
```
|
||
|
||
Old data is removed once an hour. A range picks the finest data that
|
||
covers it: per minute up to 2 days, hourly up to 90 days, then daily.
|
||
|
||
## Traffic per device
|
||
|
||
```sh
|
||
hc settings set --stats-per-device on
|
||
```
|
||
|
||
(or **Settings → Statistics** in the web UI). This is a firewall setting,
|
||
so it goes through the usual review and probation. Housecarl then adds
|
||
four sets to its table, and four rules at the top of the forward chain
|
||
(shown as the built-in `stats`):
|
||
|
||
```
|
||
iifname != "wan" oifname "wan" update @stats_up_v4 { ip saddr }
|
||
iifname != "wan" oifname "wan" update @stats_up_v6 { ip6 saddr and ffff:ffff:ffff:ffff:: }
|
||
iifname "wan" oifname != "wan" update @stats_down_v4 { ip daddr }
|
||
iifname "wan" oifname != "wan" update @stats_down_v6 { ip6 daddr and ffff:ffff:ffff:ffff:: }
|
||
```
|
||
|
||
Each inside address talking to an external zone gets a counter: what it
|
||
sent out, and what came back to it. IPv6 addresses are counted per /64,
|
||
since one device uses many addresses. Counting is done in the kernel, a
|
||
set lookup per forwarded packet. An address quiet for an hour drops out
|
||
of the sets, and a reload keeps the counts.
|
||
|
||
With the flowtable on, established connections skip the forward chain,
|
||
so only their first packets are counted (warning `stats_flowtable`). Set
|
||
names starting with `stats_` (and `meter_`, `conn_`, `quota_`) are
|
||
Housecarl's own and can't be used for your sets.
|
||
|
||
## What is collected
|
||
|
||
Every minute housecarld samples its sources and stores what changed:
|
||
|
||
| Figure | Source |
|
||
|---|---|
|
||
| `rx:IF`, `tx:IF` (bytes), `rxp:IF`, `txp:IF` (packets) | the interfaces' counters (`/sys/class/net`); the loopback is left out |
|
||
| `zrx:ZONE`, `ztx:ZONE` | the same, summed per zone: received from and sent to the zone's interfaces |
|
||
| `hits:MARK`, `hitb:MARK` | each rule's (`r12`), built-in's (`b-established`), port forward's (`f3`) and policy's (`p-input`) counter, in packets and bytes |
|
||
| `drop:CHAIN`, `drops` | the packets counted by what drops them: rules that drop or reject, the drop policies, antispoofing (`early`) and invalid packets (`invalid`) |
|
||
| `quota:r12` | bytes counted against a rule's quota |
|
||
| `g:conntrack`, `g:conntrack_max` | connections tracked, and the limit (`nf_conntrack_max`) |
|
||
| `g:rss`, `g:cpu` | housecarld's memory (bytes) and CPU (tenths of a percent of one CPU) |
|
||
| `g:mem`, `g:load` | the host's memory in use, and its 1-minute load average × 100 |
|
||
|
||
`g:` figures are gauges: the API averages them over a bucket.
|
||
|
||
Top lists, per hour and per day, the `top_n` largest:
|
||
|
||
| List | |
|
||
|---|---|
|
||
| `blocked_src` | sources of logged drops |
|
||
| `blocked_port` | what they tried: `tcp/22`, `udp/53`, `icmp/echo-request` |
|
||
| `device_up`, `device_down` | bytes per inside address, sent and received (traffic per device) |
|
||
| `quota_r12` | a per-address quota's use, per address |
|
||
|
||
The blocked lists only see what is **logged**: the input and forward
|
||
policies' drops with `hc settings set --log-drops on`, and rules with
|
||
logging. The drop counts above see everything.
|
||
|
||
A counter that goes down (an interface that came back, a quota reset)
|
||
counts again from zero. What happens while housecarld is stopped isn't
|
||
counted.
|
||
|
||
## The CLI
|
||
|
||
```sh
|
||
hc stats --since 7d # summary: traffic per zone, drops, busiest rules, top lists
|
||
hc stats status # collecting? since when, the database, figures now
|
||
hc stats traffic --since 24h # per zone over time (--interfaces: per interface)
|
||
hc stats top devices --since 30d # devices by traffic (blocked, tried, quota RULE-ID)
|
||
hc stats rules --unused # rules that matched nothing in 30 days
|
||
hc stats connections --since 24h
|
||
```
|
||
|
||
Ranges are written `90m`, `24h`, `7d` or `4w`. `-o json` gives the API's
|
||
answer.
|
||
|
||
## The API
|
||
|
||
| Endpoint | |
|
||
|---|---|
|
||
| `GET /stats/status` | `enabled`, `database`, `size`, `first_data`, `last_sample`, `sample_error`, `per_device`, `retention`, `latest` (the newest gauges) |
|
||
| `GET /stats/summary?since=24h&limit=10` | `zones` and `interfaces` (`name`, `received`, `sent`), `drops`, `drops_by_chain`, `rules` (busiest, named), `peak_connections`, `conntrack_max`, `blocked_sources`, `blocked_ports`, `devices_up`, `devices_down` |
|
||
| `GET /stats/timeline?keys=zrx:*,drops&since=24h&step=auto` | `step`, `times` (bucket starts), `series` (`key`, `values`); `GROUP:*` means every key of a group; `step` is `auto`, `1m`, `5m`, `15m`, `1h`, `6h` or `1d`; a gauge without a sample is -1 |
|
||
| `GET /stats/top?kind=device_down&since=7d&limit=20` | a top list (`key`, `count`) |
|
||
| `GET /stats/hits?since=30d` | packets per rule, built-in and policy (`key`, `count`, `name`), with the enabled rules that counted none |
|
||
| `GET /stats/heatmap?key=drops&since=28d` | `avg[weekday][hour]` (Monday first, local time; -1 where there is no data) |
|
||
|
||
With statistics off, `/stats/status` says `enabled: false` and the others
|
||
answer 503.
|
||
|
||
## Warnings
|
||
|
||
When the connection tracking table is over 80% full, the status shows
|
||
the warning `conntrack_full`: once it is full, the kernel drops new
|
||
connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens
|
||
so many on the Connections page.
|