abrown38andClaude Opus 5.5
ca5ac971e5
The established built-in passes most of the traffic, so it topped the statistics' busiest rules every time and made every rule's bar look empty. The summary now lists rules and NAT objects in rules, and built-ins and policies in builtins; the Statistics page and hc stats show them as two lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
147 lines
6.8 KiB
Markdown
147 lines
6.8 KiB
Markdown
# Statistics
|
||
|
||
Housecarl keeps what the firewall did over time:
|
||
|
||
- traffic per interface and zone, received and sent;
|
||
- packets dropped, by chain, and when drops happen (a weekday × hour
|
||
heatmap);
|
||
- hits per rule, and per built-in and policy in a list of their own (the
|
||
established built-in counts most packets), and the rules that matched
|
||
nothing;
|
||
- the sources blocked most and what they tried (from the log);
|
||
- traffic per inside device (a setting, below);
|
||
- connections tracked, against the kernel's limit;
|
||
- what each quota counted;
|
||
- housecarld's and the machine's memory, CPU and load.
|
||
|
||
All of it is in the web UI (**Traffic → Statistics**, and two charts on
|
||
the dashboard), the CLI (`hc stats`) and the API (`/api/v1/stats/...`),
|
||
for anyone with `log:read`.
|
||
|
||
Statistics are **on by default**. They cost nothing on the packets'
|
||
path: housecarld reads counters the kernel keeps anyway, once a minute.
|
||
Only traffic per device adds to the ruleset, so it is a setting you turn
|
||
on. The figures are kept as aggregates in a database of their own,
|
||
`/var/lib/housecarl/stats.db`, separate from the configuration. Deleting
|
||
it is always safe (stop housecarl first), and backups leave it out.
|
||
|
||
## Settings
|
||
|
||
In `/etc/housecarl/housecarl.yaml` (restart housecarl after a change):
|
||
|
||
```yaml
|
||
stats:
|
||
enabled: true
|
||
database: /var/lib/housecarl/stats.db
|
||
retention:
|
||
minute: 7d # per-minute figures
|
||
hour: 90d # hourly figures and top lists
|
||
day: 730d # daily figures and top lists
|
||
top_n: 100 # entries kept per top list, per hour and day
|
||
```
|
||
|
||
Old data is removed once an hour. A range picks the finest data that
|
||
covers it: per minute up to 2 days, hourly up to 90 days, then daily.
|
||
|
||
## Traffic per device
|
||
|
||
```sh
|
||
hc settings set --stats-per-device on
|
||
```
|
||
|
||
(or **Settings → Statistics** in the web UI). This is a firewall setting,
|
||
so it goes through the usual review and probation. Housecarl then adds
|
||
four sets to its table, and four rules at the top of the forward chain
|
||
(shown as the built-in `stats`):
|
||
|
||
```
|
||
iifname != "wan" oifname "wan" update @stats_up_v4 { ip saddr }
|
||
iifname != "wan" oifname "wan" update @stats_up_v6 { ip6 saddr and ffff:ffff:ffff:ffff:: }
|
||
iifname "wan" oifname != "wan" update @stats_down_v4 { ip daddr }
|
||
iifname "wan" oifname != "wan" update @stats_down_v6 { ip6 daddr and ffff:ffff:ffff:ffff:: }
|
||
```
|
||
|
||
Each inside address talking to an external zone gets a counter: what it
|
||
sent out, and what came back to it. IPv6 addresses are counted per /64,
|
||
since one device uses many addresses. Counting is done in the kernel, a
|
||
set lookup per forwarded packet. An address quiet for an hour drops out
|
||
of the sets, and a reload keeps the counts.
|
||
|
||
With the flowtable on, established connections skip the forward chain,
|
||
so only their first packets are counted (warning `stats_flowtable`). Set
|
||
names starting with `stats_` (and `meter_`, `conn_`, `quota_`) are
|
||
Housecarl's own and can't be used for your sets.
|
||
|
||
## What is collected
|
||
|
||
Every minute housecarld samples its sources and stores what changed:
|
||
|
||
| Figure | Source |
|
||
|---|---|
|
||
| `rx:IF`, `tx:IF` (bytes), `rxp:IF`, `txp:IF` (packets) | the interfaces' counters (`/sys/class/net`); the loopback is left out |
|
||
| `zrx:ZONE`, `ztx:ZONE` | the same, summed per zone: received from and sent to the zone's interfaces |
|
||
| `hits:MARK`, `hitb:MARK` | each rule's (`r12`), built-in's (`b-established`), port forward's (`f3`) and policy's (`p-input`) counter, in packets and bytes |
|
||
| `drop:CHAIN`, `drops` | the packets counted by what drops them: rules that drop or reject, the drop policies, antispoofing (`early`) and invalid packets (`invalid`) |
|
||
| `quota:r12` | bytes counted against a rule's quota |
|
||
| `g:conntrack`, `g:conntrack_max` | connections tracked, and the limit (`nf_conntrack_max`) |
|
||
| `g:rss`, `g:cpu` | housecarld's memory (bytes) and CPU (tenths of a percent of one CPU) |
|
||
| `g:mem`, `g:load` | the host's memory in use, and its 1-minute load average × 100 |
|
||
|
||
`g:` figures are gauges: the API averages them over a bucket.
|
||
|
||
Top lists, per hour and per day, the `top_n` largest:
|
||
|
||
| List | |
|
||
|---|---|
|
||
| `blocked_src` | sources of logged drops |
|
||
| `blocked_port` | what they tried: `tcp/22`, `udp/53`, `icmp/echo-request` |
|
||
| `device_up`, `device_down` | bytes per inside address, sent and received (traffic per device) |
|
||
| `quota_r12` | a per-address quota's use, per address |
|
||
|
||
The blocked lists only see what is **logged**: the input and forward
|
||
policies' drops with `hc settings set --log-drops on`, and rules with
|
||
logging. The drop counts above see everything.
|
||
|
||
In shadow mode ([shadow.md](shadow.md)) the drops, and the blocked
|
||
lists, are what Housecarl would have dropped: the packets went on. What
|
||
the policies would drop is always logged there.
|
||
|
||
A counter that goes down (an interface that came back, a quota reset)
|
||
counts again from zero. What happens while housecarld is stopped isn't
|
||
counted.
|
||
|
||
## The CLI
|
||
|
||
```sh
|
||
hc stats --since 7d # summary: traffic per zone, drops, busiest rules, top lists
|
||
hc stats status # collecting? since when, the database, figures now
|
||
hc stats traffic --since 24h # per zone over time (--interfaces: per interface)
|
||
hc stats top devices --since 30d # devices by traffic (blocked, tried, quota RULE-ID)
|
||
hc stats rules --unused # rules that matched nothing in 30 days
|
||
hc stats connections --since 24h
|
||
```
|
||
|
||
Ranges are written `90m`, `24h`, `7d` or `4w`. `-o json` gives the API's
|
||
answer.
|
||
|
||
## The API
|
||
|
||
| Endpoint | |
|
||
|---|---|
|
||
| `GET /stats/status` | `enabled`, `database`, `size`, `first_data`, `last_sample`, `sample_error`, `per_device`, `retention`, `latest` (the newest gauges) |
|
||
| `GET /stats/summary?since=24h&limit=10` | `zones` and `interfaces` (`name`, `received`, `sent`), `drops`, `drops_by_chain`, `rules` (busiest rules and NAT objects, named), `builtins` (busiest built-ins and policies, named), `peak_connections`, `conntrack_max`, `blocked_sources`, `blocked_ports`, `devices_up`, `devices_down` |
|
||
| `GET /stats/timeline?keys=zrx:*,drops&since=24h&step=auto` | `step`, `times` (bucket starts), `series` (`key`, `values`); `GROUP:*` means every key of a group; `step` is `auto`, `1m`, `5m`, `15m`, `1h`, `6h` or `1d`; a gauge without a sample is -1 |
|
||
| `GET /stats/top?kind=device_down&since=7d&limit=20` | a top list (`key`, `count`) |
|
||
| `GET /stats/hits?since=30d` | packets per rule, built-in and policy (`key`, `count`, `name`), with the enabled rules that counted none |
|
||
| `GET /stats/heatmap?key=drops&since=28d` | `avg[weekday][hour]` (Monday first, local time; -1 where there is no data) |
|
||
|
||
With statistics off, `/stats/status` says `enabled: false` and the others
|
||
answer 503.
|
||
|
||
## Warnings
|
||
|
||
When the connection tracking table is over 80% full, the status shows
|
||
the warning `conntrack_full`: once it is full, the kernel drops new
|
||
connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens
|
||
so many on the Connections page.
|