adam/housecarl
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 34 Wiki Activity

Statistics: traffic, drops, devices and connections over time

Browse Source 
internal/stats samples every minute what the kernel already counts: the
interfaces' counters (per zone too), Housecarl's table (hits per rule,
built-in and policy; drops by chain; quotas; per-device sets), the
connection tracking table, and housecarld's and the host's memory, CPU
and load. Logged drops feed top lists of sources and what they tried.
Aggregates go to a database of their own (stats.db; per minute 7 days,
per hour 90 days, per day 2 years), with the store and queries from
Jarl's.

Traffic per inside device is a setting (stats_per_device): counting sets
stats_up/down_v4/v6 and rules at the top of the forward chain, IPv6 per
/64, carried across reloads. Set names starting with meter_, conn_,
quota_ or stats_ are reserved (set_name_reserved).

GET /stats/{status,summary,timeline,top,hits,heatmap}, hc stats, a
Statistics page with Jarl's charts, two charts on the dashboard, and a
conntrack_full warning. e2e: 188 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
abrown38andClaude Opus 5.5 committed 2026-10-19 22:27:48 +01:00
1 parent 6bb82e86f5
commit cf30715c5a
37 files changed
+3652 -18
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/api.md
+8 -1
View File
@@ -99,7 +99,8 @@ Creating answers 201, other changes 200.
"builtins": {"loopback": true, "established": true, "drop_invalid": true, "icmpv6": true,
"ping_internal": true, "dhcpv6_client": true, "antispoof": true, "mss_clamp": false},
"own_prefixes": ["2001:db8::/48"], "antispoof_v4": ["10.0.0.0/8", "..."], "antispoof_v6": ["fc00::/7", "..."],
"flowtable_devices": ["wan", "eth1"], "probation_seconds": 120, "manual_confirm": false}
"flowtable_devices": ["wan", "eth1"], "probation_seconds": 120, "manual_confirm": false,
"stats_per_device": false}
```
Lists replace the old ones. `probation_seconds` 0 is the default (120).
@@ -145,6 +146,12 @@ A source NAT rule is a match (as a rule's, without the rule's other
fields) with `"to"` (empty: masquerade), a redirect a match with
`"to_port"`. Times are seconds (`timeout`).
### Statistics
`GET /stats/status`, `/stats/summary`, `/stats/timeline`, `/stats/top`,
`/stats/hits` and `/stats/heatmap` (`log:read`): see
[stats.md](stats.md#the-api).
### Changesets
Changes staged to be reviewed and applied together ([design.md](design.md#reviewing-changes-changesets)).
docs/cli.md
+2 -1
View File
@@ -138,6 +138,7 @@ by name.
| `quotas` | what each rule's quota used (each address's, for per-address quotas), and when it resets |
| `log [-f] [--search TEXT] [--limit N]` | the packets logged by rules and policies ([operations.md](operations.md#the-log)) |
| `connections [--search TEXT] [--limit N]`, `connections close --search TEXT [-y]` | tracked connections; closing makes the rules check them again ([operations.md](operations.md#connections)) |
| `stats [--since 24h]` | traffic per zone, drops, the busiest rules, blocked sources, devices; `stats status`, `stats traffic`, `stats top devices\|blocked\|tried\|quota ID`, `stats rules [--unused]`, `stats connections` ([stats.md](stats.md#the-cli)) |
| `matrix [FROM TO]` | what the rules do to new connections between zones; with FROM and TO, that pair's rules in order ([rules.md](rules.md#the-zone-matrix)) |
| `trace [--address A] [--port N] [--protocol P] [--seconds S]` | follow packets through every table's rules ([operations.md](operations.md#tracing)) |
@@ -207,7 +208,7 @@ See [nat.md](nat.md).
| Command | |
|---|---|
| `settings show` | |
| `settings set [--input-policy\|--forward-policy\|--output-policy accept\|drop] [--log-drops on\|off] [--builtin NAME=on\|off]... [--own-prefix P]... [--antispoof-v4 P]... [--antispoof-v6 P]... [--flowtable DEV]... [--probation 2m] [--auto-confirm on\|off]` | lists given replace the old ones |
| `settings set [--input-policy\|--forward-policy\|--output-policy accept\|drop] [--log-drops on\|off] [--builtin NAME=on\|off]... [--own-prefix P]... [--antispoof-v4 P]... [--antispoof-v6 P]... [--flowtable DEV]... [--probation 2m] [--auto-confirm on\|off] [--stats-per-device on\|off]` | lists given replace the old ones |
### probation, history, import
docs/configuration.md
+6 -1
View File
@@ -26,11 +26,16 @@ reads the file when it starts.
| `backup.dir` | `/var/backups/housecarl` | where `housecarld backup` writes |
| `backup.retention` | `30d` | backups older than this are deleted… |
| `backup.keep_at_least` | `7` | …keeping at least this many |
| `stats.enabled` | `true` | keep traffic, drops, hits, blocked sources, connections and quotas over time ([stats.md](stats.md)) |
| `stats.database` | `/var/lib/housecarl/stats.db` | the statistics' own database; deleting it is safe |
| `stats.retention.minute`, `.hour`, `.day` | `7d`, `90d`, `730d` | how long per-minute, hourly and daily figures are kept |
| `stats.top_n` | `100` | entries kept per top list, per hour and per day |
Durations are written `15s`, `12h` or `30d`.
The probation's length is a firewall setting, not here: `hc settings set
--probation 5m`.
--probation 5m`. So is counting traffic per device for the statistics:
`hc settings set --stats-per-device on`.
## HTTPS
docs/stats.md
+140
View File
@@ -0,0 +1,140 @@
# Statistics
Housecarl keeps what the firewall did over time:
- traffic per interface and zone, received and sent;
- packets dropped, by chain, and when drops happen (a weekday × hour
heatmap);
- hits per rule, built-in and policy, and the rules that matched nothing;
- the sources blocked most and what they tried (from the log);
- traffic per inside device (a setting, below);
- connections tracked, against the kernel's limit;
- what each quota counted;
- housecarld's and the machine's memory, CPU and load.
All of it is in the web UI (**Traffic → Statistics**, and two charts on
the dashboard), the CLI (`hc stats`) and the API (`/api/v1/stats/...`),
for anyone with `log:read`.
Statistics are **on by default**. They cost nothing on the packets'
path: housecarld reads counters the kernel keeps anyway, once a minute.
Only traffic per device adds to the ruleset, so it is a setting you turn
on. The figures are kept as aggregates in a database of their own,
`/var/lib/housecarl/stats.db`, separate from the configuration. Deleting
it is always safe (stop housecarl first), and backups leave it out.
## Settings
In `/etc/housecarl/housecarl.yaml` (restart housecarl after a change):
```yaml
stats:
enabled: true
database: /var/lib/housecarl/stats.db
retention:
minute: 7d # per-minute figures
hour: 90d # hourly figures and top lists
day: 730d # daily figures and top lists
top_n: 100 # entries kept per top list, per hour and day
```
Old data is removed once an hour. A range picks the finest data that
covers it: per minute up to 2 days, hourly up to 90 days, then daily.
## Traffic per device
```sh
hc settings set --stats-per-device on
```
(or **Settings → Statistics** in the web UI). This is a firewall setting,
so it goes through the usual review and probation. Housecarl then adds
four sets to its table, and four rules at the top of the forward chain
(shown as the built-in `stats`):
```
iifname != "wan" oifname "wan" update @stats_up_v4 { ip saddr }
iifname != "wan" oifname "wan" update @stats_up_v6 { ip6 saddr and ffff:ffff:ffff:ffff:: }
iifname "wan" oifname != "wan" update @stats_down_v4 { ip daddr }
iifname "wan" oifname != "wan" update @stats_down_v6 { ip6 daddr and ffff:ffff:ffff:ffff:: }
```
Each inside address talking to an external zone gets a counter: what it
sent out, and what came back to it. IPv6 addresses are counted per /64,
since one device uses many addresses. Counting is done in the kernel, a
set lookup per forwarded packet. An address quiet for an hour drops out
of the sets, and a reload keeps the counts.
With the flowtable on, established connections skip the forward chain,
so only their first packets are counted (warning `stats_flowtable`). Set
names starting with `stats_` (and `meter_`, `conn_`, `quota_`) are
Housecarl's own and can't be used for your sets.
## What is collected
Every minute housecarld samples its sources and stores what changed:
| Figure | Source |
|---|---|
| `rx:IF`, `tx:IF` (bytes), `rxp:IF`, `txp:IF` (packets) | the interfaces' counters (`/sys/class/net`); the loopback is left out |
| `zrx:ZONE`, `ztx:ZONE` | the same, summed per zone: received from and sent to the zone's interfaces |
| `hits:MARK`, `hitb:MARK` | each rule's (`r12`), built-in's (`b-established`), port forward's (`f3`) and policy's (`p-input`) counter, in packets and bytes |
| `drop:CHAIN`, `drops` | the packets counted by what drops them: rules that drop or reject, the drop policies, antispoofing (`early`) and invalid packets (`invalid`) |
| `quota:r12` | bytes counted against a rule's quota |
| `g:conntrack`, `g:conntrack_max` | connections tracked, and the limit (`nf_conntrack_max`) |
| `g:rss`, `g:cpu` | housecarld's memory (bytes) and CPU (tenths of a percent of one CPU) |
| `g:mem`, `g:load` | the host's memory in use, and its 1-minute load average × 100 |
`g:` figures are gauges: the API averages them over a bucket.
Top lists, per hour and per day, the `top_n` largest:
| List | |
|---|---|
| `blocked_src` | sources of logged drops |
| `blocked_port` | what they tried: `tcp/22`, `udp/53`, `icmp/echo-request` |
| `device_up`, `device_down` | bytes per inside address, sent and received (traffic per device) |
| `quota_r12` | a per-address quota's use, per address |
The blocked lists only see what is **logged**: the input and forward
policies' drops with `hc settings set --log-drops on`, and rules with
logging. The drop counts above see everything.
A counter that goes down (an interface that came back, a quota reset)
counts again from zero. What happens while housecarld is stopped isn't
counted.
## The CLI
```sh
hc stats --since 7d # summary: traffic per zone, drops, busiest rules, top lists
hc stats status # collecting? since when, the database, figures now
hc stats traffic --since 24h # per zone over time (--interfaces: per interface)
hc stats top devices --since 30d # devices by traffic (blocked, tried, quota RULE-ID)
hc stats rules --unused # rules that matched nothing in 30 days
hc stats connections --since 24h
```
Ranges are written `90m`, `24h`, `7d` or `4w`. `-o json` gives the API's
answer.
## The API
| Endpoint | |
|---|---|
| `GET /stats/status` | `enabled`, `database`, `size`, `first_data`, `last_sample`, `sample_error`, `per_device`, `retention`, `latest` (the newest gauges) |
| `GET /stats/summary?since=24h&limit=10` | `zones` and `interfaces` (`name`, `received`, `sent`), `drops`, `drops_by_chain`, `rules` (busiest, named), `peak_connections`, `conntrack_max`, `blocked_sources`, `blocked_ports`, `devices_up`, `devices_down` |
| `GET /stats/timeline?keys=zrx:*,drops&since=24h&step=auto` | `step`, `times` (bucket starts), `series` (`key`, `values`); `GROUP:*` means every key of a group; `step` is `auto`, `1m`, `5m`, `15m`, `1h`, `6h` or `1d`; a gauge without a sample is -1 |
| `GET /stats/top?kind=device_down&since=7d&limit=20` | a top list (`key`, `count`) |
| `GET /stats/hits?since=30d` | packets per rule, built-in and policy (`key`, `count`, `name`), with the enabled rules that counted none |
| `GET /stats/heatmap?key=drops&since=28d` | `avg[weekday][hour]` (Monday first, local time; -1 where there is no data) |
With statistics off, `/stats/status` says `enabled: false` and the others
answer 503.
## Warnings
When the connection tracking table is over 80% full, the status shows
the warning `conntrack_full`: once it is full, the kernel drops new
connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens
so many on the Connections page.
docs/validation.md
+4
View File
@@ -34,6 +34,8 @@ are unique per kind, ignoring case.
| `flowtable_device` | error | a flowtable device with a wildcard: name each device |
| `flowtable_device_missing` | warning, *host* | a flowtable device doesn't exist; the flowtable is left out until it does |
| `probation_range` | error | the probation isn't 10 to 3600 seconds |
| `stats_no_external` | warning | traffic per device is on, but no zone is external: it counts what goes to and comes from external zones |
| `stats_flowtable` | warning | traffic per device with the flowtable on: its fast path skips the forward chain, so only each connection's first packets are counted |
## Zones
@@ -49,6 +51,7 @@ are unique per kind, ignoring case.
| Code | | When |
|---|---|---|
| `set_name_reserved` | error | a set name starting with `meter_`, `conn_`, `quota_` or `stats_`: Housecarl's own sets (rate limits, quotas, statistics) use those |
| `set_entry` | error | an entry that isn't an address, prefix (without host bits) or range |
| `set_empty` | warning | a set (not dynamic) with no entries: rules using it match nothing |
| `set_dynamic_entries` | error | configured entries on a dynamic set: its entries come at run time |
@@ -127,4 +130,5 @@ configuration: the dashboard and `hc status` show them.
| Code | | When |
|---|---|---|
| `nftables_service` | warning, *host* | Debian's `nftables.service` is enabled: at boot it loads `/etc/nftables.conf`, and stopping or restarting it runs `nft flush ruleset`, deleting Housecarl's table too. Disable it with `systemctl disable nftables` (without `--now`) |
| `conntrack_full` | warning, *host* | the connection tracking table is over 80% full; once it is full, the kernel drops new connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens so many ([stats.md](stats.md#warnings)) |
| `takeover_pending` | warning, *host* | the table you imported from is still in force beside Housecarl's, so a packet must get past both; `hc takeover TABLE` deletes it once Housecarl's rules are right |