housecarl/docs/stats.md

Statistics

Housecarl keeps what the firewall did over time:

• traffic per interface and zone, received and sent;
• packets dropped, by chain, and when drops happen (a weekday × hour heatmap);
• hits per rule, built-in and policy, and the rules that matched nothing;
• the sources blocked most and what they tried (from the log);
• traffic per inside device (a setting, below);
• connections tracked, against the kernel's limit;
• what each quota counted;
• housecarld's and the machine's memory, CPU and load.

All of it is in the web UI (Traffic → Statistics, and two charts on the dashboard), the CLI (hc stats) and the API (/api/v1/stats/...), for anyone with log:read.

Statistics are on by default. They cost nothing on the packets' path: housecarld reads counters the kernel keeps anyway, once a minute. Only traffic per device adds to the ruleset, so it is a setting you turn on. The figures are kept as aggregates in a database of their own, /var/lib/housecarl/stats.db, separate from the configuration. Deleting it is always safe (stop housecarl first), and backups leave it out.

Settings

In /etc/housecarl/housecarl.yaml (restart housecarl after a change):

stats:
  enabled: true
  database: /var/lib/housecarl/stats.db
  retention:
    minute: 7d     # per-minute figures
    hour: 90d      # hourly figures and top lists
    day: 730d      # daily figures and top lists
  top_n: 100       # entries kept per top list, per hour and day

Old data is removed once an hour. A range picks the finest data that covers it: per minute up to 2 days, hourly up to 90 days, then daily.

Traffic per device

hc settings set --stats-per-device on

(or Settings → Statistics in the web UI). This is a firewall setting, so it goes through the usual review and probation. Housecarl then adds four sets to its table, and four rules at the top of the forward chain (shown as the built-in stats):

iifname != "wan" oifname "wan" update @stats_up_v4 { ip saddr }
iifname != "wan" oifname "wan" update @stats_up_v6 { ip6 saddr and ffff:ffff:ffff:ffff:: }
iifname "wan" oifname != "wan" update @stats_down_v4 { ip daddr }
iifname "wan" oifname != "wan" update @stats_down_v6 { ip6 daddr and ffff:ffff:ffff:ffff:: }

Each inside address talking to an external zone gets a counter: what it sent out, and what came back to it. IPv6 addresses are counted per /64, since one device uses many addresses. Counting is done in the kernel, a set lookup per forwarded packet. An address quiet for an hour drops out of the sets, and a reload keeps the counts.

With the flowtable on, established connections skip the forward chain, so only their first packets are counted (warning stats_flowtable). Set names starting with stats_ (and meter_, conn_, quota_) are Housecarl's own and can't be used for your sets.

What is collected

Every minute housecarld samples its sources and stores what changed:

Figure	Source
rx:IF, tx:IF (bytes), rxp:IF, txp:IF (packets)	the interfaces' counters (/sys/class/net); the loopback is left out
zrx:ZONE, ztx:ZONE	the same, summed per zone: received from and sent to the zone's interfaces
hits:MARK, hitb:MARK	each rule's (r12), built-in's (b-established), port forward's (f3) and policy's (p-input) counter, in packets and bytes
drop:CHAIN, drops	the packets counted by what drops them: rules that drop or reject, the drop policies, antispoofing (early) and invalid packets (invalid)
quota:r12	bytes counted against a rule's quota
g:conntrack, g:conntrack_max	connections tracked, and the limit (nf_conntrack_max)
g:rss, g:cpu	housecarld's memory (bytes) and CPU (tenths of a percent of one CPU)
g:mem, g:load	the host's memory in use, and its 1-minute load average × 100

g: figures are gauges: the API averages them over a bucket.

Top lists, per hour and per day, the top_n largest:

List
blocked_src	sources of logged drops
blocked_port	what they tried: tcp/22, udp/53, icmp/echo-request
device_up, device_down	bytes per inside address, sent and received (traffic per device)
quota_r12	a per-address quota's use, per address

The blocked lists only see what is logged: the input and forward policies' drops with hc settings set --log-drops on, and rules with logging. The drop counts above see everything.

In shadow mode (shadow.md) the drops, and the blocked lists, are what Housecarl would have dropped: the packets went on. What the policies would drop is always logged there.

A counter that goes down (an interface that came back, a quota reset) counts again from zero. What happens while housecarld is stopped isn't counted.

The CLI

hc stats --since 7d              # summary: traffic per zone, drops, busiest rules, top lists
hc stats status                  # collecting? since when, the database, figures now
hc stats traffic --since 24h     # per zone over time (--interfaces: per interface)
hc stats top devices --since 30d # devices by traffic (blocked, tried, quota RULE-ID)
hc stats rules --unused          # rules that matched nothing in 30 days
hc stats connections --since 24h

Ranges are written 90m, 24h, 7d or 4w. -o json gives the API's answer.

The API

Endpoint
GET /stats/status	enabled, database, size, first_data, last_sample, sample_error, per_device, retention, latest (the newest gauges)
GET /stats/summary?since=24h&limit=10	zones and interfaces (name, received, sent), drops, drops_by_chain, rules (busiest, named), peak_connections, conntrack_max, blocked_sources, blocked_ports, devices_up, devices_down
GET /stats/timeline?keys=zrx:*,drops&since=24h&step=auto	step, times (bucket starts), series (key, values); GROUP:* means every key of a group; step is auto, 1m, 5m, 15m, 1h, 6h or 1d; a gauge without a sample is -1
GET /stats/top?kind=device_down&since=7d&limit=20	a top list (key, count)
GET /stats/hits?since=30d	packets per rule, built-in and policy (key, count, name), with the enabled rules that counted none
GET /stats/heatmap?key=drops&since=28d	avg[weekday][hour] (Monday first, local time; -1 where there is no data)

With statistics off, /stats/status says enabled: false and the others answer 503.

Warnings

When the connection tracking table is over 80% full, the status shows the warning conntrack_full: once it is full, the kernel drops new connections. Raise net.netfilter.nf_conntrack_max, or find what opens so many on the Connections page.