# Statistics

Housecarl keeps what the firewall did over time:

- traffic per interface and zone, received and sent;
- packets dropped, by chain, and when drops happen (a weekday × hour
  heatmap);
- hits per rule, and per built-in and policy in a list of their own (the
  established built-in counts most packets), and the rules that matched
  nothing;
- the sources blocked most and what they tried (from the log);
- traffic per inside device (a setting, below);
- connections tracked, against the kernel's limit;
- what each quota counted;
- housecarld's and the machine's memory, CPU and load.

All of it is in the web UI (**Traffic → Statistics**, and two charts on
the dashboard), the CLI (`hc stats`) and the API (`/api/v1/stats/...`),
for anyone with `log:read`.

Statistics are **on by default**. They cost nothing on the packets'
path: housecarld reads counters the kernel keeps anyway, once a minute.
Only traffic per device adds to the ruleset, so it is a setting you turn
on. The figures are kept as aggregates in a database of their own,
`/var/lib/housecarl/stats.db`, separate from the configuration. Deleting
it is always safe (stop housecarl first), and backups leave it out.

## Settings

In `/etc/housecarl/housecarl.yaml` (restart housecarl after a change):

```yaml
stats:
  enabled: true
  database: /var/lib/housecarl/stats.db
  retention:
    minute: 7d     # per-minute figures
    hour: 90d      # hourly figures and top lists
    day: 730d      # daily figures and top lists
  top_n: 100       # entries kept per top list, per hour and day
```

Old data is removed once an hour. A range picks the finest data that
covers it: per minute up to 2 days, hourly up to 90 days, then daily.

## Traffic per device

```sh
hc settings set --stats-per-device on
```

(or **Settings → Statistics** in the web UI). This is a firewall setting,
so it goes through the usual review and probation. Housecarl then adds
four sets to its table, and four rules at the top of the forward chain
(shown as the built-in `stats`):

```
iifname != "wan" oifname "wan" update @stats_up_v4 { ip saddr }
iifname != "wan" oifname "wan" update @stats_up_v6 { ip6 saddr and ffff:ffff:ffff:ffff:: }
iifname "wan" oifname != "wan" update @stats_down_v4 { ip daddr }
iifname "wan" oifname != "wan" update @stats_down_v6 { ip6 daddr and ffff:ffff:ffff:ffff:: }
```

Each inside address talking to an external zone gets a counter: what it
sent out, and what came back to it. IPv6 addresses are counted per /64,
since one device uses many addresses. Counting is done in the kernel, a
set lookup per forwarded packet. An address quiet for an hour drops out
of the sets, and a reload keeps the counts.

With the flowtable on, established connections skip the forward chain,
so only their first packets are counted (warning `stats_flowtable`). Set
names starting with `stats_` (and `meter_`, `conn_`, `quota_`) are
Housecarl's own and can't be used for your sets.

## What is collected

Every minute housecarld samples its sources and stores what changed:

| Figure | Source |
|---|---|
| `rx:IF`, `tx:IF` (bytes), `rxp:IF`, `txp:IF` (packets) | the interfaces' counters (`/sys/class/net`); the loopback is left out |
| `zrx:ZONE`, `ztx:ZONE` | the same, summed per zone: received from and sent to the zone's interfaces |
| `hits:MARK`, `hitb:MARK` | each rule's (`r12`), built-in's (`b-established`), port forward's (`f3`) and policy's (`p-input`) counter, in packets and bytes |
| `drop:CHAIN`, `drops` | the packets counted by what drops them: rules that drop or reject, the drop policies, antispoofing (`early`) and invalid packets (`invalid`) |
| `quota:r12` | bytes counted against a rule's quota |
| `g:conntrack`, `g:conntrack_max` | connections tracked, and the limit (`nf_conntrack_max`) |
| `g:rss`, `g:cpu` | housecarld's memory (bytes) and CPU (tenths of a percent of one CPU) |
| `g:mem`, `g:load` | the host's memory in use, and its 1-minute load average × 100 |

`g:` figures are gauges: the API averages them over a bucket.

Top lists, per hour and per day, the `top_n` largest:

| List | |
|---|---|
| `blocked_src` | sources of logged drops |
| `blocked_port` | what they tried: `tcp/22`, `udp/53`, `icmp/echo-request` |
| `device_up`, `device_down` | bytes per inside address, sent and received (traffic per device) |
| `quota_r12` | a per-address quota's use, per address |

The blocked lists only see what is **logged**: the input and forward
policies' drops with `hc settings set --log-drops on`, and rules with
logging. The drop counts above see everything.

In shadow mode ([shadow.md](shadow.md)) the drops, and the blocked
lists, are what Housecarl would have dropped: the packets went on. What
the policies would drop is always logged there.

A counter that goes down (an interface that came back, a quota reset)
counts again from zero. What happens while housecarld is stopped isn't
counted.

## The CLI

```sh
hc stats --since 7d              # summary: traffic per zone, drops, busiest rules, top lists
hc stats status                  # collecting? since when, the database, figures now
hc stats traffic --since 24h     # per zone over time (--interfaces: per interface)
hc stats top devices --since 30d # devices by traffic (blocked, tried, quota RULE-ID)
hc stats rules --unused          # rules that matched nothing in 30 days
hc stats connections --since 24h
```

Ranges are written `90m`, `24h`, `7d` or `4w`. `-o json` gives the API's
answer.

## The API

| Endpoint | |
|---|---|
| `GET /stats/status` | `enabled`, `database`, `size`, `first_data`, `last_sample`, `sample_error`, `per_device`, `retention`, `latest` (the newest gauges) |
| `GET /stats/summary?since=24h&limit=10` | `zones` and `interfaces` (`name`, `received`, `sent`), `drops`, `drops_by_chain`, `rules` (busiest rules and NAT objects, named), `builtins` (busiest built-ins and policies, named), `peak_connections`, `conntrack_max`, `blocked_sources`, `blocked_ports`, `devices_up`, `devices_down` |
| `GET /stats/timeline?keys=zrx:*,drops&since=24h&step=auto` | `step`, `times` (bucket starts), `series` (`key`, `values`); `GROUP:*` means every key of a group; `step` is `auto`, `1m`, `5m`, `15m`, `1h`, `6h` or `1d`; a gauge without a sample is -1 |
| `GET /stats/top?kind=device_down&since=7d&limit=20` | a top list (`key`, `count`) |
| `GET /stats/hits?since=30d` | packets per rule, built-in and policy (`key`, `count`, `name`), with the enabled rules that counted none |
| `GET /stats/heatmap?key=drops&since=28d` | `avg[weekday][hour]` (Monday first, local time; -1 where there is no data) |

With statistics off, `/stats/status` says `enabled: false` and the others
answer 503.

## Warnings

When the connection tracking table is over 80% full, the status shows
the warning `conntrack_full`: once it is full, the kernel drops new
connections. Raise `net.netfilter.nf_conntrack_max`, or find what opens
so many on the Connections page.