Release 0.2.0-rc.4+git74-5348765

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -124,6 +124,16 @@ All notable changes to Housecarl are documented here. The format follows
 
 ### Changed
 
+- Moving a rule several places is one change. The Rules page has a
+  **Move…** button that puts a rule before any other of its chain (or at
+  the end) at once, and in a changeset a move right after a move of the
+  same rule takes its place, so ↑ three times stages one operation (and
+  moving it back where it was takes the operation out).
+- Statistics' "Busiest rules" (the Statistics page, `hc stats`,
+  `GET /stats/summary`) lists only rules and NAT objects. Built-ins and
+  policies have their own list ("Built-ins and policies", `builtins` in
+  the API): the established built-in passes most of the traffic, so it
+  topped the list every time and made every rule's bar look empty.
 - `hc user delete` asks first, as every other delete does, and takes
   `-y`/`--yes` not to; it deleted the user and their tokens at once.
 - The ruleset's diff (Changes, `hc changeset diff`, `--dry-run`) names
@@ -162,6 +172,9 @@ All notable changes to Housecarl are documented here. The format follows
 
 ### Fixed
 
+- An IPv4-mapped prefix (`::ffff:0:0/96`) is refused with an error that
+  says so; it was refused as "has host bits set; the prefix is invalid
+  Prefix".
 - `hc forward show` shows the forward's hits and comment, and its
   hairpin as `no`, `yes` or `yes, from …` (it printed `no -`).
 - A history entry's ruleset diff showed the object it changed as moved:

README.md

@@ -128,7 +128,7 @@ The web UI is on `http://127.0.0.1:8070/` (see `listen` in
 
 ## Status
 
-Housecarl 0.2.0 is in testing (0.2.0-rc.4+git68-b4c0988): zones, address sets and
+Housecarl 0.2.0 is in testing (0.2.0-rc.4+git74-5348765): zones, address sets and
 services with ordered rules, IPv4 and IPv6, port forwards with hairpin NAT
 and IPv6 pinholes, source NAT and redirects, built-ins, probation for
 network changes (confirmed by the client that made them), the ruleset

VERSION

@@ -1 +1 @@
-0.2.0-rc.4+git68-b4c0988
+0.2.0-rc.4+git74-5348765

cmd/hc/stats.go

@@ -33,6 +33,7 @@ type statSummary struct {
 	Drops           int64         `json:"drops"`
 	DropsByChain    []statEntry   `json:"drops_by_chain"`
 	Rules           []statEntry   `json:"rules"`
+	BuiltIns        []statEntry   `json:"builtins"`
 	PeakConnections int64         `json:"peak_connections"`
 	ConntrackMax    int64         `json:"conntrack_max"`
 	BlockedSources  []statEntry   `json:"blocked_sources"`
@@ -136,6 +137,7 @@ func printSummary(w io.Writer, s statSummary) {
 	}
 	packets := func(n int64) string { return humanCount(uint64(n)) + " packets" }
 	list("Busiest rules:", s.Rules, packets)
+	list("Built-ins and policies:", s.BuiltIns, packets)
 	list("Blocked most (logged drops):", s.BlockedSources, packets)
 	list("What they tried:", s.BlockedPorts, packets)
 	list("Devices, sent:", s.DevicesUp, humanBytes)

docs/design.md

@@ -56,7 +56,10 @@ As in Jarl, a change can be made at once or staged in a **changeset**,
 to be reviewed and applied together with others:
 
 * Each change is an **operation** (create, update, delete or move an
-  object, or change the settings), kept as the API received it.
+  object, or change the settings), kept as the API received it. A move
+  staged right after a move of the same object takes its place: moving a
+  rule up three steps is one operation, and moving it back where it was
+  takes it out.
 * Staged in a changeset, an operation is checked as it is added: the
   changeset with it must still be valid, and nftables must accept the
   ruleset it makes. Nothing is applied.

docs/rules.md

@@ -130,6 +130,11 @@ drops or rejects decides. What no rule decides gets the chain's
 **policy** (Settings): by default input and forward drop, output
 accepts.
 
+On the Rules page, ↑ and ↓ move a rule one place; **Move…** puts it
+before any other rule of its chain, or at the end, in one change
+(`hc rule move ID --before ID`). In a changeset, moves of the same rule
+one after another are kept as one operation.
+
 ```sh
 hc rule add input --from trusted --service ssh --comment "SSH from home"
 hc rule add input --in wg --service dns

docs/stats.md

@@ -5,7 +5,9 @@ Housecarl keeps what the firewall did over time:
 - traffic per interface and zone, received and sent;
 - packets dropped, by chain, and when drops happen (a weekday × hour
   heatmap);
-- hits per rule, built-in and policy, and the rules that matched nothing;
+- hits per rule, and per built-in and policy in a list of their own (the
+  established built-in counts most packets), and the rules that matched
+  nothing;
 - the sources blocked most and what they tried (from the log);
 - traffic per inside device (a setting, below);
 - connections tracked, against the kernel's limit;
@@ -127,7 +129,7 @@ answer.
 | Endpoint | |
 |---|---|
 | `GET /stats/status` | `enabled`, `database`, `size`, `first_data`, `last_sample`, `sample_error`, `per_device`, `retention`, `latest` (the newest gauges) |
-| `GET /stats/summary?since=24h&limit=10` | `zones` and `interfaces` (`name`, `received`, `sent`), `drops`, `drops_by_chain`, `rules` (busiest, named), `peak_connections`, `conntrack_max`, `blocked_sources`, `blocked_ports`, `devices_up`, `devices_down` |
+| `GET /stats/summary?since=24h&limit=10` | `zones` and `interfaces` (`name`, `received`, `sent`), `drops`, `drops_by_chain`, `rules` (busiest rules and NAT objects, named), `builtins` (busiest built-ins and policies, named), `peak_connections`, `conntrack_max`, `blocked_sources`, `blocked_ports`, `devices_up`, `devices_down` |
 | `GET /stats/timeline?keys=zrx:*,drops&since=24h&step=auto` | `step`, `times` (bucket starts), `series` (`key`, `values`); `GROUP:*` means every key of a group; `step` is `auto`, `1m`, `5m`, `15m`, `1h`, `6h` or `1d`; a gauge without a sample is -1 |
 | `GET /stats/top?kind=device_down&since=7d&limit=20` | a top list (`key`, `count`) |
 | `GET /stats/hits?since=30d` | packets per rule, built-in and policy (`key`, `count`, `name`), with the enabled rules that counted none |

internal/model/addr.go

@@ -84,7 +84,11 @@ func ParseAddress(s string) (Address, error) {
 		if err != nil {
 			return Address{}, fmt.Errorf("%q is not an address or prefix", s)
 		}
-		p = netip.PrefixFrom(p.Addr().Unmap(), p.Bits())
+		if p.Addr().Is4In6() {
+			// Unmapping would leave an IPv4 address with IPv6 bits
+			// (::ffff:0:0/96 as 0.0.0.0/96).
+			return Address{}, fmt.Errorf("%q is an IPv4-mapped prefix, which can't be used: write IPv4 prefixes as IPv4 (10.0.0.0/8)", s)
+		}
 		if p.Masked() != p {
 			return Address{}, fmt.Errorf("%q has host bits set; the prefix is %s", s, p.Masked())
 		}

internal/model/addr_test.go

@@ -0,0 +1,28 @@
+package model
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestParseAddress(t *testing.T) {
+	for _, c := range []struct{ in, want, err string }{
+		{in: "10.0.0.0/8", want: "10.0.0.0/8"},
+		{in: "2001:db8::/32", want: "2001:db8::/32"},
+		{in: "::ffff:10.0.0.1", want: "10.0.0.1"},
+		{in: "10.0.0.1/8", err: "host bits"},
+		{in: "::ffff:0:0/96", err: "IPv4-mapped"},
+		{in: "::ffff:10.0.0.0/104", err: "IPv4-mapped"},
+	} {
+		a, err := ParseAddress(c.in)
+		if c.err != "" {
+			if err == nil || !strings.Contains(err.Error(), c.err) {
+				t.Errorf("%s: error %v, want one saying %q", c.in, err, c.err)
+			}
+			continue
+		}
+		if err != nil || a.String() != c.want {
+			t.Errorf("%s: %q, %v; want %q", c.in, a.String(), err, c.want)
+		}
+	}
+}

internal/server/h_changesets.go

@@ -89,6 +89,17 @@ func (s *Server) stage(w http.ResponseWriter, r *http.Request, p *auth.Principal
 	if err := replay(before, cs.Ops); err != nil {
 		return conflict("changeset %d can't be applied as it is (%v); review it first", id, err)
 	}
+	// A move right after a move of the same object takes its place:
+	// moving a rule up three steps is one operation.
+	kept := cs.Ops
+	if n := len(kept); n > 0 && foldMove(before, &kept[n-1], &op) {
+		kept = kept[:n-1]
+		before = s.eng.State()
+		if err := replay(before, kept); err != nil {
+			return conflict("changeset %d can't be applied as it is (%v); review it first", id, err)
+		}
+	}
+	folded := len(kept) < len(cs.Ops)
 	after := before.Clone()
 	objID, err := applyOp(after, &op)
 	if err != nil {
@@ -98,6 +109,13 @@ func (s *Server) stage(w http.ResponseWriter, r *http.Request, p *auth.Principal
 	after.Sort()
 	changes := engine.Diff(before, after)
 	if len(changes) == 0 {
+		// Moved back where it was: the earlier move goes too.
+		if folded && !queryBool(r, "dry_run") {
+			cs.Ops = kept
+			if err := s.store.UpdateChangeset(r.Context(), cs); err != nil {
+				return err
+			}
+		}
 		writeJSON(w, http.StatusOK, api.MutationResponse{Changes: []model.Change{}, Changeset: cs})
 		return nil
 	}
@@ -108,7 +126,7 @@ func (s *Server) stage(w http.ResponseWriter, r *http.Request, p *auth.Principal
 		op.Summary = engine.UndoDescription(&h)
 	}
 	op.AddedAt = s.nowFn()
-	for _, o := range cs.Ops {
+	for _, o := range kept {
 		if o.Seq >= op.Seq {
 			op.Seq = o.Seq + 1
 		}
@@ -116,7 +134,7 @@ func (s *Server) stage(w http.ResponseWriter, r *http.Request, p *auth.Principal
 	if op.Seq == 0 {
 		op.Seq = 1
 	}
-	ops := append(append([]model.Op{}, cs.Ops...), op)
+	ops := append(append([]model.Op{}, kept...), op)
 	dry := queryBool(r, "dry_run")
 	// The whole changeset, checked as it would be applied.
 	res, err := s.eng.Apply(r.Context(), p, "", engine.Options{DryRun: true, Via: via(r, p)}, func(st *model.State, _ func() int64) error {
@@ -145,6 +163,31 @@ func (s *Server) stage(w http.ResponseWriter, r *http.Request, p *auth.Principal
 	return nil
 }
 
+// foldMove reports whether op, a move, can take the place of prev, the
+// changeset's last operation: a move of the same object. A move puts
+// the object relative to the others, which an earlier move of it
+// doesn't change, so only the chain it moved to has to be carried over.
+// st is the changeset so far.
+func foldMove(st *model.State, prev, op *model.Op) bool {
+	if prev.Op != model.OpMove || op.Op != model.OpMove || prev.Kind != op.Kind {
+		return false
+	}
+	a, err1 := findObject(st, prev.Kind, prev.Ref)
+	b, err2 := findObject(st, op.Kind, op.Ref)
+	if err1 != nil || err2 != nil || a.GetID() != b.GetID() {
+		return false
+	}
+	var was, now api.MoveInput
+	if json.Unmarshal(prev.Data, &was) != nil || json.Unmarshal(op.Data, &now) != nil {
+		return false
+	}
+	if now.Chain == "" && was.Chain != "" {
+		now.Chain = was.Chain
+		op.Data, _ = json.Marshal(now)
+	}
+	return true
+}
+
 func (s *Server) createChangeset(w http.ResponseWriter, r *http.Request, p *auth.Principal) error {
 	if err := require(p, auth.ResConfig, true); err != nil {
 		return err

internal/server/h_stats.go

@@ -150,8 +150,10 @@ func (s *Server) statsSummary(w http.ResponseWriter, r *http.Request, p *auth.Pr
 		return err
 	}
 	st := s.eng.State()
-	for i := range sum.Rules {
+	for _, es := range [][]stats.Entry{sum.Rules, sum.BuiltIns} {
-		sum.Rules[i].Name = markName(st, sum.Rules[i].Key)
+		for i := range es {
+			es[i].Name = markName(st, es[i].Key)
+		}
 	}
 	writeJSON(w, http.StatusOK, sum)
 	return nil

internal/server/server_test.go

@@ -1336,6 +1336,53 @@ func TestCreatePlaced(t *testing.T) {
 	}
 }
 
+// TestStagedMovesFold: moving a rule several steps in a changeset stays
+// one operation, and moving it back where it was takes it out.
+func TestStagedMovesFold(t *testing.T) {
+	e := newEnv(t)
+	var ids []int64
+	for _, svc := range []string{"ssh", "dns", "http", "https"} {
+		var mr api.MutationResponse
+		var r model.Rule
+		e.call(e.token, "POST", "/rules", model.Rule{Chain: "input", Match: model.Match{Services: []string{svc}}}, &mr)
+		json.Unmarshal(mr.Object, &r)
+		ids = append(ids, r.ID)
+	}
+	var cs model.Changeset
+	e.call(e.token, "POST", "/changesets", nil, &cs)
+	q := "?changeset=" + itoa(cs.ID)
+	var mr api.MutationResponse
+	for _, before := range []int64{ids[2], ids[1], ids[0]} { // https up three steps
+		if code := e.call(e.token, "POST", "/rules/"+itoa(ids[3])+"/move"+q, api.MoveInput{Before: before}, &mr); code != 200 || !mr.Staged {
+			t.Fatalf("move before %d: %d %+v", before, code, mr)
+		}
+	}
+	if n := len(mr.Changeset.Ops); n != 1 {
+		t.Fatalf("%d operations, want 1: %+v", n, mr.Changeset.Ops)
+	}
+	// Another operation in between: the next move is its own.
+	e.call(e.token, "PATCH", "/rules/"+itoa(ids[0])+q, map[string]any{"comment": "x"}, nil)
+	e.call(e.token, "POST", "/rules/"+itoa(ids[3])+"/move"+q, api.MoveInput{After: ids[0]}, &mr)
+	if n := len(mr.Changeset.Ops); n != 3 {
+		t.Errorf("%d operations after a patch and a move, want 3", n)
+	}
+	// Back to the start: the same place as before that move, so it goes.
+	e.call(e.token, "POST", "/rules/"+itoa(ids[3])+"/move"+q, api.MoveInput{To: "start"}, &mr)
+	if n := len(mr.Changeset.Ops); n != 2 {
+		t.Errorf("%d operations after moving back, want 2", n)
+	}
+	if code := e.call(e.token, "POST", "/changesets/"+itoa(cs.ID)+"/apply", nil, nil); code != 200 {
+		t.Fatalf("apply: %d", code)
+	}
+	var out []string
+	for _, r := range e.eng.State().Rules {
+		out = append(out, strings.Join(r.Services, ""))
+	}
+	if o := strings.Join(out, ","); o != "https,ssh,dns,http" {
+		t.Errorf("order: %s", o)
+	}
+}
+
 // TestTokenForUser: an administrator makes a token for another user (the
 // web UI's New token on a user's row); that user can't make one for
 // someone else.

internal/stats/collector_test.go

@@ -7,6 +7,7 @@ import (
 	"log/slog"
 	"os"
 	"path/filepath"
+	"slices"
 	"testing"
 	"time"
 
@@ -154,6 +155,14 @@ func TestCollector(t *testing.T) {
 		t.Fatal(err)
 	}
 	check("drops", s.Drops, 18)
+	for _, e := range s.Rules {
+		if e.Key[0] == 'b' || e.Key[0] == 'p' {
+			t.Errorf("built-in or policy %s among the busiest rules %+v", e.Key, s.Rules)
+		}
+	}
+	if !slices.ContainsFunc(s.Rules, func(e Entry) bool { return e.Key == "r1" }) || !slices.ContainsFunc(s.BuiltIns, func(e Entry) bool { return e.Key == "p-input" }) {
+		t.Errorf("rules %+v, built-ins %+v", s.Rules, s.BuiltIns)
+	}
 	if len(s.Zones) == 0 || s.Zones[0].Name != "wan" || s.Zones[0].Received != 4000 {
 		t.Errorf("zones %+v", s.Zones)
 	}

internal/stats/query.go

@@ -191,8 +191,11 @@ type Summary struct {
 	// chain.
 	Drops        int64   `json:"drops"`
 	DropsByChain []Entry `json:"drops_by_chain"`
-	// Rules are the busiest rules, built-ins and policies, by packets.
+	// Rules are the busiest rules and NAT objects, by packets; BuiltIns
-	Rules []Entry `json:"rules"`
+	// the busiest built-ins and policies, kept apart because the
+	// established built-in counts most of the traffic.
+	Rules    []Entry `json:"rules"`
+	BuiltIns []Entry `json:"builtins"`
 	// Connections: the most tracked at once (per-bucket averages), and
 	// the table's size.
 	PeakConnections int64   `json:"peak_connections"`
@@ -251,9 +254,15 @@ func (q *Querier) Summary(ctx context.Context, since time.Duration, topLimit int
 	s.Zones, s.Interfaces = traffic("zrx:", "ztx:"), traffic("rx:", "tx:")
 	s.Drops = all["drops"]
 	s.DropsByChain = sorted(all, "drop:")
-	s.Rules = sorted(all, "hits:")
+	s.Rules, s.BuiltIns = []Entry{}, []Entry{}
-	if len(s.Rules) > topLimit {
+	for _, e := range sorted(all, "hits:") {
-		s.Rules = s.Rules[:topLimit]
+		if strings.HasPrefix(e.Key, "b-") || strings.HasPrefix(e.Key, "p-") {
+			if len(s.BuiltIns) < topLimit {
+				s.BuiltIns = append(s.BuiltIns, e)
+			}
+		} else if len(s.Rules) < topLimit {
+			s.Rules = append(s.Rules, e)
+		}
 	}
 	if tl, err := q.Timeline(ctx, since, "auto", []string{"g:" + GaugeConntrack, "g:" + GaugeConntrackMax}); err == nil {
 		for _, v := range tl.Series[0].Values {

internal/web/static/assets/pages/rules.js

@@ -62,6 +62,7 @@ export async function render(el, params, query) {
           return [
           h("button", { class: "small reveal", title: "Move up", disabled: i === 0, onclick: () => move(r, { before: rs[i - 1].id }) }, "↑"), " ",
           h("button", { class: "small reveal", title: "Move down", disabled: i === rs.length - 1, onclick: () => move(r, { after: rs[i + 1].id }) }, "↓"), " ",
+          h("button", { class: "small reveal", title: "Move to another place, in one change", disabled: rs.length < 2, onclick: () => moveDialog(r, rs, (x) => live(rules).indexOf(x) + 1) }, "Move…"), " ",
           h("button", { class: "small reveal", onclick: () => ruleDialog(r, chain) }, "Edit"), " ",
           h("button", { class: "small reveal", onclick: () => toggle(r) }, r.disabled ? "Enable" : "Disable"), " ",
           h("button", { class: "small reveal", onclick: () => del(r) }, "Delete"),
@@ -245,6 +246,28 @@ async function move(r, where) {
   try { await change("POST", "/rules/" + r.id + "/move", where, { quiet: true }); rerender(); } catch (e) { modal("Not moved", errorBox(e)); }
 }
 
+// moveDialog puts a rule anywhere among rs (the rules it can move
+// among) in one move: one change, or one staged operation.
+async function moveDialog(r, rs, order) {
+  const others = rs.filter((x) => x !== r);
+  const label = (x) => order(x) + ". " + (x.comment || (x.raw ? x.raw : describe(x))) + " (#" + x.id + ")";
+  const next = rs[rs.indexOf(r) + 1];
+  const where = select([...others.map((x) => [String(x.id), "before " + label(x)]), ["end", "after " + label(others[others.length - 1])]],
+    next ? String(next.id) : "end");
+  const err = h("div");
+  await modal("Move " + (r.comment || "rule #" + r.id), h("div", null, h("div", { class: "form-grid" }, field("Put it", where)), err), [
+    { label: "Cancel", value: null },
+    { label: "Move", kind: "primary", onclick: async () => {
+      const v = where.value;
+      if (v === (next ? String(next.id) : "end")) return true; // where it is
+      try {
+        await change("POST", "/rules/" + r.id + "/move", v === "end" ? { after: others[others.length - 1].id } : { before: Number(v) }, { quiet: true });
+        rerender();
+      } catch (e) { mount(err, errorBox(e)); return false; }
+    } },
+  ]);
+}
+
 async function toggle(r) {
   try { await change("PATCH", "/rules/" + r.id, { disabled: !r.disabled }); rerender(); } catch (e) { modal("Not changed", errorBox(e)); }
 }

internal/web/static/assets/pages/stats.js

@@ -98,7 +98,8 @@ export async function render(el, params, query) {
     ? "No traffic per device " + rangeLabel + "."
     : ["Traffic per device is off. Turn it on under ", h("a", { href: "#/settings" }, "Settings"), " (it adds a counter for each inside address)."]);
   parts.push(h("div", { class: "chart-grid" },
-    listCard("Busiest rules", "packets each counted, " + rangeLabel, barList(s.rules, { label: (k) => k, href: (k) => ruleHref(k) })),
+    listCard("Busiest rules", "packets each counted, " + rangeLabel, barList(s.rules, { label: (k) => k, href: (k) => ruleHref(k), empty: "No rule counted anything " + rangeLabel + "." })),
+    listCard("Built-ins and policies", "packets each counted; 'established' passes the replies and the rest of every allowed connection", barList(s.builtins || [], { label: (k) => k, href: (k) => ruleHref(k) })),
     listCard("Dropped, by chain", "early: antispoofing · invalid: packets conntrack can't place", barList(s.drops_by_chain, { empty: "Nothing dropped " + rangeLabel + "." })),
     listCard("Blocked most", "sources of logged drops (the policies' with 'log drops' on, and rules that log)", barList(s.blocked_sources, { empty: "No logged drops " + rangeLabel + ".", href: (k) => "#/trace?address=" + encodeURIComponent(k) })),
     listCard("What they tried", "the logged drops' protocol and port", barList(s.blocked_ports, { empty: "No logged drops " + rangeLabel + "." })),