fw: a dead firewall leaves the network down, and can be restarted into it

Two things can happen when a firewall dies: the traffic it was
filtering carries on unfiltered, or it stops.  Only the second is
defensible.  A machine briefly off the network is a machine somebody
notices and fixes; a machine briefly on the network with no rules is
the thing the firewall was installed to prevent, and nobody notices it
at all.

fw already does the second, in all three modes, and by mechanism rather
than by care.  Measured rather than assumed:

	before fw   device: /net/ether1  addr: 10.9.9.1  route: 10.9.9.254
	fw running  device: pkt0         addr: 10.9.9.1  route: 10.9.9.254
	fw killed   device:              addr:           route:

pktmedium is unbindonclose, so the interface and the address go when
fw's fds close, and the card is left bound to nothing with nothing
reading it.  In a namespace it is harder still: /net answers "i/o on
hungup channel" and bind -a '#I' /net answers "mount/attach disallowed",
because the device mask was dropped before the program started.

So todo item 1 -- "a dead fw takes the network with it", open since the
first commit -- was the requirement written down as a defect.  It is
now design.md and a FAILURE section in fw(8), and the tests assert it,
which is the point: this is exactly the property a later helpful change
reverses without meaning to.  putback() was that change, written and
never run; deleting it removed a fail-open path, not just dead code.

What was actually broken is recovery, and in a way nobody had reached:
a fw that dies leaves its control filesystem mounted, and a corpse of a
mount fails everything asked of it -- including the access() check fw
makes before touching a card, which then refuses the restart:

	fw: /tmp/rdbg/ctl: clone failed; not touching /net/ether1 until it exists

That check exists so fw does not take a card it cannot then serve, and
it was keeping fw from ever coming back.  Now the dead mount is cleared
first, the same way reclaim() clears the pkt interface the same dead fw
left behind: both are its own wreckage.

With that and -a/-g in the service file -- so a restart does not need
the address it just lost -- the whole cycle works and never passes
through open: crash, network down, restart, network up and filtered.
Ten checks, three of which fail against the previous fw.c.  87 pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

6 files changed, 229 insertions, 60 deletions

diff --git a/fw/doc/design.md b/fw/doc/design.md
index 1060ba6..98eca29 100644
--- a/fw/doc/design.md
+++ b/fw/doc/design.md
@@ -3,8 +3,11 @@
 **A firewall for 9front. One program, filtering at a network card, between
 two networks, or in front of one namespace.**
 
-Status: working and tested on the init-test VM, in all three modes. Not
-yet fit to run on a machine you care about — see `todo.md`.
+Status: working on the init-test VM in all three modes, with 87 checks
+in `test/fwtest.rc` covering every bug that has shipped. What is left
+open is in `todo.md`, and none of it is a reason not to run this any
+more; what has never been tested is the wire — a second machine on the
+same segment, and IPv6 anywhere.
 
 ## Why
 
@@ -143,6 +146,41 @@ forbid are dropped. pf and iptables leave them running until they time
 out; that is a wart everyone has to learn, and "I blocked it, why is the
 transfer still going" is the wrong thing to discover at 3am.
 
+**A dead firewall takes the network with it.** There are two things
+that can happen when a firewall dies: the traffic it was filtering
+flows unfiltered, or it stops. Only the second is defensible. A machine
+that is briefly off the network is a machine an operator notices and
+fixes; a machine that is briefly on the network with no rules is a
+machine nobody notices, and it is exactly the moment you built the
+firewall for.
+
+`fw` does the second, in all three modes, and it is the mechanism
+rather than anything careful:
+
+    card:       pktmedium is unbindonclose, so the interface and the
+                address go when fw's fds close.  The card is left
+                bound to nothing and nothing is reading it.
+    gateway:    the same, on both sides at once.
+    namespace:  the mount is hung up, and #I was dropped from the
+                process group's device mask, so the program cannot
+                bind the real stack back in its place.
+
+Measured, not assumed — a card-mode fw killed leaves `device` with
+nothing after it, no address in `ipselftab` and no default route; a
+sandboxed program gets `i/o on hungup channel` from `/net` and
+`mount/attach disallowed` from `bind -a '#I' /net`.
+
+This is why `putback` was deleted rather than repaired. It meant to
+restore the card to the stack as fw exited, which is precisely the
+first option: the machine back on the network with no firewall on it.
+It never ran, so nothing was ever wrong; had it run, it would have been
+wrong every time.
+
+The cost is that recovery needs an address `fw` no longer knows, and
+that is a real cost, paid deliberately. `-a` and `-g` in the service
+file buy it back: crash, network down, restart, network up and
+filtered, and no moment in between where it is up and unfiltered.
+
 **A file server, so it detaches.** `fw` posts to `/srv`, forks the
 server and lets the process you started exit — what 42 of the 47 file
 servers in `/sys/src/cmd` do, and the five that do not are stdio
diff --git a/fw/doc/todo.md b/fw/doc/todo.md
index d101989..6c39e9b 100644
--- a/fw/doc/todo.md
+++ b/fw/doc/todo.md
@@ -4,8 +4,9 @@ Most of this came from code review. Where a finding has been fixed it
 says so; where it has not, it says what is actually true. An item never
 appears in both halves.
 
-`test/fwtest.rc` is a check for every bug that has shipped here — 77 of
-them now. Run it twice in a row after touching anything.
+`test/fwtest.rc` is a check for every bug that has shipped here, and
+for the properties that must not quietly stop being true — 87 of them
+now. Run it twice in a row after touching anything.
 
 ## Fixed, first round
 
@@ -84,7 +85,22 @@ them now. Run it twice in a row after touching anything.
 10. **Promiscuous mode had no filter behind it.** The card must be
     promiscuous for multicast, but every neighbour's unicast was then
     judged, counted and flow-tracked as if it were ours.
-11. **"fw daemonizes, so svc cannot supervise it" was wrong.** `svc`
+11. **"A dead fw takes the network with it" was the requirement, not
+    the bug.** It had been item 1 since the beginning, on the grounds
+    that a dead `fw` leaves the machine with no network. That is the
+    only defensible thing for a firewall to do: the alternative is
+    traffic flowing unfiltered, which is the moment the firewall exists
+    for. All three modes do it, by mechanism rather than by care —
+    `unbindonclose` on the card and the gateway, a hung-up mount plus a
+    dropped `#I` in a namespace — and all three are now measured.
+    `putback`, deleted earlier as dead code, was the fail-open version
+    of this; it never ran, so it was never wrong, but repairing it
+    would have been. What remained was that recovery did not work:
+    `-a` and `-g` are now in the service file so a restart does not
+    need the address it just lost, and `fw` clears the dead mount its
+    own corpse left on the control mountpoint, which was refusing the
+    restart by way of the check meant to protect the card.
+12. **"fw daemonizes, so svc cannot supervise it" was wrong.** `svc`
     has had the detaching shape from the start: `ready=srv:` watches
     the `/srv` name, and for those services the process exiting "is
     normal and never causes a restart". Nothing needed a foreground
@@ -97,29 +113,11 @@ them now. Run it twice in a row after touching anything.
 
 ## Still open
 
-### 1. Nothing puts the card back
-
-Taking a card is destructive and is not undone. The pkt interface is
-`unbindonclose`, so when `fw` stops the interface and the address both
-go, leaving the card bound to nothing and the machine with no network.
-`fw` cannot restart unaided either: the address it would read off the
-card is the address that just vanished.
-
-The cleanup that claimed to handle this was dead code and is gone. The
-answer is not more note handling — it has to be something that outlives
-`fw`, which means the supervisor, with the address stored where `fw`
-does not own it.
-
-This is what makes `restart=always` a half-measure rather than an
-answer: `svc` will start `fw` again, and the new one has no address to
-read off a card that no longer has one. It needs `-a`, or the card
-configured again first. Supervision works; recovery does not.
-
-### 2. One card per fw, and rules cannot name a card
+### 1. One card per fw, and rules cannot name a card
 
 Wants repeatable `-e` and an `ifc=` attribute, together.
 
-### 3. Tflush is not implemented
+### 2. Tflush is not implemented
 
 A request `fw` is blocked on cannot be abandoned. Only affects
 namespace mode, where waiting for an inbound connection is the one
@@ -127,11 +125,18 @@ operation that blocks indefinitely. ~80 lines, with a race that cannot
 be fully closed: between the syscall returning and the handler clearing
 its entry, a note may land on a worker that has moved on.
 
-### 4. Positional `delete n` renumbers
+### 3. Positional `delete n` renumbers
 
 Inherent to positional deletion; iptables has it too. Said out loud
 here rather than fixed.
 
+## Decided, and now written down
+
+**A dead `fw` leaves the network down, not open.** See `design.md` and
+FAILURE in `fw(8)`. The tests assert it in card and namespace mode,
+because it is exactly the kind of property a later helpful change would
+reverse without meaning to.
+
 ## Never tested
 
 - **The wire side of card mode.** The suite can take a spare card — it
diff --git a/fw/lib/svc/fw.ether0 b/fw/lib/svc/fw.ether0
index 93de412..627b1d3 100644
--- a/fw/lib/svc/fw.ether0
+++ b/fw/lib/svc/fw.ether0
@@ -8,9 +8,12 @@
 # when the last of fw exits, whether that is the wire failing, the
 # control filesystem going away, or a note.
 #
-# It has to start after the card has an address, because that is where
-# fw reads the address, mask and gateway from - and before anything
-# dials, because a connection made before fw is up was never filtered.
+# It has to start before anything dials: a connection made before fw is
+# up was never filtered.
+#
+# If fw dies the machine has no network until it is restarted.  That is
+# the intended behaviour and not a fault - see FAILURE in fw(8) - so
+# restart=always is what you want, not a supervisor that gives up.
 #
 svc=fw.ether0
 	exec=/bin/fw
@@ -20,6 +23,18 @@ svc=fw.ether0
 	args=/mnt/fw/ether0
 	args=-e
 	args=/net/ether0
+#
+# The address and gateway, rather than letting fw read them off the
+# card.  A fw that has died has taken the address with it, so a
+# restarted fw has nothing to read; told them, it can restart into the
+# bare card it left behind.  Without them restart=always brings fw back
+# only to fail, and the machine stays off the network - which is the
+# safe direction, but not a working one.
+#
+	args=-a
+	args=10.0.2.15/24
+	args=-g
+	args=10.0.2.2
 	args=/lib/fw/host.ndb
 	needs=ipconfig
 	ready=srv:fw.ether0
diff --git a/fw/man/fw b/fw/man/fw
index 9279c60..0b11ac9 100644
--- a/fw/man/fw
+++ b/fw/man/fw
@@ -348,6 +348,44 @@ everything else instead.
 .PP
 .B /lib/fw/fw.rc
 does this in the right order: private namespace, mount, drop, exec.
+.SH FAILURE
+When
+.I fw
+dies, the traffic it was filtering stops.
+It does not carry on unfiltered.
+.PP
+Filtering a card, the interface that replaced the card is unbound when
+.I fw
+stops and the address goes with it, so the machine has no network at
+all: the card is bound to nothing and nothing is reading it.
+Between two networks, the same happens on both sides and nothing
+crosses.
+Filtering a namespace, the mount is hung up and every path into
+.B /net
+fails; the program cannot put the real one back either, because it was
+denied
+.B #I
+before it started \- see
+.BR CONTAINMENT .
+.PP
+This is deliberate, and it is the reason
+.I fw
+does not try to give the card back as it exits.
+A machine that is briefly off the network is a machine somebody
+notices; a machine that is briefly on the network with no rules is the
+thing the firewall was installed to prevent.
+.PP
+The cost is that
+.I fw
+cannot restart unaided: the address it would read off the card is the
+address that has just gone.
+Give
+.B -a
+and
+.B -g
+so that it does not have to read them, and it can be restarted into a
+machine it has itself left bare \- see
+.BR SUPERVISION .
 .SH SUPERVISION
 .I Fw
 posts its filesystem on
@@ -373,6 +411,10 @@ svc=fw.ether0
 	args=/mnt/fw/ether0
 	args=-e
 	args=/net/ether0
+	args=-a
+	args=10.0.2.15/24
+	args=-g
+	args=10.0.2.2
 	args=/lib/fw/host.ndb
 	needs=ipconfig
 	ready=srv:fw.ether0
@@ -395,17 +437,20 @@ What such a mode would buy is a pid to watch, and the
 name is the better signal: it survives the process that made it and
 answers the question that matters.
 .PP
-Restarting is not free.
-.B Restart=always
-will bring
+The address and gateway are given here rather than read off the card,
+which is the one place that is worth doing.
+A
 .I fw
-back, but taking a card is destructive and is not undone, so the card
-must be configured again before the new
+that has died has taken the address with it, so a restarted
 .I fw
-can read an address off it \- see
-.B BUGS
-and
-.IR ipconfig (8).
+has nothing to read; told them, it can restart into the bare card it
+left behind, and does \- including clearing away its own control mount,
+which a dead
+.I fw
+leaves behind as a mount that fails everything asked of it.
+Without them the restart fails and the machine stays off the network,
+which is the safe direction but not a working one.
+.PP
 A copy of this file per card is the shape to use; one
 .I fw
 per card is the only shape there is.
@@ -549,26 +594,14 @@ so a fragmented IPv6 datagram does not cross.
 IPv4 fragments do: the first one carries the transport header and
 decides, and the rest of the datagram inherits what it decided.
 .PP
-Taking a card is destructive and is not undone.
-The interface that replaces it is unbound when
-.I fw
-stops, and the address goes with it, so a
-.I fw
-that is killed leaves the card bound to nothing and the machine with no
-network.
-It cannot be restarted unaided either: the address it would have read
-off the card is the address that has just been lost, so it must be told
-one with
-.BR -a ,
-or the card configured again with
-.IR ipconfig (8)
-first.
-Nothing puts the card back.
-Whatever restarts
-.I fw
-must configure the card first, with the address it had, which
+Taking a card is destructive and is not undone, on purpose; see
+.BR FAILURE .
+A
 .I fw
-no longer knows.
+that is killed therefore leaves the machine with no network, and must
+be told an address with
+.B -a
+to start again.
 .PP
 The first packet to an unresolved next hop is dropped while
 .I fw
diff --git a/fw/src/fw.c b/fw/src/fw.c
index bb97960..800e004 100644
--- a/fw/src/fw.c
+++ b/fw/src/fw.c
@@ -1463,9 +1463,26 @@ threadmain(int argc, char **argv)
 		 * that has no card behind it and no network at all, which
 		 * is a bad way to discover a typo in -m.
 		 */
-		if(access(mtpt, AEXIST) < 0)
-			sysfatal("%s: %r; not touching %s until it exists",
-				mtpt, etherdev);
+		if(access(mtpt, AEXIST) < 0){
+			/*
+			 * A fw that died left its control filesystem mounted
+			 * here and what remains is a corpse: the name is
+			 * there and every operation on it fails, including
+			 * the check above.  So the restart after a crash was
+			 * refused by the very test meant to keep fw from
+			 * taking a card it could not then serve.  Clear it,
+			 * the same way reclaim() clears the pkt interface the
+			 * same dead fw left behind; both are its wreckage,
+			 * and a firewall that cannot restart into its own
+			 * wreckage stays down for good.
+			 */
+			unmount(nil, mtpt);
+			if(access(mtpt, AEXIST) < 0)
+				sysfatal("%s: %r; not touching %s until it exists",
+					mtpt, etherdev);
+			fprint(2, "fw: cleared %s, left mounted by an earlier fw\n",
+				mtpt);
+		}
 
 		fprint(2, "fw: filtering packets on %s\n", etherdev);
 		syslog(0, "fw", "started, filtering %s for %s", etherdev, etheraddr);
diff --git a/fw/test/fwtest.rc b/fw/test/fwtest.rc
index 3f80242..bd7423d 100755
--- a/fw/test/fwtest.rc
+++ b/fw/test/fwtest.rc
@@ -547,10 +547,71 @@ echo '== a card'
 		r=`{awk '$1 == "0.0.0.0" {print $3}' $tmp/nC/iproute | sed 1q}
 		check 'and the route that went with the card' 10.9.9.254 $"r
 
+		# The question a firewall has to answer about its own death:
+		# does the network come back open, or not at all.  A note is
+		# a fair stand-in for a crash - what matters is the kernel
+		# closing what fw held, which happens either way.  No other
+		# fw is running here; the leak checks at the end say so.
+		kill fw | rc
+		sleep 3
+		r=`{ps | awk '$NF == "fw"' | wc -l}
+		check 'killing one part kills the firewall' 0 $"r
+		r=`{awk '/4u$/ {print $1}' $tmp/nC/ipselftab}
+		check 'a dead fw leaves the stack no address' '' $"r
+		r=`{awk '$1 == "0.0.0.0" {print $3}' $tmp/nC/iproute | sed 1q}
+		check 'and no route: the network is down, not open' '' $"r
+		# "device" with nothing after it: the pkt interface is gone
+		# and the card was never given back
+		r=`{sed 1q $tmp/nC/ipifc/0/status | awk '{print $2}'}
+		check 'and nothing bound to hand the card back' maxtu $"r
+
+		# recovery, without ever passing through open.  The dead fw
+		# also left its control filesystem mounted, and a corpse of a
+		# mount fails every operation on it - including the check fw
+		# makes before touching a card.  So a restart has that to get
+		# past too.
+		@{ $fw -n $tmp/nC -e /net/ether1 -m $mtpt $tmp/empty.ndb } >[2]/dev/null
+		if(~ $status '')
+			r=started
+		if not
+			r=refused
+		check 'it will not restart on a guess' refused $"r
+
+		$fw -n $tmp/nC -e /net/ether1 -a 10.9.9.1/24 -g 10.9.9.254 \
+			-m $mtpt $tmp/empty.ndb >[2] $tmp/rerr
+		sleep 2
+		r=`{grep -c 'cleared '^$mtpt $tmp/rerr}
+		check 'the control mount the dead one left is cleared away' 1 $"r
+		r=`{awk '/4u$/ {print $1}' $tmp/nC/ipselftab}
+		check 'but does when told the address the card no longer has' 10.9.9.1 $"r
+		r=`{awk '$1 == "0.0.0.0" {print $3}' $tmp/nC/iproute | sed 1q}
+		check 'route and all' 10.9.9.254 $"r
+
 		stopfw $tmp/nC
 	}
 }
 
+echo '== a dead firewall leaves a sandbox with no network either'
+@{
+	rfork n
+	$fw $tmp/empty.ndb >[2]/dev/null
+	# the drop is per process group and this block has its own, so it
+	# reaches this shell and its children and nothing else
+	echo 'chdev' '&~' 'Iluσ' > /dev/drivers
+	kill fw | rc
+	sleep 3
+	if(@{ ls /net/tcp/clone } >/dev/null >[2]/dev/null)
+		r=there
+	if not
+		r=gone
+	check 'the filtered /net dies with it' gone $"r
+	if(@{ bind -a '#I' /net } >[2]/dev/null)
+		r=allowed
+	if not
+		r=refused
+	check 'and the real one cannot be bound back in its place' refused $"r
+}
+
 echo '== supervision'
 # fw daemonizes, like every other Plan 9 file server: the process you
 # exec posts to /srv, forks the server and exits.  A supervisor watches