test: put fw on a wire, with another machine on the other end

Everything in fwtest.rc happens on one machine, which is as far as it
goes: it can take a spare card and watch what fw does to the
interfaces, but it cannot make a neighbour send anything.  Four things
were listed as never tested for that reason.  Three of them now are.

run.sh -gw and -lan join two VMs into a point-to-point segment, so the
peer is a plain overlay of the base with an address and nothing else.

ARP first, since it is the part with no other explanation: the stack
that owns 10.9.9.1 has a pkt interface and no ethernet, so nothing else
on that segment can answer for it.  The peer's arp table:

	ether  OK   10.9.9.1     52540087c8c1     10.9.9.2

which is the firewall's card.  The first ping takes about a second and
the rest are sub-millisecond -- fw ARPing for the peer before it can
reply, and dropping the first one while it asks, which is the drop no
rule caused in fw(8) BUGS, until now only reasoned about.

Then filtering as against forwarding, which cannot be seen from inside
at all.  With allow=in proto=icmp alone, a TCP connect from the peer
sits for 290 seconds and times out; prepend a rule through ctl and the
same connect is refused in 2.  Refused is the far stack's RST, so the
packet arrived; timed out is fw dropping it in silence.

And frames addressed to somebody else, which is the commit that shipped
saying it could not be tested without a second machine.  A stack only
addresses frames to the mac it resolved, so rawether.c forges one.  A
promiscuous reader on the firewall machine sees both frames, which had
to be confirmed first -- a frame that never arrived looks exactly like
one that was filtered, and my first attempt at this drew the wrong
conclusion from precisely that.  Same rig, same frames, same rules:

	                 to fw's card   to nobody's
	no check              +3            +3
	with the check        +2             0

test/wire.md is the procedure, including the two ways I wasted time:
fw's control files need -s to be reachable from another shell, and
anything that leaves fw holding a pipe waits forever for a program that
has already detached.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

3 files changed, 230 insertions, 10 deletions

diff --git a/fw/doc/todo.md b/fw/doc/todo.md
index 6c39e9b..e83167d 100644
--- a/fw/doc/todo.md
+++ b/fw/doc/todo.md
@@ -137,21 +137,39 @@ FAILURE in `fw(8)`. The tests assert it in card and namespace mode,
 because it is exactly the kind of property a later helpful change would
 reverse without meaning to.
 
+## Tested on a wire
+
+Two VMs on one ethernet segment, `run.sh -gw` and `-lan`; the procedure
+is `test/wire.md` and it is worth keeping, because none of this can be
+seen from one machine.
+
+- **ARP for an address whose card has been taken.** The peer's arp
+  table says `10.9.9.1 → 52540087c8c1`, which is the firewall's card
+  and not the peer's own. Nothing else could have answered.
+- **The first packet to an unresolved hop is dropped**, exactly as
+  fw(8) BUGS says: first ping ~1s while fw ARPs for the peer, the rest
+  sub-millisecond.
+- **Filtering rather than forwarding.** `allow=in proto=icmp` alone:
+  ping works, TCP sits for 290 seconds and times out. Add the rule
+  through `ctl` and the same connect is refused in 2 seconds — refused
+  is the far stack's RST, so the packet arrived.
+- **Frames addressed to somebody else are ignored.** Forged with
+  `test/rawether.c`. With the destination check the counter does not
+  move; without it, a frame for nobody is judged exactly like one for
+  us.
+
 ## Never tested
 
-- **The wire side of card mode.** The suite can take a spare card — it
-  does, and everything up to the wire is now covered — but it cannot
-  make a neighbour send to it. Anything that depends on another machine
-  on the same segment is unproven: the destination-address filter, ARP
-  against a real peer, broadcast.
-- **Broadcast handling.** Written, reviewed, never observed crossing
-  `fw`. The mapping is the standard one and normal traffic is
-  unaffected.
+- **Broadcast handling.** Written, reviewed, still never observed
+  crossing `fw`. The rig above could now show it — a broadcast from the
+  peer is one `rawether` call away — and it has not been done.
 - **IPv6 traffic**, in any mode. Under `-e` it cannot work at all:
   there is no neighbour discovery, so v6 unicast is dropped. IPv6
   extension headers are not walked, so v6 fragments do not cross.
-- **The gateway with two real machines**, and a real second NIC
-  carrying real traffic.
+- **The gateway between two real machines.** The wire rig proves card
+  mode; the gateway has still only been run between two synthetic
+  stacks on one machine.
+- **A real NIC on real hardware**, as against an emulated e1000.
 
 ## Deliberately not doing
 
diff --git a/fw/test/rawether.c b/fw/test/rawether.c
new file mode 100644
index 0000000..540d0b6
--- /dev/null
+++ b/fw/test/rawether.c
@@ -0,0 +1,97 @@
+/*
+ * rawether - put one ethernet frame on the wire, addressed as told.
+ *
+ * A stack will only ever address a frame to the mac it resolved, so a
+ * frame addressed to somebody else has to be forged.  That is the only
+ * way to ask a firewall on a card whether it is looking at frames that
+ * are not for it.
+ *
+ *	rawether /net/ether1 <dstmac> <srcip> <dstip>
+ *
+ * The payload is an icmp echo request, which is enough to be counted.
+ */
+#include <u.h>
+#include <libc.h>
+#include <ip.h>
+
+enum { Ehdrlen = 14, Eminlen = 60, Etip4 = 0x0800 };
+
+static ushort
+csum(uchar *p, int n)
+{
+	ulong s;
+	int i;
+
+	s = 0;
+	for(i = 0; i+1 < n; i += 2)
+		s += (p[i]<<8) | p[i+1];
+	if(i < n)
+		s += p[i]<<8;
+	while(s >> 16)
+		s = (s & 0xFFFF) + (s >> 16);
+	return ~s;
+}
+
+void
+main(int argc, char **argv)
+{
+	uchar f[Eminlen], dst[6], src[6], sip[IPaddrlen], dip[IPaddrlen];
+	char path[128], buf[64];
+	int cfd, dfd, n, conn;
+
+	if(argc != 5){
+		fprint(2, "usage: rawether /net/etherN dstmac srcip dstip\n");
+		exits("usage");
+	}
+	if(parseether(dst, argv[2]) < 0)
+		sysfatal("%s: bad ethernet address", argv[2]);
+	if(parseip(sip, argv[3]) == -1 || parseip(dip, argv[4]) == -1)
+		sysfatal("bad ip address");
+
+	snprint(path, sizeof path, "%s/clone", argv[1]);
+	if((cfd = open(path, ORDWR)) < 0)
+		sysfatal("open %s: %r", path);
+	if((n = read(cfd, buf, sizeof buf - 1)) <= 0)
+		sysfatal("read %s: %r", path);
+	buf[n] = '\0';
+	conn = atoi(buf);
+	if(fprint(cfd, "connect -1") < 0)
+		sysfatal("connect -1: %r");
+	snprint(path, sizeof path, "%s/%d/data", argv[1], conn);
+	if((dfd = open(path, ORDWR)) < 0)
+		sysfatal("open %s: %r", path);
+	snprint(path, sizeof path, "%s/addr", argv[1]);
+	if((n = open(path, OREAD)) < 0)
+		sysfatal("open addr: %r");
+	if(read(n, buf, 12) != 12)
+		sysfatal("read addr: %r");
+	close(n);
+	buf[12] = '\0';
+	if(parseether(src, buf) < 0)
+		sysfatal("unparseable card address %s", buf);
+
+	memset(f, 0, sizeof f);
+	memmove(f, dst, 6);
+	memmove(f+6, src, 6);
+	f[12] = Etip4 >> 8;
+	f[13] = Etip4;
+
+	f[14] = 0x45;			/* v4, 20 byte header */
+	hnputs(f+16, 28);		/* total length */
+	f[22] = 64;			/* ttl */
+	f[23] = 1;			/* icmp */
+	memmove(f+26, sip+IPv4off, 4);
+	memmove(f+30, dip+IPv4off, 4);
+	hnputs(f+24, csum(f+14, 20));
+
+	f[34] = 8;			/* echo request */
+	hnputs(f+38, 0x1234);		/* id */
+	hnputs(f+40, 1);		/* seq */
+	hnputs(f+36, csum(f+34, 8));
+
+	if(write(dfd, f, sizeof f) != sizeof f)
+		sysfatal("write: %r");
+	fmtinstall('E', eipfmt);
+	print("sent %d bytes to %E\n", (int)sizeof f, dst);
+	exits(nil);
+}
diff --git a/fw/test/wire.md b/fw/test/wire.md
new file mode 100644
index 0000000..b4494f3
--- /dev/null
+++ b/fw/test/wire.md
@@ -0,0 +1,105 @@
+# Testing fw on a wire
+
+`fwtest.rc` runs on one machine and can go no further than the card: it
+can take a spare one and watch what fw does to the interfaces, but it
+cannot make a neighbour send anything. Everything past that — ARP
+against something that will answer, real traffic being filtered, frames
+addressed to somebody else — needs a second machine on the same
+segment.
+
+`run.sh` will build one. `-gw` listens, `-lan` connects, and qemu joins
+the two into a point-to-point ethernet segment the host is not on. The
+LAN card is `ether1` on both.
+
+    ./newvm.sh peer
+    ./run.sh -headless -gw  init-test.qcow2 &
+    ./run.sh -headless -lan peer.qcow2 &
+
+The peer can be a plain overlay of the base: it needs no fw, only an
+address. Both cards must have distinct ethernet addresses, which
+`run.sh` derives from the VM name — before it did that, every guest had
+qemu's default and two machines on one segment shared a MAC, which is
+invisible point-to-point and makes every check below meaningless.
+
+## The rig
+
+On the firewall machine, a stack of its own so the real network is
+untouched, and fw on the spare card:
+
+    bind -a '#l1' /net
+    bind -a '#I63' /tmp/nW
+    @{ conv=`{cat /fd/0}
+       echo -n 'bind ether /net/ether1' >[1=0]
+       echo -n 'add 10.9.9.1 255.255.255.0' >[1=0] } <>[0] /tmp/nW/ipifc/clone
+    echo 'allow=in	proto=icmp' > /tmp/r.ndb
+    fw -s fw.wire -n /tmp/nW -e /net/ether1 -m /tmp/ctl /tmp/r.ndb \
+        </dev/null >/dev/null >[2]/dev/null
+
+`-s` matters: fw's control files are otherwise only reachable through
+the mount in the namespace that started it, and that namespace is gone
+by the time you want to look. With a `/srv` name any shell can
+`mount /srv/fw.wire /tmp/wctl`.
+
+Redirect all three descriptors. fw daemonizes, so anything that leaves
+it holding a pipe — `fw ... | grep`, or a command substitution around
+it — waits for a firewall that is not going to exit.
+
+On the peer, the other end of the segment:
+
+    bind -a '#l1' /net
+    bind -a '#I64' /tmp/np
+    @{ conv=`{cat /fd/0}
+       echo -n 'bind ether /net/ether1' >[1=0]
+       echo -n 'add 10.9.9.2 255.255.255.0' >[1=0] } <>[0] /tmp/np/ipifc/clone
+    rfork n; bind /tmp/np /net
+
+## What it shows
+
+**ARP for an address whose card has been taken.** Nothing else is
+answering for 10.9.9.1 — the stack that owns it has a `pkt` interface
+and no ethernet — so if `ip/ping 10.9.9.1` works at all, fw answered.
+The peer's `/net/arp` says whose:
+
+    ether  OK   10.9.9.1     52540087c8c1     10.9.9.2
+
+which is the firewall's *card*, not the peer's own. The first ping
+takes about a second and the rest are sub-millisecond: fw has to ARP
+for the peer before it can send the reply, and drops the first one
+while it asks. That is the one drop no rule caused, in fw(8) BUGS.
+
+**Filtering, as against forwarding.** With `allow=in proto=icmp` alone,
+a TCP connect from the peer sits for 290 seconds and gives up:
+
+    connection timed out
+
+Add the rule through the running firewall and the same connect answers
+in 2 seconds:
+
+    echo -n 'prepend allow=in proto=tcp lport=17019' > /tmp/wctl/ctl
+    connection refused
+
+"Refused" is the far stack's RST, so the packet arrived; "timed out" is
+fw dropping it silently. That difference is the whole point of the
+program, and it is the only way to see it from outside.
+
+**Frames addressed to somebody else.** A stack only ever addresses a
+frame to the MAC it resolved, so this one has to be forged:
+`rawether.c`, in this directory, writes a single frame with whatever
+destination you name.
+
+    rawether /net/ether1 52540087c8c1 10.9.9.2 10.9.9.1   # fw's card
+    rawether /net/ether1 525400aabbcc 10.9.9.2 10.9.9.1   # nobody's
+
+Watch `passed` in `/tmp/wctl/stats` across each. The card is
+promiscuous and has to be, so both frames arrive — a promiscuous reader
+on the firewall machine sees both, which is worth confirming first, or
+a frame that never arrived looks exactly like one that was filtered:
+
+	52 54 00 aa bb cc 52 54 00 37 df 23 08 00 45 00
+
+With the destination check in `etherin`, the second frame moves nothing.
+Without it, both move the counter by the same amount:
+
+	                 to fw's card   to nobody's
+	no check              +3            +3
+	with the check        +2             0