diff options
Diffstat (limited to 'svc/doc/design.md')
| -rw-r--r-- | svc/doc/design.md | 437 |
1 files changed, 437 insertions, 0 deletions
diff --git a/svc/doc/design.md b/svc/doc/design.md new file mode 100644 index 0000000..343d23a --- /dev/null +++ b/svc/doc/design.md @@ -0,0 +1,437 @@ +# init + +**init is the service manager. It provides svcfs if you want to interact with services.** + +Status: design. Nothing implemented yet. + +## Why + +A machine is a set of running services. That's all it is. 9front instead has +three half-mechanisms that each know a piece of that and never talk to each +other. + +`service=` in `plan9.ini` picks between `/rc/bin/termrc` and `/rc/bin/cpurc`. +One word decides "what kind of machine is this." + +`/rc/bin/service/` is a directory of scripts named by port, where the enable bit +is a `!` prefix on the filename: + + /rc/bin/service/!tcp23 disabled + /rc/bin/service/tcp17019 enabled + +Everything else is a bare line in a shell script — `plumber`, `webfs`, `ndb/cs`, +`ndb/dns`, `aux/timesync` — started once, owned by nobody, restarted never. + +So: nothing is supervised, nothing has state, nothing has logs, nothing has +dependencies, and nothing can be managed remotely without a shell and an editor. + +A real example from this project. The serial control shell was one line in +`termrc`, non-interactive: + + rc </dev/eia0 >/dev/eia0 >[2=1] & + +One malformed command produced a syntax error, rc exited, and the machine was +unreachable over serial for the rest of the session. The fix was to make it +interactive and wrap it in a restart loop: + + while(){ rc -i </dev/eia0 >/dev/eia0 >[2=1]; sleep 1 }& + +That line is a service manager for exactly one service. Every daemon on the box +wants the same thing and none of them get it. + +## Roles are vocabulary + +"Terminal", "cpu server", "auth server", "file server" are not kernel concepts. +The kernel does not know which one it is running. Each is a bundle of userspace +programs: + +| Role | What it actually is | +|---|---| +| auth server | `auth/keyfs` + `auth/authsrv` on 567 | +| cpu server | a listener on 17010, plus `exportfs` | +| file server | `hjfs`/`cwfs`/`gefs` + a 9P listener on 564 | +| terminal | a window system, if you want one | + +Nothing ever stopped one machine being all four. The words are the obstacle: +they imply a machine *is* one thing, and that exactly one machine holds each +role in a single administrative domain. + +We delete the words. Role is emergent and plural. + +## How it works + +`init` runs services. It: + +1. reads service definitions from `/lib/svc` +2. starts the enabled ones in dependency order +3. supervises them, restarting per policy +4. serves `svcfs` — posted at `/srv/svc`, mounted at `/mnt/svcs` +5. handles `reset` and `reexec` + +None of that depends on how it was started, and an instance can be run by +anyone over any directory of services in its own namespace. Run as the program +that brings a machine up, it does four more things: builds the base namespace, +offers a rescue console, halts the root file server last, and carries out +`halt` and `reboot`. + +**init is not pid 1 on 9front today.** Pid 1 is `rc` running `/bin/bootrc`, +sitting in `Await`; init is an ordinary child — pid 143 on the machine this was +checked against. Since the boot chain is being replaced anyway, whether init +runs as pid 1 is a decision rather than an inherited fact, and nothing in this +design requires it. + +`svcfs` is a namespace, not a program. There is no second daemon. It's what init +looks like from outside, the same way `/proc` is what the kernel's process table +looks like from outside. + +**The control plane is optional.** Services start at boot whether or not anything +ever mounts `/mnt/svcs`. If posting the filesystem fails, init logs it and keeps +supervising. A bug in the 9P layer must never be a boot failure. + +## A service + +`/lib/svc/dns`, in ndb format — the parser already exists and it's the same +shape as factotum keys and `/lib/ndb/local`: + + svc=dns + exec=/bin/ndb/dns + args=-r + needs=cs + ready=srv:dns + restart=always + enable=yes + +Ten attributes exist in total: `svc exec args needs ready restart ns user +enable stop`. There are deliberately no tuning knobs — restart delays and +readiness timeouts are constants in the code until something proves it needs to +be configurable. + +An unknown attribute is an **error**, not a warning. ndb has no schema, so +`exce=/bin/ndb/dns` parses fine as ndb and means nothing. Ignoring unknowns is +how a typo silently does nothing forever. + +## Three shapes of service + +Not everything is a daemon that stays in the foreground, and the difference is +declared rather than guessed: + +| Shape | Liveness is | On exit | +|---|---|---| +| foreground | the pid | restart per policy | +| detaching | the `/srv` file | not a failure — don't restart | +| oneshot | nothing, it's done | never restart | + +Detaching matters because the `postmountsrv` idiom forks a server proc and lets +the original exit *on purpose*. A supervisor that assumes exit means death will +restart-loop something that is running fine. + +Which is why **readiness and liveness are separate questions**. `ready=srv:dns` +answers "is it up yet"; the same file answers "is it still up" for a service +whose pid is already gone. + +## Namespaces and identity + +Each service gets `rfork(RFNAMEG)` and a namespace built from its `ns=` file via +`newns` — the same machinery `cpu` already uses. + +This is the part that isn't a systemd transliteration. Two services on one +machine can hold completely different views of the filesystem: different `/net`, +different `/srv`, different roots. That is what would let one box participate in +two grids at once, each with its own `/mnt/factotum` and its own identity — +impossible today, because a machine has one hostowner, one `authdom` and one +namespace. + +Identity comes from the capability device (`/dev/caphash`, `/dev/capuse`), the +path `cpu` and `rx` already use, since Plan 9 has no setuid. That means init runs +as hostowner and is security-critical. Not a detail to discover later. + +**Services consume namespaces; they never construct the shared one.** A service +that runs `bind` inside its own namespace group affects nobody, and would appear +to work while doing nothing. Base namespace construction belongs to init, before +any service starts. + +## Talking to it + +The filesystem is the mechanism, commands are what people type. Both. + + svc list services, their state, and why + svc start dns + svc restart dns + svc log dns + halt stop services, halt the fs, power off + shutdown alias for halt + reboot ... and restart the kernel + reset stop services, then start them again + +Commands are pure translation — a few lines that open a file and write a word. +If `svc restart dns` can ever do something a write to `/mnt/svcs/dns/ctl` +cannot, we've built two interfaces and they will drift. + +`reset` is worth naming because it's the cheap one: it's the stopping half +followed by the starting half, both of which already exist. No new machinery. + +## Ctl files say what they accept + +The worst thing about Plan 9 ctl files is that they're write-only channels for +magic strings. `/net/tcp/0/ctl`, `#S/sdctl`, `/dev/mousectl` — the only way to +learn the verbs is to read source. + +So every ctl file here reads back its own command list: + + % cat /mnt/svcs/dns/ctl + start + stop + restart + note <string> post a note to the process + +and every rejected write names the alternatives: + + % echo frobnicate >/mnt/svcs/ctl + echo: write error: unknown command "frobnicate"; try + halt reboot reset start stop restart enable disable reexec + +This deviates from convention — most Plan 9 ctl files return nothing or return +state on read. We can afford to spend the read on documentation because `status` +carries the state. Given that the convention is the complaint, it's the right +one to break. + +## State is ndb too + +`/proc/mdstat` on Linux is the canonical disease: positional, bracket-encoded, +with an ASCII progress bar inside the data. Plan 9 does it too — +`/net/ipifc/0/status` opens with genuine attribute/value pairs (`maxtu 1514 +sendra 0`) and then collapses into unlabelled positional columns the moment +there's a list. + +So status output is ndb, not columns: + + svc=dns state=running pid=231 + svc=authsrv state=failed restarts=3 exit='cannot open /adm/keys' + svc=listen state=waiting needs=cs + +Self-describing, quoting handles the exit string, and fields can be added +without breaking readers. `svc` with no arguments prints aligned columns for +humans; the file stays ndb for programs. + +Nothing in the tree currently serves ndb — all 15 libndb consumers are readers +of static config. This would be the first, which also means it shouldn't be +called a convention until a second, genuinely different server carries it. + +## Configuration goes one way + +`/lib/svc` is the only source of truth. The filesystem accepts *verbs*, not +configuration. No writing `exec=` into `/mnt/svcs/dns/args` and having init +persist it back — that's two paths to one state, and they will disagree. It's +also how the system already behaves: you configure an interface by writing +`/net/ipifc/0/ctl`, not by editing `status`. + +`reload` re-reads `/lib/svc`, applies what parses, and reports what doesn't: + + % echo reload >/mnt/svcs/ctl + echo: write error: /lib/svc/dns:4: unknown attribute "exce" + +One bad file never costs you the whole machine — same behaviour at boot. + +## Permissions + +init is the file server, so it checks the 9P attach identity itself. One +mechanism covers local writes and remote mounts; there's no separate "is this +allowed remotely" path to get wrong later. + + /mnt/svcs/ctl 0644 hostowner halt, reboot, reset + /mnt/svcs/<n>/ctl 0664 hostowner:user start, stop, restart + /mnt/svcs/<n>/status 0444 anyone + /mnt/svcs/<n>/log 0640 + +Note the ctl files are world *readable*. Restricting reads was the first +thing tried and it immediately broke the rule above — reading a ctl file is +how you learn what it accepts, which is no use if only one user may do it. +Writing is the thing that wants restricting. + +This isn't cosmetic: if `/srv/svc` is exported, anyone who can mount it can halt +the machine. + +## Shutdown + +`fshalt` is the proof that this is needed. It spends ninety lines rediscovering +the system at shutdown — a hardcoded glob for every file server 9front has +shipped (`cwfs*cmd`, `hjfs*cmd`, `ext4*cmd`, `gefs*cmd`, `fscons*`), plus +hardcoded knowledge of each one's stop protocol (`echo halt` for most, +`echo fsys all halt` and a `sleep 2` for fscons). + +Then the strangest part: + + ramfs + cp /bin/echo /tmp + cp /bin/rc /tmp + # put this in a shell function so this rc script doesn't get read + # when it's no longer accessible + fn x { ... } + +It copies its own tools into a ramfs and hides its body in a function so rc +won't re-read the script — because the shutdown logic lives *on the filesystem +it is shutting down*. init has none of that problem: it's resident, it's process +1, and it re-reads nothing. + +So shutdown is: + +1. stop services in reverse dependency order — `hangup` note, wait, then `kill` +2. halt the adopted file server +3. power off or reboot + +A wedged service must never stall shutdown, because the thing being protected is +a clean filesystem. Timeout, hard kill, proceed. + +**Adopted services.** The root file server is started by `bootrc` before init +exists. init isn't its parent, can't `await` it, can't `kill` it — it can only +stop it the way fshalt does, by writing its `/srv` ctl file. So it's a second +category: known but not owned, stop-only, always last. Each file server declares +its own `stop=` rather than init carrying a table of how to kill everyone else. + +`fshalt` goes away. + +## Logging + +Service output is appended to `/log/<name>`. `/mnt/svcs/<name>/log` is a view of +that file, and a blocking read at EOF gives live tailing. + +**Rotation is a non-goal.** Appending is twenty lines; rotation is a subsystem, +and it isn't going in init. Files grow until something else deals with them. + +Two places this doesn't work, both the same problem: there's no filesystem yet, +or there won't be one shortly. Early boot messages go to console. Once teardown +starts, logging to disk stops and the rest goes to console. + +## Testing without booting + +Nothing about supervising services requires being process 1: + + init -d /tmp/svc.test -s svc.test -m /mnt/svcs.test + +runs the real init as an ordinary user, supervising toy services, serving a test +filesystem. No reboot, no risk. + +That only works if it's designed in, so it's a rule: **no hardcoded paths, no +assumption of being pid 1, every path a flag.** The only parts needing a real +boot are base namespace construction, the rescue console, and adopted-fs +shutdown — which is the argument for keeping those three thin and separable. + +This is where recoverability actually comes from, not from splitting init into +two programs. + +## Implementation + +C, and it isn't close: the program that brings a machine up should need nothing but libc at runtime. + +- **lib9p** — the file server. `threadpostmountsrv`, fill in a `Srv`. +- **libndb** — service definitions and status output. +- **libauth** — `newns`. +- **libthread** — `threadwaitchan()` hands you a channel of `Waitmsg*`, so the + main loop is literally a select over "a child exited" and "a 9P request + arrived." The concurrency model matches the problem. + +Roughly 1500–2500 lines. Small enough to hold in your head, which for +something this hard to restart is the point. + +Services written in rc are just `exec=/bin/rc args=/bin/foo`. **An rc service +script must `exec` its final program, not background it** — otherwise init +supervises rc, rc exits immediately, and the real work is orphaned. + +## Getting there + +Staged because you need a working machine tomorrow, not because the end state +keeps any of it. First: init starts `svcfs` and supervises a handful of services +alongside the existing `termrc`, fully reversible. Then service definitions +replace `/rc/bin/service/` and the loose daemon lines. Then `termrc`, `cpurc` +and `service=` are deleted. + +## Open questions + +- **`reexec` state handover.** Init replacing itself without dropping services + means supervision state survives the exec — handed over in memory, or + re-derived from `/proc` and `/srv`. The genuinely hard part. +- **Detaching liveness.** Plan 9 has no file-change notification, so noticing + that a detached service's `/srv` file vanished means polling, or only + noticing on a `status` read. +- **Who restarts init?** If init dies, nothing catches it. Confirm `bootrc` + honours `init=` in `plan9.ini` — that's the escape hatch for a bad init. +- **A machine-readable interface description.** An ndb file next to `ctl` + describing fields, actions, and which files block, so a generic client can + render a UI over any 9P server. Not part of init, but init is a reasonable + first carrier — and it shouldn't be frozen until a second server carries it + too. + +## Namespaces are the capability set + +This started as a federation feature and turned out to be the privilege +mechanism, which is a better justification and changes the defaults. + +In Plan 9 there is no access control for devices beyond what is in your +namespace. If `#S` is not bound, you cannot touch storage — not "you get +permission denied", but there is no name to open. That is stronger than +Unix, where `/dev/sda` exists for everyone and mode bits are the only guard. + +But termrc binds all nine kernel devices into `/dev` for everything: + + for(i in P S f æ t L A J '$') + bind -qa '#'^$i /dev + +So today `timesync` — which talks to an NTP server — can reach raw disk and +PCI config space. `#p` is worse: every process can walk `/proc`, read other +processes' memory, and post notes to them, so any service can kill init. + +### Consequences for the design + +**No default namespace.** A service with no namespace declaration is a load +error, like an unknown attribute. Inheriting by default means granting +everything by default to services that asked for nothing. + +**Ship profiles so explicit is not verbose.** The base is ~40 lines and most +services want most of it; hand-written namespaces per service would be +copy-pasted and would drift. + + /lib/ns/net #I #c #e #d — dials out, no disk, no /proc + /lib/ns/disk net plus #S + /lib/ns/draw net plus #i #m + /lib/ns/full everything termrc currently binds + +A service says `nsfile=/lib/ns/net` plus optional inline `ns=bind …` lines. + +**`full` will be abused, and that is survivable, because it is greppable.** +`grep -l ns/full /lib/services/*` is a list of over-privileged services, which +is a work queue. Inheritance produces no such list. + +### /srv is a hole + +Binding `#s` undoes most of the above. A service that can see `/srv` can +`mount /srv/boot` and get the whole root filesystem back, write +`/srv/hjfs.cmd` to halt the machine, or mount factotum and reach keys. + +Three layers, in order of strength: + +1. **Do not bind `#s`.** The only airtight one — the channels are not nameable. +2. **`rfork(RFNOMNT)` before exec.** Plan 9's pledge: a one-way drop after + which the process cannot mount, bind or unmount. Default on; `nsmount=yes` + is the visible opt-out for services that genuinely mount things. +3. **`user=`.** File permissions on whatever remains. + +RFNOMNT is not a wall. It blocks the `mount` call, not a process that opens +`/srv/foo` and speaks 9P down the fd by hand. It stops accidents and casual +escalation; only omitting `#s` stops intent. + +### Unresolved: how this interacts with everything else + +- **Providers of namespaces need mount privilege.** A service that imports a + remote `/net` for others must mount, and must post to `/srv` so dependents + can reach it — so the exception to rule 1 and the exception to rule 2 are + the same services. Circular and not thought through. +- **Namespace changes do not propagate.** A provider's mount dies with its + process group, so `/srv` is the only rendezvous, which is the thing we are + trying to restrict. +- **init's own bootstrap** predates all of this: it needs `/lib/services` and + `#s` before it can read a profile. +- **`user=` ordering.** The capability device must still be in the namespace + at the moment we drop privilege, so profiles cannot omit `#¤` blindly. +- **Adopted services** were started by bootrc in a namespace we never chose. +- **Migration.** `cs`, `dns` and `timesync` work today with no namespace + declaration. Making it mandatory breaks them until profiles exist. |