The death switch in detail
The shared page, the heartbeat, the guard's rules and timings, and the reason codes, for developers who write a client or a tool.
The death switch is implemented in the deathswitch crate; the runtime and the
guard that use it are in the robot crate.
The page
/run/nao-rs/deathswitch, or the path in NAO_RS_DEATHSWITCH when that is
set (tests and tools). It is a plain C struct of atomics, mapped shared by
every process that opens it:
| Field | Type | Meaning |
|---|---|---|
magic | u32 | 0x6e656473 ("neds") once the page is initialized |
version | u32 | Page layout version, 1 |
tripped | u32 | 0 = armed, otherwise the reason code below |
generation | u32 | Increments on every trip and every re-arm, so a reader notices changes it missed |
driver_pid | u32 | Process id of the runtime that sends frames |
driver_cycle | u64 | Cycle number of the runtime's last frame |
driver_beat_ns | u64 | CLOCK_MONOTONIC time of that frame, in ns |
driver_all_off_cycle | u64 | The last cycle whose frame was all-off |
tripped_at_ns | u64 | CLOCK_MONOTONIC time of the trip, in ns |
Opening rules:
- An existing page is opened without
O_CREAT. Withfs.protected_regularthe kernel refusesO_CREATon another user's file in a sticky directory, even for root. - A new page is created with mode
0660, whatever the umask, sohulk, the guard andnaoctl(all in grouphal) can share it./run/nao-rsitself is created at boot by systemd-tmpfiles, ownerroot:hal, mode2770. - A page without the magic value is initialized tripped.
- An existing page that is shorter than the page layout is made whole and
tripped with
PageLost; a surviving magic value is not trusted.
Page integrity
A stop that lands in a file the runtime no longer reads would be lost, so nao-rs treats any damage to the page file as a stop:
| What happened to the file | Detected by | Result |
|---|---|---|
| Truncated, zeroed or overwritten (the magic value is gone, or the file is too short) | The runtime before every frame, the guard before every poll | The frame is all-off at once. The page is made whole again and tripped with PageLost, so the lost part, which reads as zeros ("armed"), never re-arms by accident |
| Deleted, or another file now has its name | The same checks (link count, device and inode of the path) | Tripped with PageLost. Runtime and guard keep their shared mapping, but stops written to the path would miss it, so everything stays off until nao-rs is restarted |
A page without its magic value always counts as tripped. Truncating the file
under a live mapping makes every access raise SIGBUS; a signal handler
re-extends the file and clears the magic, so the access is retried and the
page reads as tripped. The guard reports these cases in its log as
PageRepaired and PageReplaced.
Reason codes
| Code | Reason | Set by |
|---|---|---|
| 0 | None | Armed |
| 1 | ChestButton | Runtime: chest button press |
| 2 | HeadTouch | Runtime: all three head pads |
| 3 | Api | POST /v1/safety/stop, the WebSocket stop and the MessagePack API's Stop (the brain's stop tool): each writes the page directly and also sends the UDP stop |
| 4 | Udp | Guard: the UDP stop alone, for example cargo nao-rs stop <address> from a computer |
| 5 | Cli | naoctl stop, and every newly created page |
| 6 | DriverFault | Runtime: cycle overrun, failed exchange, shutdown |
| 7 | Overtemperature | Runtime: a joint at 75 °C; fan thread: CPU above 95 °C |
| 8 | GuardHeartbeatLost | Guard: heartbeat older than the timeout |
| 9 | GuardNoConfirmation | Guard: trip not confirmed in time |
| 10 | Test | Tests |
| 11 | PageLost | The page file was truncated, zeroed, overwritten, deleted or replaced on disk (see Page integrity) |
trip stores the reason only when the switch is armed (compare-and-swap from
0), so the first reason stays until rearm. Both are safe from any thread or
process, including signal handlers.
The heartbeat
After each frame the runtime writes its heartbeat in a fixed order:
confirmation first (driver_all_off_cycle, only when the frame was all-off),
then the time, then the cycle number. The guard reads in the opposite order.
A guard that sees cycle N therefore also sees N's confirmation, so a
half-written heartbeat never looks like an unconfirmed all-off. A test with
two million concurrent reads holds this; the old order made the guard kill a
healthy runtime on the robot.
The guard's rules
The guard decides from the page alone, with a pure function that is tested case by case:
- No heartbeat since the page was cleared: fine. Nothing sends frames.
- Last beat older than the beat timeout: take over (
GuardHeartbeatLost). - Tripped, and the confirmation doesn't cover the latest cycle, or the last
beat is older than the trip, for longer than the confirmation window:
take over (
GuardNoConfirmation). A confirmation from before the trip doesn't count. - Otherwise: fine.
| Setting | Default |
|---|---|
| Beat timeout | 60 ms (the runtime beats every ~12 ms) |
| Confirmation window after a trip | 40 ms |
| Poll interval | 2 ms |
| UDP stop port | 7701, bound on all addresses |
| Real-time priority | SCHED_FIFO 40 |
While it holds the chest board after a takeover, the guard writes its own all-off heartbeat into the page, so its own rules don't fire a second time. It gives the board up in two cases:
- A restarted runtime registers (a new process id in
driver_pid): the guard hands the board over, clears the heartbeat fields but keeps the new process id, and judges the new runtime from its first beat. The switch stays tripped. The new runtime retries opening the chest board every 50 ms for up to 5 s while the guard lets go. A runtime owned by another user, which the guard can't signal (EPERM), counts as alive for this. - A person re-arms with no new runtime: the guard releases the board and clears the driver fields.
The UDP stop
A datagram whose payload starts with the 12 bytes NAO-RS STOP (ASCII)
trips the switch with reason Udp. Anything after them is ignored, and so is
any other datagram. There is no reply. Send it several times: UDP can drop a
datagram, and one arriving is enough.
What all-off sends
With the switch tripped, the runtime still runs its normal cycle in HAL's order, but every joint's stiffness is 0. With stiffness 0, each joint is commanded to its own measured position, the way HAL does it, so nothing pushes. LEDs and sonar keep working. The guard's takeover sends stiffness 0 on both motor buses from its first frame and skips the start-up handshake; the boards keep the configuration the runtime gave them.
Safety deviations from HAL
The runtime limits what reaches the boards even when the switch is armed. These are deliberate deviations from HAL, covered by tests:
- A position or stiffness that is not a finite number makes that joint limp. HAL would turn NaN into a count of 0 on the wire, a full-range position command.
- Stiffness is limited to 0..1.
Joint positions are not limited to the joint's range, as on HAL: with
stiffness on, a position goes to the board as commanded, and at stiffness 0 the
joint follows its own sensor. hulk's stand-up and sit-down motions command
some joints a few degrees past the range (up to 22.5° on an elbow) on purpose,
to press against the end stops; a limit would change those motions.
Safety model
How nao-rs stops the motors: a latched death switch checked before every frame, a guard process that takes over a hung runtime, the triggers, and the chest board's own hard power-off.
Temperatures, fan and thermal protection
The head fan curve, the CPU alert and halt, the motor stop at 75 °C, and every temperature nao-rs reads, with its unit and source.