Safety

The death switch in detail

The shared page, the heartbeat, the guard's rules and timings, and the reason codes, for developers who write a client or a tool.

The death switch is implemented in the deathswitch crate; the runtime and the guard that use it are in the robot crate.

The page

/run/nao-rs/deathswitch, or the path in NAO_RS_DEATHSWITCH when that is set (tests and tools). It is a plain C struct of atomics, mapped shared by every process that opens it:

FieldTypeMeaning
magicu320x6e656473 ("neds") once the page is initialized
versionu32Page layout version, 1
trippedu320 = armed, otherwise the reason code below
generationu32Increments on every trip and every re-arm, so a reader notices changes it missed
driver_pidu32Process id of the runtime that sends frames
driver_cycleu64Cycle number of the runtime's last frame
driver_beat_nsu64CLOCK_MONOTONIC time of that frame, in ns
driver_all_off_cycleu64The last cycle whose frame was all-off
tripped_at_nsu64CLOCK_MONOTONIC time of the trip, in ns

Opening rules:

  • An existing page is opened without O_CREAT. With fs.protected_regular the kernel refuses O_CREAT on another user's file in a sticky directory, even for root.
  • A new page is created with mode 0660, whatever the umask, so hulk, the guard and naoctl (all in group hal) can share it. /run/nao-rs itself is created at boot by systemd-tmpfiles, owner root:hal, mode 2770.
  • A page without the magic value is initialized tripped.
  • An existing page that is shorter than the page layout is made whole and tripped with PageLost; a surviving magic value is not trusted.

Page integrity

A stop that lands in a file the runtime no longer reads would be lost, so nao-rs treats any damage to the page file as a stop:

What happened to the fileDetected byResult
Truncated, zeroed or overwritten (the magic value is gone, or the file is too short)The runtime before every frame, the guard before every pollThe frame is all-off at once. The page is made whole again and tripped with PageLost, so the lost part, which reads as zeros ("armed"), never re-arms by accident
Deleted, or another file now has its nameThe same checks (link count, device and inode of the path)Tripped with PageLost. Runtime and guard keep their shared mapping, but stops written to the path would miss it, so everything stays off until nao-rs is restarted

A page without its magic value always counts as tripped. Truncating the file under a live mapping makes every access raise SIGBUS; a signal handler re-extends the file and clears the magic, so the access is retried and the page reads as tripped. The guard reports these cases in its log as PageRepaired and PageReplaced.

Reason codes

CodeReasonSet by
0NoneArmed
1ChestButtonRuntime: chest button press
2HeadTouchRuntime: all three head pads
3ApiPOST /v1/safety/stop, the WebSocket stop and the MessagePack API's Stop (the brain's stop tool): each writes the page directly and also sends the UDP stop
4UdpGuard: the UDP stop alone, for example cargo nao-rs stop <address> from a computer
5Clinaoctl stop, and every newly created page
6DriverFaultRuntime: cycle overrun, failed exchange, shutdown
7OvertemperatureRuntime: a joint at 75 °C; fan thread: CPU above 95 °C
8GuardHeartbeatLostGuard: heartbeat older than the timeout
9GuardNoConfirmationGuard: trip not confirmed in time
10TestTests
11PageLostThe page file was truncated, zeroed, overwritten, deleted or replaced on disk (see Page integrity)

trip stores the reason only when the switch is armed (compare-and-swap from 0), so the first reason stays until rearm. Both are safe from any thread or process, including signal handlers.

The heartbeat

After each frame the runtime writes its heartbeat in a fixed order: confirmation first (driver_all_off_cycle, only when the frame was all-off), then the time, then the cycle number. The guard reads in the opposite order. A guard that sees cycle N therefore also sees N's confirmation, so a half-written heartbeat never looks like an unconfirmed all-off. A test with two million concurrent reads holds this; the old order made the guard kill a healthy runtime on the robot.

The guard's rules

The guard decides from the page alone, with a pure function that is tested case by case:

  1. No heartbeat since the page was cleared: fine. Nothing sends frames.
  2. Last beat older than the beat timeout: take over (GuardHeartbeatLost).
  3. Tripped, and the confirmation doesn't cover the latest cycle, or the last beat is older than the trip, for longer than the confirmation window: take over (GuardNoConfirmation). A confirmation from before the trip doesn't count.
  4. Otherwise: fine.
SettingDefault
Beat timeout60 ms (the runtime beats every ~12 ms)
Confirmation window after a trip40 ms
Poll interval2 ms
UDP stop port7701, bound on all addresses
Real-time prioritySCHED_FIFO 40

While it holds the chest board after a takeover, the guard writes its own all-off heartbeat into the page, so its own rules don't fire a second time. It gives the board up in two cases:

  • A restarted runtime registers (a new process id in driver_pid): the guard hands the board over, clears the heartbeat fields but keeps the new process id, and judges the new runtime from its first beat. The switch stays tripped. The new runtime retries opening the chest board every 50 ms for up to 5 s while the guard lets go. A runtime owned by another user, which the guard can't signal (EPERM), counts as alive for this.
  • A person re-arms with no new runtime: the guard releases the board and clears the driver fields.

The UDP stop

A datagram whose payload starts with the 12 bytes NAO-RS STOP (ASCII) trips the switch with reason Udp. Anything after them is ignored, and so is any other datagram. There is no reply. Send it several times: UDP can drop a datagram, and one arriving is enough.

What all-off sends

With the switch tripped, the runtime still runs its normal cycle in HAL's order, but every joint's stiffness is 0. With stiffness 0, each joint is commanded to its own measured position, the way HAL does it, so nothing pushes. LEDs and sonar keep working. The guard's takeover sends stiffness 0 on both motor buses from its first frame and skips the start-up handshake; the boards keep the configuration the runtime gave them.

Safety deviations from HAL

The runtime limits what reaches the boards even when the switch is armed. These are deliberate deviations from HAL, covered by tests:

  • A position or stiffness that is not a finite number makes that joint limp. HAL would turn NaN into a count of 0 on the wire, a full-range position command.
  • Stiffness is limited to 0..1.

Joint positions are not limited to the joint's range, as on HAL: with stiffness on, a position goes to the board as commanded, and at stiffness 0 the joint follows its own sensor. hulk's stand-up and sit-down motions command some joints a few degrees past the range (up to 22.5° on an elbow) on purpose, to press against the end stops; a limit would change those motions.

On this page