Troubleshooting

What to check when nao-rs doesn't start, the death switch won't stay armed, the guard takes over, or a tool can't open the chest board.

Start with these two commands on the robot; most problems show up in one of them.

naoctl status
journalctl -u hulk -u nao-rs-guard -n 50 --no-pager

The motors don't move

That is expected today. nao-rs rejects the robot's version 11 motor boards, as HAL 2.8.5 does, until the motor session on the robot (planned). See Status.

The switch is tripped and I didn't trip it

naoctl status names the reason. The first reason since the last re-arm is kept.

Reason	Cause	What to do
`Cli` right after boot	Every boot starts with a new, tripped page	Re-arm when the robot is ready: `naoctl rearm`
`ChestButton`	The chest button was pressed	Check the robot, re-arm
`HeadTouch`	All three head pads were touched at once	Check the robot, re-arm
`Overtemperature`	A joint at 75 °C, or the CPU above 95 °C (the system halts then)	Let the robot cool down; check the fan
`DriverFault`	The cycle fell more than 3 periods behind, an exchange with the chest board failed, or the runtime stopped	Read `journalctl -u hulk`; look for `cycle overran` or `cycle failed`
`Api`	A stop from the REST API, its WebSocket, the MessagePack API or the brain's `stop` tool	Find out which client; re-arm
`Udp`	Someone sent the UDP stop alone (for example `cargo nao-rs stop`)	Find out who; re-arm
`GuardHeartbeatLost`	The runtime stopped beating for more than 60 ms	The guard killed `hulk`; see below
`PageLost`	The switch page in `/run/nao-rs` was truncated, overwritten, deleted or replaced	Find what touched `/run/nao-rs`. After a deletion or replacement, restart nao-rs (`sudo systemctl restart nao-rs-guard`, then `systemctl restart hulk`) so every stop path uses one page again; then re-arm
`GuardNoConfirmation`	After a trip, the runtime didn't confirm an all-off frame within 40 ms	The guard killed `hulk`; see below

journalctl -u nao-rs-guard shows TakeOver, KilledDriver and Holding. The guard holds the chest board and sends all-off. systemd restarts hulk after one second (Restart=on-failure); the guard hands the board to the new hulk, and the switch stays tripped.

Read why hulk hung or died: journalctl -u hulk -n 200 --no-pager.
Check that the new hulk beats: naoctl status.
When the robot is safe, re-arm: naoctl rearm.

If hulk keeps failing, systemd gives up after repeated failures; fix the cause and run systemctl start hulk.

`hulk` doesn't start

failed to start the robot runtime: chest board: …: another program holds the chest board. HAL may still run (systemctl status hal), or the guard holds it after a takeover (see above), or a naoctl hardware command runs.
death switch: Permission denied or No such file or directory: /run/nao-rs is missing or has the wrong owner. It should be root:hal, mode 2770. Run sudo systemd-tmpfiles --create /etc/tmpfiles.d/nao-rs.conf.
head bus unavailable or head fan unavailable in the log: the runtime keeps running without the head or the fan. Check that LoLA is stopped (systemctl status lola) and that /dev/i2c-head and /dev/cgos exist.

The installer rolled back

Exit code 1 means the health check failed and the previous installation runs again. The installer printed the last 30 journal lines of both services; the failed installation is kept in /home/nao/hulk.failed. Exit code 2 means the rollback failed too: run cargo nao-rs robot restore <address>, or reflash.

`naoctl` can't open the chest board

The hardware commands (prefs, run, guard, head, replay-sensors) need the device to themselves. Stop hulk (systemctl stop hulk) and, for guard, the guard service. On the original stack, stop HAL, and LoLA for head. status, stop and rearm need nothing stopped.

LoLA can't reconnect after a restore

On HULKs-OS, LoLA can't connect to a HAL that is already running ("No HAL found"). Restart the whole chain in order: HAL, then LoLA, then HULA, then hulk. nao-rs robot restore does this.

The robot doesn't react to the UDP stop

Check the guard runs: systemctl status nao-rs-guard.
The datagram must start with exactly NAO-RS STOP and go to UDP port 7701.
A firewall between you and the robot may drop UDP. On the robot, naoctl stop always works.

The robot doesn't respond at all

Hold the chest button for 8 seconds: the chest board switches the robot off, whatever runs on the head.