The brain

The program on your Mac that talks, looks, remembers and moves the robot through the robot API, its live and agent modes, and its settings.

The robot runs only a small real-time core. Everything heavy runs on a Mac in the brain, nao-rs-brain. It talks with people through the robot's own microphones and speakers, looks at them through its camera, remembers them and moves the robot. It reaches the robot only through the robot API on port 7700, so it works the same with a real NAO and with the simulator app.

The conversation runs on your own ElevenLabs agent: nao-rs has no agent of its own. You create one at ElevenLabs, set its prompt, voice, language and model there, and give the brain its ID. Voice agent walks through it.

cargo brain --robot <robot-address>:7700 --agent-id <agent-id> --mode live

No motion in this phase

nao-rs doesn't drive the motors yet, and the robot must not move during bring-up without a person present. On nao-rs today, motion requests from the brain change hulk's state without moving a joint.

Modes

Mode	What the robot does	Tools the agent may use
live (default)	Talks, with subtle movement: stands, turns its head toward the person it talks to, small head movements while it speaks, glances around now and then when nobody is in view. Only the head moves, only while the robot stands; it never walks on its own, and a sitting robot stays sitting	`halt`, `stop`, `learn_name`, `remember`. Motion tools are refused with a hint to agent mode
agent	Talks, and the agent drives the robot with client tools: walk, turn, turn in a circle, raise the arms, squat, sit, stand, look, turn the head, colour the eyes	All tools; see Voice agent

halt (stand still) and stop (the death switch) work in every mode.

What else it does

Part	How
Conversation	Full duplex with the ElevenLabs agent: the robot's microphones to the agent, the agent's voice to the robot's speakers, 16 kHz mono PCM both ways. The robot's microphones are noise-reduced; they are muted while it speaks, so it doesn't hear itself (see Its own voice)
Seeing people	When the camera picture changes, Gemini 3.8 Flash on Vertex AI looks at the frame and the stored faces of known people, within a hard daily cost cap. Needs `gcloud auth application-default login`; `--camera none` turns it off
Memory	People, their names and facts about them in SQLite at `~/Library/Application Support/nao-rs/companion.db`. When someone comes back, the agent picks up where they left off
Reactions	Touch, falls and a low battery change the eye colour at once and are passed to the agent as context
Languages	English or German for the brain's own words (`language`). The robot's own announcements, such as its IP address after a chest-button stop, follow the same `NAO_RS_LANGUAGE` (What the robot says). The conversation's language is set on your agent at ElevenLabs
Party mode	`--downbeat <room code>` plays a Downbeat room on the robot instead of talking; it needs no agent. See Party mode

Waking and stopping. The robot wakes only from the app or an API (Power wake, or --wake). The chest button is the stop: one press trips the latched death switch, and a person re-arms it on the robot with naoctl rearm. A re-arm doesn't wake the robot: it stays limp until the app or an API wakes it again. See Gestures.

When the brain goes away (the Mac sleeps, Wi-Fi drops, the program stops), the robot's watchdog makes it stand still after 0.5 s and sit down after 3 s. Its feet show the link: dark before a brain connected, green while connected, red after the link was lost.

Its own voice

The robot's speakers sit a few centimeters from its microphones, so it must not hear itself. By default (self_voice: mute in the robot's etc/parameters/hardware.json) the microphones are silent while the robot speaks and for about 300 ms of room echo after. Wait until it has finished before you answer: you can't interrupt it. The brain mutes the same way, while the robot speaks and for half a second after, whenever the robot does not cancel its own voice, and always with the Mac's own microphone and speakers in the simulator app.

Echo cancellation (self_voice: cancel, WebRTC's AEC3 on the robot) is experimental and not yet good on the NAO: in the first live test, interrupting did not work, speech recognition got worse and the robot answered its own voice. When it is on, the robot says so (echo_cancellation in the MessagePack Hello and in REST GET /v1/robot) and the brain keeps the microphones open while the robot speaks. Change self_voice on the robot and restart hulk.

Settings

The brain takes each setting from the command line first, then the environment, then a config file. There is no default agent: without an agent ID it stops at once and says how to set one.

Setting	Command line	Environment	Config file	Default
Agent ID	`--agent-id agent_…`	`NAO_RS_AGENT_ID`	`agent_id`	none, required
Mode	`--mode live` or `--mode agent`	`NAO_RS_MODE`	`mode`	`live`
Language of the brain's own words	`--language en` or `de`	`NAO_RS_LANGUAGE`	`language`	`en`
Robot	`--robot <address:port>`	`NAO_RS_ROBOT`	`robot`	`nao.local:7700`
Camera	`--camera robot`, `mac` or `none` (also `--mac-camera`, `--no-camera`)		`camera`	`robot`
Gemini budget per day	`--daily-cap-usd <usd>`		`daily_cap_usd`	`1`
API key (agents with authentication only)		`XI_API_KEY` or `ELEVENLABS_API_KEY`	`api_key_command`	none
Config file	`--config <file>`	`NAO_RS_CONFIG`		`~/.config/nao-rs/brain.toml`

The config file lives at ~/.config/nao-rs/brain.toml, or under $XDG_CONFIG_HOME when that is set. Every key is optional, and a misspelled key is an error instead of being ignored:

agent_id = "agent_0123456789abcdefghijklmnop"
mode = "agent"              # "live" (default) or "agent"
language = "en"             # "en" (default) or "de"
robot = "nao.local:7700"
camera = "robot"            # "robot" (default), "mac" or "none"
daily_cap_usd = 1.0
# Only for agents with authentication on: a command that prints the ElevenLabs API key.
# The key itself never goes into this file, and the brain never logs it.
api_key_command = "op read op://Private/ElevenLabs/credential"

The simulator app reads the same environment and file.

Language sets the brain's own words, not the agent's (the agent speaks whatever its prompt says): the instructions to Gemini, so it describes people and scenes in that language, the conversation summaries in memory, and the notes about what the robot sees and feels. Those notes reach the agent as contextual updates tagged [Perception] and [Feeling] in English, or [Wahrnehmung] and [Gefühl] in German; mention the tags of your language in the agent's prompt. NAO_RS_LANGUAGE also picks the language of the robot's own spoken lines.

Other options

Option	Default	Meaning
`--wake`	off	Send a wake command (motors stiff) on connecting. Only for the simulator or a robot that is safe to move. On the nao-rs drivers a chest press trips the death switch, so don't press it to wake the robot; see Gestures
`--downbeat <room code>`	off	Party mode
`--downbeat-url <url>`	`https://downbeat.lukaloehr.com`, env `DOWNBEAT_URL`	The Downbeat server
`--downbeat-latency-ms <ms>`	`250`	Party mode lead time until the robot's speakers play

Debugging

Variable	Effect
`NAO_RS_MIC_DUMP=<file>`	Appends what the agent hears (16 kHz mono, 16-bit little-endian)
`NAO_RS_EVENT_DUMP=<file>`	Appends every event from the agent as one JSON line, audio shortened
`NAO_RS_DENOISE=0`	Turns off the noise reduction of the robot's microphones

cargo test --manifest-path tools/Cargo.toml -p nao_rs_brain runs the brain's tests without a robot: settings, every tool's validation, the mapping of tool calls to robot commands on a fake robot, live mode's movement and recorded agent events.