> Documentation index: https://nao-rs.lukaloehr.com/llms.txt, a list of every page in this documentation.

# The brain

> The program on your Mac that talks, looks, remembers and moves the robot through the robot API, its live and agent modes, and its settings.

The robot runs only a small real-time core. Everything heavy runs on a Mac in
the **brain**, `nao-rs-brain`. It talks with people through the robot's own
microphones and speakers, looks at them through its camera, remembers them and
moves the robot. It reaches the robot only through the [robot
API](https://nao-rs.lukaloehr.com/reference/api.md) on port 7700, so it works the same with a real NAO and
with the simulator app.

The conversation runs on **your own ElevenLabs agent**: nao-rs has no agent
of its own. You create one at ElevenLabs, set its prompt, voice, language and
model there, and give the brain its ID. [Voice agent](https://nao-rs.lukaloehr.com/brain/voice-agent.md)
walks through it.

```bash
cargo brain --robot <robot-address>:7700 --agent-id <agent-id> --mode live
```

> **Warning:** No motion in this phase
>
> nao-rs doesn't drive the motors yet, and the robot must not move during
> bring-up without a person present. On nao-rs today, motion requests from the
> brain change `hulk`'s state without moving a joint.

## Modes

| Mode | What the robot does | Tools the agent may use |
| --- | --- | --- |
| **live** (default) | Talks, with subtle movement: stands, turns its head toward the person it talks to, small head movements while it speaks, glances around now and then when nobody is in view. Only the head moves, only while the robot stands; it never walks on its own, and a sitting robot stays sitting | `halt`, `stop`, `learn_name`, `remember`. Motion tools are refused with a hint to agent mode |
| **agent** | Talks, and the agent drives the robot with client tools: walk, turn, turn in a circle, raise the arms, squat, sit, stand, look, turn the head, colour the eyes | All tools; see [Voice agent](https://nao-rs.lukaloehr.com/brain/voice-agent.md#tools) |

`halt` (stand still) and `stop` (the death switch) work in every mode.

## What else it does

| Part | How |
| --- | --- |
| **Conversation** | Full duplex with the ElevenLabs agent: the robot's microphones to the agent, the agent's voice to the robot's speakers, 16 kHz mono PCM both ways. The robot's microphones are noise-reduced; they are muted while it speaks, so it doesn't hear itself (see [Its own voice](#its-own-voice)) |
| **Seeing people** | When the camera picture changes, Gemini 3.8 Flash on Vertex AI looks at the frame and the stored faces of known people, within a hard daily cost cap. Needs `gcloud auth application-default login`; `--camera none` turns it off |
| **Memory** | People, their names and facts about them in SQLite at `~/Library/Application Support/nao-rs/companion.db`. When someone comes back, the agent picks up where they left off |
| **Reactions** | Touch, falls and a low battery change the eye colour at once and are passed to the agent as context |
| **Languages** | English or German for the brain's own words (`language`). The robot's own announcements, such as its IP address after a chest-button stop, follow the same `NAO_RS_LANGUAGE` ([What the robot says](https://nao-rs.lukaloehr.com/get-started/voice.md)). The conversation's language is set on your agent at ElevenLabs |
| **Party mode** | `--downbeat <room code>` plays a Downbeat room on the robot instead of talking; it needs no agent. See [Party mode](https://nao-rs.lukaloehr.com/brain/party-mode.md) |

**Waking and stopping.** The robot wakes only from the app or an API
(`Power` wake, or `--wake`). The chest button is the stop: one press trips the
latched death switch, and a person re-arms it on the robot with
`naoctl rearm`. A re-arm doesn't wake the robot: it stays limp until the app or
an API wakes it again. See [Gestures](https://nao-rs.lukaloehr.com/safety.md#gestures).

When the brain goes away (the Mac sleeps, Wi-Fi drops, the program stops),
the robot's watchdog makes it stand still after 0.5 s and sit down after 3 s.
Its feet show the link: dark before a brain connected, green while connected,
red after the link was lost.

## Its own voice

The robot's speakers sit a few centimeters from its microphones, so it must
not hear itself. By default (`self_voice: mute` in the robot's
`etc/parameters/hardware.json`) the microphones are silent while the robot
speaks and for about 300 ms of room echo after. Wait until it has finished
before you answer: you can't interrupt it. The brain mutes the same way, while
the robot speaks and for half a second after, whenever the robot does not
cancel its own voice, and always with the Mac's own microphone and speakers in
the simulator app.

Echo cancellation (`self_voice: cancel`, WebRTC's AEC3 on the robot) is
**experimental and not yet good on the NAO**: in the first live test,
interrupting did not work, speech recognition got worse and the robot answered
its own voice. When it is on, the robot says so (`echo_cancellation` in the
MessagePack `Hello` and in REST `GET /v1/robot`) and the brain keeps the
microphones open while the robot speaks. Change `self_voice` on the robot and
restart `hulk`.

## Settings

The brain takes each setting from the command line first, then the
environment, then a config file. There is **no default agent**: without an
agent ID it stops at once and says how to set one.

| Setting | Command line | Environment | Config file | Default |
| --- | --- | --- | --- | --- |
| Agent ID | `--agent-id agent_…` | `NAO_RS_AGENT_ID` | `agent_id` | none, required |
| Mode | `--mode live` or `--mode agent` | `NAO_RS_MODE` | `mode` | `live` |
| Language of the brain's own words | `--language en` or `de` | `NAO_RS_LANGUAGE` | `language` | `en` |
| Robot | `--robot <address:port>` | `NAO_RS_ROBOT` | `robot` | `nao.local:7700` |
| Camera | `--camera robot`, `mac` or `none` (also `--mac-camera`, `--no-camera`) |  | `camera` | `robot` |
| Gemini budget per day | `--daily-cap-usd <usd>` |  | `daily_cap_usd` | `1` |
| API key (agents with authentication only) |  | `XI_API_KEY` or `ELEVENLABS_API_KEY` | `api_key_command` | none |
| Config file | `--config <file>` | `NAO_RS_CONFIG` |  | `~/.config/nao-rs/brain.toml` |

The config file lives at `~/.config/nao-rs/brain.toml`, or under
`$XDG_CONFIG_HOME` when that is set. Every key is optional, and a misspelled
key is an error instead of being ignored:

```text
agent_id = "agent_0123456789abcdefghijklmnop"
mode = "agent"              # "live" (default) or "agent"
language = "en"             # "en" (default) or "de"
robot = "nao.local:7700"
camera = "robot"            # "robot" (default), "mac" or "none"
daily_cap_usd = 1.0
# Only for agents with authentication on: a command that prints the ElevenLabs API key.
# The key itself never goes into this file, and the brain never logs it.
api_key_command = "op read op://Private/ElevenLabs/credential"
```

The simulator app reads the same environment and file.

**Language** sets the brain's own words, not the agent's (the agent speaks
whatever its prompt says): the instructions to Gemini, so it describes people
and scenes in that language, the conversation summaries in memory, and the
notes about what the robot sees and feels. Those notes reach the agent as
contextual updates tagged `[Perception]` and `[Feeling]` in English, or
`[Wahrnehmung]` and `[Gefühl]` in German; mention the tags of your language in
the agent's prompt. `NAO_RS_LANGUAGE` also picks the language of the robot's
own spoken lines.

### Other options

| Option | Default | Meaning |
| --- | --- | --- |
| `--wake` | off | Send a wake command (motors stiff) on connecting. Only for the simulator or a robot that is safe to move. On the nao-rs drivers a chest press trips the death switch, so don't press it to wake the robot; see [Gestures](https://nao-rs.lukaloehr.com/safety.md#gestures) |
| `--downbeat <room code>` | off | Party mode |
| `--downbeat-url <url>` | `https://downbeat.lukaloehr.com`, env `DOWNBEAT_URL` | The Downbeat server |
| `--downbeat-latency-ms <ms>` | `250` | Party mode lead time until the robot's speakers play |

### Debugging

| Variable | Effect |
| --- | --- |
| `NAO_RS_MIC_DUMP=<file>` | Appends what the agent hears (16 kHz mono, 16-bit little-endian) |
| `NAO_RS_EVENT_DUMP=<file>` | Appends every event from the agent as one JSON line, audio shortened |
| `NAO_RS_DENOISE=0` | Turns off the noise reduction of the robot's microphones |

`cargo test --manifest-path tools/Cargo.toml -p nao_rs_brain` runs the brain's
tests without a robot: settings, every tool's validation, the mapping of tool
calls to robot commands on a fake robot, live mode's movement and recorded
agent events.

## Related topics

- [Voice agent](https://nao-rs.lukaloehr.com/brain/voice-agent.md)
- [Party mode](https://nao-rs.lukaloehr.com/brain/party-mode.md)
- Previous: [Temperatures, fan and thermal protection](https://nao-rs.lukaloehr.com/safety/thermal.md)
- Next: [Voice agent](https://nao-rs.lukaloehr.com/brain/voice-agent.md)