> Documentation index: https://nao-rs.lukaloehr.com/llms.txt, a list of every page in this documentation.

# Voice agent

> Create your own ElevenLabs agent for the robot, pick the realtime model, put nao-rs's client tools on it, and what each tool does and refuses.

The conversation runs on **your own ElevenLabs conversational agent**. Its
prompt, voice, language and model are set at ElevenLabs; nao-rs never changes
them. The brain needs only the agent's ID, and for agents with authentication
on, your API key.

## Create the agent

### 1. Create a blank agent

At [elevenlabs.io/app/agents](https://elevenlabs.io/app/agents), create a
blank agent. Write its prompt, pick a voice and a language. The nao-rs
repository has an example prompt in `docs/examples/agent-prompt.md`. Tell the
prompt about the brain's context notes in your language: `[Perception]` and
`[Feeling]` in English, `[Wahrnehmung]` and `[Gefühl]` in German (see
[Settings](https://nao-rs.lukaloehr.com/brain.md#settings)).

### 2. Pick the model and the audio format

| Setting | Value | Why |
| --- | --- | --- |
| Model | the realtime model `eleven_realtime_v1_max` | Answers in 0.6 to 0.9 s; a cascaded LLM such as Gemini took 2 to 7 s in our measurements |
| Output and input audio format (Voice, Advanced) | PCM 16000 Hz | The robot's own format. Other PCM rates and µ-law 8000 Hz are converted by the brain; MP3 and Opus don't work |

### 3. Give the brain its ID

Copy the agent ID (`agent_…`) from the agent's page and set it in one of three
ways: `--agent-id`, `NAO_RS_AGENT_ID`, or `agent_id` in
`~/.config/nao-rs/brain.toml` (see [Settings](https://nao-rs.lukaloehr.com/brain.md#settings)).

If the agent has authentication on, the brain also needs your API key: from
`XI_API_KEY` or `ELEVENLABS_API_KEY`, or from a command named in
`api_key_command`. The key never goes into the config file and is never
logged. A public agent needs no key, but anyone who knows its ID can talk to
it on your account.

### 4. Put the tools on the agent

```bash
cargo nao-rs agent tools <agent-id>              # agent mode: all tools
cargo nao-rs agent tools <agent-id> --mode live  # live mode: halt, stop and memory only
cargo nao-rs agent tools <agent-id> --dry-run    # show what would change
```

This needs your ElevenLabs API key (`ELEVENLABS_API_KEY`, or the macOS
Keychain item `elevenlabs-api-key`). It adds or updates only nao-rs's tools,
by name, leaves every other tool on the agent alone, and never changes the
prompt, voice or model: it sends the prompt back unchanged and checks it
afterwards. Run it again after updating nao-rs.

## Tools

The agent calls ElevenLabs **client tools**; the brain checks each call,
checks the robot's state, sends the robot API commands and answers with what
the robot does now, or why not. Every tool is asynchronous: the agent says
what it does and keeps talking while the robot acts. Every call and its result
appear in the brain's log. The definitions live in
`tools/brain/agent-tools.json` in the nao-rs repository, the same file the
brain checks calls against.

> **Warning:** No motion in this phase
>
> nao-rs doesn't drive the motors yet, and the robot must not move during
> bring-up without a person present. The motion tools are listed so you can set
> up an agent; on nao-rs today they change `hulk`'s state without moving a joint.

| Tool | Arguments | What the robot does | Modes |
| --- | --- | --- | --- |
| `walk` | `meters` (−1 to 1, negative: backward), optional `sideways` (positive: left) | Walks there and stops, keeping its heading. Longer walks are capped at 1 m and the agent is told so | agent |
| `turn` | `degrees` (−360 to 360, positive: left) | Turns on the spot; more than 180° in two parts | agent |
| `circle` | optional `direction` (`left`, `right`) | Turns once all the way around in four quarter turns, then stands | agent |
| `arms_up` | none | Raises both arms for 3 s, then stands | agent |
| `squat` | none | Squats with raised arms for 3 s, then stands up | agent |
| `sit_down` | none | Sits down | agent |
| `stand_up` | none | Stands up and stands still (`stand` is accepted too) | agent |
| `look` | `direction`: `ahead`, `left`, `right`, `up`, `down`, `around` | Turns the head; only while standing | agent |
| `turn_head` | `degrees` (−90 to 90, positive: left) | Turns the head sideways; only while standing | agent |
| `eyes` | `color` (`red`, `orange`, `yellow`, `green`, `cyan`, `blue`, `purple`, `pink`, `white`, `off`, `default`), optional `seconds` (0.1 to 60) | Colours both eyes, for a while or until changed; `default` gives them back to the robot | agent |
| `halt` | none | Stops whatever the body does and stands still | every mode |
| `stop` | none | **Emergency stop**: trips the [death switch](https://nao-rs.lukaloehr.com/safety.md). All motors off at once; the robot goes limp and may fall. Someone re-arms it on the robot | every mode |
| `learn_name` | `name` | Remembers the name of the person in front of it | every mode |
| `remember` | `fact` | Remembers something about the person | every mode |

Waving is not possible yet: the robot API has no arm motion besides arms up
and squat.

## Safety rules of the tools

- Arguments are checked before anything moves. Wrong types, missing values and
  out-of-range numbers are refused with a message the agent can act on.
- Motion is refused while the robot rests (motors off), falls or gets up, and
  head movements while it sits (they would stand it up).
- A refused call stops a walking robot and cancels the running plan; a sitting
  robot stays sitting.
- `halt` and `stop` cancel everything and go out at once. `stop` can't be
  undone from the Mac.
- The robot's own safety comes first: fall protection, the watchdog that sits
  it down when the Mac goes away, and the death switch.

## Related topics

- [The brain](https://nao-rs.lukaloehr.com/brain.md)
- [Party mode](https://nao-rs.lukaloehr.com/brain/party-mode.md)
- Previous: [The brain](https://nao-rs.lukaloehr.com/brain.md)
- Next: [Party mode](https://nao-rs.lukaloehr.com/brain/party-mode.md)