Chris and I wanted to build a low cost (sub 100$) desk robot that feels alive and isn’t overly talkative. We were inspired by the Sesame quadruped and Reachy Mini.
At first we built Sesame by following the instructions to see what limitations exist and write out a requirements spec for our V1 of walnut. These include:
- natural gait and agility
- sub 100$ cost
- hearing, speaking, feeling
- real time adaptive facial expressions and voice/noises
- ‘always-on’
The last req came from us never having seen a Reachy Mini turned on, since it mostly appeared in short demos where the user is actively engaging with it. This means that even if two people nearby Walnut are having a conversation, it could be turned on and uninterrupting.
architecture
We went for a dual-compute system where the mobile components (SCS0009 motors, BNO055 IMU, 3 layer MLP PPO) were controlled by a Xiao ESP-S3 board, and all remaining peripherals and general intelligence were managed by a Raspberry Pi 2 Zero W. We're also exploring using the Radxa Zero 3W instead, but haven't gotten around to it.
Side note: we used an Iphone X speaker for crisp audio output since it was cheap (about 3 euros) and the right size for the robot.
For power delivery, we needed to choose a battery and a converter that could handle 8 servos running at 5V as well as the Raspberry Pi and other peripherals. The motors have a stall current of about 1A each, so we needed to reach close to 8A peak supply to avoid major voltage drops under load. Ultimately, we settled on a 1S LiPo with separate charge and discharge paths: a charger-only USB-C board connects to the battery, while a 10–15 A path feeds a 5.2 V boost converter. We did this to avoid the ~3 A cutoff of common DW01/8205A protection boards, which would brown out the eight servos. A 2S LiPo with a buck converter was the alternative, offering lower battery-side current and more headroom under servo load, but we chose the former due to space efficiency.
When we build a distro board we may integrate the charger and boost converter into the board itself.
packaging
To fit all these new components into a form factor that was the same size as the original Sesame, we had to be highly space efficient (especially without a distro board). Our approach was to carefully sketch out the components such that we could move them around neatly in a component map in Onshape, and use component variables to adapt for any downstream design or component changes.
This worked super well but took a long time. Eventually we ended up with a complete design that we 3D printed and tested out.
motor control
Since the documentation for the SCS0009 Feetech motors was quite poor, we decided to build a motor control suite on top of the sts-suite by Binh Pham. It's called scs-suite though I plan on merging this into the sts-suite with Binh at some point.
This also required adding support for new motors in the pollen-robotics/rustypot project, which the sts-suite uses under the hood. Fortunately, it made it a lot easier to detect and test motors instead of having to write a series of unique test scripts in our repository.
locomotion
For locomotion, we trained a small PPO policy in MuJoCo with mjlab. The main constraint was that it had to run directly on the Xiao ESP32-S3 and only use things Walnut can measure in real life. The actor gets two 33-value snapshots of the IMU, joint encoders, previous action and movement command, then outputs targets for the 8 motors at 50 Hz.
We trained 16,384 randomized Walnuts in parallel on an A100. The curriculum starts with walking forward, then gradually adds turning, foot clearance, reverse and harder variations in friction, center of mass, servo response and bus latency. This worked much better than introducing everything at once, which kept killing the gait before it had properly formed.
Most of the work was stopping the policy from finding cheap tricks. In simulation it discovered that lifting a foot by 2 mm counted as a step, even though backlash and compliance meant the same foot would just slide on the real robot. We ended up gating the main gait reward on actual clearance:
lifted = peak_height >= min_clearance
reward = (air_time_in_range & lifted).sum()No lift, no reward. The final actor is a 66 → 128 → 128 → 8 MLP of about 100 kB, including its normalizer, which leaves enough flash for several policies that we can swap between live.
personality
We split Walnut's personality into three speeds instead of putting an LLM directly in charge of everything. Reflexes react to being picked up, falling over or being told to stop without waiting for the model; an idle loop adds occasional winks, chirps and sleepy eyes; and a slower model loop handles context, camera frames and more deliberate actions.
The layers share the same body through a small priority system: idle < cognition < human < reflex. This means a random wink can't interrupt a gesture, we can always take control, and Walnut stops immediately if someone picks it up. Reflexes also tell the model what they did, so it can remember its reaction afterwards rather than repeating it.
The face is a procedural eye system rather than a set of bitmaps. Expressions are a few values for lids, gaze, size and tilt, with blinks, and micro drifts layered on top.
The model mostly communicates through these expressions, tiny movements and short WALL-E-like sounds. We decided that speech should be rare to make it less intrusive too. Also note that the textual output of the VLM was seen as internal dialogue between Walnut and itself. Only the content in <say> tags was actually spoken out loud.