Introduction

If you have ever tried to get a robot arm to pick up a cube, you know where the time goes: training a policy, writing a planner, and handling every way a grasp can fail. Embodied-reasoning models change that. In this post, we connect Google’s Gemini Robotics-ER 2 to a low-cost SO-101 arm, with one USB camera and four ArUco markers, and get it to follow instructions in natural language. You can run all of it in simulation first, and the code is open source on GitHub.

Figure 1: The web UI driving the SO-101 from a natural-language instruction

Why we built this

Robotics has a reputation for being hard, and much of that difficulty is exactly what embodied-reasoning models like Gemini Robotics-ER 2 are built to take on: recognising objects in a camera image, understanding an instruction, deciding what to do next, and noticing when something went wrong.

We wanted to showcase what the model can do, and show how little it takes to put it to work. Can a developer with a desk, a low-cost arm and an API key get a robot to follow natural-language instructions? We think the answer is yes: in minutes in simulation, and after a short calibration session on a real arm.

We set three rules for ourselves:

  1. Affordable hardware. An SO-101 arm (the open-source LeRobot design), a single USB Arducam and ArUco markers. There is no depth camera, motion capture or force sensor. Anyone can reproduce it.
  2. Let the model do the AI part. The model sees the scene, understands the instruction, decides what to touch and when, and checks the result. Deterministic code only turns its answers into joint angles and keeps the arm safe.
  3. Nothing more. No reinforcement learning, no policy training and no large simulation stack. The project is Gemini, kinematics and a friendly web UI (powered by Gradio).

The AI side turned out to be the easy part. Most of our effort went into the plumbing around it: calibration, geometry and delivering camera frames to the model in the right order. That plumbing is what the repository packages up, so you can skip it. Clone it, run it in simulation, then take it to a real arm and instruct a robot manipulator in natural language.

The brain: Gemini Robotics-ER 2

Everything in this project hangs on one idea: let the model be the brain. We use gemini-robotics-er-2-streaming-preview over the Gemini Live API (a WebSocket session) and give it function calling as its only way to affect the world. Every movement the arm makes starts as a tool call from the model.

What makes ER 2 remarkable for a low-cost arm is how much it understands from a single camera image:

  • It can point. Given an image, the model returns points as normalised [y, x] coordinates in [0, 1000]. It doesn’t need to know anything about the camera or the robot; it only needs to say where in the picture the thing is. That one ability replaces a trained object detector.
  • It understands the scene and the request. It can tell the cube from the tape and from the gripper. It splits “place it on the left” into a sequence of tool calls, and asks for clarification when an instruction is ambiguous.
  • It closes the loop. After every action it receives a fresh frame and checks whether the world now looks the way it expected, then corrects course.

We expose six tools to the model:

Tool Purpose
move_to_spatial_point(point=[y, x], z_height_cm) Top-down move to an image point at a given height
open_gripper() / close_gripper() Gripper control; closing measures the jaw aperture
move_to_preset(pose_name) Named poses such as HOME
move_to_pose(qpos, duration_sec) Direct joint-space move
get_joint_positions() Read back the arm state

The system prompt enforces a strict Move → Look → Correct loop. The model performs one physical action per step, looks at the post-action frame, reads the measured values the controller returns, and only then decides the next step. Pretty much like a state machine, but with reasoning!

One design choice turned out to matter more than any other, and it is about playing to strengths: the model never estimates millimetres. It points at the object. The controller measures how far the gripper is from the target and reports it back (for example perception.measured_offset = dx +6.2, dy −1.0 mm). The model then narrates and decides, and never offsets a point by itself. Gemini is excellent at where and whether; sub-centimetre arithmetic on pixels is better left to geometry calculations. Each side does what it does best.

Architecture

The model is the brain; the rest of the repository exists to give it eyes, hands and a safety net. The code is split into five layers. Each layer may import only from the layers below it, and import-linter enforces that rule in the pre-commit hooks:



      ┌─────────────────────────────────────────────────────────────────┐
      │  ui/          FastAPI + Gradio web app, Three.js 3D viewer      │
      ├─────────────────────────────────────────────────────────────────┤
      │  agent/       Gemini Live session, tools, pick loop             │
      ├─────────────────────────────────────────────────────────────────┤
      │  vision/      camera capture, frame buffers, jaw-tag locator    │
      ├─────────────────────────────────────────────────────────────────┤
      │  simulation/  MuJoCo SO-101 model, synthetic calibration        │
      ├─────────────────────────────────────────────────────────────────┤
      │  robot/       servo driver, IK, homography, camera model        │
      └─────────────────────────────────────────────────────────────────┘
      each layer imports only from the layers below it  (import-linter)
      tools/  operator utilities (setup wizard, preflight) sit on top

The pick loop

Gemini decides what to do, and the physical world is unforgiving, so the controller backs it up with a few promises the model can rely on:

                ┌─────────────────────┐
         ┌─────►│       HOVER         │◄────┐  not aligned:
         │      │   3 cm above target │─────┘  correct and re-hover
         │      └──────────┬──────────┘
         │                 │ aligned: measured offset within tolerance
         │                 ▼
         │      ┌─────────────────────┐
         │      │  DESCEND and CLOSE  │
         │      └──────────┬──────────┘
         │                 │ servo aperture: something in the jaws
         │                 ▼
         │      ┌─────────────────────┐
         └──────┤  LIFT and VERIFY    │
   missed,      │  straight up        │
   up to 3      └──────────┬──────────┘
   attempts                │ confirmed by aperture (vision may veto)
                           ▼
                ┌─────────────────────┐
                │         MOVE        │──► place, open, HOME
                └─────────────────────┘
  1. Hover. The model points at the object, and the controller moves the arm 3 cm above it. The target is locked to the object’s footprint, so later moves keep tracking the same object.
  2. Measure. The jaw tag gives the real gripper position. If the gripper isn’t within tolerance, the controller applies a bounded correction and the model re-hovers.
  3. Descend and grasp. Descent is only allowed after an aligned hover. close_gripper() reads the servo aperture to tell “closed on a cube” from “closed on air”.
  4. Lift and verify. The arm lifts straight up and checks the aperture again. Vision can veto the result if the object is still on the table.
  5. Move around. Moving away is only allowed with a confirmed grasp.

Retries are bounded: three grasp attempts per object. Beyond that, the arm stops and the model explains the situation to the user.

Figure 2: The SO-101 running the pick loop

One step of the loop

Every physical action is one round trip through Gemini. The model looks at a frame, points at something and calls a tool. The controller turns the point into a motion, measures the result, and sends the answer back together with a fresh frame:

  ┌────────────┐  "pick up the red cube"   ┌──────────────────────┐
  │  Operator  │ ────────────────────────► │  agent/              │
  │  (web UI)  │ ◄──────────────────────── │  VLAController       │
  └────────────┘   narration, reasoning    └──┬────────────────▲──┘
                                              │                │
                         (1) one ordered turn │                │ (4) FunctionResponse:
                             [frame, text]    │                │     result, measured
                                              ▼                │     offset, NEW frame
                                  ┌────────────────────────────┴──┐
                                  │   Gemini Robotics-ER 2        │
                                  │   Live API, function calling  │
                                  └───────────────┬───────────────┘
                                                  │ (2) tool call
                                                  │     move_to_spatial_point([y, x], 3 cm)
                                                  ▼
  ┌───────────────────────────────────────────────────────────────────┐
  │  agent/tools.py  (3)  point ─► mm ─► IK ─► move ─► measure        │
  └─────────────────────────────────┬─────────────────────────────────┘
                                    ▼
                     SO-101 arm  (real servos or MuJoCo)

The same code drives the simulation and the real arm. Both implement one RobotInterface, so everything above the robot layer is identical in both modes, and we could develop and test the whole loop without touching hardware.

How frames move

One capture thread feeds two buffers: the model always gets the untouched frame, and only the operator sees overlays.

    ┌──────────────────────┐          ┌──────────────────────┐
    │  USB camera  (real)  │    or    │  MuJoCo render (sim) │
    └──────────┬───────────┘          └───────────┬──────────┘
               └─────────────────┬────────────────┘
                                 │ raw RGB, one capture thread
                                 ▼
           ┌───────────────────────────────────────────────┐
           │               vision/frames.py                │
           │   ┌──────────────────┐  ┌──────────────────┐  │
           │   │  pristine buffer │  │ annotated buffer │  │
           │   └────────┬─────────┘  └─────────┬────────┘  │
           └────────────┼──────────────────────┼───────────┘
                        │ untouched JPEG       │ MJPEG + overlays
                        ▼                      ▼
              ┌───────────────────┐  ┌───────────────────┐
              │  Gemini ER 2      │  │  Operator UI      │
              │  never sees any   │  │  markers, IDs,    │
              │  overlay          │  │  target pointers  │
              └───────────────────┘  └───────────────────┘

From a point in an image to a joint angle

With one RGB camera and no depth sensor, the bridge between what the model sees and where the arm goes is a short chain of classical geometry, and Gemini’s pointing is what keeps it short:

   [y, x]   normalised 0..1000      ◄── Gemini points at the object's
      │                                  contact centre on the table
      │  × image width / height
      ▼
   (u, v)   pixels
      │
      │  homography H, fitted to the 4 ArUco corner markers
      ▼
   (X, Y)   mm on the table, robot base frame
      │
      │  + hover height, + bounded bias correction
      ▼
   (X, Y, Z)   grasp-site target
      │
      │  damped least-squares IK + null-space posture
      ▼
   q = (pan, lift, elbow, wrist flex, wrist roll)
      │
      │  cosine S-curve, zero velocity at both ends
      ▼
   servos  /  MuJoCo actuators

Pixels to millimetres. Four ArUco markers at the corners of a rectangle on the table define a homography that maps any image point on the table plane to robot millimetres. That is why Gemini is asked to point at an object’s bottom contact centre, where the table plane applies.

Where is the gripper, really? Hobby servos are not precise enough to close the loop on their own, so a 25 mm ArUco tag on the fixed jaw is located in every frame. Because the tag sits above the table, perspective shifts it. We recover the camera centre C from the homography and the lens field of view, and correct for the tag’s height h:

xytable=Cxy+(H(u,v)−Cxy) Cz−hCzxy_{table} = C_{xy} + \big(H(u, v) - C_{xy}\big)\,\frac{C_z - h}{C_z}

The controller compares the corrected gripper position with the target, and that difference is the measured_offset the model reads:

   jaw tag (u, v) ──► H ──► parallax fix ──► − tag offset ──► gripper (X, Y)
                            using camera C                            │
                            and height h                              │
                                                                      ▼
                         measured_offset  =  target (X, Y)  −  gripper (X, Y)
                                         reported back to the model

Millimetres to joints. Damped least-squares inverse kinematics, with null-space projection to stay near a comfortable posture, turns a target into five joint angles. Moves follow a cosine S-curve, and approaches from low poses rise to a safe travel height first, so the jaws don’t sweep objects across the table.

Setup in five numbers. python main.py --setup asks for five tape-measure values from the shoulder-pan axis (near X, depth, width, lateral offset and marker size), writes the calibration, reports which parts of the rectangle the arm can reach, and prints an A4 placement guide.

Your turn

Everything above fits in five commands. You don’t need a robot to start: the simulation runs on your computer, and all it asks for is a Gemini API key.

git clone https://github.com/Ekumen-OS/GeminiRobotics-SO101.git
cd GeminiRobotics-SO101
uv sync --extra ui
cp .env.example .env               # add your GOOGLE_API_KEY
uv run python main.py --mode sim   # http://localhost:7860

Type “pick up the cube and place it on the left” and watch Gemini look at the scene, point at the cube and guide the arm there, one careful step at a time. When you’re ready for the real thing, the README walks you through --setup, from printing the markers to verifying the calibration overlay.

Conclusion

Not long ago, a robot that could see a table, understand a sentence and decide what to do next was a research-lab project. Now it starts with a conversation: you describe what you want, and the model works out how to get there.

We hope this is only the beginning. Swap the cube for something new, give the arm a harder task, mount a different camera, or fork the code and make it your own. The model can see, reason and plan, and where it goes next depends on what you ask of it.

Try it, break it, and tell us what you find. Star the repo, open an issue, or send a pull request (see CONTRIBUTING.md).

If your team wants to take embodied reasoning from a desk demo to robots in the field, our engineers can help. Talk to us: contact@ekumenlabs.com

Credits: the SO-101 hardware and MuJoCo model come from TheRobotStudio/SO-ARM100 and Hugging Face LeRobot. The physics engine is MuJoCo by Google DeepMind. Embodied reasoning is by Gemini Robotics.