Harnessing Robot-Use Agents
via Perception Augmentation
Overview
Foundation vision-language models recognize objects, interpret instructions, and reason about spatial relations. Yet a robot also needs to know where to move, how to orient its gripper, and whether an action actually worked. Inferring these details from RGB images alone leaves fundamental uncertainty about depth, contact, and 3D spatial relationships. We identify accessible perceptual evidence as the critical missing piece between VLM competence and manipulation capability.
Robo-Harness K1 exposes perception as tools. The agent queries calibrated depth, inspects visual anchors with persistent object identities, evaluates grasp hypotheses, and selects generic motions from the returned evidence. Visual reference lines overlay calibrated measurements onto the VLM's existing RGB views, making 3D object relations, orientations, and distances visible without a depth encoder. The K1 harness supplies calibrated spatial evidence, not task-specific pick/place primitives or hidden object poses. The VLM remains responsible for choosing the target, the motion, and the recovery strategy.
The same interface supports both frontier models without fine-tuning and training compact student models: tool-call traces naturally align with next-token prediction, so successful interactions become supervision for what to observe, what to measure, and which tool to call next.
In action
Gemini 3.7 Flash with Robo-Harness K1. Current-view perception overlays show what the agent observed, alongside its function calls.
Locate the black bowl between the plate and the ramekin, then place it on the plate.
Pick the target object and transfer it into the basket.
Ground the drawer handle and control contact while pulling the drawer open.
Grasp the square nut, align it above the peg, and lower it into place.
Pick up a cube and stack it on the other cube.
Lift the pot with two arms. Native success requires the pot bottom to rise more than 10 cm above the table.
Decision observations pause briefly to show the recorded perception overlays and tool calls (key arguments shown). Motion segments retain every recorded control step at simulator speed; overlays are not carried onto changed views. Each clip ends with a two-second hold of the successful terminal state. API waiting time is omitted. Use fullscreen to read the annotations.
The method
K1 connects a vision-language model to calibrated perception, motion, and memory tools through the VLM's existing image-and-language interface. The agent selects what to measure, acts on the evidence, and observes what actually happened.
A text query returns labeled candidate regions with persistent identifiers for the agent to inspect. Calibrated depth queries and region geometry convert RGB-D observations into metric surface positions, principal directions, and relative displacements. Evidence is projected back into the existing external and wrist-camera views as visual reference lines. The agent reads 3D relations directly from annotated images without interpreting raw point clouds or depth arrays.
The agent selects a grasp hypothesis or specifies a motion target. Generic tools control translation, rotation, and gripper opening at adaptive granularity: fine-grained bounded steps for precise alignment, and direct target moves for longer transits that reduce model calls and keep the context history clean. Every action reports achieved motion and residual error, so a requested pose is not confused with a completed movement.
Each detected region receives a persistent identity anchor that survives across observations. Anchor tracking provides frame-to-frame correspondences lifted into world coordinates, separating apparent camera motion from actual surface displacement. Validity flags distinguish live measurements from uncertain or historical references. A sliding context window, searchable full history, and agent-maintained progress records carry information through longer episodes.
Teacher interactions are filtered into executable next-tool-call examples, paired with the observations and tool evidence available at each decision. Qwen3.5-9B is fine-tuned with language-side LoRA and a frozen vision encoder. The student uses the same frozen harness at evaluation without a separate action head. Tool-call traces align directly with the VLM's native next-token prediction objective, enabling a training paradigm with better sample efficiency and generalization than action regression.
Measurements describe observed surfaces, not hidden geometry. Tracking can become uncertain, and a grasp candidate is a hypothesis, not a guarantee of collision-free motion or secure contact. Re-observation and execution feedback remain part of the control loop.
Experiments
On matched configurations, K1 augments GPT-6 Astra from 61.1% to 88.9%, a gain of 27.8 percentage points. Gemini 3.7 Flash with K1 reaches 77.8%, surpassing RGB-only Astra despite being a weaker model. This shows that perception tools and model capability are complementary: a weaker model equipped with K1 can outperform a stronger model without it, and K1 further lifts an already strong model.
At epoch 5, Qwen3.5-9B with K1 reaches 51.2% on training-matched configurations (A), 44.2% on new initial states (B), and 13.9% on held-out task conditions (C), versus OpenVLA's 20.9%, 30.2%, and 0.0% respectively. K1 RUA is the only method to succeed on held-out tasks where every VLA baseline scores 0.0%.
Gemini with K1 transfers to RoboSuite without any target-environment fine-tuning. On the four shared tasks (cube lifting, restacking, stacking, and nut assembly):
90.0%Panda
88.8%UR5e
86.2%IIWA
The small spread across arms supports separating decisions from actuation: the agent reasons about measured geometry, while embodiment-specific adapters control each arm. All arms use PandaGripper to isolate changes in arm kinematics. The broader Panda evaluation includes wiping and two-arm tasks, reaching 72.1% overall. Two-arm lifting reaches 75.0%, while handover reaches only 5.0%, exposing coordination limits beyond geometric target-reaching.