Pro-0: Towards Dexterous Policies with Zero On-Robot Data via Hardware–Software Co-Design

Policy Videos

Autonomous rollouts on the robot at 1× speed, with policies trained only on human demonstrations.

Abstract

Introduction

Imitation learning trades off data-collection throughput against transfer to the robot. Teleoperation records data on the target robot but is slow and needs robot hardware, while human videos are plentiful but hard to turn into executable robot actions. UMI showed that co-designing the collection device, its observations, and the policy interface lets portable human demonstrations transfer directly to a two-finger gripper. Generalist shows the scale that wearable human-data collection can reach, and HiFi-UMI shows that sufficiently accurate demonstrations can support direct deployment without robot data. Pro-0 asks how to extend this co-design principle to dexterous hands, reaching >80% success on a real robot from human demonstrations alone.

Existing dexterous systems each give something up. Exoskeletons such as DexUMI match the robot hand’s kinematics but constrain natural movement. Motion-capture gloves leave the hand free, but embodiment and observation gaps hinder direct transfer: DexCap relies on point clouds augmented with robot-hand geometry, DexWild reports poor human-only performance, and DexViTac reaches only 17.5% success with vision alone, versus 85.8% with multimodal input. Pro-0 instead aligns the human and robot at collection time:

• A thin tactile glove fits both the human hand and the ProHand, so demonstrators move naturally.
• A glove-mounted wrist camera reproduces the robot’s wrist view, so no image editing is needed.
• The glove’s tactile signals correct the tracked hand during grasps, where vision alone underestimates finger curl.
• Tracked hand motion is retargeted to the arm and hand with inverse kinematics, turning each demonstration into robot training data.

Method

A human demonstration recorded from the robot’s viewpoint and mapped into its joint space can be treated as a robot demonstration. Pro-0 turns each demonstration into wrist-camera images paired with executable robot actions, the same form as teleoperation data, in four steps.

1. Data capture

The capture system is built so that a human demonstration already looks like robot data:

• Matched viewpoint. A wrist camera on the glove sits where the robot’s wrist camera sits relative to its palm, so a policy trained on human data needs no image adaptation on the robot.
• Fixed tracking. A Meta Quest 3 on a stand, placed at a pose taken from the robot cell’s CAD model, tracks the demonstrator’s arm, wrist, and fingers in the robot’s workspace.
• Natural hands. The thin glove is the only thing worn on the hand, so demonstrators keep full dexterity and can feel contact.
• Shared touch. The glove’s tactile sensors use the same layout on the human hand and on the ProHand, so tactile observations can transfer to the robot in future work.

Modalities recorded per episode, for both hands, timestamped on a common clock.

Besides hand tracking, the Quest’s body tracking fits a skeleton to the demonstrator, giving arm, torso, and head poses as well. The glove’s tactile unit and wrist camera connect to the headset over USB, so every stream is captured in sync, and each episode is saved as a single time-indexed log.

2. Tactile-based hand pose refinement

Visual hand tracking is least reliable exactly when it matters most: during a grasp, the object hides most of the fingers, the tracker falls back on temporal history, and it underestimates how far the fingers are curled. Retargeted directly, the robot hand would stop short of the object. The glove’s tactile sensors detect contact regardless of occlusion, so we use them to tell which fingers are touching the object, and we register a grasp whenever the thumb and at least one other finger are in contact. During each grasp, the finger keypoints are offset inward: each contacting finger moves toward the thumb, and the thumb moves toward the average position of the contacting fingertips, closing each finger by up to 25°. The refined motion closes around the object and gives the robot a stable grasp.

3. Human-to-robot action transfer

Because the headset sits at a known pose in the robot cell, every tracked keypoint can be mapped directly from the headset frame into the robot base frame with a single rigid transform taken from the CAD model, with no per-session calibration. Retargeting then solves, offline and for every timestep, the joints of a 7-DoF arm and a 22-DoF ProHand (2 wrist and 20 finger joints) that reproduce the human motion.

• Arm. Differential inverse kinematics drives the base of the robot wrist to the human wrist pose, taken from the last joint of the tracked arm chain. A posture term anchored to the current configuration keeps the redundant elbow still unless tracking needs it, and each timestep is warm-started from the previous solution for smooth trajectories.
• Hand. With the wrists aligned, keypoints at every finger joint and fingertip of the robot hand are matched to their human counterparts, using the same solver with the posture term anchored to the initial hand configuration. Matching every joint, not just the fingertips, preserves the shape of each finger.
• Executable by construction. Joint limits and velocity bounds are enforced in the solver and the hand’s mechanical couplings after each step, so the resulting trajectory replays on the robot without further processing.

4. Policy learning

Each retargeted demonstration pairs wrist-camera images with robot joint trajectories, which we use to train a single multi-task diffusion policy.

• Image-only input. The policy sees only the current wrist image, with no image history or proprioception. Because the glove camera matches the robot camera, this input looks the same in training and deployment.
• Relative actions. Each action is the end-effector pose plus the 22 wrist and finger joints, expressed relative to the current state, so the policy describes how the hand should move rather than where it is.
• Architecture. A frozen DINOv2 encoder turns the image into a grid of patch tokens, kept as a sequence so the policy can attend to where objects and fingers appear. A diffusion transformer cross-attends to these tokens, with the diffusion step injected through adaptive layer norm.
• Deployment. The policy denoises an action chunk in 10 DDIM steps; the robot executes the first half through inverse kinematics, then replans from a new image.

Experiments & Results

We test whether a policy trained only on human demonstrations can perform dexterous manipulation on a real robot, using three short tasks that cover reaching, pressing, grasping, and releasing.

• Buzzer: press a buzzer firmly enough to trigger it.
• Bucket: grasp a bucket and lift it off the table.
• Beanbag: starting with a beanbag in hand, release it into a bucket.

We recorded 250 demonstrations per task (750 total) and ran 20 evaluation trials per task, with object positions varied over the same 40 cm × 40 cm area. A trial scores 0.5 for reaching the object or target and 1.0 for also completing the interaction.

Real-robot results: 20 trials per task, with a policy trained only on human demonstrations.

Analysis

Pro-0 reaches an average full success rate of 81.7% and completes the approach in 98.3% of trials, so the policy reliably finds its target from the wrist view alone and nearly every failure happens during the interaction. Those failures are small and physical rather than navigational: in Buzzer, both misses made contact but the fingertip slid off the button without triggering it; in Beanbag, the one miss was a premature release that dropped the beanbag outside the bucket. Bucket is the hardest task: its only approach failure came from an incorrect wrist roll, and the grasp succeeded in 12 of the 19 trials that reached the bucket. We attribute this mainly to the data: fingers wrapped around the handle are partly hidden from the headset, so tracked finger curl tends to be underestimated and the retargeted grasps are less consistent across demonstrations than the approach motion.

Limitations

• Fixed tracking. Hand poses still come from a Quest headset on a fixed stand. This works in our current setting, but collecting demonstrations in the wild will require mobile tracking.
• Task scope. We evaluated only three short tasks. Scaling up both data collection and the range of tasks is needed to assess how well the approach scales.
• Unused touch. The gloves record tactile signals, but the policies in this study do not use them. Tactile sensing as an additional policy input is a natural next step.