Ego2Robot Learning Paper PDF
A SURVEY / NOVEMBER 2026

From to Robot Manipulation.

In the era of embodied intelligence

Yi Wang1,†,*Jingqiu Wu1,†Lei Yao1,†Zhaoshuo Li2Haoyu Zhu1
Yirui Zhou1Huaiyuan Xu1Ying Zheng1Chuang Yang1Lap-Pui Chau1

1 Department of EEE, The Hong Kong Polytechnic University

2 Cosmos Lab, NVIDIA

† Equal contribution * Corresponding author

wang1241@e.ntu.edu.sgyi-eie.wang@polyu.edu.hk

PaperGitHub

TL;DR

Egocentric human video captures close-range hand–object interactions and how tasks unfold, without recording robot actions.

We review four ways to learn or adapt robot manipulation policies from EHV: what information they extract, how policies use it, and what additional data they need for training or deployment.

Taxonomy

TaxonomyPDF

Learning paradigms

Training data

01 / HUMAN22 resources
Egocentric human hand–object interaction in an everyday kitchen, from EgoDex

Egocentric human videos

EPIC-KITCHENS · Ego4D
EgoDex · HD-EPIC
Browse resources
02 / ROBOT17 resources
Robot manipulation demonstration with recorded embodiment-specific actions, from RoboMimic

Robot demonstration data

DROID · RH20T
Open X-Embodiment · RoboMIND 2.0
Browse resources
03 / SYNTHETIC16 resources
A simulated robot interaction illustrating synthetic and simulated training data, from RoboTurk

Synthetic & simulated data

MimicGen · RoboTwin
EgoSim · TASTE-Rob
Browse resources

Images: EgoDex, RoboMimic, RoboTurk

Literature

BibTeX
Research landscapePDF

Loading references…

Literature cited in the Ego2Robot survey
REF.WORKDISCUSSIONYEAR

Open challenges

  1. 01

    AI agents for Ego2Robot learning

    Check whether video-derived steps are feasible with available robot skills, and verify their outcomes through execution feedback.

  2. 02

    In-context adaptation from demonstrations & feedback

    Compare matched, mismatched, and absent context with fixed policy weights, and report adaptation gains against robot interaction budgets.

  3. 03

    Cross-embodiment alignment & physical feasibility

    Preserve task outcomes across embodiments while accounting for reconstruction uncertainty, missing contact information, and the cost of extra sensing.

  4. 04

    Evaluation, attribution & interpretability

    Measure robot-data efficiency and generalization against equally sized non-egocentric video baselines, using diagnostics and robot trials to explain the gains.

  5. 05

    Reliable prediction & closed-loop control

    Test whether predictions respond correctly to action changes and help the robot replan or recover within closed-loop latency limits.

Citation

BIBTEX
@misc{wang2026egocentrichumanvideorobot,
  title={From Egocentric Human Video to Robot Manipulation in the Era of Embodied Intelligence},
  author={Yi Wang and Jingqiu Wu and Lei Yao and Zhaoshuo Li and Haoyu Zhu and Yirui Zhou and Huaiyuan Xu and Ying Zheng and Chuang Yang and Lap-Pui Chau},
  year={2026},
  eprint={2611.13610},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2611.13610},
}

Original PDF