Skip to content

[python] Support distributed LeRobot import with Ray - #10483

Draft
XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/lerobot-ray-import
Draft

XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/lerobot-ray-import

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

Allow LeRobot v3 imports to run across Ray workers instead of a single process:

conn.load_from_lerobot(
    "robot_data", "oss://bucket/lerobot_dataset",
    engine="ray", concurrency=8, source_options=source_options,
)

Workers reuse the existing validation/writer path and prepare complete Episode groups. Episodes sharing video files stay together. The driver commits frames only after every task succeeds, then publishes companion metadata, the initial BTree and optional tags. Existing calls remain single-process.

Requires an initialized Ray cluster and shared source/target storage. Supports FileIO URIs and shared directories; Hub IDs remain on the single-process path. Tasks are not automatically retried. Publication across tables is not atomic; failed imports may leave partial tables and uncommitted files.

Tests

72 passed, 29 optional-dependency skips. Six new tests cover grouping, API validation, real Ray-worker image/video imports against serial results, tags and worker failure without a frame commit. Flake8 and git diff --check passed.

Validated on a local two-CPU Ray cluster (Ray 2.55.1); multi-node OSS performance has not been measured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant