RoboCasa

RoboCasa is the kitchen-scale, long-horizon manipulation environment. In RPent it is driven by the RLDX-1 VLA policy, served over HTTP RPC by default (matching LIBERO); a pickle-framed socket transport is also supported. See robots/robocasa/vla_server.py and robots/robocasa/robot_spec.py for the wire/transport selection.

Note

The task/global Target50 protocol is in robots/robocasa/eval/target50_v2.json; target50.json retains v1. RPent uses ordinary single-task rpent --robot robocasa commands for its 340 cells.

Runtime flow

RoboCasa365 uses a PandaOmron mobile manipulator and the frozen RLDX-1 policy. The integration is planner-agnostic: API planners, Claude Code and Codex use the same RoboCasa toolkit. See Agentic Planner for credentials and backend configuration; users supply credentials outside the repository.

rpent CLI -> task-memory sync -> environment and VLA servers
          -> planner toolkit -> final environment state.success

Unless external endpoints are supplied, RPent starts an environment server and a VLA server for each run. The planner selects primitives using the live task language and observations; RLDX-1 executes manipulation skills. Only the environment’s own _check_success() result, surfaced as state.success, determines evaluation success. The public protocol uses ordinary single-cell commands, not an included batch launcher. See Harness VLA for the Harness VLA overview.

Installation

RLDX-1 requires Python 3.10. Create a dedicated environment and install the complete RoboCasa365 stack with .[robocasa]:

uv venv --python 3.10
source .venv/bin/activate

First install a matching CUDA-enabled PyTorch and torchvision pair using the PyTorch installation selector for your GPU, driver and Python version. Run the selected command in this environment (use uv pip in place of pip). The RLDX dependency requires Torch >= 2.7 and torchvision >= 0.22; choose a mutually compatible pair, not two independent versions. Then install RPent:

uv pip install -e ".[robocasa]" \
   --constraint robots/robocasa/eval/target50-constraints.txt
uv pip check

The RoboCasa-specific constraints file pins the compatibility-sensitive package versions validated for Target50 reproduction without narrowing RPent’s shared LIBERO or RoboTwin dependencies. The robocasa extra tracks the maintained rpent branches of RoboCasa, RLDX and Robosuite for both ordinary runs and Target50. The manifest records these branches, not frozen source commits. RoboCasa’s rpent branch declares the distribution name rpent-robocasa365; do not co-install the rlinf-robocasa365 distribution, which provides the same import package. No second source-install step is required. Branches can advance, so record the resolved Git commits and installed versions with each evaluation:

uv pip freeze > installed-requirements.txt

Keep this environment record with the experiment artifacts. Installing the same branch later is not a guarantee of identical source code. The checkpoint, backbone support resources remain fixed below; task/global memory follows the selected branch. The constraints do not pin Torch, torchvision or a CUDA backend. Installation retains a compatible installed pair; dependency conflicts must be resolved before running. The manifest’s reference_accelerator records the previously used Torch 2.7.0 / torchvision 0.22.0 / CUDA 12.6 combination as provenance only, not an installation requirement. Record your actual versions with your results and run the component checks below; other combinations are not presumed to have identical numerical results. Package mirrors are optional user configuration, not part of the evaluation protocol.

Note

flash-attn is optional; RLDX-1 uses PyTorch SDPA when it is absent. If installing it, follow the FlashAttention installation guidance and select a build compatible with your Python, Torch, CUDA and GPU. This guide does not prescribe a machine-specific wheel.

Post-install setup

Download the kitchen assets (~10 GB) outside site-packages so they survive reinstalls. Target50 does not use RoboCasa dataset or teleop macros, so skip the optional private-macros setup:

robocasa-download-assets --assets-path ~/.robocasa/assets --no-macros -y

It prints the environment variable to export afterwards; add it to the shell that launches rpent:

export ROBOCASA_ASSETS_PATH=~/.robocasa/assets

The external root requires the six downloaded collections and the bundled static scene, arena and fixture files. The corrected installer supplements the latter without replacing different existing content. Re-run with --skip-existing to verify successful download inventories; a nonempty directory alone is not a complete installation. Keep official attribution files and finish interrupted downloads before starting experiments.

Resource publication is atomic: interrupted copies do not leave half-written final files. To repair conflicting resources left by an earlier installer, rerun the same command with explicit overwrite permission:

robocasa-download-assets --assets-path ~/.robocasa/assets --no-macros --overwrite -y

--overwrite takes precedence over --skip-existing and replaces only resource files in the installation scope, not unrelated files or whole directories. Without it, different existing content is preserved. Atomic no-overwrite publication requires hard-link support on the destination filesystem. Temporary files left by a killed process do not block retries.

New collections require space for the ZIP and one unpacked copy: staging is published without copying the payload again. Existing installations need additional space during replacement. --skip-existing avoids downloading and comparing completed collections; bundled static files are checked separately.

Navigation camera

The robocasa extra installs the rpent branch of RLinf/robosuite, which provides the Omron base’s fixed navview camera. Its composed MuJoCo name is mobilebase0_navview. Navigation RGB-D and world-map rendering validate the camera when they first request it, and report an error if it is missing. No manual site-packages XML patch is required. Target50 uses this same maintained branch; record the resolved revision with the environment information above.

RLDX-1 checkpoint

The --vla-model-path flag on the run commands below expects a local path to the RLDX-1-FT-RC365 checkpoint (the RoboCasa365 fine-tune). Download it from HuggingFace:

hf download RLWRLD/RLDX-1-FT-RC365 \
   --revision 587e9ecdcc5e7184fcc17f58713908edff5af041 \
   --local-dir ./checkpoints/rldx-1-ft-rc365

If the download is slow, use the HF mirror:

HF_ENDPOINT=https://hf-mirror.com hf download RLWRLD/RLDX-1-FT-RC365 \
   --revision 587e9ecdcc5e7184fcc17f58713908edff5af041 \
   --local-dir ./checkpoints/rldx-1-ft-rc365

RLDX-1 backbone support files

The FT checkpoint contains the weights but also references RLWRLD/RLDX-1-VLM for architecture, processor and tokenizer metadata. Target50 freezes revision 4b9f870d1287e0d38d7eb1445e6d8c60afe66dd7: 15 non-weight files, about 16.4 MB including documentation and images. Download these into the same cache used when launching RPent:

export HF_HOME="$PWD/.cache/huggingface"
export HF_HUB_CACHE="$HF_HOME/hub"
hf download RLWRLD/RLDX-1-VLM \
   --revision 4b9f870d1287e0d38d7eb1445e6d8c60afe66dd7 \
   --include "*.json" "*.txt" "*.jinja" "*.md" "*.png" ".gitattributes" \
   --exclude "*.safetensors.index.json"

No additional base weights are required. Keep these cache variables in the launch shell; do not shadow them with an empty TRANSFORMERS_CACHE. The RoboCasa VLA worker automatically uses this same support revision for both ordinary and Target50 runs, including separately started RPent VLA servers. There is no extra revision flag or manual cache-ref edit. The pin applies only to backbone metadata, not the weights selected by --vla-model-path. Model and asset licenses apply separately from RPent’s code license.

Task and global memory

With --memory-profile hf (the default), the CLI and Dashboard synchronize robocasa/** from the current main branch of the RLinf/RPent-memory dataset. Memory is not pinned to a commit. The layout is:

memory/robocasa/
├── task_only/
│   ├── <Task>_s0.json
│   ├── <Task>_s0_recipe.jsonl
│   └── <Task>.md              # optional
└── global/
    └── GLOBAL_MEMORY.md

The default --memory-policy task-global selects the current task’s available JSON, recipe and Markdown, together with global/GLOBAL_MEMORY.md. The planner must read every selected file completely through the RPent read_text_file tool before the first robot action. The prompt and file tools share this selection. RPent file tools deny other tasks’ memory; this is a tool restriction, not an operating-system sandbox.

Use --memory-policy task-only for a comparison using the same task files. This disables the global layer in both the prompt and file tools. A missing JSON/JSONL pair is allowed: the planner continues with live observations and the enabled global layer. A half-present pair is an error. Missing optional Markdown is logged. The global file must exist in task-global mode. Files are discovered by task name and directory; no extra index is needed.

Live task_language, RGB-D observations, task progress and tool results take precedence over memory. Apply a global strategy only when its visible preconditions hold. Continue VLA calls while contact, a held object, fixture progress or a counter increase shows progress. After two consecutive calls without contact or visible progress, re-ground and make a bounded pose adjustment. Every VLA call uses the full, verbatim live task language. Historical vla_act entries describe strategies; use current tools and never replay historical coordinates. Reset remains unavailable.

To use local memory, download into a fresh directory and select the local profile. This also avoids retaining deleted files in an older download directory:

hf download RLinf/RPent-memory --repo-type dataset \
   --include 'robocasa/**' --local-dir ./target50-memory

rpent --robot robocasa \
      --task-name OpenDrawer --split target --seed 1 \
      --vla-model-path /path/to/rldx \
      --planner codex --model gpt-5.5 --reasoning-effort xhigh \
      --memory-profile local --memory-dir ./target50-memory/robocasa \
      --memory-policy task-global

For the maintained reproduction memory branch, add --revision reproduce/memory to the download command. This selects a mutable branch, not a fixed data version. Use a fresh directory when switching branches. Both branches use the same RoboCasa task/global layout. Future memory updates require only editing the appropriate task or global file; code changes are not needed.

Custom memory sources

To use another HF dataset with the same robocasa/ layout, set RPENT_MEMORY_HF_REPO=<owner>/<dataset> when launching RPent with --memory-profile hf. This accepts a dataset repository ID, not a browser URL.

For a custom subdirectory or a maintained branch, download that subtree into a fresh directory and use the local profile:

hf download <owner>/<dataset> --repo-type dataset \
   --include '<subpath>/**' --local-dir ./custom-memory

# Add to the RPent run command:
# --memory-profile local --memory-dir ./custom-memory/<subpath>

The selected directory must contain task_only/ and, for task-global, global/GLOBAL_MEMORY.md. Add --revision <branch> to the HF download command when selecting a branch. No delivery package or migration script is required.

Harness VLA Target50 reproduction protocol

The current robots/robocasa/eval/target50_v2.json protocol (robocasa-harness-vla-v2) uses task/global memory without pinning its data version. It preserves the target task/seed matrix, cell time limits, no-reset rule, environment success predicate, and 40/999/8 RLDX settings. The protocol ID identifies the result format and evaluation rules; it lets the validator distinguish v1 from v2 and does not select a memory data version.

Results record the memory policy, selected/missing files and complete reads. The validator rejects mixed policies and incomplete or cross-task reads. Keep memory unchanged across cells of a comparison and retain the downloaded files and validation evidence locally.

The historical target50.json v1 manifest and published task-only results remain available. A new task-only run uses v2 and is a new comparison; it does not reproduce the historical corpus. Validate historical result records with --manifest robots/robocasa/eval/target50.json. Source dependencies follow the recorded rpent branches.

RoboCasa Target50 matrix

Split	Tasks	Seeds per task	Cell timeout	Cells
Atomic	18	1–10	1800 s	180
Composite-Seen	16	1–5	3600 s	80
Composite-Unseen	16	1–5	3600 s	80
Total	50			340

The tasks split into three groups:

• Atomic (18) — single-primitive articulation and pick-place tasks: CloseBlenderLid, CloseFridge, CloseToasterOvenDoor, CoffeeSetupMug, NavigateKitchen, OpenCabinet, OpenDrawer, OpenStandMixerHead, PickPlaceCounterToCabinet, PickPlaceCounterToStove, PickPlaceDrawerToCounter, PickPlaceSinkToCounter, PickPlaceToasterToCounter, SlideDishwasherRack, TurnOffStove, TurnOnElectricKettle, TurnOnMicrowave, TurnOnSinkFaucet.

• Composite seen (16) — multi-step tasks on kitchen layouts seen during training: ScrubCuttingBoard, StackBowlsCabinet, WashLettuce, RinseSinkBasin, PreSoakPan, StirVegetables, LoadDishwasher, SteamInMicrowave, SetUpCuttingStation, GetToastedBread, DeliverStraw, KettleBoiling, PrepareCoffee, StoreLeftoversInBowl, SearingMeat, PackIdenticalLunches.

• Composite unseen (16) — multi-step tasks on layouts not seen during training (generalization eval): ArrangeBreadBasket, ArrangeTea, BreadSelection, CategorizeCondiments, CuttingToolSelection, GarnishPancake, GatherTableware, HeatKebabSandwich, MakeIceLemonade, PanTransfer, PortionHotDogs, RecycleBottlesByType, SeparateFreezerRack, WaffleReheat, WashFruitColander, WeighIngredients.

Pass any of these to --task-name. The full RoboCasa catalog is larger; see the RoboCasa upstream.

Running a task

HTTP RPC endpoints whose hostname is 127.0.0.1 or localhost are reached directly, whether RPent starts the worker or the user supplies the endpoint. Every other hostname and IP uses the standard proxy environment. Codex applies the same two-host exception only to its child process for the local MCP connection. Leave HTTP_PROXY and HTTPS_PROXY unchanged when Hugging Face, a remote planner, or another remote service requires them; the default runtime does not require a shell-wide NO_PROXY setup.

If a user-supplied local service uses another hostname or IP and should be reached directly, add that exact value to the user’s existing NO_PROXY and no_proxy configuration.

The RoboCasa CLI flags are registered by robots/robocasa/__init__ and are visible under rpent --robot robocasa --help:

rpent --robot robocasa \
      --task-name OpenDrawer \
      --split target \
      --seed 1 \
      --vla-model-path /path/to/rldx \
      --planner claude_code \
      --model claude-opus-4-8

RoboCasa does not select a planner implementation; any planner supported by RPent can run this robot. See Agentic Planner for configuration.

For Target50, first download the fixed resources above, then invoke one ordinary command for each manifest cell. The Codex reference profile is gpt-5.5, xhigh, and max_turns=100; RoboCasa itself remains planner-agnostic. For the scene identity, use the ordinary --seed argument and do not set RLDX_RESET_SEED. Ordinary RoboCasa uses max_chunks=70; Target50 alone overrides it to 40. Freeze the Target50 RLDX execution values first:

export RLDX_MAX_CHUNKS=40
export RLDX_SETTLE_PATIENCE=999
export RLDX_ACTION_STEPS_PER_CHUNK=8
unset RLDX_RESET_SEED

The first OpenDrawer Atomic cell is:

rpent --robot robocasa \
      --task-name OpenDrawer --split target --seed 1 \
      --vla-model-path ./checkpoints/rldx-1-ft-rc365 --cuda-device 0 \
      --planner codex --model gpt-5.5 --reasoning-effort xhigh \
      --max-turns 100 --planner-timeout-s 1800 \
      --memory-profile local \
      --memory-dir ./target50-memory/robocasa \
      --output-dir ./runs/target50/atomic/OpenDrawer_s1

Use --planner-timeout-s 3600 for either composite split. Execute Atomic, Composite-Seen, and Composite-Unseen in that order. A cell succeeds only when the final recorded environment state has state.success=true; the planner’s finish(status=...) argument is not an evaluation label. Valid task failures and planner timeouts are not retried. Retry an infrastructure failure only when no valid environment result was produced for that cell.

Every completed command atomically writes <output-dir>/result.json using the final environment state.success. The record includes the effective protocol values but omits provider errors and credentials. Once all cells are present under <results-root>/<manifest-split>/<Task>_s<seed>/result.json, validate the fixed denominator and print the task-weighted score with:

python -m robots.robocasa.eval.validate_target50 ./runs/target50

For a separate task-only comparison, pass --memory-policy task-only to both the run command and validator, and use a separate results directory.

Note

Use --env-endpoint / --vla-endpoint to point at already-running servers ([protocol://]host:port); when omitted, RPent spawns the env and VLA daemons in-process and writes their logs to <output_dir>/env_server.log and <output_dir>/vla_server.log.

Historical task-only results

The published v1 task-only Codex evaluation covers 340 cells and reports a 57.00% task-weighted success rate. These are historical aggregates, not results for the current task-global policy. The complete historical table is linked at its recorded code commit. It contains no per-seed traces or failure classifications.

Environment smoke tests

After installing RoboCasa and its assets, run the opt-in environment smoke suite to check simulator installation and interfaces. It requires no planner credentials or VLA checkpoint:

uv pip install pytest pytest-timeout
RPENT_RUN_ROBOCASA_INTEGRATION=1 \
   pytest tests/integration_tests/robots/robocasa/test_target50_runtime_smoke.py -v

The four cases cover OpenDrawer, NavigateKitchen, and PickPlaceCounterToCabinet at seed 1, plus mobile-camera movement. Task checks verify construction/reset, 12D actions, operation cameras, navigation RGB-D/world map, the success predicate, and clean close. The camera check verifies pose and image changes after eight base steps. These real-simulator tests require a working GPU/EGL setup and are separate from offline CPU CI; skipped tests are not passes.

Troubleshooting

Run the environment smoke tests first. After downloading all four resources, use the existing RoboCasa E2E component test to verify VLA worker startup, HTTP RPC and first inference. Install .[test] if needed, select one available GPU and use a fresh output directory:

CUDA_VISIBLE_DEVICES=0 \
RLDX_MODEL_PATH="$PWD/checkpoints/rldx-1-ft-rc365" \
RPENT_E2E_OUTPUT_DIR="$PWD/e2e-robocasa" \
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 NO_ALBUMENTATIONS_UPDATE=1 \
MUJOCO_GL=egl python -m pytest -q \
   tests/e2e_tests/robocasa/test_components.py::test_rldx_component --timeout=300

These checks do not run a planner or create benchmark results. Skipped tests are not passes. Offline variables apply only to the check; ordinary HF memory sync needs network access. Keep remote planner proxies unchanged. The lightweight protocol tests still validate all 50 tasks and the fixed 340-cell denominator; full benchmark execution is a separate procedure.

• For slow package downloads, use UV_HTTP_TIMEOUT=600 and put caches and temporary files on a sufficiently large filesystem. Retry the pinned HF download; apparent shard size is not a completeness check. Do not disable TLS.

• A read-only asset failure needs the corrected RoboCasa dependency, not writable canonical assets. Transformed XML uses the temporary directory.

• For an RLDX offline cache miss, check the support snapshot and cache variables above. NO_ALBUMENTATIONS_UPDATE=1 disables only an import-time version check, not image processing. Keep the existing image-geometry fallback.

• Test the selected Torch/CUDA build with a GPU operation and EGL render, not just the driver’s version display. Use a build compatible with the host.

• For shared read-only installations, set NUMBA_CACHE_DIR to a writable per-user directory instead of making package code writable.

• If navigation RGB-D or world-map rendering reports a missing mobilebase0_navview, reinstall .[robocasa] to refresh the RLinf/robosuite rpent branch. Do not patch installed XML files manually.

• If read_text_file reports a missing current-task result, check the memory/robocasa/task_only/ directory or the selected local directory. RPent does not fall back to another task’s memory. Markdown is optional; Atomic tasks have no published <Task>.md.

• Environment and VLA startup failures are recorded in <output_dir>/env_server.log and <output_dir>/vla_server.log; also inspect <output_dir>/run.log for the run-level error.

• Only the exact 127.0.0.1 and localhost hostnames bypass HTTP proxies automatically. Other hostnames and IPs use the standard proxy environment; add the exact host to NO_PROXY and no_proxy only when it should be reached directly.

Toolkit design vs. LIBERO

The RoboCasa toolkit exposes the same shape of tools as LIBERO (a primitive call, a state view, a finish), with two RoboCasa-specific aspects:

• Env-side helpers. Grasp checks and action assembly need the live simulator env, so they live in env_server as RPCs. The agent-side skill holds both clients: the env client for render/step, the model client for RLDX-1 inference. See Add a New Robot for the rationale.

• Observation shape. RLDX-1 sees 3 camera video tensors (1, T, H, W, 3) stacked over history T, plus state.* and annotation.* fields. The session id is not part of the observation — it is managed automatically by the RPC framework: RpcClient generates a private rpc_ + uuid hex session id, wait_for_ready registers it with the server on connect; the server tracks each session’s idle time and a background sweep thread reaps sessions idle longer than the timeout (default 3600s), and the client sends session.close via atexit on process exit. Business code (rldx_skill / vla_client) never sees the session id directly; the server injects it into predict / reset_session to isolate per-client RLDX memory/RTC policy state.