Update Wall-X to 1.1.0 (#104)

workspace/README.md

@@ -1,160 +1,387 @@
-# Training Guide
+# Wall-X-OSS Usage Guide
 
-This document explains the key configuration parameters and memory requirements for Wall-X training.
+This guide explains how to fine-tune, evaluate in simulation, and deploy on real robots with the **Wall-OSS-0.5** pretrained model on LeRobot-format datasets.
 
-## Quick Start Checklist
+> All commands below assume you are in the **repository root** (`wall-x/`).
 
-### 🚀 **Step 1: Prepare Model**
-Choose one of our pretrained models:
-- **WALL-OSS-FLOW**: https://huggingface.co/x-square-robot/wall-oss-flow
-- **WALL-OSS-FAST**: https://huggingface.co/x-square-robot/wall-oss-fast
-Or from Qwen-2.5-VL
-- Download https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct, settings refer to `config_qact_from_vlm.yml`
+> **Note:** This open-source release targets **Wall-OSS-0.5**. If you are using **Wall-OSS-FLOW** or **Wall-OSS-FAST** instead, switch back to the previous codebase version:
+>
+> ```bash
+> git checkout 97406f2ab5de414c79b091873f946c112d105c72
+> ```
 
-### ⚙️ **Step 2: Configure Environment**
-- Update `run.sh`: Set `code_dir` and `config_path` to your actual paths
-- Set `CUDA_VISIBLE_DEVICES` for your available GPUs
+---
 
-### 📝 **Step 3: Update Configuration Files**
-- Replace all `/path/to/` placeholders in `config_qact.yml` with actual paths
-- Configure robot settings: `dof_config` and `agent_pos_config`
-- Set dataset: Choose appropriate `repo_id`
-- Adjust `batch_size_per_gpu` based on your GPU memory
+## Environment Setup
 
-### ▶️ **Step 4: Start Training**
 ```bash
-bash ./workspace/lerobot_example/run.sh
+conda create --name wallx python=3.10
+conda activate wallx
+
+pip install -r requirements.txt
+pip install "dmuon @ git+https://github.com/X-Square-Robot/dmuon.git"
+
+git clone https://github.com/huggingface/lerobot.git
+cd lerobot
+git checkout c66cd401767e60baece16e1cf68da2824227e076
+pip install --no-deps -e .
+cd -
+
+# Optional: only needed for LIBERO simulator evaluation.
+pip install -r requirements-libero.txt
+mkdir -p third_party
+git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git third_party/LIBERO
+
+# Install wall-x (from repo root)
+MAX_JOBS=8 pip install --no-build-isolation -e .
 ```
 
-## Enable FAST tokenizer
-To fine-tune using the FAST tokenizer, please download the repository and update the `action_tokenizer_path`. Make sure to set `use_fast_tokenizer` to `true` and q01 and q99 to normalize the dataset, refer to `wall-x/scripts/compute_norm_stats.py`:
+---
+
+## Part 1: Download Wall-OSS-0.5 Weights
+
+Wall-OSS-0.5 is a VLA foundation model built on Qwen2.5-VL-3B. Fine-tuning requires both the **pretrained weights** and the **VLM processor**.
+
+### 1.1 Download Wall-OSS-0.5
+
 ```bash
-git clone https://huggingface.co/physical-intelligence/fast
+# Option A: huggingface-cli
+huggingface-cli download X-Square-Robot/wall-oss-0.5 \
+  --local-dir /path/to/wall-oss-0.5
+
+# Option B: Python
+python -c "
+from huggingface_hub import snapshot_download
+snapshot_download('X-Square-Robot/wall-oss-0.5', local_dir='/path/to/wall-oss-0.5')
+"
 ```
 
-## Required Paths (Must Modify)
+After download, the directory should contain at least:
+
+| File | Purpose |
+|------|---------|
+| `config.json` | Model architecture; maps to `model.config_path` in training YAML |
+| `model.safetensors` | Pretrained weights; maps to `checkpoint.resume_from` |
+| tokenizer / processor files | Used at inference time |
+
+HuggingFace: <https://huggingface.co/X-Square-Robot/wall-oss-0.5>
+
+### 1.2 Download Qwen2.5-VL-3B-Instruct (processor)
+
+Set `model.processor_path` and `model.pretrained_path` to the Qwen2.5-VL processor directory:
+
+```bash
+huggingface-cli download Qwen/Qwen2.5-VL-3B-Instruct \
+  --local-dir /path/to/Qwen2.5-VL-3B-Instruct
+```
+
+---
+
+## Part 2: Simulation Tasks (LIBERO)
+
+### 2.1 Download the LIBERO Dataset
+
+Download the LeRobot-format LIBERO dataset from HuggingFace:
+
+```bash
+huggingface-cli download lerobot/libero \
+  --repo-type dataset \
+  --local-dir /path/to/libero_all
+```
+
+Dataset page: <https://huggingface.co/datasets/lerobot/libero>
+
+### 2.2 Edit Your Training Config (`config.yml`)
+
+Copy an example config and replace every `/path/to/*` placeholder:
+
+```bash
+cp workspace/example/libero.yml /path/to/my_libero_config.yml
+```
+
+**Required paths:**
+
 ```yaml
-pretrained_wallx_path: "/path/to/wallx_model/"      # Path to pretrained wallx model
-save_path: "/path/to/workspace/"                    # Path to save training outputs
-use_fast_tokenizer: False                           # True: train FAST, False: train Flow
-action_tokenizer_path: "/path/to/fast/"             # Must set if use_fast_tokenizer is True
-norm_stats_path: "/path/to/stats/"                  # Must set for normalize dataset
+model:
+  config_path: /path/to/wall-oss-0.5/config.json
+  processor_path: /path/to/Qwen2.5-VL-3B-Instruct
+  pretrained_path: /path/to/Qwen2.5-VL-3B-Instruct
+
+data:
+  lerobot_config:
+    repo_id: /path/to/libero_all          # local LeRobot dataset root
+  norm_stats_path: /path/to/libero_all_norm_stats.json
+  key_mappings:                           # must match your dataset keys
+    camera:
+      observation.images.faceImg: face_view
+      observation.images.rightImg: right_wrist_view
+    state: observation.state
+    action: action
+
+checkpoint:
+  save_path: /path/to/libero_training_output
+  resume_from: /path/to/wall-oss-0.5/model.safetensors
 ```
-## Customize your robot configuration
-Ensure that the sum of the configuration dimensions corresponds to the values specified in norm_stats.json, and that each key is unique. The maximum dimensionality is set to 20, consistent with our robot configuration.
+
+See `workspace/example/libero.yml` for the full example. LIBERO uses a 7-dim single-arm action; pad to 26 dims with `action_padding` to match the Wall-OSS-0.5 pretraining space (see comments in the YAML).
+
+### 2.3 Compute LIBERO Normalization Stats
+
+Generate `norm_stats.json` from the dataset before training:
+
+```bash
+python scripts/compute_norm_stats.py \
+  --train_config /path/to/my_libero_config.yml \
+  --data_root /path/to/libero_all \
+  --output_path /path/to/libero_all_norm_stats.json
+```
+
+Then set `data.norm_stats_path` to the generated JSON file.
+
+### 2.4 Start Training
+
+```bash
+# Single GPU
+CUDA_VISIBLE_DEVICES=0 \
+  python wall_x/trainer/fsdp_trainer/train_fsdp.py \
+    --config /path/to/my_libero_config.yml
+
+# Multi-GPU (recommended)
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+  torchrun --nproc_per_node=4 \
+    wall_x/trainer/fsdp_trainer/train_fsdp.py \
+    --config /path/to/my_libero_config.yml
+```
+
+Logs and checkpoints are written to `checkpoint.save_path`. If training saved FSDP-sharded checkpoints, merge them before inference:
+
+```bash
+python scripts/merge_sharded_weights.py \
+  /path/to/sharded_checkpoint \
+  /path/to/merged_checkpoint
+```
+
+Single-GPU training needs at least **48 GB** VRAM. For multi-GPU runs, enable `distributed.use_fsdp: true`.
+
+### 2.5 Run Inference (LIBERO Simulation)
+
+`scripts/run_libero.sh` requires the optional LIBERO simulator stack:
+
+```bash
+pip install -r requirements-libero.txt
+mkdir -p third_party
+git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git third_party/LIBERO
+```
+
+The script checks for LIBERO, robosuite, MuJoCo, PyOpenGL, BDDL, Gym, and h5py
+before loading the model, so missing simulator dependencies fail fast with
+installation instructions.
+
+Use `scripts/run_libero.sh` for batch evaluation in the LIBERO simulator:
+
+```bash
+CHECKPOINT_PATH=/path/to/checkpoint \
+TRAIN_CONFIG_PATH=/path/to/my_libero_config.yml \
+TASK_SUITE_NAME=libero_spatial \
+NUM_TRIALS_PER_TASK=50 \
+bash scripts/run_libero.sh
+
+# Quick smoke test (1 trial per task)
+SMOKE=1 CHECKPOINT_PATH=/path/to/checkpoint bash scripts/run_libero.sh
+```
+
+Useful environment variables:
+
+| Variable | Description |
+|----------|-------------|
+| `CHECKPOINT_PATH` | Checkpoint directory |
+| `TRAIN_CONFIG_PATH` | Training YAML |
+| `TASK_SUITE_NAME` | `libero_spatial` / `libero_object` / `libero_goal` / `libero_10` |
+| `ALL_SUITES=1` | Run all 4 standard suites sequentially |
+| `TASK_INDICES` | Task indices, e.g. `0,1,2` |
+| `CUDA_ID` | GPU index |
+
+---
+
+## Part 3: Real-Robot Tasks
+
+### 3.1 Prepare Your Real-Robot Training Dataset
+
+Convert your robot data to **LeRobot v3** format and note the local dataset root path (used for `data.lerobot_config.repo_id`).
+
+Reference config: `workspace/example/maniparena_example.yml` (dual-arm, 448px, 3 cameras).
+
+### 3.2 Edit Your Training Config (`config.yml`)
+
+```bash
+cp workspace/example/maniparena_example.yml /path/to/my_robot_config.yml
+```
+
+**Required paths:**
+
 ```yaml
-  customized_dof_config:
-    "action_eef": 6
-    "action_gripper": 1
+model:
+  config_path: /path/to/wall-oss-0.5/config.json
+  processor_path: /path/to/Qwen2.5-VL-3B-Instruct
+  pretrained_path: /path/to/Qwen2.5-VL-3B-Instruct
 
-  customized_agent_pos_config:
-    "state_eef_with_gripper": 7
+data:
+  lerobot_config:
+    repo_id: /path/to/your_robot_dataset
+  norm_stats_path: /path/to/your_robot_norm_stats.json
+  key_mappings:                           # match your dataset camera / state / action keys
+    camera:
+      observation.images.faceImg: face_view
+      # ...
+    state: observation.state
+    action: action
+
+checkpoint:
+  save_path: /path/to/robot_training_output
+  resume_from: /path/to/wall-oss-0.5/model.safetensors
 ```
 
-## Using Lerobot Dataset
-- Each dataset employs distinct keys; please specify the corresponding key mappings as described in `wall-x/wall_x/data/utils.py`.
-```python
-"lerobot/aloha_mobile_cabinet": {
-      "camera": {
-          "observation.images.cam_high": "face_view",
-          "observation.images.cam_left_wrist": "left_wrist_view",
-          "observation.images.cam_right_wrist": "right_wrist_view",
-      },
-      "state": "observation.state",
-      "action": "action",
-  }
-```
+`task.dof_config` defines **predicted action dimensions**; `task.agent_pos_config` defines **observation state dimensions**. The sum of each config must match `norm_stats.json`. If your robot has fewer than 26 DOF, pad with `action_padding` (see comments in the example YAML).
+
+### 3.3 Compute Dataset Normalization Stats
 
-## Compute stats
 ```bash
-    python wall-x/scripts/compute_norm_stats.py
+python scripts/compute_norm_stats.py \
+  --train_config /path/to/my_robot_config.yml \
+  --data_root /path/to/your_robot_dataset \
+  --output_path /path/to/your_robot_norm_stats.json
 ```
 
-## Configuration Explain
-- `agent_pos_config` corresponds to `obs_action_keys` and subsequently to state, while `dof_config` corresponds to `predict_action_keys` and subsequently to action. Note that the state and action may not necessarily share the same set of DoF.
+### 3.4 Start Training
 
-## Training Parameters (Commonly Modified)
-
-### Learning Rate Settings
-- `learning_rate`: Initial learning rate (default: 0.00009)
-- `min_lr`: Minimum learning rate for scheduler (default: 0.00005)
-- `num_warmup_steps`: Number of warmup steps (default: 100)
-
-### Batch Size and Memory
-- `batch_size_per_gpu`: Batch size per GPU - adjust based on GPU memory
-- `gradient_accumulation_steps`: Gradient accumulation steps
-- `num_training_steps`: Total training steps
-- `num_epoch`: Number of training epochs
-
-### Training Optimization Settings
-- `FSDP2`: Enable FSDP2 for distributed training (default: True) - **Recommended for multi-GPU**
-- `torch_compile`: Enable PyTorch compilation optimization (default: False)
-
-**⚠️ Important Note on torch_compile:**
-- **Benefits**: Enabling `torch_compile` can significantly improve training efficiency
-- **Requirements**: Requires that the data input shape is always consistent throughout training
-- **Caution**: If you don't have sufficient understanding of torch compile, please **DO NOT** enable it as it may cause unexpected issues with dynamic input shapes
-
-## Robot Configuration (Modify for Your Robot)
-
-### DOF Configuration
-Modify `dof_config` to match your robot's action space:
-- Add/remove action keys based on your robot's capabilities
-- Ensure DOF numbers match your robot's action dimensions
-
-### Agent Position Configuration
-Keep `agent_pos_config` consistent with `dof_config`.
-
-### Action Keys
-- `obs_action_keys`: Actions used as observation context
-- `predict_action_keys`: Actions to predict/control
-
-## Data Configuration
-
-### Dataset
-- `repo_id`: LeRobot dataset identifier
-- `train_test_split`: Training/validation split ratio (default: 0.95)
-- `action_horizon`: Number of future actions to predict (default: 32)
-
-### Image Settings
-- `resolution`: Image resolution for different camera views
-- `download_videos`: Whether to download video files (true/false)
-
-## Resume Training (Optional)
-- `resume.ckpt`: Path to checkpoint for resuming training
-- `resume.load_ckpt_only`: Only load model weights, not optimizer state
-
-## Merge checkpoint
-- If FSDP SHARDED_STATE_DICT is used, please run command below to merge checkpoint into a single safetensors
 ```bash
-    # refer to accelerate/commands/merge.py
-    accelerate merge-weights /path/to/sharded_tensors /path/to/model.safetensors
-    # copy the saved processor files
-    cp /path/to/saved_processor_dir/* /path/to/model.safetensors
-
-    # In earlier versions of PyTorch, errors may occur. You can use our provided script to address this issue; refer to wall-x/scripts/merge_sharded_weights.py for details.
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+  torchrun --nproc_per_node=4 \
+    wall_x/trainer/fsdp_trainer/train_fsdp.py \
+    --config /path/to/my_robot_config.yml
 ```
 
-## Memory Usage
+To resume training, point `checkpoint.resume_from` to a checkpoint **directory** (not a single `.safetensors` file).
 
-Below are the memory consumption benchmarks for different training configurations using the `lerobot/aloha_mobile_cabinet` dataset:
+### 3.5 Start Inference (WebSocket Server)
 
-| Dataset | Batch Size | FSDP2 | Torch Compile | Num GPUs | Max Allocated Memory |
-|---------|------------|--------|---------------|----------|---------------------|
-| lerobot/aloha_mobile_cabinet | 1 | ❌ | ❌ | 1 | 40.11G |
-| lerobot/aloha_mobile_cabinet | 1 | ❌ | ❌ | 8 | 48.02G |
-| lerobot/aloha_mobile_cabinet | 1 | ✅ | ❌ | 2 | 43.70G |
-| lerobot/aloha_mobile_cabinet | 1 | ✅ | ❌ | 8 | 24.96G |
-| lerobot/aloha_mobile_cabinet | 1 | ✅ | ✅ | 8 | 24.21G |
+Use `scripts/run_serving.sh` to launch the inference server for real-robot clients or open-loop evaluation:
 
+```bash
+bash scripts/run_serving.sh \
+  --checkpoint-path /path/to/checkpoint \
+  --train-config-path /path/to/my_robot_config.yml \
+  --port 32195
+```
 
-**Hardware Recommendations:**
+By default this wrapper returns raw model action chunks, which is the expected
+mode for open-loop evaluation. Pass `--serialize-actions` if your client expects
+robot-serialized actions.
 
-- For single GPU training: Ensure at least 48GB VRAM (e.g., RTX 6000 Ada, A6000)
-- For multi-GPU training: Enable FSDP2 for optimal memory distribution
+Or invoke the serving module directly (adjust parameters as needed):
 
-## Reproduce
+```bash
+export ENABLE_CUDA_GRAPH=True
+export ENABLE_EXPERIMENTAL_INFERENCE_ENGINE=True
 
-Openloop plot `wall-x/workspace/lerobot_example/evaluation/lerobot_openloop.png`
+CKPT_PATH=/path/to/checkpoint
 
-To reproduce the results, use the config file wall-x/workspace/lerobot_example/config_qact_from_vlm.yml with a global batch size of 128, adjusted via `gradient_accumulation_steps` and numbers of gpu.
+python -m wall_x._vendor.harrix.serving.launch_serving \
+  --env X2ROBOT \
+  --port 32195 \
+  --no-serialize-actions \
+  model-config:server-model-config \
+  --model-config.checkpoint-path "$CKPT_PATH" \
+  --model-config.train-config-path /path/to/my_robot_config.yml \
+  --model-config.action-horizon 32 \
+  --model-config.robot-action-interpolate-multiplier 1 \
+  --model-config.robot-action-end-ratio 1.0 \
+  --model-config.robot-type desktop
+```
+
+Clients connect at `ws://127.0.0.1:32195` to send observations and receive predicted actions.
+
+### 3.6 Plot Open-Loop Results to Verify the Server
+
+Open-loop evaluation compares model-predicted action trajectories against dataset ground truth **without executing actions or feeding back state**.
+
+**Terminal 1:** Keep the inference server from section3.5 running.
+
+**Terminal 2:** Run the open-loop plotting script:
+
+```bash
+python scripts/draw_openloop_plot.py \
+  --uri ws://127.0.0.1:32195 \
+  --dataset-root /path/to/your_robot_dataset \
+  --train-config /path/to/my_robot_config.yml \
+  --episode-indices 0,1,2 \
+  --save-dir ./openloop_plots
+```
+
+The script loads episodes from the LeRobot dataset, queries the WebSocket server frame by frame, and saves **predicted vs. ground-truth** comparison plots (PNG) under `--save-dir`.
+`--dataset-root` and `--train-config` are both required; pass the same training
+config used for the checkpoint so dataset layouts and action dimensions match.
+
+Common options:
+
+| Option | Description |
+|--------|-------------|
+| `--episode-indices` | Comma-separated episode indices to evaluate |
+| `--start-ratio` | Start position as a fraction of episode length (0.0 = from the beginning) |
+| `--stride` | Frames between inference calls (default: `action_horizon`) |
+| `--max-inferences` | Maximum inference requests per episode |
+
+---
+
+## Appendix
+
+### Example Config Files
+
+| File | Scenario |
+|------|----------|
+| `workspace/example/libero.yml` | LIBERO single-arm fine-tuning |
+| `workspace/example/maniparena_example.yml` | Real-robot dual-arm fine-tuning |
+
+### Helper Scripts
+
+| Script | Purpose |
+|--------|---------|
+| `scripts/compute_norm_stats.py` | Compute dataset normalization statistics |
+| `scripts/fake_inference.py` | Inference smoke test |
+| `scripts/run_libero.sh` | Batch LIBERO simulation evaluation |
+| `scripts/run_serving.sh` | Launch WebSocket inference server |
+| `scripts/draw_openloop_plot.py` | Open-loop evaluation and plotting |
+| `scripts/merge_sharded_weights.py` | Merge FSDP sharded checkpoints |
+
+See [`scripts/README.md`](../scripts/README.md) for more script details.
+
+### Training Hyperparameters
+
+| Parameter | Description | Default (libero example) |
+|-----------|-------------|--------------------------|
+| `hyperparams.batch_size_per_gpu` | Batch size per GPU | 4 |
+| `hyperparams.gradient_accumulation_steps` | Gradient accumulation steps | 4 |
+| `hyperparams.optimizer.learning_rate` | Learning rate | 5e-5 |
+| `hyperparams.num_epoch` | Number of training epochs | 100 |
+| `distributed.use_fsdp` | Multi-GPU FSDP training | true |
+| `logging.save_interval` | Checkpoint save interval (steps) | 2000 |
+
+### Quick Start Checklist
+
+**Simulation (LIBERO)**
+
+- [ ] Downloaded `wall-oss-0.5` and `Qwen2.5-VL-3B-Instruct`
+- [ ] Downloaded the `lerobot/libero` dataset
+- [ ] Wrote `config.yml` and replaced all `/path/to/*` placeholders
+- [ ] Ran `compute_norm_stats.py` to generate norm stats
+- [ ] Launched `train_fsdp.py` and completed fine-tuning
+- [ ] Ran simulation evaluation with `run_libero.sh`
+
+**Real Robot**
+
+- [ ] Prepared a LeRobot v3 real-robot dataset
+- [ ] Wrote `config.yml` and configured `key_mappings` / DOF
+- [ ] Ran `compute_norm_stats.py` to generate norm stats
+- [ ] Launched `train_fsdp.py` and completed fine-tuning
+- [ ] Started the inference server with `run_serving.sh`
+- [ ] Verified server output with `draw_openloop_plot.py`

workspace/example/lerobot/qwen2_5_lerobot_template.yml

@@ -0,0 +1,93 @@
+# Public Wall-X Qwen2.5 + LeRobot training template.
+#
+# Replace all /path/to/... values before launching training.
+model_type: qwen2_5
+
+task:
+  dof_config:
+    master_right_ee_cartesian_pos: 3
+    master_right_ee_rotation: 3
+    master_right_gripper: 1
+  agent_pos_config:
+    follow_right_ee_cartesian_pos: 3
+    follow_right_ee_rotation: 3
+    follow_right_gripper: 2
+  action_horizon: 32
+  action_horizon_flow: 32
+  use_state_string_representation: false
+
+model:
+  backbone: qwen2_5
+  config_path: workspace/models_config/qwen2_5_moe_flash.json
+  processor_path: Qwen/Qwen2.5-VL-3B-Instruct
+  pretrained_path: Qwen/Qwen2.5-VL-3B-Instruct
+  attn_deterministic: true
+  use_ema: false
+  flow_loss_weight: 1.0
+  enable_customized_robot_config: true
+  customized_robot_config:
+    name: your_robot
+    customized_dof_config:
+      action: 7
+    customized_agent_pos_config:
+      observation.state: 8
+
+hyperparams:
+  num_epoch: 100
+  batch_size_per_gpu: 8
+  gradient_accumulation_steps: 2
+  seed: 10233
+  optimizer:
+    optimizer_type: dmuon
+    learning_rate: 0.0001
+    max_grad_norm: 1.0
+    enable_grad_clip: true
+    betas: [0.9, 0.95]
+    weight_decay: 1.0e-8
+    eps: 1.0e-8
+  scheduler:
+    scheduler_type: cosine
+    num_warmup_steps: 1000
+    num_training_steps: 64000000
+    min_lr: 1.0e-5
+
+distributed:
+  use_fsdp: true
+  use_mixed_precision: true
+  bf16: true
+
+data:
+  dataset_type: lerobot
+  lerobot_config:
+    repo_id: /path/to/your/lerobot_dataset
+    root: null
+  key_mappings:
+    camera:
+      observation.images.image: face_view
+      observation.images.image2: right_wrist_view
+    state: observation.state
+    action: action
+  norm_stats_path: /path/to/your/norm_stats.json
+  train_test_split: 0.95
+  num_workers: 4
+  resolution:
+    face_view: 256
+    left_wrist_view: 256
+    right_wrist_view: 256
+
+logging:
+  log_name: wallx_lerobot_train
+  log_project: wallx_public
+  log_entity: null
+  use_wandb: false
+  log_interval: 10
+  save_interval: 1000000
+  val_interval: 1000000
+  epoch_save_interval: 1
+
+checkpoint:
+  save_path: /path/to/output/checkpoints
+
+debug:
+  profile: false
+  nvtx: false

workspace/example/libero.yml

@@ -0,0 +1,125 @@
+# LIBERO single-arm finetune example (Euler delta action, 256px, 2 cameras).
+#
+# Replace every /path/to/* placeholder before training:
+#
+#   model.config_path           -> model architecture JSON (mot_flash_mask_causal_xloss.json)
+#   model.processor_path        -> Qwen2.5-VL-3B-Instruct directory
+#   model.pretrained_path       -> same as processor_path, or HuggingFace cache path
+#   data.lerobot_config.repo_id -> local LeRobot dataset root (libero_all)
+#   data.norm_stats_path        -> q01/q99 normalization JSON
+#   checkpoint.save_path        -> writable directory for training checkpoints
+#   checkpoint.resume_from      -> Wall-OSS-0.5 pretrained .safetensors or checkpoint directory
+#
+# Compute norm stats first:
+#
+#   python scripts/compute_norm_stats.py \
+#     --train_config workspace/example/libero.yml \
+#     --data_root /path/to/libero_all \
+#     --output_path /path/to/libero_all_norm_stats.json
+#
+# Launch training (from repo root):
+#
+#   torchrun --nproc_per_node=<N> wall_x/trainer/fsdp_trainer/train_fsdp.py \
+#     --config workspace/example/libero.yml
+#
+# Strategy: keep dof / agent_pos totals at 26 to match the pretraining action space
+# via ``action_padding``. The lerobot collator right-pads libero's 7-dim action /
+# 8-dim state with zeros; loss does not flow through the padded tail.
+
+model_type: qwen2_5
+
+task:
+  # Libero delta action: pos3 + rot3 + gripper1 = 7, plus action_padding(19) = 26.
+  dof_config:
+    master_right_ee_cartesian_pos: 3     # delta position
+    master_right_ee_rotation: 3          # delta rotation (ZYX euler)
+    master_right_gripper: 1
+    action_padding: 19
+  ar_dof_config:
+    master_right_ee_cartesian_pos: 3
+    master_right_ee_rotation: 3
+    master_right_gripper: 1
+    action_padding: 19
+  # State: pos3 + rot3 + gripper2 = 8, plus action_padding(18) = 26.
+  agent_pos_config:
+    follow_right_ee_cartesian_pos: 3
+    follow_right_ee_rotation: 3
+    follow_right_gripper: 2
+    action_padding: 18
+  action_horizon: 10
+  action_horizon_flow: 10
+  use_state_string_representation: false
+
+model:
+  backbone: qwen2_5
+  config_path: /path/to/wall-oss-0.5/config.json
+  processor_path: /path/to/Qwen2.5-VL-3B-Instruct
+  pretrained_path: /path/to/Qwen2.5-VL-3B-Instruct
+  attn_deterministic: true
+  use_ema: false
+  flow_loss_weight: 1.0
+  ar_loss_weight: 0.01
+
+hyperparams:
+  num_epoch: 100
+  batch_size_per_gpu: 4
+  gradient_accumulation_steps: 4
+  seed: 10222
+  optimizer:
+    optimizer_type: adamw
+    learning_rate: 5.0e-05
+    max_grad_norm: 1.0
+    enable_grad_clip: true
+    betas: [0.9, 0.95]
+    weight_decay: 1.0e-8
+    eps: 1.0e-8
+  scheduler:
+    scheduler_type: cosine
+    num_warmup_steps: 1000
+    num_training_steps: 200000
+    min_lr: 1.0e-6
+
+distributed:
+  use_fsdp: true
+  use_mixed_precision: true
+  bf16: true
+
+data:
+  dataset_type: lerobot
+  lerobot_config:
+    repo_id: /path/to/libero_all
+    root: null
+  key_mappings:
+    # libero_all v3.0 only has faceImg + rightImg (no leftImg).
+    camera:
+      observation.images.faceImg: face_view
+      observation.images.rightImg: right_wrist_view
+    state: observation.state
+    action: action
+  norm_stats_path: /path/to/libero_all_norm_stats.json
+  train_test_split: 0.95
+  num_workers: 4
+  max_length: 1024
+  resolution:
+    face_view: 256
+    right_wrist_view: 256
+
+logging:
+  log_name: libero_ft
+  log_project: lerobot_libero_ft
+  log_entity: your_wandb_entity
+  use_wandb: true
+  log_interval: 10
+  save_interval: 2000
+  val_interval: 1000000
+  epoch_save_interval: 1
+
+checkpoint:
+  save_path: /path/to/libero
+  # Single-file .safetensors loads as pretrain weights before FSDP wrapping.
+  # Use a checkpoint directory for full resume (optimizer / scheduler / RNG).
+  resume_from: /path/to/wall-oss-0.5/model.safetensors
+
+debug:
+  profile: false
+  nvtx: false

workspace/example/maniparena_example.yml

@@ -0,0 +1,134 @@
+# ManipArena / CVPR dual-arm finetune example (6D relative action, 448px, 3 cameras).
+#
+# Replace every /path/to/* placeholder before training:
+#
+#   model.config_path           -> model architecture JSON (mot_flash_mask_causal_xloss.json)
+#   model.processor_path        -> Qwen2.5-VL-3B-Instruct directory
+#   model.pretrained_path       -> same as processor_path, or HuggingFace cache path
+#   data.lerobot_config.repo_id -> local LeRobot dataset root (cvpr_4tasks)
+#   data.norm_stats_path        -> q01/q99 normalization JSON
+#   checkpoint.save_path        -> writable directory for training checkpoints
+#   checkpoint.resume_from      -> Wall-OSS-0.5 pretrained .safetensors or checkpoint directory
+#
+# Compute norm stats first:
+#
+#   python scripts/compute_norm_stats.py \
+#     --train_config workspace/example/maniparena_example.yml \
+#     --data_root /path/to/cvpr_4tasks \
+#     --output_path /path/to/cvpr_4tasks_norm_stats.json
+#
+# Launch training (from repo root):
+#
+#   torchrun --nproc_per_node=<N> wall_x/trainer/fsdp_trainer/train_fsdp.py \
+#     --config workspace/example/maniparena_example.yml
+#
+# Strategy: keep dof / agent_pos totals at 26 to match the pretraining action space
+# via ``action_padding``. The lerobot collator right-pads real action / state dims;
+# loss does not flow through the padded tail.
+
+model_type: qwen2_5
+
+task:
+  # Dual-arm 6D relative action: 10 + 10 = 20, plus action_padding(6) = 26.
+  dof_config:
+    follow_left_ee_cartesian_pos_relative: 3
+    follow_left_ee_rotation_6D_relative: 6
+    follow_left_gripper: 1
+    follow_right_ee_cartesian_pos_relative: 3
+    follow_right_ee_rotation_6D_relative: 6
+    follow_right_gripper: 1
+    action_padding: 6
+  ar_dof_config:
+    follow_left_ee_cartesian_pos_relative: 3
+    follow_left_ee_rotation_6D_relative: 6
+    follow_left_gripper: 1
+    follow_right_ee_cartesian_pos_relative: 3
+    follow_right_ee_rotation_6D_relative: 6
+    follow_right_gripper: 1
+    action_padding: 6
+  agent_pos_config:
+    follow_left_ee_cartesian_pos: 3
+    follow_left_ee_rotation_6D: 6
+    follow_left_gripper: 1
+    follow_right_ee_cartesian_pos: 3
+    follow_right_ee_rotation_6D: 6
+    follow_right_gripper: 1
+    action_padding: 6
+  action_horizon: 32
+  action_horizon_flow: 32
+  use_state_string_representation: false
+
+model:
+  backbone: qwen2_5
+  config_path: /path/to/wall-oss-0.5/config.json
+  processor_path: /path/to/Qwen2.5-VL-3B-Instruct
+  pretrained_path: /path/to/Qwen2.5-VL-3B-Instruct
+  attn_deterministic: true
+  use_ema: false
+  flow_loss_weight: 1.0
+  ar_loss_weight: 0.01
+
+hyperparams:
+  num_epoch: 100
+  batch_size_per_gpu: 4
+  gradient_accumulation_steps: 4
+  seed: 10222
+  optimizer:
+    optimizer_type: adamw
+    learning_rate: 5.0e-05
+    max_grad_norm: 1.0
+    enable_grad_clip: true
+    betas: [0.9, 0.95]
+    weight_decay: 1.0e-8
+    eps: 1.0e-8
+  scheduler:
+    scheduler_type: cosine
+    num_warmup_steps: 1000
+    num_training_steps: 200000
+    min_lr: 1.0e-6
+
+distributed:
+  use_fsdp: true
+  use_mixed_precision: true
+  bf16: true
+
+data:
+  dataset_type: lerobot
+  lerobot_config:
+    repo_id: /path/to/cvpr_4tasks
+    root: null
+  key_mappings:
+    camera:
+      observation.images.faceImg: face_view
+      observation.images.leftImg: left_wrist_view
+      observation.images.rightImg: right_wrist_view
+    state: observation.state
+    action: action
+  norm_stats_path: /path/to/cvpr_4tasks_norm_stats.json
+  train_test_split: 0.95
+  num_workers: 4
+  max_length: 1024
+  resolution:
+    face_view: 448
+    left_wrist_view: 448
+    right_wrist_view: 448
+
+logging:
+  log_name: maniparena_ft
+  log_project: lerobot_maniparena_ft
+  log_entity: your_wandb_entity
+  use_wandb: true
+  log_interval: 10
+  save_interval: 2000
+  val_interval: 1000000
+  epoch_save_interval: 1
+
+checkpoint:
+  save_path: /path/to/cvpr_4tasks
+  # Single-file .safetensors loads as pretrain weights before FSDP wrapping.
+  # Use a checkpoint directory for full resume (optimizer / scheduler / RNG).
+  resume_from: /path/to/wall-oss-0.5/model.safetensors
+
+debug:
+  profile: false
+  nvtx: false

workspace/lerobot_example/config_qact.yml

@@ -1,151 +0,0 @@
-# Training Configuration for Wall-X Robotic Multi-Modal Learning
-# This configuration supports multi-modal learning with vision, language, and action data
-
-# Model and paths configuration
-log_name: "robotic_training"
-log_project: "vla_training"
-model_type: wall-oss
-pretrained_wallx_path: "/path/to/wallx_model/"  # Must set
-save_path: "/path/to/workspace/"                # Must set
-use_fast_tokenizer: False                       # True: train FAST, False: train Flow
-action_tokenizer_path: "/path/to/fast/"         # Must set if use_fast_tokenizer is true
-
-# Torch Profile
-profile: False
-profile_save_path: /path/to/profile/
-profile_wait_iters: 10
-profile_warmup_iters: 5
-profile_active_iters: 2
-
-# Training hyperparameters
-num_warmup_steps: 100
-num_training_steps: 64000000
-learning_rate: 0.00005
-min_lr: 0.00005
-num_epoch: 100
-gradient_accumulation_steps: 32
-batch_size_per_gpu: 8
-padding_side: left
-epoch_save_interval: 10
-
-# Training optimization settings
-FSDP2: True
-torch_compile: False
-
-# Robot configuration - Define degrees of freedom for each component
-dof_config:
-  follow_left_ee_cartesian_pos: 3      # Left end-effector Cartesian position
-  follow_left_ee_rotation: 3           # Left end-effector rotation
-  follow_left_gripper: 1               # Left gripper control
-  follow_right_ee_cartesian_pos: 3     # Right end-effector Cartesian position
-  follow_right_ee_rotation: 3          # Right end-effector rotation
-  follow_right_gripper: 1              # Right gripper control
-  head_actions: 2                      # Head/camera movement
-  height: 1                            # Mobile base height control
-  car_pose: 3                          # Mobile base pose (x, y, theta)
-
-# Agent proprioception configuration (typically matches DOF config)
-agent_pos_config:
-  follow_left_ee_cartesian_pos: 3
-  follow_left_ee_rotation: 3
-  follow_left_gripper: 1
-  follow_right_ee_cartesian_pos: 3
-  follow_right_ee_rotation: 3
-  follow_right_gripper: 1
-  head_actions: 2
-  height: 1
-  car_pose: 3
-
-# # Checkpoint resuming configuration
-# resume:
-#   ckpt: "/path/to/resume_model/"
-#   load_ckpt_only: true
-
-norm_stats_path: "/path/to/norm_stats.json"
-
-enable_customized_robot_config: true
-customized_robot_config:
-  name: "lerobot/aloha_mobile_cabinet"
-  customized_dof_config:
-    "action_left_shoulder" : 1
-    "action_left_elbow" : 1
-    "action_left_forearm_roll" : 1
-    "action_left_wrist_angle" : 1
-    "action_left_wrist_rotate" : 1
-    "action_left_gripper" : 1
-    "action_right_waist" : 1
-    "action_right_shoulder" : 1
-    "action_right_elbow" : 1
-    "action_right_forearm_roll" : 1
-    "action_right_wrist_angle" : 1
-    "action_right_wrist_rotate" : 1
-    "action_right_gripper" : 1
-
-  customized_agent_pos_config:
-    "state_left_shoulder" : 1
-    "state_left_elbow" : 1
-    "state_left_forearm_roll" : 1
-    "state_left_wrist_angle" : 1
-    "state_left_wrist_rotate" : 1
-    "state_left_gripper" : 1
-    "state_right_waist" : 1
-    "state_right_shoulder" : 1
-    "state_right_elbow" : 1
-    "state_right_forearm_roll" : 1
-    "state_right_wrist_angle" : 1
-    "state_right_wrist_rotate" : 1
-    "state_right_gripper" : 1
-
-# Data configuration
-data:
-  use_lerobot: true
-
-  # LeRobot dataset configuration
-  lerobot_config:
-    repo_id: "lerobot/aloha_mobile_cabinet"
-    root: null
-    episodes: null
-    image_transforms: null
-    delta_timestamps: null
-    tolerance_s: 1e-4
-    revision: null
-    force_cache_sync: false
-    download_videos: true
-    video_backend: null
-
-  action_horizon: 32
-  train_test_split: 0.95
-
-  # Action keys for observation and prediction
-  obs_action_keys:
-    - follow_left_ee_cartesian_pos
-    - follow_left_ee_rotation
-    - follow_left_gripper
-    - follow_right_ee_cartesian_pos
-    - follow_right_ee_rotation
-    - follow_right_gripper
-    - head_actions
-    - height
-    - car_pose
-
-  predict_action_keys:
-    - follow_left_ee_cartesian_pos
-    - follow_left_ee_rotation
-    - follow_left_gripper
-    - follow_right_ee_cartesian_pos
-    - follow_right_ee_rotation
-    - follow_right_gripper
-    - head_actions
-    - height
-    - car_pose
-
-  # Image resolution configuration for different camera views
-  resolution:
-    face_view: 256
-    left_wrist_view: 256
-    right_wrist_view: 256
-    move1_view: 256
-    move2_view: 256
-    top_view: 256
-    wall_view: 256
-    multi_modal: 256

workspace/lerobot_example/config_qact_from_vlm.yml

@@ -1,152 +0,0 @@
-# Train from Qwen-2.5-VL
-
-# Model and paths configuration
-log_name: "robotic_training"
-log_project: "vla_training"
-model_type: qwen2_5
-pretrained_wallx_path: "/path/to/wallx_model/"  # Must set
-save_path: "/path/to/workspace/"                # Must set
-use_fast_tokenizer: True                       # True: train FAST, False: train Flow
-action_tokenizer_path: "/path/to/fast/"         # Must set if use_fast_tokenizer is true
-qwen_vl_act_config_path: "wall-x/workspace/lerobot_example/qwen25_config.json"
-
-
-# Torch Profile
-profile: False
-profile_save_path: /path/to/profile/
-profile_wait_iters: 10
-profile_warmup_iters: 5
-profile_active_iters: 2
-
-# Training hyperparameters
-num_warmup_steps: 100
-num_training_steps: 64000000
-learning_rate: 0.00009
-min_lr: 0.00005
-num_epoch: 100
-gradient_accumulation_steps: 1
-batch_size_per_gpu: 8
-padding_side: left
-epoch_save_interval: 10
-
-# Training optimization settings
-FSDP2: True
-torch_compile: False
-
-# Robot configuration - Define degrees of freedom for each component
-dof_config:
-  follow_left_ee_cartesian_pos: 3      # Left end-effector Cartesian position
-  follow_left_ee_rotation: 3           # Left end-effector rotation
-  follow_left_gripper: 1               # Left gripper control
-  follow_right_ee_cartesian_pos: 3     # Right end-effector Cartesian position
-  follow_right_ee_rotation: 3          # Right end-effector rotation
-  follow_right_gripper: 1              # Right gripper control
-  head_actions: 2                      # Head/camera movement
-  height: 1                            # Mobile base height control
-  car_pose: 3                          # Mobile base pose (x, y, theta)
-
-# Agent proprioception configuration (typically matches DOF config)
-agent_pos_config:
-  follow_left_ee_cartesian_pos: 3
-  follow_left_ee_rotation: 3
-  follow_left_gripper: 1
-  follow_right_ee_cartesian_pos: 3
-  follow_right_ee_rotation: 3
-  follow_right_gripper: 1
-  head_actions: 2
-  height: 1
-  car_pose: 3
-
-# # Checkpoint resuming configuration
-# resume:
-#   ckpt: "/path/to/resume_model/"
-#   load_ckpt_only: true
-
-norm_stats_path: "/path/to/norm_stats.json"
-
-enable_customized_robot_config: true
-customized_robot_config:
-  name: "physical-intelligence/libero"
-  customized_dof_config:
-    "action_left_shoulder" : 1
-    "action_left_elbow" : 1
-    "action_left_forearm_roll" : 1
-    "action_left_wrist_angle" : 1
-    "action_left_wrist_rotate" : 1
-    "action_left_gripper" : 1
-    "action_right_waist" : 1
-    "action_right_shoulder" : 1
-    "action_right_elbow" : 1
-    "action_right_forearm_roll" : 1
-    "action_right_wrist_angle" : 1
-    "action_right_wrist_rotate" : 1
-    "action_right_gripper" : 1
-
-  customized_agent_pos_config:
-    "state_left_shoulder" : 1
-    "state_left_elbow" : 1
-    "state_left_forearm_roll" : 1
-    "state_left_wrist_angle" : 1
-    "state_left_wrist_rotate" : 1
-    "state_left_gripper" : 1
-    "state_right_waist" : 1
-    "state_right_shoulder" : 1
-    "state_right_elbow" : 1
-    "state_right_forearm_roll" : 1
-    "state_right_wrist_angle" : 1
-    "state_right_wrist_rotate" : 1
-    "state_right_gripper" : 1
-
-# Data configuration
-data:
-  use_lerobot: true
-
-  # LeRobot dataset configuration
-  lerobot_config:
-    repo_id: "lerobot/aloha_mobile_cabinet"
-    root: null
-    episodes: null
-    image_transforms: null
-    delta_timestamps: null
-    tolerance_s: 1e-4
-    revision: null
-    force_cache_sync: false
-    download_videos: true
-    video_backend: null
-
-  action_horizon: 32
-  train_test_split: 0.95
-
-  # Action keys for observation and prediction
-  obs_action_keys:
-    - follow_left_ee_cartesian_pos
-    - follow_left_ee_rotation
-    - follow_left_gripper
-    - follow_right_ee_cartesian_pos
-    - follow_right_ee_rotation
-    - follow_right_gripper
-    - head_actions
-    - height
-    - car_pose
-
-  predict_action_keys:
-    - follow_left_ee_cartesian_pos
-    - follow_left_ee_rotation
-    - follow_left_gripper
-    - follow_right_ee_cartesian_pos
-    - follow_right_ee_rotation
-    - follow_right_gripper
-    - head_actions
-    - height
-    - car_pose
-
-  # Image resolution configuration for different camera views
-  resolution:
-    face_view: 256
-    left_wrist_view: 256
-    right_wrist_view: 256
-    move1_view: 256
-    move2_view: 256
-    top_view: 256
-    wall_view: 256
-    multi_modal: 256

workspace/lerobot_example/evaluation/lerobot_openloop.png

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:aae8646e566b79b64a669956d6d2c779d72809010c802f75b2623ee371444b47
-size 985979

workspace/lerobot_example/libero/config_qact_libero_from_vlm.yml

@@ -1,100 +0,0 @@
-# Training Configuration for Wall-X Robotic Multi-Modal Learning
-# This configuration supports multi-modal learning with vision, language, and action data
-
-# Model and paths configuration
-log_name: "opensource_training"
-log_project: "libero"
-model_type: qwen2_5
-use_fast_tokenizer: true
-pretrained_wallx_path: "/path/to/qwen/"
-action_tokenizer_path: "/path/to/fast/"
-qwen_vl_act_config_path: "/path/to/qwen25_config.json"
-
-save_path: "/path/to/save"
-# Torch Profile
-profile: False
-profile_save_path: /path/to/profile/
-profile_wait_iters: 10
-profile_warmup_iters: 5
-profile_active_iters: 2
-
-# Training hyperparameters
-num_warmup_steps: 100
-num_training_steps: 64000000
-learning_rate: 0.00005
-min_lr: 0.00005
-num_epoch: 100
-gradient_accumulation_steps: 1
-batch_size_per_gpu: 8
-padding_side: left
-epoch_save_interval: 1
-
-# Robot configuration - Define degrees of freedom for each component
-dof_config:
-  master_right_ee_cartesian_pos: 3     # Right end-effector Cartesian position
-  master_right_ee_rotation: 3          # Right end-effector rotation
-  master_right_gripper: 1              # Right gripper control
-
-# Agent proprioception configuration (typically matches DOF config)
-agent_pos_config:
-  follow_right_ee_cartesian_pos: 3
-  follow_right_ee_rotation: 3
-  follow_right_gripper: 1
-
-norm_stats_path: "/path/to/libero_norm_stats.json"
-
-enable_customized_robot_config: true
-customized_robot_config:
-  name: "libero_all"
-  customized_dof_config:
-    "panda_action_eef_with_gripper": 7
-
-  customized_agent_pos_config:
-    "panda_state_eef_with_gripper": 7
-
-# Checkpoint resuming configuration
-# resume:
-#   ckpt: "/path/to/ckpt"
-#   load_ckpt_only: false
-
-# Data configuration
-data:
-  use_lerobot: true
-
-  # LeRobot dataset configuration
-  lerobot_config:
-    repo_id: "libero_all"
-    root: null
-    episodes: null
-    image_transforms: null
-    delta_timestamps: null
-    tolerance_s: 1e-4
-    revision: null
-    force_cache_sync: false
-    download_videos: true
-    video_backend: null
-
-  action_horizon: 10
-  train_test_split: 0.95
-
-  # Action keys for observation and prediction
-  obs_action_keys:
-    - follow_right_ee_cartesian_pos
-    - follow_right_ee_rotation
-    - follow_right_gripper
-
-  predict_action_keys:
-    - master_right_ee_cartesian_pos
-    - master_right_ee_rotation
-    - master_right_gripper
-
-  # Image resolution configuration for different camera views
-  resolution:
-    face_view: 256
-    left_wrist_view: 256
-    right_wrist_view: 256
-    move1_view: 256
-    move2_view: 256
-    top_view: 256
-    wall_view: 256
-    multi_modal: 256

workspace/lerobot_example/qwen25_config.json

@@ -1,106 +0,0 @@
-{
-    "architectures": [
-      "Qwen2_5_VLForConditionalGeneration"
-    ],
-    "attention_dropout": 0.0,
-    "bos_token_id": 151643,
-    "eos_token_id": 151645,
-    "vision_start_token_id": 151652,
-    "vision_end_token_id": 151653,
-    "vision_token_id": 151654,
-    "image_token_id": 151655,
-    "video_token_id": 151656,
-    "hidden_act": "silu",
-    "hidden_size": 2048,
-    "initializer_range": 0.02,
-    "intermediate_size": 11008,
-    "max_position_embeddings": 128000,
-    "max_window_layers": 70,
-    "model_type": "qwen2_5_vl",
-    "num_attention_heads": 16,
-    "num_hidden_layers": 36,
-    "num_key_value_heads": 2,
-    "rms_norm_eps": 1e-06,
-    "rope_theta": 1000000.0,
-    "sliding_window": 32768,
-    "tie_word_embeddings": true,
-    "torch_dtype": "bfloat16",
-    "transformers_version": "4.41.2",
-    "_attn_implementation": "flash_attention_2",
-    "use_cache": true,
-    "use_sliding_window": false,
-    "vision_config": {
-      "depth": 32,
-      "hidden_act": "silu",
-      "hidden_size": 1280,
-      "intermediate_size": 3420,
-      "num_heads": 16,
-      "in_chans": 3,
-      "out_hidden_size": 2048,
-      "patch_size": 14,
-      "spatial_merge_size": 2,
-      "spatial_patch_size": 14,
-      "window_size": 112,
-      "fullatt_block_indexes": [
-        7,
-        15,
-        23,
-        31
-      ],
-      "tokens_per_second": 2,
-      "temporal_patch_size": 2
-    },
-    "rope_scaling": {
-      "type": "mrope",
-      "mrope_section": [
-        16,
-        24,
-        24
-      ]
-    },
-    "vocab_size": 151936,
-    "num_experts": 2,
-    "experts":[
-      {
-        "hidden_size": 2048,
-        "intermediate_size": 11008,
-        "hidden_act": "silu"
-      },
-      {
-        "hidden_size": 2048,
-        "intermediate_size": 2048,
-        "hidden_act": "silu"
-      }
-    ],
-    "dof_config": {
-        "follow_left_ee_cartesian_pos": 3,
-        "follow_left_ee_rotation": 3,
-        "follow_left_gripper": 1,
-        "follow_right_ee_cartesian_pos": 3,
-        "follow_right_ee_rotation": 3,
-        "follow_right_gripper": 1,
-        "head_actions": 2,
-        "height": 1,
-        "car_pose": 3
-    },
-    "agent_pos_config": {
-      "follow_left_ee_cartesian_pos": 3,
-      "follow_left_ee_rotation": 3,
-      "follow_left_gripper": 1,
-      "follow_right_ee_cartesian_pos": 3,
-      "follow_right_ee_rotation": 3,
-      "follow_right_gripper": 1,
-      "head_actions": 2,
-      "height": 1,
-      "car_pose": 3
-    },
-    "noise_scheduler": {
-      "beta_alpha": 1.5,
-      "beta_beta": 1.0,
-      "s": 0.999,
-      "num_inference_timesteps": 5
-    },
-    "dim_inputs": [2048,2048],
-    "attention_moe": false,
-    "mlp_moe": true
-  }

workspace/lerobot_example/run.sh

@@ -1,23 +0,0 @@
-#!/bin/bash
-export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
-NUM_GPUS=$(echo $CUDA_VISIBLE_DEVICES | tr ',' '\n' | wc -l)
-
-# print current time
-echo "[current time: $(date +'%Y-%m-%d %H:%M:%S')]"
-
-code_dir="/path/to/wall-x"
-config_path="/path/to/wall-x/workspace/lerobot_example"
-
-# Use a fixed port instead of a random one
-export PORT=$((21000 + $RANDOM % 30000))
-
-MASTER_PORT=10239 # use 5 digits ports
-
-export LAUNCHER="accelerate launch --num_processes=$NUM_GPUS --main_process_port=$PORT"
-
-export SCRIPT="${code_dir}/train_qact.py"
-export SCRIPT_ARGS="--config ${config_path}/config_qact.yml --seed $MASTER_PORT"
-
-echo "Running command: $LAUNCHER $SCRIPT $SCRIPT_ARGS"
-
-$LAUNCHER $SCRIPT $SCRIPT_ARGS

workspace/models_config/qwen2_5_moe_flash.json

@@ -0,0 +1,108 @@
+{
+  "architectures": [
+    "Qwen2_5_VLForConditionalGeneration"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "vision_start_token_id": 151652,
+  "vision_end_token_id": 151653,
+  "vision_token_id": 151654,
+  "image_token_id": 151655,
+  "video_token_id": 151656,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 128000,
+  "max_window_layers": 70,
+  "model_type": "qwen2_5_vl",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 36,
+  "num_key_value_heads": 2,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 1000000.0,
+  "sliding_window": 32768,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.41.2",
+  "_attn_implementation": "flash_attention_2",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vision_config": {
+    "depth": 32,
+    "hidden_act": "silu",
+    "hidden_size": 1280,
+    "intermediate_size": 3420,
+    "num_heads": 16,
+    "in_chans": 3,
+    "out_hidden_size": 2048,
+    "patch_size": 14,
+    "spatial_merge_size": 2,
+    "spatial_patch_size": 14,
+    "window_size": 112,
+    "fullatt_block_indexes": [
+      7,
+      15,
+      23,
+      31
+    ],
+    "tokens_per_second": 2,
+    "temporal_patch_size": 2
+  },
+  "rope_scaling": {
+    "type": "mrope",
+    "mrope_section": [
+      16,
+      24,
+      24
+    ]
+  },
+  "vocab_size": 151936,
+  "num_experts": 2,
+  "experts": [
+    {
+      "hidden_size": 2048,
+      "intermediate_size": 11008,
+      "hidden_act": "silu"
+    },
+    {
+      "hidden_size": 2048,
+      "intermediate_size": 2048,
+      "hidden_act": "silu"
+    }
+  ],
+  "dof_config": {
+    "follow_left_ee_cartesian_pos": 3,
+    "follow_left_ee_rotation": 3,
+    "follow_left_gripper": 1,
+    "follow_right_ee_cartesian_pos": 3,
+    "follow_right_ee_rotation": 3,
+    "follow_right_gripper": 1
+  },
+  "agent_pos_config": {
+    "follow_left_ee_cartesian_pos": 3,
+    "follow_left_ee_rotation": 3,
+    "follow_left_gripper": 1,
+    "follow_right_ee_cartesian_pos": 3,
+    "follow_right_ee_rotation": 3,
+    "follow_right_gripper": 1,
+    "follow_left_arm_joint_cur": 1,
+    "follow_right_arm_joint_cur": 1
+  },
+  "noise_scheduler": {
+    "beta_alpha": 1.5,
+    "beta_beta": 1.0,
+    "t_eps": 0.001,
+    "s": 0.999,
+    "num_inference_timesteps": 10
+  },
+  "dim_inputs": [
+    2048,
+    2048
+  ],
+  "attention_moe": false,
+  "mlp_moe": true,
+  "ar_loss_weight": 1,
+  "causal_action_attention_mask": true
+}