Wall-X

Overview

Wall-X is a multimodal foundation model designed for robotics applications, combining vision, language, and action capabilities. The model architecture is built upon Qwen2.5-3B-VL with specialized adaptations for robotic control tasks.

Environment Setup

Create and activate conda environment:

conda create --name wallx python=3.10
conda activate wallx

Install requirements:

pip install -r requirements.txt
MAX_JOBS=4 pip install flash-attn==2.7.4.post1 --no-build-isolation

Install lerobot:

git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e .

Install wall_x:

git submodule update --init --recursive
MAX_JOBS=4 pip install --no-build-isolation --verbose .

Training

Finetune on LeRobot Datasets

Before training, please refer to workspace/README.md for detailed configuration instructions including:

Training script path configuration

GPU setup
Model and data paths
Robot DOF configuration
Training hyperparameters

bash ./workspace/lerobot_example/run.sh

Inference

For model inference, please refer to:

python ./scripts/fake_inference.py

This script demonstrates how to:

Load the Wall-OSS model using Qwen2_5_VLMoEForAction.from_pretrained()
Prepare input data including proprioceptive information, attention masks, and dataset specifications
Run inference in validation mode with proper data types (bfloat16)
Validate model outputs and check for numerical stability

To generate an open-loop comparison plot, please follow:

python ./scripts/draw_openloop_plot.py

1.6 KiB Raw Blame History