Starrick Liu
421db17d53
This release introduces significant performance optimizations, memory efficiency improvements, and enhanced system robustness: 🚀 Performance Optimizations: - Add three new fused CUDA kernels (rope_index, rot_pos_emb, get_window_index) for accelerated multimodal preprocessing - Implement FSDP2 support for distributed training with improved memory efficiency - Add Torch.compile integration for additional performance gains - Optimize memory usage: reduce peak allocation from 48GB to 24GB on 8-GPU setup 🔧 System Robustness: - Fix missing token position inputs in prediction pipeline - Add type-robust negation operations in RoPE CUDA kernels (half/bfloat16 support) - Fix dataset root parameter initialization in LeRobot data loader - Enhanced error handling and input validation across fusion operators 📚 Documentation & Usability: - Add comprehensive memory usage benchmarks and hardware recommendations - Update citation format with proper arXiv reference - Improve training configuration documentation with quick start guide - Add detailed API documentation for new fusion operators 🛠️ Technical Details: - Version bump to 1.0.1 - New CUDA kernels: rope_index.cu, rot_pos.cu, window_index.cu - FSDP2 state dict loading with distribute_tensor support - Enhanced multimodal RoPE with 3D position encoding - Window attention optimization for Vision Transformers Breaking Changes: None - all changes are backward compatible
24 lines
701 B
Bash
24 lines
701 B
Bash
#!/bin/bash
|
|
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
|
NUM_GPUS=$(echo $CUDA_VISIBLE_DEVICES | tr ',' '\n' | wc -l)
|
|
|
|
# print current time
|
|
echo "[current time: $(date +'%Y-%m-%d %H:%M:%S')]"
|
|
|
|
code_dir="/path/to/wall-x"
|
|
config_path="/path/to/wall-x/workspace/lerobot_example"
|
|
|
|
# Use a fixed port instead of a random one
|
|
export PORT=$((21000 + $RANDOM % 30000))
|
|
|
|
MASTER_PORT=10239 # use 5 digits ports
|
|
|
|
export LAUNCHER="accelerate launch --num_processes=$NUM_GPUS --main_process_port=$PORT"
|
|
|
|
export SCRIPT="${code_dir}/train_qact.py"
|
|
export SCRIPT_ARGS="--config ${config_path}/config_qact.yml --seed $MASTER_PORT"
|
|
|
|
echo "Running command: $LAUNCHER $SCRIPT $SCRIPT_ARGS"
|
|
|
|
$LAUNCHER $SCRIPT $SCRIPT_ARGS
|