International Conference on Learning Representation (ICLR), 2024
[Project Page] [Arxiv] [Openreview]
Yinan Zheng*, Jianxiong Li*, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, Jingjing Liu
The official implementation of FISOR, which represents a pioneering effort in considering hard constraints (Hamilton-Jacobi Reachability) within the safe offline RL setting.
FISOR transforms the original tightly-coupled safety-constrained offline RL problem into three decoupled simple supervised objectives:
- Offline identification of the largest feasible region;
- Optimal advantage learning;
- Optimal policy extraction via time-independent classifier-guided diffusion model, enhancing both performance and stability.
Branch name | Usage |
---|---|
master | FISOR implementation for Point Robot , Safety-Gymnasium and Bullet-Safety-Gym ; data quantity experiment; feasible region visualization. |
metadrive_imitation | FISOR implementation for MetaDrive ; data quantity experiment; imitation learning experiment. |
conda create -n env_name python=3.9
conda activate FISOR
git clone https://github.com/ZhengYinan-AIR/FISOR.git
cd FISOR
pip install -r requirements.txt
Run
# OfflineCarButton1Gymnasium-v0
export XLA_PYTHON_CLIENT_PREALLOCATE=False
python launcher/examples/train_offline.py --env_id 0 --config configs/train_config.py:fisor
where env_id
serves as an index for the list of environments.
We can run filter_data.py to generate offline data of varying volumes. We also can download the necessary offline datasets (Download link). Then run
python launcher/examples/train_offline.py --env_id 17 --config configs/train_config.py:fisor --ratio 0.1
where ratio
refers to the proportion of the processed data to the original dataset.
We need to download the necessary offline dataset for Point Robot
environment (Download link). Training FISOR in the Point Robot
environment
python launcher/examples/train_offline.py --env_id 29 --config configs/train_config.py:fisor
Then visualize the feasible region by running viz_map.py.
If you find our code and paper can help, please cite our paper as:
@inproceedings{
zheng2024safe,
title={Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model},
author={Yinan Zheng and Jianxiong Li and Dongjie Yu and Yujie Yang and Shengbo Eben Li and Xianyuan Zhan and Jingjing Liu},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=j5JvZCaDM0}
}