INF581 - Reinforcement Learning on a ShootEm Up

This repository contains the code for a simple ShootEm Up game, and the implementation of various reinforcement learning agents in order to play it.

Setups for VSCode and Pycharm are included. Instructions on how to use the project build tools in command line can be accessed here.

Project structure

├── .idea     # PyCharm configuration
├── .vscode   # VSCode configuration
│
├── game      # Main source directory
│   ├── backend            # Game backend logic
│   ├── frontend           # Game frontend display using PyGame
│   ├── rl_agents          # Reinforcement Learning agent definition
│   └── rl_environment     # Reinforcement Learning game environment wrapper
│
├── notebook  # Demo Jupyter notebook
│
└── scripts   # Ready-to-run scripts
    ├── example.py         # Example script
    ├── run_game.py        # Run the game in order to play it yourself
    └── setup.py           # Python PATH setup to import at the top of each script

Quickstart

Poetry

This project manages its dependencies and python version with poetry. You can find installation instructions here.

Install the dependencies with poetry: You must first install Python 3.11 on your machine (preferably using a manager like conda, pyenv...)

poetry install

If this command fails, (install is stuck on 'Pending...') you can run the following command before retrying :

export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring

The issue is described here. Otherwise, you can install the dependencies manually from the pyproject.toml file.

This will create a virtual environment in the .venv root folder. You can activate it with:

poetry shell

Add, remove and update dependencies with the following commands

poetry add <package>
poetry remove <package>
poetry update

You can then run scripts with:

python scripts/example.py  # Replace with any script

Launch the game with:

python scripts/run_game.py

You can control the player with the arrow keys. It will look in the direction of your mouse.

Note on TorchRL

It is way easier to define a classic torch Gym-like environment and then use torchrl's converters to make it compatible. Trying to directly subclass torchrl's EnvBase class is a pain because of the way its internals are designed and the lack of existing examples. Our GameEnv class would need to be refactored this way before attempting to complete the run_ppo.py script for PPO training.

Build tools

This project uses Black for code formatting, PyLint for linting and isort for import statements sorting. Checks are run using a Gitlab-CI pipeline in order to ensure that the code is properly formatted, linted and sorted.

Add the recommended extensions from the .vscode/extensions.json file to your VSCode workspace. This project is based on the Python extension from Microsoft. It uses Black for formatting and Pylint for linting.

Use the following shortcuts for everything formatting and linting related: These shortcuts are written in the [tool.poe.tasks] section of the pyproject.toml file.

Format the code:

poe format-write

Check that the code has been formatted (used only in gitlab-ci checks):

poe format-check

Lint the code / check that there are no errors or warnings:

poe lint

Sort the imports:

poe sort-imports

Check that the imports are correctly sorted:

poe sort-check