addeding files

README.md

@@ -1,36 +1,58 @@
-# MultiAgent and QualityDiversity ReinforcementLearning
-This repository contains all the main informations about the university and thesis project on Multi Agent Reinforcemnt Learning and Quality Diversity.
+# Multi-Agent Reinforcement Learning with Quality Diversity
+The aim of the project is to develop a multi-agent reinforcement learning algorithm that uses quality diversity to create the sets of agents to solve a given multi agent task. The project is part of the master thesis developed by [Nielsen Erik](github.com/NielsenErik) at the University of Trento.
+This repo contains the code and relevant sources used to developed the thesis project. 
 The project is supervised by Giovanni Iacca and Andrea Ferigo from University of Trento and follows their current researches.
 
-## Papers and References
-In [references](/references) there is a comprehensive list of references of the studied papers to complete the project.
-
-## Source codes
-In [src](/src) are stored all the scripts developed during the project. The produced scripts are based and continue the work developed in the following papers by Giovanni Iacca, Marco Crespi, Andrea Ferigo, Leonardo Lucio Custode:
+## Introduction
+In [src](/src) are stored all the scripts developed during the project. The produced scripts and code are based on the following papers:
 - [A Population-Based Approach for Multi-Agent Interpretable Reinforcement Learning](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4467882)
 - [Quality Diversity Evolutionary Learning of Decision Trees](https://arxiv.org/abs/2208.12758)
 
-It is possible to run the aforementioned code by following the instructions in the README.md file in the [src/base](/src/base/) folder.
-Otherwise by changing the working direcotry `cd src` and running the following command:
+## Installation
+The project is developed in python 3.11. Here are the steps to install the project
 ```bash
-chmod +x script.sh
-source script.sh
+git clone https://github.com/NielsenErik/MultiAgent_and_QualityDiversity_ReinforcementLearning
+cd MultiAgent_and_QualityDiversity_ReinforcementLearning
+pip install -r requirements.txt
 ```
-It will appear the following menu:
+### Note
+1. The project is developed in python 3.11. It is recommended to use a virtual environment to install the project and its dependencies.
+
+2. Magent2 is the test environment of the project. To install it, first is required to clone the project and then install the downloaded repository:
 ```bash
-On the terminal output will appear the following menu:
-Hello! Here you can set environment and run codes
-Please enter an integer to select an option:
-[1]. Activate environment
-[2]. Deactivate environment
-[3]. Run code dts4marl
-[4]. Run code marldts
-[5]. Run code qd_marl
-[6]. Run code qd_marl with debug mode
-[7]. Run test environment
-[8]. Exit
+git clone https://github.com/Farama-Foundation/MAgent2
+cd MAgent2
+pip install -e .
 ```
-Press 1 to activate the python venv.
-Then run `./script.sh` again and select one of the possible experiment.
-If 3 or 4 is selected it will run the projects developed by Giovanni Iacca, Marco Crespi, Andrea Ferigo, Leonardo Lucio Custode.
-if 5 or 6 (for debug and serialized mode) is selected it will run the project developed in this repository which apply a Quality Diversity approach to a Multi Agent Reinforcement Learning task.
+This solution was proposed by [Issue #19](https://github.com/Farama-Foundation/MAgent2/issues/19) of the MAgent2 repository
+
+## Running the project
+To run the project, if the installation is done by creating a virtual environment, first is required to activate the virtual environment:
+```bash
+source venv/bin/activate
+```
+Then, to run the project, execute the following command:
+```bash
+chmod +x script.sh
+./script.sh
+```
+The script will pop different running options, choose the desired option and the project will start running.
+
+## Structure
+The project is structured as follows:
+1. [src](/src): Contains the source code of the project
+    1. [agents](/src/agents): Contains the agents classes and algorithms used in the project
+    2. [algorithm](/src/algorithm): Contains the algorithm regarding Map-Elites and Quality Diversity, developed using PyRibs, and the classes for Genetic Algorithm and Genetic Programming
+    3. [config](/src/config): Contains the configuration files used in the project, such as the configuration of the environment, the algorithm, and the agents and most importantly the configuration of Map-Elites archive
+    4. [decisiontrees](/src/decisiontrees): Contains the classes for create and manage the Decision Trees, RL-Decision Trees, Leaves and the Conditions on the Trees Nodes
+    5. [utils](/src/utils): Contains the utility functions used in the project
+2. [logs](/logs): Contains the logs files generated during the execution of the project
+3. [hpc_scripts](/hpc_scripts): Contains the scripts used to run the project on the High-Performance Computing (HPC) cluster
+
+## Papers and References
+In [references](/references) there is a comprehensive list of references of the studied papers to complete the project.
+
+## Source codes
+In [src](/src) are stored all the scripts developed during the project. The produced scripts work only if linked to the code developed in the following papers by Giovanni Iacca, Marco Crespi, Andrea Ferigo, Leonardo Lucio Custode:
+- [A Population-Based Approach for Multi-Agent Interpretable Reinforcement Learning](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4467882)
+- [Quality Diversity Evolutionary Learning of Decision Trees](https://arxiv.org/abs/2208.12758)

hpc_run_all.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+cd ${PBS_O_WORKDIR}/MARL-QD/Marl-QD_Private/
+for file in $(ls $PWD/hpc_scripts/with_set/); do
+    qsub $PWD/hpc_scripts/with_set/$file
+done

hpc_scripts/no_set/hpc_script_pyribsCMA_best.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=6:00:00
+
+#execution queue configs
+#PBS -q short_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher_no_sets.py $PWD/src/QD_MARL/configs/hpc/no_sets/battlefield_hpc_pyribsCMA_best.json 4

hpc_scripts/no_set/hpc_script_pyribsCMA_coach.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=6:00:00
+
+#execution queue configs
+#PBS -q short_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher_no_sets.py $PWD/src/QD_MARL/configs/hpc/no_sets/battlefield_hpc_pyribsCMA_coach.json 4

hpc_scripts/no_set/hpc_script_pyribsCMA_random.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=6:00:00
+
+#execution queue configs
+#PBS -q short_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher_no_sets.py $PWD/src/QD_MARL/configs/hpc/no_sets/battlefield_hpc_pyribsCMA_random.json 4

hpc_scripts/no_set/hpc_script_pyribs_best.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=6:00:00
+
+#execution queue configs
+#PBS -q short_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher_no_sets.py $PWD/src/QD_MARL/configs/hpc/no_sets/battlefield_hpc_pyribs_best.json 4

hpc_scripts/no_set/hpc_script_pyribs_coach.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=6:00:00
+
+#execution queue configs
+#PBS -q short_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher_no_sets.py $PWD/src/QD_MARL/configs/hpc/no_sets/battlefield_hpc_pyribs_coach.json 4

hpc_scripts/no_set/hpc_script_pyribs_random.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=6:00:00
+
+#execution queue configs
+#PBS -q short_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher_no_sets.py $PWD/src/QD_MARL/configs/hpc/no_sets/battlefield_hpc_pyribs_random.json 4

hpc_scripts/with_set/hpc_script_pyribsCMA_best.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=30:00:00
+
+#execution queue configs
+#PBS -q common_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher.py $PWD/src/QD_MARL/configs/hpc/with_sets/battlefield_hpc_pyribsCMA_best.json 4

hpc_scripts/with_set/hpc_script_pyribsCMA_coach.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=30:00:00
+
+#execution queue configs
+#PBS -q common_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher.py $PWD/src/QD_MARL/configs/hpc/with_sets/battlefield_hpc_pyribsCMA_coach.json 4

hpc_scripts/with_set/hpc_script_pyribsCMA_random.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=30:00:00
+
+#execution queue configs
+#PBS -q common_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher.py $PWD/src/QD_MARL/configs/with_sets/hpc/battlefield_hpc_pyribsCMA_random.json 4

hpc_scripts/with_set/hpc_script_pyribs_best.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=30:00:00
+
+#execution queue configs
+#PBS -q common_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher.py $PWD/src/QD_MARL/configs/hpc/with_sets/battlefield_hpc_pyribs_best.json 4

hpc_scripts/with_set/hpc_script_pyribs_coach.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=30:00:00
+
+#execution queue configs
+#PBS -q common_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher.py $PWD/src/QD_MARL/configs/hpc/with_sets/battlefield_hpc_pyribs_coach.json 4

hpc_scripts/with_set/hpc_script_pyribs_random.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+#resources allcoation
+#PBS -l select=1:ncpus=12:mem=100gb -l place=pack:excl
+
+#set max execution time
+#PBS -l walltime=30:00:00
+
+#execution queue configs
+#PBS -q common_cpuQ
+
+#execution outpust name
+#PBS -N marl_battlefield_hpc
+
+#set mail notification
+#PBS -M erik.nielsen@studenti.unitn.it
+
+cd ${PBS_O_WORKDIR}
+
+module load python-3.8.13
+source $PWD/pyenv_hpc/bin/activate
+python $PWD/src/QD_MARL/marl_qd_launcher.py $PWD/src/QD_MARL/configs/hpc/with_sets/battlefield_hpc_pyribs_random.json 4