Microsoft MLADS Fall 2020

Reinforcement Learning with Azure ML: Beyond Games

This repository contains the files and code for the Microsoft MLADS Fall 2020 lab.

Lab Goals

• Gain a basic understanding of the Reinforcement Learning (RL) functionality and APIs available in the Azure ML Platform.
• Understand a typical use case for optimization and how it is framed as a RL problem.
• Learn how to use the RLLib APIs available in Azure ML to integrate a custom RL environment, a custom policy model, and train the RL agent in a scalable fashion.

Target Audience

You will benefit most from this lab if you:

• Have a good understanding about the Reinforcement Learning paradigm in Machine Learning, including the main approaches and techniques.
• Have a good understanding about basic neural network architectures and basic concepts of deep learning.
• Are comfortable with reading and writing Python code.
• Know the basic concepts of Azure ML, and how to setup a workspace and compute instance.

Lab Requirements

To reproduce this lab you will need:

1. Access to a Microsoft Azure subscription.
2. Create an Azure ML Workspace and a Compute Instance to run the code for this lab. You can follow the instructions here: Create and manage Azure Machine Learning workspaces and Create and manage an Azure Machine Learning compute instance. A VM with 2 or 4 CPU cores should be enough for your compute instance.
3. Access the Azure ML studio UI in your workspace and select Notebooks. Then click on Terminal (preview). This will open a terminal window in your running compute instance. From there you can clone this GitHub repository into your workspace.

To gain a better understanding of the RL capabilities on Azure ML, please see this documentation.

You may also want to refer to the Ray and especially the RLLib documentations, as they form the basis of the RL functionality provided by Azure ML.

Use Case Overview

We are going to train RL agents that learn to optimize resource allocation for tasks execution.

The lab is based on the following paper and code:

• Paper: Resource Management with Deep Reinforcement Learning
• Code: DeepRM GitHub repository *

* We are using only the code needed to reproduce the RL environment: environment.py, job_distribution.py, and parameters.py after minor modifications as explained in the lab notebooks.

Lab Overview

The lab is divided in 4 parts and should be executed in the following order:

1. Azure ML Setup: this will configure the necessary networking objects and permissions in your Azure ML workspace, to allow the proper instantiation of the RL infrastructure on Azure ML to perform the agent training.

2. Environment Exploration: here we explore the environment by acting and observing the corresponding states. We also analyze how a random policy and a heuristics-based policy behave, to have baselines to compare with trained agents. You can also check the environment rollouts for those environments in the videos provided in the environment_rollouts subfolder.

3. Agent Training: here we train RL agents on Azure ML, using one of the available RL algorithms from RLLib and showing how to integrate the training with a custom environment and a custom model for the policy. We also show how to scale out the training process, using multiple worker processes for the environment rollout simulations.

4: Agent Testing: here we test the trained agents, analyzing the corresponding learned policies by observing the state transitions in the environment during a rollout.