notebooks/3DBall_parallel_environment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3D Balance Ball in parallel environments\n",
    "\n",
    "In this notebook, we will run the [3D Balance Ball example](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Learning-Environment-Examples.md#3dball-3d-balance-ball) from [Unity ML Agents](https://unity.com/products/machine-learning-agents) in parallel environments. Please check the README file to setup this project.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from parallel_unity_environment import ParallelUnityEnvironment\n",
    "from model import Actor, Critic\n",
    "from ddpg_agents import DDPGAgents\n",
    "from ddpg_agent import DDPGAgent\n",
    "from replay_buffer import ReplayBuffer\n",
    "from utilities import convert_to_tensor\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn.functional as f\n",
    "import random\n",
    "from collections import deque\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment. Before running the code cell below, change the `ENV_FILE_NAME` parameter to match the location of the Unity environment that you [downloaded](README.md) or [created](../README.md#creating-a-custom-unity-executable) yourself. For example:\n",
    "```\n",
    "ENV_FILE_NAME = \"3DBall_Windows_x86_64/UnityEnvironment.exe\"\n",
    "```\n",
    "Four new windows should pop up, one for each environment. Don't worry if the windows become unresponsive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ENV_FILE_NAME = \"3DBall_Windows_x86_64/UnityEnvironment.exe\"\n",
    "NUM_ENVS = 4  # number of environments to run in parallel\n",
    "CHECKPOINT_FILENAME = \"checkpoint-3dball-parallel.pth\" # this is used for saving and loading the model\n",
    "DISPLAY_SIZE = [1024, 768] # The width and height of the Unity windows\n",
    "\n",
    "test_env = ParallelUnityEnvironment(num_envs=NUM_ENVS, seeds=list(range(NUM_ENVS)),\n",
    "                                    file_name=ENV_FILE_NAME, no_graphics=False)\n",
    "test_env.set_timescale(1.0)\n",
    "test_env.set_display_size(width=DISPLAY_SIZE[0], height=DISPLAY_SIZE[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, an agent must balance a ball on its head for as long as possible.\n",
    "\n",
    "**Agent Reward Function:**\n",
    "- +0.1 for every step the ball remains on its head.\n",
    "- -1.0 if the ball falls off.\n",
    "\n",
    "**Behavior Parameters:**\n",
    "- Vector Observation space: 8 variables corresponding to rotation of the agent cube, and position and velocity of ball.\n",
    "- Actions: 2 continuous actions, with one value corresponding to X-rotation, and the other to Z-rotation.\n",
    "\n",
    "Run the code cell below to print some information about the environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 1\n",
      "Size of each action: 2\n",
      "States look like: [-0.01467304 -0.01468306 -0.52082086  4.         -0.79952097  0.\n",
      "  0.          0.        ]\n",
      "States have shape: (8,)\n"
     ]
    }
   ],
   "source": [
    "def examine_environment(env: ParallelUnityEnvironment):\n",
    "    # number of agents in the first behavior:\n",
    "    print('Number of agents:', env.num_agents_list[0])\n",
    "\n",
    "    # number of actions\n",
    "    print('Size of each action:', env.behavior_specs[0].action_spec.continuous_size)\n",
    "\n",
    "    # examine the state space\n",
    "    print('States look like:', env.get_observations(0, 0)[0])\n",
    "    print('States have shape:', env.behavior_specs[0].observation_specs[0].shape)\n",
    "\n",
    "examine_environment(test_env)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Parallel Environment\n",
    "\n",
    "Run the code cell below, to watch a random agent in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score from environment 3, episode 0: 1.40\n",
      "Score from environment 0, episode 1: 1.90\n",
      "Score from environment 2, episode 2: 2.10\n",
      "Score from environment 1, episode 3: 2.70\n",
      "Score from environment 3, episode 4: 1.40\n",
      "Score from environment 2, episode 5: 2.10\n",
      "Score from environment 1, episode 6: 1.70\n",
      "Score from environment 3, episode 7: 1.30\n",
      "Score from environment 0, episode 8: 3.40\n",
      "Score from environment 2, episode 9: 0.70\n",
      "Time elapsed: 8.16\n"
     ]
    }
   ],
   "source": [
    "def test_random_agents(env: ParallelUnityEnvironment, n_episodes: int, max_t: int):\n",
    "    start_time = time.time()\n",
    "    current_episode = 0\n",
    "    current_timestep_list = [0] * env.num_envs\n",
    "    scores = np.zeros(env.num_envs)\n",
    "    reset_list = [True] * env.num_envs\n",
    "    reset_env = True\n",
    "    while current_episode < n_episodes:\n",
    "        # reset environments if needed:\n",
    "        if reset_env:\n",
    "            env.reset(reset_list)\n",
    "            for env_index, reset in enumerate(reset_list):\n",
    "                reset_list[env_index] = False\n",
    "\n",
    "        # set actions for each environment:\n",
    "        for env_index in range(env.num_envs):\n",
    "            actions = np.random.randn(env.num_agents_list[0], env.behavior_specs[0].action_spec.continuous_size)\n",
    "            actions = np.clip(actions, -1, 1)\n",
    "            env.set_actions(behavior_index=0, env_index=env_index, continuous=actions)\n",
    "\n",
    "        # step forward in all environments:\n",
    "        env.step()\n",
    "\n",
    "        for env_index in range(env.num_envs):\n",
    "            # collect experiences from environment:\n",
    "            _, rewards, dones = env.get_experiences(behavior_index=0, env_index=env_index)\n",
    "            scores[env_index] += rewards.squeeze()\n",
    "            current_timestep_list[env_index] += 1\n",
    "\n",
    "            # check if episode has ended:\n",
    "            if current_timestep_list[env_index] >= max_t or np.any(dones):\n",
    "                print(f\"Score from environment {env_index}, episode {current_episode}: \"\n",
    "                      f\"{scores[env_index]:.2f}\")\n",
    "                current_timestep_list[env_index] = 0\n",
    "                reset_list[env_index] = True\n",
    "                reset_env = True\n",
    "                scores[env_index] = 0.0\n",
    "                current_episode += 1\n",
    "\n",
    "    print(f\"Time elapsed: {time.time() - start_time:.2f}\")\n",
    "\n",
    "test_random_agents(test_env, n_episodes=10, max_t=200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [],
   "source": [
    "test_env.close()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 4. Train the Agent with DDPG\n",
    "\n",
    "Run the code cells below to train the agent from scratch.\n",
    "\n",
    "Alternatively, you can skip to the next step below (**5. Watch a Smart Agent**), to load the saved model weights from a pre-trained agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class DDPGAgentsTester:\n",
    "    def __init__(self, ddpg_agents: DDPGAgents,\n",
    "                 env: ParallelUnityEnvironment,\n",
    "                 buffer_size=int(1.0e6),  # replay buffer size\n",
    "                 noise_start=1.0\n",
    "                 ):\n",
    "        self.ddpg_agents = ddpg_agents\n",
    "        self.env = env\n",
    "        self.buffer_size = buffer_size\n",
    "        self.scores = []\n",
    "        self.scores_deque = deque(maxlen=100)\n",
    "        self.episode = 0\n",
    "        self.noise = noise_start\n",
    "        self.replay_buffer = ReplayBuffer(buffer_size)\n",
    "\n",
    "    def train_agents(self, n_episodes, max_t, goal=float(\"inf\"), print_every=1000, update_every=1,\n",
    "                     num_updates=1, batch_size=64, noise_decay=6.93e-6):\n",
    "        \"\"\" Multi Agent Deep Deterministic Policy Gradient algorithm.\n",
    "\n",
    "        Params\n",
    "        ======\n",
    "            n_episodes (int): maximum number of training episodes\n",
    "            max_t (int): maximum number of timesteps per episode\n",
    "            goal (float): the algorithm will stop when the goal is reached\n",
    "            print_every (int) : print intermediate results every %print_every episodes\n",
    "            update_every (int): update the neural networks every %update_every time steps\n",
    "            num_updates: How many updates to do in a row\n",
    "            batch_size (int): minibatch size\n",
    "            noise_decay (float): noise decay factor = 1.0 - %noise_decay\n",
    "        \"\"\"\n",
    "        noise_decay = 1.0 - noise_decay\n",
    "        stop_episode = self.episode + n_episodes\n",
    "        timesteps = 0\n",
    "        start_time = time.time()\n",
    "        last_print_time = 0\n",
    "        current_timestep_list = [0] * self.env.num_envs\n",
    "        scores = np.zeros((self.env.num_envs, len(self.ddpg_agents)))\n",
    "        states_list = [np.ndarray((0,))] * self.env.num_envs\n",
    "        actions_list = [np.ndarray((0,))] * self.env.num_envs\n",
    "        reset_list = [True] * self.env.num_envs\n",
    "        reset_env = True\n",
    "        while self.episode < stop_episode:\n",
    "            # reset environments if needed:\n",
    "            if reset_env:\n",
    "                self.env.reset(reset_list)\n",
    "                for env_index, reset in enumerate(reset_list):\n",
    "                    if reset:\n",
    "                        states_list[env_index] = self.env.get_observations(behavior_index=0, env_index=env_index)\n",
    "                        reset_list[env_index] = False\n",
    "\n",
    "            # get a batch of states from all environments:\n",
    "            env_states = np.stack([states for env_index, states in enumerate(states_list)], axis=1)\n",
    "            # get actions from all agents:\n",
    "            env_actions = self.ddpg_agents.act(convert_to_tensor(env_states), self.noise)\n",
    "\n",
    "            # set actions for each environment:\n",
    "            for env_index in range(self.env.num_envs):\n",
    "                actions_list[env_index] = env_actions[:, env_index, :]\n",
    "                self.env.set_actions(behavior_index=0, env_index=env_index, continuous=actions_list[env_index])\n",
    "\n",
    "            # step forward in all environments:\n",
    "            self.env.step()\n",
    "\n",
    "            for env_index in range(self.env.num_envs):\n",
    "                # collect experiences from environment:\n",
    "                next_states, rewards, dones = self.env.get_experiences(behavior_index=0, env_index=env_index)\n",
    "\n",
    "                # add sample to replay buffer:\n",
    "                sample = (states_list[env_index].copy(), actions_list[env_index].copy(), rewards, next_states, dones)\n",
    "                self.replay_buffer.add(sample)\n",
    "\n",
    "                # update networks every %update_every time steps:\n",
    "                if timesteps % update_every == 0 and len(self.replay_buffer) > batch_size * 100:\n",
    "                    for _ in range(num_updates):\n",
    "                        samples = [self.replay_buffer.sample(batch_size) for _ in range(len(self.ddpg_agents))]\n",
    "                        self.ddpg_agents.step(samples)\n",
    "                        #soft update the target network towards the actual networks:\n",
    "                        self.ddpg_agents.update_target_networks()\n",
    "\n",
    "                states_list[env_index] = next_states\n",
    "                self.noise *= noise_decay\n",
    "                scores[env_index] += rewards.squeeze()\n",
    "                current_timestep_list[env_index] += 1\n",
    "                timesteps += 1\n",
    "\n",
    "                # check if episode has ended:\n",
    "                if current_timestep_list[env_index] >= max_t or np.any(dones):\n",
    "                    self.scores_deque.append(scores[env_index, :].copy())\n",
    "                    self.scores.append(scores[env_index, :].copy())\n",
    "                    current_timestep_list[env_index] = 0\n",
    "                    reset_list[env_index] = True\n",
    "                    reset_env = True\n",
    "                    scores[env_index, :] = 0.0\n",
    "                    self.episode += 1\n",
    "\n",
    "                    average_scores = np.mean(self.scores_deque, 0)  # average score over last 100 episodes for each agent\n",
    "                    if time.time() - last_print_time > 1.0:\n",
    "                        time_per_step = (time.time() - start_time) / timesteps\n",
    "                        print('\\rEpisode {}\\tSteps: {}\\tTime per step: {:.6f}\\tAverage Scores: {:.3f}'\n",
    "                              .format(self.episode, timesteps, time_per_step, *average_scores), end=\"\")\n",
    "                        last_print_time = time.time()\n",
    "                    if self.episode % print_every == 0:\n",
    "                        print(\"\\r\" + \" \" * 80, end=\"\")\n",
    "                        print('\\rEpisode {}\\tAverage Scores: {:.3f}'.format(self.episode, *average_scores))\n",
    "                    if np.max(average_scores) >= goal:\n",
    "                        print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}\\tTime elapsed: {}'.format(\n",
    "                            self.episode, np.max(average_scores), time.time() - start_time))\n",
    "                        return\n",
    "\n",
    "    def test_agents(self, n_episodes, max_t):\n",
    "        current_episode = 0\n",
    "        current_timestep_list = [0] * self.env.num_envs\n",
    "        scores = np.zeros((self.env.num_envs, len(self.ddpg_agents)))\n",
    "        states_list = [np.ndarray((0,))] * self.env.num_envs\n",
    "        actions_list = [np.ndarray((0,))] * self.env.num_envs\n",
    "        reset_list = [True] * self.env.num_envs\n",
    "        reset_env = True\n",
    "        while current_episode < n_episodes:\n",
    "            # reset environments if needed:\n",
    "            if reset_env:\n",
    "                self.env.reset(reset_list)\n",
    "                for env_index, reset in enumerate(reset_list):\n",
    "                    if reset:\n",
    "                        states_list[env_index] = self.env.get_observations(behavior_index=0, env_index=env_index)\n",
    "                        reset_list[env_index] = False\n",
    "\n",
    "            # get a batch of states from all environments:\n",
    "            env_states = np.stack([states for env_index, states in enumerate(states_list)], axis=1)\n",
    "            # get actions from all agents:\n",
    "            env_actions = self.ddpg_agents.act(convert_to_tensor(env_states), self.noise)\n",
    "\n",
    "            # set actions for each environment:\n",
    "            for env_index in range(self.env.num_envs):\n",
    "                actions_list[env_index] = env_actions[:, env_index, :]\n",
    "                self.env.set_actions(behavior_index=0, env_index=env_index, continuous=actions_list[env_index])\n",
    "\n",
    "            # step forward in all environments:\n",
    "            self.env.step()\n",
    "\n",
    "            for env_index in range(self.env.num_envs):\n",
    "                # collect experiences from environment:\n",
    "                next_states, rewards, dones = self.env.get_experiences(behavior_index=0, env_index=env_index)\n",
    "                states_list[env_index] = next_states\n",
    "                scores[env_index] += rewards.squeeze()\n",
    "                current_timestep_list[env_index] += 1\n",
    "\n",
    "                # check if episode has ended:\n",
    "                if current_timestep_list[env_index] >= max_t or np.any(dones):\n",
    "                    print(f\"Score from environment {env_index}, episode {current_episode}: \"\n",
    "                          f\"{scores[env_index, 0]:.2f}\")\n",
    "                    current_timestep_list[env_index] = 0\n",
    "                    reset_list[env_index] = True\n",
    "                    reset_env = True\n",
    "                    scores[env_index, :] = 0.0\n",
    "                    current_episode += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [],
   "source": [
    "random_seed = 1\n",
    "np.random.seed(random_seed)\n",
    "torch.manual_seed(random_seed)\n",
    "random.seed(random_seed)\n",
    "train_env = ParallelUnityEnvironment(num_envs=NUM_ENVS, seeds=list(range(random_seed, random_seed + NUM_ENVS)),\n",
    "                                     file_name=ENV_FILE_NAME, no_graphics=True, worked_id_start=10)\n",
    "train_env.set_timescale(100.0)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [],
   "source": [
    "actor1 = Actor(state_size=8, action_size=2, hidden_layer_sizes=[400, 300], activation_func=f.leaky_relu)\n",
    "critic1 = Critic(state_size=8, action_size=2, hidden_layer_sizes=[400, 300], activation_func=f.leaky_relu,\n",
    "                 inject_layer=0)\n",
    "ddpg_agent1 = DDPGAgent(actor1, critic1, gamma=0.99, tau=1.0e-3, lr_actor=1.0e-4, lr_critic=1.0e-3, weight_decay=1.0e-2)\n",
    "ddpg_agent_list = [ddpg_agent1]\n",
    "ddpg_agents = DDPGAgents(ddpg_agent_list)\n",
    "ddpg_agents_tester = DDPGAgentsTester(ddpg_agents, train_env, buffer_size=int(1.0e6), noise_start=1.0)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "You can skip this cell, if you don't want to train the agent from scratch. It may take 30 to 45 minutes:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1000\tAverage Scores: 1.270                                              \n",
      "Episode 2000\tAverage Scores: 1.220                                              \n",
      "Episode 3000\tAverage Scores: 1.101                                              \n",
      "Episode 4000\tAverage Scores: 2.310                                              \n",
      "Episode 5000\tAverage Scores: 5.038                                              \n",
      "Episode 6000\tAverage Scores: 4.930                                              \n",
      "Episode 6708\tSteps: 234123\tTime per step: 0.009527\tAverage Scores: 9.930\n",
      "Environment solved in 6710 episodes!\tAverage Score: 10.00\tTime elapsed: 2231.088708639145\n"
     ]
    }
   ],
   "source": [
    "ddpg_agents_tester.env = train_env\n",
    "ddpg_agents_tester.train_agents(n_episodes=int(1.0e5), max_t=100, goal=10.0, update_every=1,\n",
    "                                num_updates=1, batch_size=64, noise_decay=6.93e-6)\n",
    "ddpg_agents.save_checkpoint(filename=CHECKPOINT_FILENAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEGCAYAAACQO2mwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAy7ElEQVR4nO3deXgUVdYH4N/Jxr4Fwr6EHUQQMICAIAiMSBRGHAdxXBgdcUGR0W8UlBll3BjHcRx3GQQdFUQQAdlkB0G2AGELWyCBhASSAAlryHa/P6q6093p7lR319p13ufhoVPprjpJd+rUvXXvuSSEAGOMMeZNhNEBMMYYMy9OEowxxnziJMEYY8wnThKMMcZ84iTBGGPMpyijA1CiQYMGIj4+3ugwGGPMUnbt2pUnhIgLZR+WSBLx8fFISkoyOgzGGLMUIjoZ6j64u4kxxphPnCQYY4z5xEmCMcaYT5wkGGOM+cRJgjHGmE+aJQkimkVEOUR0wGVbLBGtJqJj8v/1tDo+Y4yx0GnZkvgSwHCPbZMBrBVCtAewVv6aMcaYSWk2T0IIsYmI4j02jwIwSH78FYANAF7SKgbGtHD5egnWpJzFb3s0AwCsO3wWnRrXRtO61YLeZ2FxKZbuy8a9PZuBiILax+qUszhwugB/HtYBP+3NQnz9Gvhi8wksSs7CuH7x+E2XRsi/Wox1h3Nw4HQBDp+5hCcGtsFNLeqiRpUo5F8twpmCQjxxW9uAj73+cA46NK6FSCKMm70Dh89cAgB0bVYHjWpXRXJGPvIuX6/wurG9W2LT0Vw8Pbgtrl4vxZvLD7l9v0pUBBY/0x+dGtcO6ncSjNIygbEztqFp3apYlJwFADj6xp2IidLmmjq74BoOZV/EzF/SQATc3NK9g+Wens3RukENTY6tBGm5noScJJYKIW6Uv84XQtR1+f4FIYTXLiciGg9gPAC0bNny5pMnQ54TwpgqnvtuDxYnZ2HJM/3RrXldxE9ehgY1Y5A0dVjQ+/z7TymYtSUNs8Yl4PZOjYLaR/zkZQCANc/fhqHvbQw6luUTB+CGpoGdlOMnL0OdatEouFYc9HH9SZ+eqMl+vflicxpeX5riti2hVT0seKqfJsdLeGONWwL1vEaYPa4XBnVsGNS+iWiXECIhlPhMO+NaCDEDwAwASEhI4JWRmGlkFxQCAK5cL3Vuy7tcFNI+c+WTxKXCkpD2A0itklBcC/L1WiUIvZ3z0uJJP3dVs+N5trDS3tYvISqh9+ims0TUBADk/3N0Pj5jIQuuM4hZm32vU/VOEksAPCI/fgTAYp2Pz5hqhI1PHMw+tBwCOxfAVgAdiSiTiB4DMB3AMCI6BmCY/DVjluLsM+YcwWxAy9FNY318a4hWx2RMDyR3OHGOsBP7djLyjGvGAhTkCFVmEZz83XGSYCxIGo4eZ8w0OEkwFiBHS4JvXNuJfd9rThKMBYhs3D/N7IeTBGNB4u4mZgecJBgLUHl3E2Phj5MEY0HSsu4ZY2bBSYKxAAVbpZUxK+IkwViQuB3B7ICTBGMB4nYEsxNOEowxVin7XhpwkmAsWNzfZCP2fbM5STAWIJ5xHd540Jo7ThKMBci+HQ/2ZefEwUmCsSDZ+cQRzry1EO086pmTBGMBcsyT4CTB7ICTBGMB4oXpwhsXcHTHSYKxANm568HhwOkC/JqaZ3QYTAeaLV/KWLizc+2muz7cDABIn55ocCRMa9ySYCwA20+cw5pDOQC4u8lObHw9wEmCsUCMmbHN6BBsKS3vitEh2BYnCcaCpObVJd/m8K+4tMzoEGyLkwRjQVMvS5ipN4NvzDNXnCQYC5JZ+6nNGhezJk4SjDE3nGQqulZcanQIhuEkwRhjLryV5bBz4uQkwViQBOw9V8JO7HyfhpMEY4xVwsY5gpMEY8HiRgSzA04SjMmKSsrwxeY0lPCYfNMx+kreztcDnCQYk83cfAKvL03Bt9tPKX4Ntyb0wb9m4xiSJIjoz0R0kIgOENFcIqpqRByMubpUWAIAuHy9RNHzy4TArC1pWobETMLoloyRdE8SRNQMwEQACUKIGwFEArhf7zgYC9VPe7PwxrJDRofBmKaM6m6KAlCNiKIAVAeQZVAcjAXNrBOs7Dxck6lP9yQhhDgN4F0ApwBkAygQQqzyfB4RjSeiJCJKys3N1TtMxipl1vsRZo2LWZMR3U31AIwC0BpAUwA1iOhBz+cJIWYIIRKEEAlxcXF6h8mYrvji30Q4yboxortpKIA0IUSuEKIYwEIA/QyIgzHTMNN5yYzdVUaHRGb8pejEiCRxCsAtRFSdpN/8EAB8948xlXirPcRC41p+JfPCVcxPyjAwGn3pvsa1EGI7ES0AsBtACYA9AGboHQdjnrgvnykx5vNtOJ1/DSO7N0WVqEijw9Gc7kkCAIQQrwJ41YhjM6aWcO2B4GRZ0ZUiaSTb+StFOJ1/zeBo9MUzrhmTBXrS55Op/Uyal2x0CLrjJMGYLFxO+ltSzxkdgurM8tbkXy0yOgTdcZJgzIPVu5H+sfKw0SGwMGL7JFFcWobPNh7H9RJzzp5l5sWjiMKUxS8S1Gb7JPHttpOYvuIw/rvphNGhMJMIl26ncKLreZvffze2TxKOUQuO/xlTisL0kvPkuStGh2AofzkiPN9x/2yfJBjzpPSehJm6m07kXlZtX89/v1e1fTHr4yTBWBgoKTNPwrI6X9cIC3Zl6hqHWXCSYMwE7NiNYTX/N9+eLSxOEox52HSUS9Mz71KyLxodgu44STDmYduJ87ofkzuLzMPfe1Fcar93ipMEY4wxnzhJMBYknk/B7ICTBGNhgBMW0wonCRmPLmGBsnqNJxaacJ1M6YmTBGMAysoEPtt4PKDXmOnq/UpRidEh2I6ZJlNqiZOEzB5vN/Pl0BlrD2189+cjRocQNoSZsr8JcJJgDEBhse/aXUIIfLrhuKnXErh8nVsSauEc4c6Q5UsZM5v5Sb5LLmw9fg7/WHkY+0/n6xdQgPjEph6+1+SOWxKMwX9Loqi0DABwqdD9at3zxCyEdF8j7/L1CvsoLi3DB2uP4ZpG1YbTDarcaueumQ/XpqKopMzoMDTHSYKxSpDCS8u9mQWYvuIwXvBSRXXezgy8t/ooPlp/TO3wAFRMYHpJzVGv+qw/Zry6/2h9Kr7edtLoMDTHSYIxKFtDoLKL5hK5xeHt/oCjpXKtyPuVpwnPgYqU2rglAfhvgYYLThKMVcJxFWuXIY8AkHnhqtEhuNEzFwVyLDO2cNTGSYKxSkTIZwI7XTQLAeRfLcJH646hjNeq8MkOE+o4ScjC/61m/vhLAEq7m5Sw0pXn1EUH8O6qo9icmmd0KKZlpfczWJwkGIOyyZSe3U0Vv3a34UiO4rUpzHitfkW+t1JSFv4jeIJlgxzB8yQYq5TjnoTCM7njxDFu9k4AQPr0RPVjYpoJJGFzS4Ixm/D3t+7od/Y8efhKGmZsFWjl840nkHOx0OgwmIa4JSGz0x82C4zzarGSD4kNLior+HHPaZy+cE3z4+h5xW7H99EfbkkwVglfJ41ATlxqjYxadfAMftXhRnIgP1u41Y0KqLvJBimFWxKMKVTZPAklJ5dQTynjv94FQPv7HEJw61oJviehESKqS0QLiOgwER0ior5GxMFYIHamX1D0PDXPG2VlAu+vOWrqCrRldppAYkNGdTf9B8BKIUQnADcBOGRQHIwBCO6qWY9z48ajuXh/zTH8dfFB7Q8WpMNnLhkdgmGU1vWyMt27m4ioNoCBAMYBgBCiCIDhl0nh/1Yzf4KpZro97bwGkbgrlutBaVU91heznfv0bKwE0jIK9dd09Kz5E6wRLYk2AHIBzCaiPUQ0k4hqeD6JiMYTURIRJeXmKpuQxJgW7HC1yMptVDgBEgg9mQ5/f1NoO9CBEUkiCkBPAJ8KIXoAuAJgsueThBAzhBAJQoiEuLg4vWNkNldwrVjV/VV205vTkH965mlH602JUMOyQlksI5JEJoBMIcR2+esFkJIGY6bx1rLy22RqnqCs0iixc+uJ78O70z1JCCHOAMggoo7ypiEAUvSOgzF/CkvK7wEoPWn8e/VR1eM4ec74kt12mAvgipOEO6PmSTwL4FsiigFwAsAfDYqDMdX8evyc6vt8c7nUormi84Q1Oy9LGgg7tLgMGQIrhEiW7zd0E0L8VgihbAA6YwbQ4zxg5lPy+2uPeV23O1ydzldeZmTO9lMaRmIOXJZDZuY/UqY9s7//el+wul4h783Ix1/mV1y3mwFHLDCENVScJBhTmbcTuq/em1mb05Cac1nbgFRwzQZrOQfL0RV4vaQU76w8jKtF4VXLSnGSIKJqLjebGQtr5OOxEn5XuXPJIEII/H1pCkZ9tDnAIzAz+XBdKgDg222n8MmG4/hk/XGDI1KXoiRBRHcDSAawUv66OxEt0TAu3YX/7SdmVld0nk2thPn+HswXkcN1eSScY35FkcJ5FheuGF5oQhGlLYnXAPQGkA9IN54BxGsREGN2EcgAIiMG0ZhrgJOpgnET7O/p1SXmrcflSmmSKBFCFGgaCWMmsig5CyUBzLwNRiA1gsxyws4uuIb/rDlmdBim98OuTOxM91/bq9Ai93mUJokDRPQAgEgiak9EHwL4VcO4GNOXl5Pw0n3ZQe1K6VX/1hPqz6vQiiNJTfh2N/69Rv1Jg+Hmhfl7cd9nW40OQxVKk8SzALoAuA5gDoACAJM0ikkTZWUC7/58BLmX7DPem4WmpEwg7/J1/GtVYCfFnekX8M22k86vF+7OxDYvCaGkNIBqoybpkr9WrG3ryk4ydVj2VQ2VzrgmokgAS4QQQwG8on1I2tiWdg4frU/FoeyL+GJcL6PDYSbjqwDf5B/2BXXFP3XRAefj578vn2PgcynUgI+gL0eSMnucahvXLx5f/pru9zmO2emB9gimZF8MLiidVdqSEEKUArhKRHV0iEczpXK5xeslfCXElCvU8Mq5ssqwZnTojDVObGrpFR9rdAiGU1q7qRDAfiJaDam0NwBACDFRk6gY05m3G8Na1y8KaHSTztfwvrq3zHIDnelHaZJYJv8LO1zIjOkpOSMfc7afwgN9WrptX7I3y6CIrMKYji4lrT3PZ4Rbl5yiJCGE+Equ2NpB3nRECKHuqiw6sWITn2nP25WzFhU+t6edx/a083igT0vTz5Ngyni+j+F2hlE643oQgGMAPgbwCYCjRDRQu7DUUVJahrdXHEL+1SKfzXU7lPpl5hTIySQ5I9/n99Yfzgk5Fk/vrDziVg3Vrg1uJd18X287ics6l3LXk9Lupn8B+I0Q4ggAEFEHAHMB3KxVYGr4+eBZfL7xBHIvXcfoHs2NDodZjJm6Ii8V+j4J/fHLnaof78c9p1XfZ2jM81548+HaY6hXIwZA+HU3KZ0nEe1IEAAghDgKIFqbkNRTUiaNTCkuFdzNxEwnlCT0xtIUXDNhzSe7Kg5gzovVKG1JJBHRFwC+lr/+A4Bd2oTEmP5M1GhQZObmNNSrEYMJg9sZHUpYG9ChgaLnhXOvtdKWxFMADgKYCOA5SGtSP6lVUFow6zq910tK8cbSFFwqtOQ4gLDhK0lo+cfvLy99siEVxypZ0Ka0TGDGJn3KUm9P81+HKFzViFF2HW3Os4s6lLYkogD8RwjxHuCchV1Fs6hsZMGuTMzcnIZSIfDq3V2MDod50LKF4Wvf0uI1R/DphsoTwFvLD6scFWPulLYk1gKo5vJ1NQBr1A/HfhwzwQOp48PChf/3/DrXSXJh7mt11xbn55tOGBeIBpQmiapCCOcai/Lj6tqEpC2r9T0zZkcvj+hkdAhMpjRJXCGino4viCgBgDVKGMp8lxmQssYnG46jrIwzCHMXzjckzax9w1pGhwBAeftlUXIWvqqkEKBVKb0nMQnAfCLKgtRGbgpgjFZBGWVn+nn0aVPf6DCYSWh9yRBqqzac81esPOfAKsJ5CQK/LQki6kVEjYUQOwF0AjAPQAmkta7TdIhPNUr+ILkdYU+bjuZi5cEzFba/vjRF99XDtqTmYc72UwAqXytZ78/rtJ/0W24zMqLyFHgwqwCfbdRndJedVdaS+BzAUPlxXwAvQ1qAqDuAGQB+p1lkNsH3SIz38KwdXrdfKizBzvQLmh3X21v/fVKm8tfr/NmZvSVd3wNWIvGDzQCAJ29ra3Ak4a2yJBEphHAMkB4DYIYQ4gcAPxBRsqaRqcxX3zLXbmJG4QsE3yr+bso3pOZcwvL9FVt+WuDTg4IkQURRQogSAEMAjA/gtUwB/hCyYNmp1Ixr0hjz+Tacu1JkXDA2U9mJfi6AjUSUB2k00y8AQETtIK1zzRgLkp1O8oHyvHhy/U0VWWh1ycLiUlSNjjQ6jJD4vXEthHgTwAsAvgRwqyivSBYB6d6E5XATn4ULO32WrfqzOgYhWFmlXUZCiG1eth3VJhztcK8OM5tAT3zL9mW7fX3q/FUVozG3U+evYnHyafzljo66HjfUe5ZlPt7kkkpGrpmJLe4r+HubjV4zwKpXSCx0gb71E+bsdvvafGs+aOfx/yUBAEb3bGZwJIHxlWTWabBQlFaUzrhWHRFFEtEeIlqq9bEEzD8Hgm9gM2YfvloYZmRYkoBUcvyQgcdnzFBGt2LN6uG+rYwOQTXhcO1nSJIgouYAEgHM1OV4KH+ztp44h5Ssi3ocNiB8vmBMMrhTQ6NDYC6Makm8D+BFAD7v3hDReCJKIqKk3NzckA7mef4d9fHmkPanJu5mYsx+rHRRqHuSIKK7AOQIIfwufyqEmCGESBBCJMTFxakaQ3GpCLo18eWWNGw9fk7VeBhjyvxr1dEKF31m7rZzvQgsKxN4Y2kKMi9Ya1SaES2J/gBGElE6gO8A3E5E32h5QG8X68G2Jl77KQVj/1thVHDQTPz5Zsx0Vhw4g8vXS9y2mflvyPXcs/90AWZuTsPEuXsMiycYuicJIcQUIURzIUQ8gPsBrBNCPKjpMYEKmaJYXglulkmKlnG3k72UlJaZ+uTG1PHN9lPYfCwPQHm39+5T+fjSQmtPGDm6yRTOu9SA4fM008svqXlGhxA2zJxrU3Mu48EvtgNwP79sTzvv/QUmZOhkOiHEBgAbtD4OAYo+SWb+sLEwI7h2ky9GXqxtOKLNJLenvtmF47mXK3+iCdlixrUVcNcDY8GRblyrk1rGzd6pyn48rTigrLT5Y7e21uT4obBNkkjJNt/cCIDvRdjV8v3ZOBvGS16y4Iy8qanRIVRgmyTxxjJzTu7mFoQ9zd+lfAU65h//CWnL1jeuv9thnjK+3KJgTBLoSf/dVUcCmiux8Wguvt6aHuBR9GHGhGfrJDF54X6jQ2CMhejzjScqzJ3w55FZO/DXxQc1jCi82DpJhJsle7OwONk+5aNZeCIE3rL+2+KDKA5yjYZdJ8/jkw2pQb3WDjhJhJGJc/fgue+SjQ6DsZAFeq/uxz2nsfZQcMNX7/10K95ZeSSo19oBJwnGWJgIrUd/vLywEXPHSYIxFhZCHSm4KuWsOoGEGU4SLniAEWPmYNfRfmasaBvWSSLQ37f53h7GmFL896uNsE4SDqUqZefSMnU/hkUlZXh1CQ/FY8wVBdmMeH1pimXrI5lZWCcJx2dtm0qLBCWlq1u5cd1h7gNlTC3ZBYX401fWvvnctVkdo0OoIKyThNrde2o3Z7edKE86qTmX8frSFFP2STJmFY6/HyEE3l5+CIdMWrPNl6hI852SzReRBpS2Xit7mtr30lwXHvn1+Dl8sTkNmReuqXwUxuzn4rUSfL7pBO6fod4qknYV1knCiiMkrBgzY2aRfu4q4icvw6nz5etIf7j2GHZ66Sren1mgZ2iWFdZJQu3RTcHeUGOMKdevbf2Q9/HATKkFQQT8a/VR3PfZ1grPGRnkOvd2E9ZJolxgJ/eci4WY/MM+FJUEVwuGMRa8aBX65aMipL/5/KvFbttzXdbwEALIuVQY8rHCnU2SRGBeXXIQ3+3MwJpD7qOPuCHBmD4oxDuAkRHeT21vLXdfV+aNpeZcZ8ZMbJEk8i4rWwHM8bF0dFM9/e1unDp31efzWbmyMoFXFx/gcerMFHz9zf+4h6skB8oWSSIUk+btcT7mhoRvx3Mv46utJ7lIGlOF4PnTpsFJwoW3j+XuU/kQQuDY2UuY9lOKc/uvqXn4aN0xxfteuDsT85MyVIjSnBy/uwgL9cllF1zDSwv2GR0G08l7q49W2LZkb5YBkViLbda4DsXVolI8MmsHsgrKb3I9MHM7AOCZ29sr2sfz3+8FANyX0EL9AE3A0UVnoRyBqT8ewNrDwa1BwLQV6j0Jbz5Yq/yijpXjloSLA6elcdOpHv3qROYuHiaEQPzkZYYce8rCfej39lqUyVni6Fm+J8FYOAnrJBFov+Yby6SRDqk51jrRnc43bpb23B0ZyCootORMcTMnfsbMIqyTRDB8jWZSq6TSrpMX8PYK38Pu1JiwZ0T9J645xVh4CuskEcx56/nvkytsI5Bqoy3u/fRXfL7xhCr7cvD8Oa8Ulaq6f0Ux6H5EFs58THNgBgjrtyKYJOHtJdkF/rtSDmVfxN8WH6hwNb31+DmM/GhzQKOgJv+wD99sO6n4+Wbh+qPziBEWKiuNkgt3PLrJg7duk2k/pfhNOA/P2oHcS9cxYXA7NKpd1bl97H+l+jH7Aigk9suxPPxyLA8P3tJKedCmUP4Lmjh3D0be1NTAWJjVRRicIx66pRUa1a5ibBAmEdYtCTV51oBxFcznedA/11f6nD2nLmDaTwedieudlYexVaUFlEKRXXANf56XjOsl5d1ary1J8fMKY328PhVreJF7SzG6JdG7dazi4e3hLqyTRDD95N5eQwQUlfou9uf4PAfSvZWuoNzHvZ/+itlb0uFYNfWTDcedrRMjvbbkIH7ccxrrXeYYnLlo3kJp//z5CP7EM8EtJdLgpgT3dpXTPUkQUQsiWk9Eh4joIBE9p9WxghlxE8wy1o6JP2qWEsg4f9UZy9RF+51zONTa96Tv9gRd5dYR1/HcKwG97sDpArzy435NRkItTj6NLzanOb++VlSKZ+fu8Vrl858/H8bmY3luLSEWuJeGd9Js30a3JLSYzGdVRtyTKAHwghBiNxHVArCLiFYLIUzbX6H046Lmue+xr3Y6H8/dkYHVKnaXTF10ABuP5mJUj2YY3LFhwK/fm5EPQLpCD8TDs3bg/JUiPD+sA+rXVLe/97nvkgEAj93aGgCwZO9p/LQ3C9WiK14Hfbz+OD5ef1zV4zN11K4qnZKMbkkMu6GR83GTOlWRXaBtS3nikPa4tV0DTY8RLN1bEkKIbCHEbvnxJQCHADTT5FhBvajiq9YfyfX59Oe/T3bONtZyGOi1AIa17kiT7lvkXrqOCXN248r1Euf3dp+6gI1Hff88SgR7lef4PU1fcTig183ekoYf92QGdUwm6dioltEhKDLn8VsAGJ8kYqLKT41jemlfSuf5YR3Qu3Ws5scJhqH3JIgoHkAPANu9fG88ESURUVJubmgntUAEeqJfuPs0cuSFTNTsRvFs7gYy9+HRL6X+9/dWH8Wyfdlu5ZF/72WFrkAF+/fruPk/f1dgJ/xpP6Xgz/P2BndQBkDqY7/v5uZGh6FYw1pV0KmxORLbnwa0MToEQxmWJIioJoAfAEwSQlz0/L4QYoYQIkEIkRAXFxfUMTLPB74WRCDDVT255oi5O04FvZ/KfL01XdHzHDFMXXQAS/dl4fukDJQEc9PFQzCzwvVc5e/NZSleR4Et3ZeFTzak6haH2cQ3qKHq/rS8bUBEWDlpoHYHCEDNKvaeKWDIT09E0ZASxLdCiIVaHeeDdcadEKYs3B/S6/39Af518UE81Dc+oP09M2dPxY06TpPedkK/obv//SXN63avvwOb8EzsLWKrIeN8aPW2Hu7bKuCuQ2Y9RoxuIgBfADgkhHhP7+NbxeEzlxQ9L/fSdTzx9S7Vjvv60hRsSc1z21ZWJvB/8/eGNMIqSqc+Zs9quN8n8b0MAPBcNvqJgW1D3mf1GPWuMW9oUlu1fTF1GdHd1B/AQwBuJ6Jk+d8IA+IICx+tO4aU7Aq9dcp4OW9/sTkNf5jpfovodP41LNiVGVIyijB6Cq3NvTyis9vXvw/TdU3U8ECflkaHYCpGjG7aLIQgIUQ3IUR3+d9yvePQgqM4YJkK/f6VmTBnN15csNfnvI5g1peYstB9lbbC4lI8/e0uZMj3dk7nX8OFK0WK9uW4if/B2mNYui+rwoz1TzccxyIv6w0XFpdiwre7kXnhKkrLBJ6fl+z83qI9p/HpBmno6sXCYjzxdRLyLl/n+Q4+pE9PdD6OrRHjNirNdfQOc9c2rqbRIZiKve/IqGxn+gUAcI520tKyfdkAgLtVrJE0d4f78qqbj+Vh+f4zbgsJzdrivb/fU2mZQFQkOZeM7Nmyrtv3/7FS6sv+bQ/30c8bjuRg2f5sFJeWYfKdnbDQJZFMkhPGU4PaYt6ODPx88Cxa1KuOW9ubc3y5mXRoWAstY6s7f++MKcVJQmUv/7gfP+lYBTWkThwBrE45i8f/l4Tm9ap5+7b0fxBDez1vlO4+le/1eUv2ZiE97woiIwjzdmY4k8mqlLNYpWAC4aqUs5i5WVnisrOICFL1HoLaoiLN0x1pnkjMwbyfGouas127oa9aeFyuaeRvZbmgamAJASV/bhPnuo84OhXgsOVAn293Xz3a29l9aCYfP9ATX/2azjewTYiThMWFsnZDWp7v2kvJGfl4Qb7HcsKlRpPSq6zOf1uJDX8ZHHRs/kz76SAOZgV5s97mbusQ3JwjrbWIrY6pd91gdBjMC757ZWN/X+q7XNa42TtwsbDE5/crU1wq8I5G/d+zt6RjR9p5TfbNlHn9tzfqcpybW9VD49pV8c7vuulyPEC/CrCv3W2NpMhJgnnla/2MMiGNclJicTKvUBeuHtJpUawfnuqHbS8P0XXIrl4VaMf1b63LcULFSYIFJPOC+fqzzaJJnaqVP0lnn/yhJ/5zf3ev35s+uitmj+vl/DquVhUseLJvpft0PYf+79HezsqtC57si6XP3hpSvGbgraDfpKH2XYCI70mwgCzi1oFPPVvVcw5NNosRXZv4/N79vd0njd15Y2MkxMeianQECot919pyrSg7sEMc9r12R+iBmkjV6MgK2/q1bYD31yhfqz6ccJJgzOb+ckdHrDiQjReGdQRQ+boo88b7b21ERhBKdZhQamVTE6UZ8HMe74PUnMuVPNtY3N3EmEqMXk0tWBMGt8PSZwegTvVot+2eXUfTR3dF+vTECs/z9IhcfDKulroLSxnJUVVGrZ+paV1pXlK/tg3wcIDFOvXGSYIxFTw+oHWFWeVGULOQYpu4GnhqUFv0kRfDUZoDJw1rj0f6tsKqSQPxcF/1b3D/sX+81+0fPdBD8T6eGuRe4NDXPh16tqyHxwe0xuIJ/Z3bJt8Z/PKtGqzgqxlOEox5CKZB8EriDYpaEh+OVX4iC8afh3UIeR8xcslYAuGl4Z3QJk5ah0JpS6l21WhMG3Uj6tWIwd9HqT9U9uZW9bxuv6ube4kafwMJOjQqr890e6eGGFBJaZeICMIriTc4WwAA8ORtwVfStVKjk5MEYx4a11Y2Sslz5TQlf/g3NK04o7h3fKyq8wAWT+iPl0cEf5X7w9P98PywDqgWI93AfWl4J4zrF49R3TVZZVgxR4vmji6N8f6Y7vjvwwl4fIA0jHS0XAMsxrMmug+JXcsTyi1tYjGwfRzqVIvGu/fdpFq8b4/u6nX7o/1bu62hbXZ845oxD77O9SueG4Bn5uzGcXkGeodGtRSv++HP90/2RWmZwIsL9lX+5EpERRBualEXN7WoG/Q+OjSqhQ4uI5jqVo/BayO7hBxbqOY9UX7D3FEYctgNjfBKYvmktIEdGmDNoRwAQOM6VVFwrRhXvSz9GxMVgfED22DGphOIiohAVGQE9r76G1XjHdu7pdfFx/5mkUl0DmHdkrjzxsZGh8AsyNfyrJ0a18JfXUpHjB8orX3cqLbym5m++qIjIwgJcjdKfP3qivfnqmVsddPfBNWa4/fbvF41vHVPVyx8up/ze/+UW2uO+lDP3t4O4/rFq7J+RN1KbuYD0n2rf49Rr6Wil7BOEo1NOLmJWUtTl88QESG2RgwA4MZmtVGnmnRicKyB7EgtD97i+6Tjr6LuhMHtAChfi7p3fKzb16+NvMHZRWRXjmqyfx/VBZ2b1EanxuXde/cltED69EQsf24AAKBW1Wi8NrKL13kRSjnGCcRWj6n0ua8k3oB7ejQP+lhGCevuJiuNIGDmMn10V0xeuB+lHh8icumMal6vGibe3g6je7r/4Qsh1Tbq0LAm0vKuYLJLl4Pr3qYmdnareioCrLcrIPCPe7vipR9CW089nLx5T1e0qFcdA9uXFzJ87/c3oZ6Ck7hSMx9OwJUiqa6ZYzrI5w/djIUui2I57kfMf7Iv7vtsq2rHNkJYJwme0MOCdXvnhgCAqAj3xrajJyo6MgJEhOd/09H5vSj5pmlUBDlrG/VpU98tSTjc2Kw2/jSgjddjKx34EhURgTG9WmLtoRysSjnrs5vM7DxbRKFoULNKhWqynkk8VEO93HRuEVsdLw3v5EwSY+XZ7L1U/NmMEtZJoqPH6JNQje7RzG2lNBa+4mpWwaSh7XH3TU1x5XqJs+psl6a18dSgtnjQS4G70T2b4ejZS5g01Pcw1PYNa2LC4La4v1dg/eCf/qEnYmvE4MN1qTiUfRHDb2yMZ2+X6gm9eU9XtKrvfvVsJZUNP7UCf/NTXhnRGbe0qa9jNOoK6yTx4C2tMHXRAdX293JiZ9WTxE3N62BvZoGq+2ShIyK3k3235nWd218a7n14aZWoSLx6t/9RQESEv9zh/fWOeQhRXoZx3inXYOrj5WQTV6uK2wgfK/jdzc2xYFcmACA6DNbbjvSTJB4f6L3FaBVhnSTUVqZB91WtqpWPimD2MKB9HMYPbIPHB7TBhMHtsPvkBVSNjnROZrOqeeNvwc708/h620mcvSit/z60cyPEREWgsKgU4/rFGxugCqza1acEJ4kAaHGLI9pEa/syY0VGEF4eIRV+i6tVBd1DmOtgJn3a1EefNvXxzO3tET95GQDpc//WPd4nmzFzsX47TyO1qlbMn6VCOIdAquUtH7MygzG6pzEzYt8f092Q46rl6UFtnWPoASBC5b+K5+VSGa+PMn5CmlkM6tjQ6BBUsfTZW50VXcOV7ZLEkE7KPpzexk4LIZDopz5/MKpEqTeuXe3YlHLMfrWqF4d3wtDO5SNW1K7mOnFIe6RPT8RDNp/o5spfH76V3Nisjs9RauHCVkmiU+NaePverhUqPnrOghzauSHmPt7Hbdukoe3RrG41/N8d5UMeB3UMbTSJr9ouSs146Ga3r6uFMCkoWPfKwwu/fqy37sfWSnicvsxp9rhemhc5NNL/Hu1t+Za1J1sliRub1UHDWlXx6t1dnF0z//xdtwqzIJ+4rS3aNXQfPjtpaAcQkXOWLQB8+cfQTox339S08if58ZsujTHFtVwxSTcJ9dQiVqqKOcCiwy8dXBsPVl0XwgoGd2oY8ufezAZ2iLN8y9qTbZLEEwPbuPUdOkYqOZq9Hz/QE2+P7oqnB7XFzS2lGjrfPNan4o48fP1Yb5+li4Hy+j6e/nJHR9SsEoWyIKaFTxvZxXnl/ojLyBACIcJPM95bLINDbA25+tGlTo6egjmnTxvZBff3aoH58prOdavH4B75j5tzBGPlbJMkpozojLouU/Oj5bHojv8TuzXB2N4t8eLwTs4T7a0KJvkMaB+H6X66jVzXA3Zo17Cms07P9RLfawkDQHWXWjxVoiKQPj0Rj/SLd165V42OxMAO0uPICPI5qWdU96Z4eURnpE9PxDSXip7eruqm+FhMpU61aBx+fbjbtmiXMf09WtZDqyCL04Ui7e1EHH9rRIXtVaN9f7wf6ReP6fd2c5sR63hPQqnlw1i4CfshsJ/+oafXyUmvJHZGbM2YSivFLp7QH/sy8/0+p13Dmpg4pD2O51xGxoWrKBMCB05fBCBNFHppeCc0ql0FlwpLkH+12HnFCkgF5CYNbY/RPZpj6uID2HQ0Fwuf7oeYyAjsSDuPAe0bYNi/NwEAfvJYTtLh3d91w1db051VRJ+9vR1+n9ACX287idgaMViTchavuUzyGtOrBV5dchCAVGeoXcOaSM25jAiS7jH8aUAbLNmbhYNZ0s+w7oXbsCrlLIZ2boiq0ZF4aXgnDGjfAIuTT+PR/q3dYvnqj73x+abjmLsjw+/vzFWf1rH4bY9mXssq+/Jo/9aYtSXN+XVkBGHKnZ3w9orDmHJnJ+Rdvo5rxaX4ZtspVI2OwBeP9EJyRj7++fMRn/tsG1cDzw1pj/sSrFeEjTGtkL+qlGaRkJAgkpKSjA7DyTHWO316os/nPDNnN5buy8YHY3tgZIh9sEqOF6gpC/dh7o4MvD+mO778NR3JGflY8dwAdHYpOBfKceMnL0NMVASWPNMfw9//BTe1qIu9GfmoHhOJEV2bOGfbAsBzQ9rjz8M6OI9XGUc8lcX31vJDmLHpBF4f1cU5skiL3yVjZkVEu4QQCaHsI+xbElpYNvFWbD9xXrfjLX32ViSlq3u8yXd2Ru2q0Ujs1gS9W8fiu50ZFVZa+9tdN6BPm+AKlL0yojMGdohDh0Y18dyQ9vh9rxZYti8Lgzs2RFytKqhfMwa1qkRh24nzeOI26V7Jgif7YsORXHy0PhUA8OLwjii4WowOjWrhhfl7AQCfPVg+ouute7qiUxPf9bkmDpFqG/2+VwvntqmJndG/nfVrBTGmF0NaEkQ0HMB/AEQCmCmEmO7v+WZrSSjxwvd78cPuTHz8QE8kdjNm/oJV8dU+Y+qwZEuCiCIBfAxgGIBMADuJaIkQIkXvWLT017s6I65WFdzRxTpr2ZrFiucG4Nfj54wOgzEGY7qbegNIFUKcAAAi+g7AKABhlSTqVo/BZB+jhJh/nZvUdrs3whgzjhFDYJsBcB36kilvc0NE44koiYiScnNzdQuOMcZYOSOShLeB/BVujAghZgghEoQQCXFx1p7NyxhjVmVEksgE0MLl6+YAsgyIgzHGWCWMSBI7AbQnotZEFAPgfgBLDIiDMcZYJXS/cS2EKCGiZwD8DGkI7CwhxEG942CMMVY5QybTCSGWA1huxLEZY4wpZ5sCf4wxxgLHSYIxxphPlijwR0S5AE4G+fIGAPJUDEcPVowZsGbcVowZsGbcHLN+HHG3EkKENIfAEkkiFESUFGrtEr1ZMWbAmnFbMWbAmnFzzPpRM27ubmKMMeYTJwnGGGM+2SFJzDA6gCBYMWbAmnFbMWbAmnFzzPpRLe6wvyfBGGMseHZoSTDGGAsSJwnGGGM+hXWSIKLhRHSEiFKJaLLBscwiohwiOuCyLZaIVhPRMfn/ei7fmyLHfYSI7nDZfjMR7Ze/9wEReSu9rlbMLYhoPREdIqKDRPSc2eMmoqpEtIOI9soxTzN7zC7HiySiPUS01EIxp8vHSyaiJCvETUR1iWgBER2WP9t9LRBzR/l37Ph3kYgm6RK3ECIs/0EqHngcQBsAMQD2ArjBwHgGAugJ4IDLtncATJYfTwbwD/nxDXK8VQC0ln+OSPl7OwD0hbQuxwoAd2oYcxMAPeXHtQAclWMzbdzy/mvKj6MBbAdwi5ljdon9eQBzACy1wudDPl46gAYe20wdN4CvAPxJfhwDoK7ZY/aIPxLAGQCt9Ihb8x/IqH/yL+Fnl6+nAJhicEzxcE8SRwA0kR83AXDEW6yQKub2lZ9z2GX7WACf6xj/Ykhrk1sibgDVAewG0MfsMUNaV2UtgNtRniRMHbN8jHRUTBKmjRtAbQBpkAftWCFmLz/DbwBs0SvucO5uUrRMqsEaCSGyAUD+v6G83VfszeTHnts1R0TxAHpAujI3ddxyt00ygBwAq4UQpo8ZwPsAXgRQ5rLN7DED0qqSq4hoFxGNl7eZOe42AHIBzJa79mYSUQ2Tx+zpfgBz5ceaxx3OSULRMqkm5St2Q34mIqoJ4AcAk4QQF/091cs23eMWQpQKIbpDujrvTUQ3+nm64TET0V0AcoQQu5S+xMs2oz4f/YUQPQHcCWACEQ3081wzxB0Fqdv3UyFEDwBXIHXT+GKGmJ1IWqhtJID5lT3Vy7ag4g7nJGGFZVLPElETAJD/z5G3+4o9U37suV0zRBQNKUF8K4RYaJW4AUAIkQ9gA4DhMHfM/QGMJKJ0AN8BuJ2IvjF5zAAAIUSW/H8OgB8B9DZ53JkAMuXWJQAsgJQ0zByzqzsB7BZCnJW/1jzucE4SVlgmdQmAR+THj0Dq83dsv5+IqhBRawDtAeyQm5OXiOgWeUTCwy6vUZ18jC8AHBJCvGeFuIkojojqyo+rARgK4LCZYxZCTBFCNBdCxEP6nK4TQjxo5pgBgIhqEFEtx2NIfeUHzBy3EOIMgAwi6ihvGgIgxcwxexiL8q4mR3zaxq3HjRaj/gEYAWlEznEArxgcy1wA2QCKIWXzxwDUh3Sz8pj8f6zL81+R4z4Cl9EHABIg/SEeB/ARPG7AqRzzrZCaovsAJMv/Rpg5bgDdAOyRYz4A4G/ydtPG7BH/IJTfuDZ1zJD69/fK/w46/sYsEHd3AEnyZ2QRgHpmj1k+XnUA5wDUcdmmedxcloMxxphP4dzdxBhjLEScJBhjjPnESYIxxphPnCQYY4z5xEmCMcaYT5wkWFgjolKP6pl+qwET0ZNE9LAKx00nogZBvO4OInqNiOoR0fJQ42AsVFFGB8CYxq4JqUSHIkKIzzSMRYkBANZDqhq8xeBYGOMkwexJLoExD8BgedMDQohUInoNwGUhxLtENBHAkwBKAKQIIe4nolgAsyBNJLsKYLwQYh8R1Yc0YTIOUilmcjnWgwAmQipLvR3A00KIUo94xkCq3NkGwCgAjQBcJKI+QoiRWvwOGFOCu5tYuKvm0d00xuV7F4UQvSHNOn3fy2snA+ghhOgGKVkAwDQAe+RtLwP4n7z9VQCbhVQ0bgmAlgBARJ0BjIFUCK87gFIAf/A8kBBiHsrXG+kKaUZsD04QzGjckmDhzl9301yX///t5fv7AHxLRIsglW8ApFIl9wKAEGIdEdUnojqQuodGy9uXEdEF+flDANwMYKe8AFg1lBdh89QeUqkEAKguhLhU2Q/HmNY4STA7Ez4eOyRCOvmPBPBXIuoC/6WWve2DAHwlhJjiLxCSlv5sACCKiFIANJHXxHhWCPGL35+CMQ1xdxOzszEu/291/QYRRQBoIYRYD2kxoLoAagLYBLm7iIgGAcgT0hobrtvvhFQ0DpCKrv2OiBrK34slolaegQghEgAsg3Q/4h1IxfK6c4JgRuOWBAt31eQrcoeVQgjHMNgqRLQd0sXSWI/XRQL4Ru5KIgD/FkLkyze2ZxPRPkg3rh1lmqcBmEtEuwFsBHAKAIQQKUQ0FdLqbRGQqgBPAHDSS6w9Id3gfhrAe16+z5juuAossyV5dFOCECLP6FgYMzPubmKMMeYTtyQYY4z5xC0JxhhjPnGSYIwx5hMnCcYYYz5xkmCMMeYTJwnGGGM+/T/L2Gn3hHrqlwAAAABJRU5ErkJggg==\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_scores():\n",
    "    scores = np.vstack(ddpg_agents_tester.scores)\n",
    "    scores = np.max(scores, 1)\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    plt.plot(np.arange(1, len(scores) + 1), scores)\n",
    "    plt.ylabel('Score')\n",
    "    plt.xlabel('Episode #')\n",
    "    plt.show()\n",
    "\n",
    "plot_scores()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Watch a Smart Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "test_env = ParallelUnityEnvironment(num_envs=NUM_ENVS, seeds=list(range(NUM_ENVS)),\n",
    "                                     file_name=ENV_FILE_NAME, no_graphics=False)\n",
    "test_env.set_timescale(1.0)\n",
    "test_env.set_display_size(width=DISPLAY_SIZE[0], height=DISPLAY_SIZE[1])\n",
    "ddpg_agents_tester.env = test_env\n",
    "ddpg_agents.load_checkpoint(filename=CHECKPOINT_FILENAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score from environment 0, episode 0: 20.00\n",
      "Score from environment 1, episode 1: 20.00\n",
      "Score from environment 2, episode 2: 20.00\n",
      "Score from environment 3, episode 3: 20.00\n",
      "Score from environment 0, episode 4: 20.00\n",
      "Score from environment 1, episode 5: 20.00\n",
      "Score from environment 2, episode 6: 20.00\n",
      "Score from environment 3, episode 7: 20.00\n",
      "Score from environment 0, episode 8: 20.00\n",
      "Score from environment 1, episode 9: 20.00\n",
      "Score from environment 2, episode 10: 20.00\n",
      "Score from environment 3, episode 11: 20.00\n"
     ]
    }
   ],
   "source": [
    "ddpg_agents_tester.test_agents(n_episodes=10, max_t=200)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "When finished, you can close the environments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "train_env.close()\n",
    "test_env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "rl39",
   "language": "python",
   "display_name": "rl39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}