Scripts ux (#53)

* scripts module, readme updates, setup updates

* tmp log default

* impala exec command readme

* minor readme tweaks

* remove requirements file

* eval bugfix

* minor bugfix

* small bugfix

* small bugfix

* mts for resume_local

Author: jtatusko

README.md

@@ -23,50 +23,55 @@ Execution Modes
 
 Environments
 * OpenAI Gym
-* StarCraft 2 (alpha)
+* StarCraft 2 (alpha, impala mode does not work with SC2 yet)
 
 We designed this library to be flexible and extensible. Plugging in novel research ideas should be doable.
 
 ## Major Dependencies
 * gym
-* PyTorch 0.4.x (excluding 0.4.1 due to the [unbind bug](https://github.com/pytorch/pytorch/pull/9995))
+* PyTorch 0.4.x (excluding 0.4.1 due to an [unbind bug](https://github.com/pytorch/pytorch/pull/9995))
 * Python 3.5+
 
 ## Installation
 * Follow instructions for [PyTorch](https://pytorch.org/)  
 * (Optional) Follow instructions for [StarCraft 2](https://github.com/Blizzard/s2client-proto#downloads)
-* More optional dependencies in requirements.txt
 
 ```
 # Remove mpi, sc2, profiler if you don't plan on using these features:
 pip install adept[mpi,sc2,profiler]
 ```
 
 ## Performance
-TODO
+* Used to win a [Doom competition](http://vizdoom.cs.put.edu.pl/competition-cig-2018/competition-results) (Ben Bell / Marv2in)
+* ~2500 training frames per second single-GPU performance on a Dell XPS 15" laptop (Geforce 1050Ti)
+* Will post Atari/SC2 baseline scores here at some point
 
 ## Examples
 If you write your own scripts, you can provide your own agents or networks, but we have some presets you can run out of the box.
-If you pip installed, these scripts are on your classpath and can be run with the commands below.
-If you cloned the repo, put a python in front of each command.
+Logs go to `/tmp/adept_logs/` by default.
+The log directory contains the tensorboard file, saved models, and other metadata.
 
 ```
 # Local Mode (A2C)
 # We recommend 4GB+ GPU memory, 8GB+ RAM, 4+ Cores
-local.py --env-id BeamRiderNoFrameskip-v4 --agent ActorCritic --vision-network FourConv --network-body LSTM
+python -m adept.scripts.local --env-id BeamRiderNoFrameskip-v4
 
-# Towered Mode (A3C Variant)
-# We recommend 2x+ GPUs, 8GB+ GPU memory, 32GB+ RAM, 4+ Cores
-towered.py --env-id BeamRiderNoFrameskip-v4 --agent ActorCritic --vision-network FourConv --network-body LSTM
+# Towered Mode (A3C Variant, requires mpi4py)
+# We recommend 2+ GPUs, 8GB+ GPU memory, 32GB+ RAM, 4+ Cores
+python -m adept.scripts.towered --env-id BeamRiderNoFrameskip-v4
 
 # IMPALA (requires mpi4py and is resource intensive)
-# We recommend 2x+ GPUs, 8GB+ GPU memory, 32GB+ RAM, 4+ Cores
-mpirun -np 3 -H localhost:3 python -m mpi4py `which impala.py` -n 8
+# We recommend 2+ GPUs, 8GB+ GPU memory, 32GB+ RAM, 4+ Cores
+mpiexec -n 3 python -m adept.scripts.impala --env-id BeamRiderNoFrameskip-v4
+
+# StarCraft 2 (IMPALA not supported yet)
+# Warning: much more resource intensive than Atari
+python -m adept.scripts.local --env-id CollectMineralShards
 
 # To see a full list of options:
-local.py -h
-towered.py -h
-impala.py -h
+python -m adept.scripts.local -h
+python -m adept.scripts.towered -h
+python -m adept.scripts.impala -h
 ```
 
 ## API Reference
@@ -77,7 +82,8 @@ Currently only ActorCritic is supported. Other agents, such as DQN or ACER may b
 ### Containers
 Containers hold all of the application state. Each subprocess gets a container in Towered and IMPALA modes.
 ### Environments
-Environments work using the OpenAI Gym wrappers.
+Environments run in subprocesses and send their observation, rewards, terminals, and infos to the host process.
+They work pretty much the same way as OpenAI's code.
 ### Experience Caches
 An Experience Cache is a Rollout or Experience Replay that is written to after stepping and read before learning.
 ### Modules
@@ -90,5 +96,5 @@ The Body network operates on the flattened embedding and would typically be an L
 The Head depends on the Environment and Agent and is created accordingly.
 
 ## Acknowledgements
-We borrow pieces of OpenAI's (gym)[https://github.com/openai/gym] and (baselines)[https://github.com/openai/baselines] code.
+We borrow pieces of OpenAI's [gym](https://github.com/openai/gym) and [baselines](https://github.com/openai/baselines) code.
 We indicate where this is done.

adept/containers/evaluation.py

@@ -86,7 +86,7 @@ def run(self, nb_episode):
             next_obs, rewards, terminals, infos = self.environment.step(actions)
 
             self.agent.reset_internals(terminals)
-            episode_rewards = self.update_buffers(rewards, terminals, infos)
+            episode_rewards, _ = self.update_buffers(rewards, terminals, infos)
             for reward in episode_rewards:
                 self._episode_count += 1
                 results.append(reward)

adept/scripts/__init__.py

@@ -0,0 +1,16 @@
+"""
+Copyright (C) 2018 Heron Systems, Inc.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""

scripts/benchmark_atari.py adept/scripts/benchmark_atari.py

@@ -38,11 +38,12 @@ def main(args):
     network_file = args.network_file
     optimizer_file = args.optimizer_file
     args_file_path = args.args_file
+    mts = args.max_train_steps
     with open(args.args_file, 'r') as args_file:
         args = dotdict(json.load(args_file))
 
     print_ascii_logo()
-    log_id = make_log_id(args.tag, args.mode_name, args.agent, args.network)
+    log_id = make_log_id(args.tag, args.mode_name, args.agent, args.vision_network + args.network_body)
     log_id_dir = os.path.join(args.log_dir, args.env_id, log_id)
 
     os.makedirs(log_id_dir)
@@ -59,7 +60,7 @@ def main(args):
 
     # construct network
     torch.manual_seed(args.seed)
-    network_head_shapes = get_head_shapes(env.action_space, env.engine, args)
+    network_head_shapes = get_head_shapes(env.action_space, env.engine, args.agent)
     network = make_network(env.observation_space, network_head_shapes, args)
     network.load_state_dict(torch.load(network_file))
 
@@ -76,9 +77,19 @@ def make_optimizer(params):
             opt.load_state_dict(torch.load(optimizer_file))
         return opt
 
-    container = Local(agent, env, device, make_optimizer, args.epoch_len, args.nb_env, logger, summary_writer, saver)
+    container = Local(
+        agent,
+        env,
+        make_optimizer,
+        args.epoch_len,
+        args.nb_env,
+        logger,
+        summary_writer,
+        args.summary_frequency,
+        saver
+    )
     try:
-        container.run(args.max_train_steps + initial_count, initial_count)
+        container.run(mts + initial_count, initial_count)
     finally:
         env.close()
 
@@ -99,6 +110,10 @@ def make_optimizer(params):
         '--optimizer-file', default=None,
         help='path to args file (.../logs/<env-id>/<log-id>/<epoch>/optimizer.pth)'
     )
+    parser.add_argument(
+        '-mts', '--max-train-steps', type=int, default=10e6, metavar='MTS',
+        help='number of steps to train for (default: 10e6)'
+    )
     args = parser.parse_args()
     args.mode_name = 'Local'
     main(args)

scripts/evaluation.py adept/scripts/evaluation.py

@@ -117,7 +117,6 @@ def get_head_shapes(action_space, engine, agent_name):
 
 
 def add_base_args(parser):
-    root_dir = os.path.abspath(os.pardir)
     """
     Common Arguments
     """
@@ -146,8 +145,8 @@ def add_base_args(parser):
         help='environment to train on (default: PongNoFrameskip-v4)'
     )
     parser.add_argument(
-        '--log-dir', default=os.path.join(root_dir, 'logs/'),
-        help='folder to save logs. (default: adept/logs)'
+        '--log-dir', default='/tmp/adept_logs/',
+        help='folder to save logs. (default: /tmp/adept_logs)'
     )
     parser.add_argument(
         '-mts', '--max-train-steps', type=int, default=10e6, metavar='MTS',

scripts/impala.py adept/scripts/impala.py

@@ -13,16 +13,6 @@
     license='GNU',
     python_requires='>=3.5.0',
     packages=find_packages(),
-    scripts=[
-        'scripts/benchmark_atari.py',
-        'scripts/evaluation.py',
-        'scripts/impala.py',
-        'scripts/local.py',
-        'scripts/render.py',
-        'scripts/replay_gen.py',
-        'scripts/resume_local.py',
-        'scripts/towered.py'
-    ],
     install_requires=[
         'numpy>=1.14',
         'gym[atari]>=0.10',