Merge pull request #884 from Unity-Technologies/release-v0.4

Release v0.4

Author: xiaomaogy

.gitattributes

@@ -2,4 +2,6 @@
 *.unity binary
 *.prefab binary
 *.meta binary
+*/CommunicatorObjects/* binary
+*/communicator_objects/* binary
 *.md text

.gitignore

@@ -6,6 +6,8 @@
 /unity-environment/[Pp]ackages/
 /unity-environment/[Uu]nity[Pp]ackage[Mm]anager/
 /unity-environment/Assets/AssetStoreTools*
+/unity-environment/Assets/Plugins*
+/unity-environment/Assets/Gizmos*
 
 # Tensorflow Model Info
 /models

Dockerfile

@@ -129,4 +129,7 @@ RUN pip install --trusted-host pypi.python.org -r requirements.txt
 WORKDIR /execute
 COPY python /execute/python
 
+# port 5005 is the port used in in Editor training.
+EXPOSE 5005
+
 ENTRYPOINT ["python", "python/learn.py"]

README.md

@@ -29,14 +29,17 @@ to the wider research and game developer communities.
 * Visualizing network outputs within the environment
 * Simplified set-up with Docker
 
-## Documentation and References
+## Documentation
 
-**For more information, in addition to installation and usage
-instructions, see our [documentation home](docs/Readme.md).** If you have
-used a version of ML-Agents prior to v0.3, we strongly recommend 
-our [guide on migrating to v0.3](docs/Migrating-v0.3.md).
+* For more information, in addition to installation and usage
+instructions, see our [documentation home](docs/Readme.md).
+* If you have
+used a version of ML-Agents prior to v0.4, we strongly recommend 
+our [guide on migrating from earlier versions](docs/Migrating.md).
 
-We have also published a series of blog posts that are relevant for ML-Agents:
+## References
+
+We have published a series of blog posts that are relevant for ML-Agents:
 - Overviewing reinforcement learning concepts
 ([multi-armed bandit](https://blogs.unity3d.com/2017/06/26/unity-ai-themed-blog-entries/)
 and [Q-learning](https://blogs.unity3d.com/2017/08/22/unity-ai-reinforcement-learning-with-q-learning/))

docs/Basic-Guide.md

@@ -0,0 +1,105 @@
+# Basic Guide
+
+This guide will show you how to use a pretrained model in an example Unity environment, and show you how to train the model yourself.
+
+If you are not familiar with the [Unity Engine](https://unity3d.com/unity),
+we highly recommend the [Roll-a-ball tutorial](https://unity3d.com/learn/tutorials/s/roll-ball-tutorial) to learn all the basic concepts of Unity. 
+
+## Setting up ML-Agents within Unity
+
+In order to use ML-Agents within Unity, you need to change some Unity settings first. Also [TensorFlowSharp plugin](https://s3.amazonaws.com/unity-ml-agents/0.4/TFSharpPlugin.unitypackage) is needed for you to use pretrained model within Unity, which is based on the [TensorFlowSharp repo](https://github.com/migueldeicaza/TensorFlowSharp). 
+
+1. Launch Unity
+2. On the Projects dialog, choose the **Open** option at the top of the window.
+3. Using the file dialog that opens, locate the `unity-environment` folder within the ML-Agents project and click **Open**.
+4. Go to **Edit** > **Project Settings** > **Player**
+5. For **each** of the platforms you target 
+(**PC, Mac and Linux Standalone**, **iOS** or **Android**):
+    1. Option the **Other Settings** section.
+    2. Select **Scripting Runtime Version** to 
+    **Experimental (.NET 4.6 Equivalent)**
+    3. In **Scripting Defined Symbols**, add the flag `ENABLE_TENSORFLOW`. 
+    After typing in the flag name, press Enter.
+6. Go to **File** > **Save Project**
+
+![Project Settings](images/project-settings.png)
+
+[Download](https://s3.amazonaws.com/unity-ml-agents/0.4/TFSharpPlugin.unitypackage) the TensorFlowSharp plugin. Then import it into Unity by double clicking the downloaded file.  You can check if it was successfully imported by checking the TensorFlow files in the Project window under **Assets** > **ML-Agents** > **Plugins** > **Computer**. 
+
+**Note**: If you don't see anything under **Assets**, drag the `ml-agents/unity-environment/Assets/ML-Agents` folder under **Assets** within Project window.
+
+![Imported TensorFlowsharp](images/imported-tensorflowsharp.png)
+
+## Running a Pre-trained Model
+
+1. In the **Project** window, go to `Assets/ML-Agents/Examples/3DBall` folder and open the `3DBall` scene file. 
+2. In the **Hierarchy** window, select the **Ball3DBrain** child under the **Ball3DAcademy** GameObject to view its properties in the Inspector window.
+3. On the **Ball3DBrain** object's **Brain** component, change the **Brain Type** to **Internal**.
+4. In the **Project** window, locate the `Assets/ML-Agents/Examples/3DBall/TFModels` folder.
+5. Drag the `3DBall` model file from the `TFModels` folder to the **Graph Model** field of the **Ball3DBrain** object's **Brain** component.
+5. Click the **Play** button and you will see the platforms balance the balls using the pretrained model.
+
+![Running a pretrained model](images/running-a-pretrained-model.gif)
+
+## Using the Basics Jupyter Notebook
+
+The `python/Basics` [Jupyter notebook](Background-Jupyter.md) contains a 
+simple walkthrough of the functionality of the Python 
+API. It can also serve as a simple test that your environment is configured
+correctly. Within `Basics`, be sure to set `env_name` to the name of the 
+Unity executable if you want to [use an executable](Learning-Environment-Executable.md) or to `None` if you want to interact with the current scene in the Unity Editor.
+
+More information and documentation is provided in the 
+[Python API](Python-API.md) page.
+
+## Training the Brain with Reinforcement Learning
+### Setting the Brain to External
+Since we are going to build this environment to conduct training, we need to 
+set the brain used by the agents to **External**. This allows the agents to 
+communicate with the external training process when making their decisions.
+
+1. In the **Scene** window, click the triangle icon next to the Ball3DAcademy 
+object.
+2. Select its child object **Ball3DBrain**.
+3. In the Inspector window, set **Brain Type** to **External**.
+
+![Set Brain to External](images/mlagents-SetExternalBrain.png)
+
+### Training the environment
+1. Open a command or terminal window. 
+2. Nagivate to the folder where you installed ML-Agents. 
+3. Change to the `python` directory. 
+4. Run `python3 learn.py --run-id=<run-identifier> --train`
+Where:
+- `<run-identifier>` is a string used to separate the results of different training runs
+- And the `--train` tells learn.py to run a training session (rather than inference)
+5. When the message _"Start training by pressing the Play button in the Unity Editor"_ is displayed on the screen, you can press the :arrow_forward: button in Unity to start training in the Editor.
+
+**Note**: Alternatively, you can use an executable rather than the Editor to perform training. Please refer to [this page](Learning-Environment-Executable.md) for instructions on how to build and use an executable.
+
+![Training command example](images/training-command-example.png)
+
+**Note**: If you're using Anaconda, don't forget to activate the ml-agents environment first.
+
+If the learn.py runs correctly and starts training, you should see something like this:
+
+![Training running](images/training-running.png)
+
+### After training
+You can press Ctrl+C to stop the training, and your trained model will be at `ml-agents/python/models/<run-identifier>/editor_<academy_name>_<run-identifier>.bytes` where `<academy_name>` is the name of the Academy GameObject in the current scene. This file corresponds to your model's latest checkpoint. You can now embed this trained model into your internal brain by following the steps below, which is similar to the steps described [above](#play-an-example-environment-using-pretrained-model).  
+
+1. Move your model file into 
+`unity-environment/Assets/ML-Agents/Examples/3DBall/TFModels/`.
+2. Open the Unity Editor, and select the **3DBall** scene as described above.
+3. Select the **Ball3DBrain** object from the Scene hierarchy.
+4. Change the **Type of Brain** to **Internal**.
+5. Drag the `<env_name>_<run-identifier>.bytes` file from the Project window of the Editor
+to the **Graph Model** placeholder in the **Ball3DBrain** inspector window.
+6. Press the :arrow_forward: button at the top of the Editor.
+
+## Next Steps
+
+* For more information on ML-Agents, in addition to helpful background, check out the [ML-Agents Overview](ML-Agents-Overview.md) page.
+* For a more detailed walk-through of our 3D Balance Ball environment, check out the [Getting Started](Getting-Started-with-Balance-Ball.md) page.
+* For a "Hello World" introduction to creating your own learning environment, check out the [Making a New Learning Environment](Learning-Environment-Create-New.md) page.
+* For a series of Youtube video tutorials, checkout the [Machine Learning Agents PlayList](https://www.youtube.com/playlist?list=PLX2vGYjWbI0R08eWQkO7nQkGiicHAX7IX) page. 

docs/FAQ.md

@@ -0,0 +1,91 @@
+# Frequently Asked Questions
+
+
+### Scripting Runtime Environment not setup correctly
+
+If you haven't switched your scripting runtime version from .NET 3.5 to .NET 4.6, you will see such error message:
+
+```
+error CS1061: Type `System.Text.StringBuilder' does not contain a definition for `Clear' and no extension method `Clear' of type `System.Text.StringBuilder' could be found. Are you missing an assembly reference?
+```
+
+This is because .NET 3.5 doesn't support method Clear() for StringBuilder, refer to [Setting Up ML-Agents Within Unity](Installation.md#setting-up-ml-agent-within-unity) for solution. 
+
+### TensorFlowSharp flag not turned on. 
+
+If you have already imported the TensorFlowSharp plugin, but havn't set ENABLE_TENSORFLOW flag for your scripting define symbols, you will see the following error message:
+
+```
+You need to install and enable the TensorFlowSharp plugin in order to use the internal brain. 
+```
+
+This error message occurs because the TensorFlowSharp plugin won't be usage without the ENABLE_TENSORFLOW flag, refer to [Setting Up ML-Agents Within Unity](Installation.md#setting-up-ml-agent-within-unity) for solution. 
+
+### Tensorflow epsilon placeholder error
+
+If you have a graph placeholder set in the internal Brain inspector that is not present in the TensorFlow graph, you will see some error like this:
+
+```
+UnityAgentsException: One of the Tensorflow placeholder could not be found. In brain <some_brain_name>, there are no FloatingPoint placeholder named <some_placeholder_name>. 
+```
+
+Solution: Go to all of your Brain object, find `Graph placeholders` and change its `size` to 0 to remove the `epsilon` placeholder. 
+
+Similarly, if you have a graph scope set in the internal Brain inspector that is not correctly set, you will see some error like this:
+
+```
+UnityAgentsException: The node <Wrong_Graph_Scope>/action could not be found. Please make sure the graphScope <Wrong_Graph_Scope>/ is correct
+```
+
+Solution: Make sure your Graph Scope field matches the corresponding brain object name in your Hierachy Inspector when there is multiple brain. 
+
+### Environment Permission Error
+
+If you directly import your Unity environment without building it in the 
+editor, you might need to give it additional permissions to execute it. 
+
+If you receive such a permission error on macOS, run:
+
+`chmod -R 755 *.app` 
+
+or on Linux:
+
+`chmod -R 755 *.x86_64` 
+
+On Windows, you can find 
+[instructions](https://technet.microsoft.com/en-us/library/cc754344(v=ws.11).aspx).
+
+### Environment Connection Timeout
+
+If you are able to launch the environment from `UnityEnvironment` but 
+then receive a timeout error, there may be a number of possible causes.
+ * _Cause_: There may be no Brains in your environment which are set 
+ to `External`.  In this case, the environment will not attempt to 
+ communicate with python. _Solution_: Set the Brains(s) you wish to 
+ externally control through the Python API to `External` from the 
+ Unity Editor, and rebuild the environment.
+ * _Cause_: On OSX, the firewall may be preventing communication with 
+ the environment. _Solution_: Add the built environment binary to the 
+ list of exceptions on the firewall by following 
+ [instructions](https://support.apple.com/en-us/HT201642). 
+ * _Cause_: An error happened in the Unity Environment preventing 
+ communication. _Solution_: Look into the 
+ [log files](https://docs.unity3d.com/Manual/LogFiles.html) 
+ generated by the Unity Environment to figure what error happened. 
+
+### Communication port {} still in use
+
+If you receive an exception `"Couldn't launch new environment because 
+communication port {} is still in use. "`, you can change the worker 
+number in the Python script when calling 
+
+`UnityEnvironment(file_name=filename, worker_id=X)`
+
+### Mean reward : nan
+
+If you receive a message `Mean reward : nan` when attempting to train a 
+model using PPO, this is due to the episodes of the learning environment 
+not terminating. In order to address this, set `Max Steps` for either 
+the Academy or Agents within the Scene Inspector to a value greater 
+than 0. Alternatively, it is possible to manually set `done` conditions 
+for episodes from within scripts for custom episode-terminating events.

docs/Feature-Monitor.md

@@ -5,14 +5,16 @@
 The monitor allows visualizing information related to the agents or training process within a Unity scene. 
 
 You can track many different things both related and unrelated to the agents themselves. To use the Monitor, call the Log function anywhere in your code :
+
 ```csharp
-Monitor.Log(key, value, displayType , target)
+Monitor.Log(key, value, target)
 ```
+
  * *`key`* is the name of the information you want to display.
- * *`value`* is the information you want to display.
- * *`displayType`* is a MonitorType that can be either `text`, `slider`, `bar` or `hist`.
-   * `text` will convert `value` into a string and display it. It can be useful for displaying error messages!
-   * `slider` is used to display a single float between -1 and 1. Note that value must be a float if you want to use a slider. If the value is positive, the slider will be green, if the value is negative, the slider will be red.
-   * `hist` is used to display multiple floats. Note that value must be a list or array of floats. The Histogram will be a sequence of vertical sliders.
-   * `bar` is used to see the proportions. Note that value must be a list or array of positive floats. For each float in values, a rectangle of width of value divided by the sum of all values will be show. It is best for visualizing values that sum to 1.
+ * *`value`* is the information you want to display. *`value`* can have different types : 
+   * *`string`* - The Monitor will display the string next to the key. It can be useful for displaying error messages.
+   * *`float`* - The Monitor will display a slider. Note that the values must be between -1 and 1. If the value is positive, the slider will be green, if the value is negative, the slider will be red.
+   * *`float[]`* - The Monitor Log call can take an additional argument called `displayType` that can be either `INDEPENDENT` (default) or `PROPORTIONAL` :
+   		* *`INDEPENDENT`* is used to display multiple independent floats as a histogram. The histogram will be a sequence of vertical sliders.
+   		* *`PROPORTION`* is used to see the proportions between numbers. For each float in values, a rectangle of width of value divided by the sum of all values will be show. It is best for visualizing values that sum to 1.
  * *`target`* is the transform to which you want to attach information. If the transform is `null` the information will be attached to the global monitor.