Skip to content

[bug-fix] Make Python able to deal with 0-step episodes #3671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Mar 23, 2020
Merged

[bug-fix] Make Python able to deal with 0-step episodes #3671

merged 4 commits into from
Mar 23, 2020

Conversation

ervteng
Copy link
Contributor

@ervteng ervteng commented Mar 23, 2020

Proposed change(s)

The Python code wouldn't clear dicts properly if episodes had 0 steps (Done immediately on reset). This changes that.

Types of change(s)

  • Bug fix
  • New feature
  • Code refactor
  • Breaking change
  • Documentation update
  • Other (please describe)

Checklist

  • Added tests that prove my fix is effective or that my feature works
  • Updated the changelog (if applicable)
  • Updated the documentation (if applicable)
  • Updated the migration guide (if applicable)

Other comments

@ervteng ervteng requested a review from vincentpierre March 23, 2020 20:42
ervteng
ervteng commented Mar 23, 2020
View reviewed changes
ml-agents/mlagents/trainers/policy/tf_policy.py
@@ -174,17 +174,6 @@ def get_action(
if batched_step_result.n_agents() == 0:
return ActionInfo.empty()

agents_done = [
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was left over from when we used to do it here (and not in AgentProcessor).

ervteng
ervteng commented Mar 23, 2020
View reviewed changes
ml-agents/mlagents/trainers/policy/tf_policy.py
@@ -379,9 +370,11 @@ def _initialize_tensorflow_references(self):

def create_input_placeholders(self):
with self.graph.as_default():
self.global_step, self.increment_step_op, self.steps_to_increment = (
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, black.

ervteng
ervteng commented Mar 23, 2020
View reviewed changes
ml-agents/mlagents/trainers/agent_processor.py
@@ -143,11 +143,12 @@ def add_experiences(
traj_queue.put(trajectory)
self.experience_buffers[global_id] = []
if curr_agent_step.done:
# Record episode length for agents which have had at least
Copy link
Contributor Author

@ervteng ervteng Mar 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise we'd have a bunch of 0-length episodes if people change models

vincentpierre
vincentpierre approved these changes Mar 23, 2020
View reviewed changes
ml-agents/mlagents/trainers/agent_processor.py

for terminated_id in terminated_agents:
self._clean_agent_data(terminated_id)
# Delete all done agents, regardless of if they had a 0-length episode.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely simpler.

@ervteng ervteng merged commit 7bdaac5 into develop-fix-wall-jump Mar 23, 2020
@delete-merged-branch delete-merged-branch bot deleted the develop-fix-wall-jump-python branch March 23, 2020 21:03
@ervteng ervteng mentioned this pull request Mar 23, 2020
Merged
10 tasks
vincentpierre added a commit that referenced this pull request Mar 24, 2020
@vincentpierre
Hotfix memory leak on Python (#3664)
1dd6cb6 
* Hotfix memory leak on Python

* Fixing

* Fixing a bug in the heuristic policy. A decision should not be requested when the agent is done

* [bug-fix] Make Python able to deal with 0-step episodes (#3671)

* adding some comments

Co-authored-by: Ervin T <ervin@unity3d.com>
vincentpierre added a commit that referenced this pull request Mar 26, 2020
@vincentpierre
Hotfix memory leak on Python (#3664)
dc88653 
* Hotfix memory leak on Python

* Fixing

* Fixing a bug in the heuristic policy. A decision should not be requested when the agent is done

* [bug-fix] Make Python able to deal with 0-step episodes (#3671)

* adding some comments

Co-authored-by: Ervin T <ervin@unity3d.com>
vincentpierre added a commit that referenced this pull request Mar 26, 2020
@vincentpierre
Hotfix memory leak on Python (#3664)
22b3654 
* Hotfix memory leak on Python

* Fixing

* Fixing a bug in the heuristic policy. A decision should not be requested when the agent is done

* [bug-fix] Make Python able to deal with 0-step episodes (#3671)

* adding some comments

Co-authored-by: Ervin T <ervin@unity3d.com>
vincentpierre added a commit that referenced this pull request Mar 30, 2020
@vincentpierre @xstreck1
Hotfixes for Release 0.15.1 (#3698)
377a25c 
* [bug-fix] Increase height of wall in CrawlerStatic (#3650)

* [bug-fix] Improve performance for PPO with continuous actions (#3662)

* Corrected a typo in a name of a function (#3670)

OnEpsiodeBegin was corrected to OnEpisodeBegin in Migrating.md document

* Add Academy.AutomaticSteppingEnabled to migration (#3666)

* Fix editor port in Dockerfile (#3674)

* Hotfix memory leak on Python (#3664)

* Hotfix memory leak on Python

* Fixing

* Fixing a bug in the heuristic policy. A decision should not be requested when the agent is done

* [bug-fix] Make Python able to deal with 0-step episodes (#3671)

* adding some comments

Co-authored-by: Ervin T <ervin@unity3d.com>

* Remove vis_encode_type from list of required (#3677)

* Update changelog (#3678)

* Shorten timeout duration for environment close (#3679)

The timeout duration for closing an environment was set to the
same duration as the timeout when waiting for a response from the
still-running environment.  This led to long waits for the error
response when communication version wasn't matching.

This change forces a timeout duration of 0 when handling errors.

* Bumping the versions

* handle multiple dones in a single step (#3700)

* handle multiple dones in a single step

* [tests] Make end-to-end tests more stable (#3697)

* [bug-fix] Fix entropy computation for GaussianDistribution (#3684)

* Fix how we set logging levels (#3703)

* cleanup logging

* comments and cleanup

* pylint, gym

* [skip-ci] Update changelog for logging fix. (#3707)

* [skip ci] Update README

* [skip ci] Fixed a typo

Co-authored-by: Ervin T <ervin@unity3d.com>
Co-authored-by: Adam Streck <adam.streck@gmail.com>
Co-authored-by: Chris Elion <chris.elion@unity3d.com>
Co-authored-by: Jonathan Harper <jharper+moar@unity3d.com>
chriselion pushed a commit that referenced this pull request Apr 8, 2020
@vincentpierre @xstreck1
Merge 0.15.1 to master (#3755)
725650f 
* Bumping version on the release (#3615)

* Update examples project to 2018.4.18f1 (#3618)

From 2018.4.14f1.  An internal package dependency was updated as
a side effect.

* Remove dead components from the examples scenes (#3619) (#3624)

* Improve warnings and exception if using unsupported combo

* add meta file

* fix unit test

* enforce onnx conversion (expect tf2 CI to fail) (#3600)

* Update error message

* Updated the release branch docs (#3621)

* Updated the release branch docs

* Edited the README

* make sure top-level timer is closed before writing

* Remove space from Product Name for examples

In #2588 it was suggested that the space in the Product Name for
our example environments causes confusion when using a default build
because of the need to escape the space in the build filename.

This change removes the space from the Product Name in the project's
player settings.

* [bug-fix] Increase 3dballhard and GAIL default steps (#3636)

* Updating the NN models (#3632)

* Updating the NN models

* Update gridworld

* [skip ci] Update BallHard

* Update hallway

* Hotfixes for Release 0.15.1  (#3698)

* [bug-fix] Increase height of wall in CrawlerStatic (#3650)

* [bug-fix] Improve performance for PPO with continuous actions (#3662)

* Corrected a typo in a name of a function (#3670)

OnEpsiodeBegin was corrected to OnEpisodeBegin in Migrating.md document

* Add Academy.AutomaticSteppingEnabled to migration (#3666)

* Fix editor port in Dockerfile (#3674)

* Hotfix memory leak on Python (#3664)

* Hotfix memory leak on Python

* Fixing

* Fixing a bug in the heuristic policy. A decision should not be requested when the agent is done

* [bug-fix] Make Python able to deal with 0-step episodes (#3671)

* adding some comments

Co-authored-by: Ervin T <ervin@unity3d.com>

* Remove vis_encode_type from list of required (#3677)

* Update changelog (#3678)

* Shorten timeout duration for environment close (#3679)

The timeout duration for closing an environment was set to the
same duration as the timeout when waiting for a response from the
still-running environment.  This led to long waits for the error
response when communication version wasn't matching.

This change forces a timeout duration of 0 when handling errors.

* Bumping the versions

* handle multiple dones in a single step (#3700)

* handle multiple dones in a single step

* [tests] Make end-to-end tests more stable (#3697)

* [bug-fix] Fix entropy computation for GaussianDistribution (#3684)

* Fix how we set logging levels (#3703)

* cleanup logging

* comments and cleanup

* pylint, gym

* [skip-ci] Update changelog for logging fix. (#3707)

* [skip ci] Update README

* [skip ci] Fixed a typo

Co-authored-by: Ervin T <ervin@unity3d.com>
Co-authored-by: Adam Streck <adam.streck@gmail.com>
Co-authored-by: Chris Elion <chris.elion@unity3d.com>
Co-authored-by: Jonathan Harper <jharper+moar@unity3d.com>

* fix changelog

* keep master gridworld

Co-authored-by: Vincent-Pierre BERGES <vincentpierre@unity3d.com>
Co-authored-by: Jonathan Harper <jharper+moar@unity3d.com>
Co-authored-by: Ervin T <ervin@unity3d.com>
Co-authored-by: Adam Streck <adam.streck@gmail.com>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@vincentpierre vincentpierre vincentpierre approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ervteng @vincentpierre