Finalize the khr-3hv environment. #49

NikosKokkinis · 2021-08-20T09:46:30Z

Finalize the khr-3hv environment for the GSoC 2021.

Create the environment that works with Ray and Stable-Baseline PPO.
Logging on Wandb website.
Step-by-step documentation.

Add example with cartpole_discrete using stable-baseline PPO

Add a first working version of the khr-3hv robot

Finilize the Stable baseline with the cartpole env

… dev

Add a first working version of the khr-3hv robot Finilize the Stable baseline with the cartpole env Add a first working version of the khr-3hv robot Finilize the Stable baseline with the cartpole env

… dev

Add example with cartpole_discrete using stable-baseline PPO

…ctions. Using stable baseline and Ray.

Better version of the khr-3hv enviroment. Reducing observations and actions

Finalize the khr-3hv environment for the GSoC 2021. - Create the environment that works with Ray and Stable-Baseline PPO. - Logging on Wandb website. - Step-to-step documentation.

Finalize the Nao environment for thr GSoC 2021

… dev

Fix path error on the khr-3hv robot readme and add reward plots

KelvinYang0320 · 2021-08-30T14:30:37Z

Hello, @NickKok. I am reviewing the khr-3hv part of your work.
Did you get AttributeError: module 'aioredis' has no attribute 'create_redis_pool' when USE_Ray = True?
Check out this link.
I downgraded aioredis to v1.3.1 to run and keep reviewing your work for now.

KelvinYang0320

Thank you for this khr-3hv example and the nice tutorial you provided!
I think users will enjoy this example.
I've left a few comments for khr-3hv.
I didn't review cartpole & nao since the description of this PR is about khr-3h.

KelvinYang0320 · 2021-08-30T09:15:15Z

examples/khr-3hv/full project/controllers/supervisorsManager/PPORun.py

@@ -0,0 +1,154 @@
+import numpy as np
+from scipy.ndimage.interpolation import shift


I think we can remove this unused shift.

KelvinYang0320 · 2021-08-30T09:18:06Z

examples/khr-3hv/full project/controllers/supervisorsManager/PPORun.py

+import ray
+from ray import tune
+from ray.tune.registry import register_env
+from ray.tune import grid_search


I think we can remove this unused import.

When using Ray. It is needed the tune, register_env. Regarding the ray and grid_serach, I could remove those.

@NickKok Hi, sorry for the confusion.
I mean that we can remove grid_search & ray and keep tune. Thanks!

KelvinYang0320 · 2021-08-30T09:18:14Z

examples/khr-3hv/full project/controllers/supervisorsManager/PPORun.py

+import ray.rllib.agents.ppo as ppo
+from ray.tune.logger import DEFAULT_LOGGERS
+from ray.tune.integration.wandb import WandbLoggerCallback
+from ray.tune.integration.wandb import wandb_mixin


I think we can remove this unused import.

I need the ppo and the WandbLoggerCallback to log the results on wandb.

Sorry for the confusion.
I think we can remove wandb_mixin only. Thanks!

KelvinYang0320 · 2021-08-30T09:18:21Z

examples/khr-3hv/full project/controllers/supervisorsManager/PPORun.py

+from ray.tune.registry import register_env
+from ray.tune import grid_search
+import ray.rllib.agents.ppo as ppo
+from ray.tune.logger import DEFAULT_LOGGERS


I think we can remove this unused DEFAULT_LOGGERS.

KelvinYang0320 · 2021-08-30T09:28:40Z

examples/khr-3hv/full project/worlds/khr-3hv.wbt

+Viewpoint {
+  fieldOfView 0.660595
+  orientation -0.9627454208252096 0.19381572750322326 0.1885648918873455 0.25970754728137774
+  position 0.22818627812370207 1.6236991612459881 10.33705776276167
+}


Suggested change

Viewpoint {

fieldOfView 0.660595

orientation -0.9627454208252096 0.19381572750322326 0.1885648918873455 0.25970754728137774

position 0.22818627812370207 1.6236991612459881 10.33705776276167

}

Viewpoint {

fieldOfView 0.660595

position 0.05 1.4 9

}

The viewpoint is a little bit too inclined.
Please update the gif in the readme file as well, thanks!

KelvinYang0320 · 2021-08-31T01:21:15Z

examples/khr-3hv/full project/controllers/supervisorsManager/PPORun.py

+# Scheduler for the learning rate when using the stable baselines
+def linear_schedule(initial_value: float) -> Callable[[float], float]:
+    """
+    Linear learning rate schedule.
+    :param initial_value:
+    :return: current learning rate depending on remaining progress
+    """
+    if isinstance(initial_value, str):
+        initial_value = float(initial_value)
+
+    # TODO  decrease the lr by .10
+    def func(progress_remaining: float) -> float:
+        """
+        Progress will decrease from 1 (beginning) to 0
+        :param progress_remaining: (float)
+        :return: (float)
+        """
+        return progress_remaining * initial_value
+
+    return func         


Remove these lines?

I believe this is something that we could use in the future. It's for the learning rate scheduler when using Stable-Baseline PPO.

Okay. I think we can keep these lines then.

KelvinYang0320 · 2021-08-31T01:59:12Z

examples/khr-3hv/full project/controllers/supervisorsManager/PPORun.py

+            model.learn(total_timesteps=100000000, callback=checkpoint_callback)
+        else:
+            # Load the weights of a trained Agent
+            model = PPO.load("./results/Deepbots-khr3hv-SB-reward-weight,prev_pos,obs-x,y,z,vel-done-0.5-RNN-5,clip_range-0.2,lr-0.0001_6778500_steps")


Maybe we can pass a file path in supervisorsManager.py.

Yes it could be possible

Please address this on a following PR or this PR. Thank you!

KelvinYang0320 · 2021-08-31T02:03:50Z

examples/khr-3hv/full project/controllers/supervisorsManager/PPORun.py

+                            )
+            ray.shutdown()
+        else:
+            # Crate an agent having the predifined model hyperparameters


Add ray.init()?
When TRAIN = False and USE_Ray = True, I get System error: Ray has not been started yet. You can start Ray with 'ray.init()'.
Adding ray.init() do fix this error.

KelvinYang0320 · 2021-08-31T02:05:21Z

examples/khr-3hv/full project/controllers/supervisorsManager/PPORun.py

+            # Crate an agent having the predifined model hyperparameters
+            agent = ppo.PPOTrainer(config=model_config)
+            # Restore the train weights
+            agent.restore("./results/Deepbots-khr3hv-Ray-reward-weight,prev_pos,obs-x,y,z,vel-done-0.5-RNN-5,clip_range-0.2,lr-0.0001_6778500_steps")


Maybe we can pass a file path in supervisorsManager.py.
Then it will be visible to for end-user.
Please address this on a following PR or this PR. Thank you!

KelvinYang0320 · 2021-08-31T02:08:14Z

examples/khr-3hv/full project/controllers/supervisorsManager/khr3hvEnv.py

+
+from RobotUtils import RobotFunc
+from deepbots.supervisor.controllers.robot_supervisor import RobotSupervisor
+from gym.spaces import Box, Discrete


I think we can remove this unused Discrete.

Suggested change

from gym.spaces import Box, Discrete

from gym.spaces import Box

KelvinYang0320 · 2021-08-31T03:03:36Z

@NickKok Thank you for contributing many examples to deepworlds!
Also, the code quality is great!

This PR includes

khr-3hv
nao
cartpole with stable baselines

I only do the code review on khr-3hv as the description of this PR.
Could you separate these examples into different branches and open PR for each?
And write the title & description for each PR according to changed files.
In this way, I think we can discuss or record these examples more clearly. Thanks! 😄

NikosKokkinis · 2021-09-06T12:07:55Z

Hello @KelvinYang0320.

No I didn't get any error on : AttributeError: module 'aioredis' has no attribute 'create_redis_pool' when USE_Ray = True?

When you downgraded aioredis to v1.3.1, does it work with Ray?

KelvinYang0320 · 2021-09-06T15:03:10Z

When you downgraded aioredis to v1.3.1, does it work with Ray?

@NickKok Yes, everything goes fine when downgrading aioredis to v1.3.1, as suggested on this website.
I follow your readme to pip install all packages.
Could you start from pip install these packages again to see if you can reproduce this problem?
Thank you

tsampazk · 2022-06-29T12:56:09Z

Hey @NickKok! We would like to take over this PR and finalize it if you don't mind, unless you have some free time to do so yourself. You could resolve whatever comments exist right now, merge it and we can take over perfecting it later.

This needs to be closed in combination with other fixes we've been doing for #58, so as to bring all existing examples up to speed with the latest Webots release.

NikosKokkinis · 2022-06-29T14:56:19Z

Hello @tsampazk. It won't be possible to work on it till the end of August. After that It might be possibl during weekends. If you would like to continue the work feel free. I could help you from September.

tsampazk · 2022-06-29T20:48:37Z

Alright @NickKok, thanks for the quick response, we will probably continue working on this and finalize it soon then. There's always more work to be done so if you feel like it you can contribute in the future 😃

tsampazk · 2022-11-16T09:39:04Z

Hey @KelvinYang0320, could we close this PR as i think all the goals were met by #76 ?

KelvinYang0320 · 2022-11-16T14:15:02Z

@tsampazk There are three examples, cartpole, khr-3hv, and nao, in this PR.

Finalize the khr-3hv environment. #49 (comment)

I think we can close this PR and open an issue for it. What do you think?

tsampazk · 2022-11-16T14:19:29Z

Oh yes i didn't notice the other two, sorry. The cartpole stable-baselines stuff are already present in the dev branch 😕, so we need a new PR that adds the nao example? Bottom line i think it's better to close it and open an issue as you suggested!

KelvinYang0320 · 2022-11-16T14:31:42Z

@tsampazk I have a nao branch on my fork. Do you want me to open a draft PR or push it to a new branch on aidudezzz/deepworlds directly?
As for cartpole, I think he also modified our existing cartpole stable-baselines. Unfortunately, I cannot remember the detail. 😕

tsampazk · 2022-11-16T14:45:52Z

@tsampazk I have a nao branch on my fork. Do you want me to open a draft PR or push it to a new branch on aidudezzz/deepworlds directly?

I don't mind, whatever you feel is easier for you!

As for cartpole, I think he also modified our existing cartpole stable-baselines. Unfortunately, I cannot remember the detail. confused

It's ok, by taking a quick look at the sb cartpole we need to refactor and update that example anyway!

NikosKokkinis and others added 22 commits May 28, 2021 15:21

Add example with cartpole_discrete using stable-baseline PPO

cf39828

Merge pull request #1 from NickKok/dev

0ca941c

Add example with cartpole_discrete using stable-baseline PPO

Add a first working version of the khr-3hv robot

3d6b0fe

Merge pull request #3 from NickKok/dev

8c9a0b6

Add a first working version of the khr-3hv robot

Finilize the Stable baseline with the cartpole env

43007c2

Merge pull request #4 from NickKok/dev

d1c6bff

Finilize the Stable baseline with the cartpole env

Add a first working version of the khr-3hv robot

fef7ec0

Finilize the Stable baseline with the cartpole env

Merge branch 'dev' of https://github.com/NickKok/deepbots-stable into…

2250025

… dev

Add example with cartpole_discrete using stable-baseline PPO

3f4bc4f

Add a first working version of the khr-3hv robot Finilize the Stable baseline with the cartpole env Add a first working version of the khr-3hv robot Finilize the Stable baseline with the cartpole env

Merge branch 'dev' of https://github.com/NickKok/deepbots-stable into…

56000fb

… dev

Merge pull request #5 from NickKok/dev

97c828e

Add example with cartpole_discrete using stable-baseline PPO

Better version of the khr-3hv enviroment. Reducing observations and a…

7f1d2a0

…ctions. Using stable baseline and Ray.

Merge pull request #6 from NickKok/dev

31b0206

Better version of the khr-3hv enviroment. Reducing observations and actions

Finalize the khr-3hv enviroment

031d6c8

Finalize the khr-3hv environment

b0aa322

Finalize the khr-3hv environment for the GSoC 2021. - Create the environment that works with Ray and Stable-Baseline PPO. - Logging on Wandb website. - Step-to-step documentation.

Finilize the Nao env for GSoC

b02f1dc

Merge branch 'eellak-gsoc2021:dev' into dev

6e38989

Merge pull request #8 from NickKok/dev

667fb6d

Finalize the Nao environment for thr GSoC 2021

Add the two reward plots

849ae67

Merge branch 'dev' of https://github.com/NickKok/deepbots-stable into…

6258fd5

… dev

Fix path typo error on the the khr-3hv robot readme

e4f06f6

Merge pull request #9 from NickKok/dev

f0f697b

Fix path error on the khr-3hv robot readme and add reward plots

KelvinYang0320 self-requested a review August 26, 2021 08:30

KelvinYang0320 requested changes Aug 31, 2021

View reviewed changes

KelvinYang0320 marked this pull request as draft July 4, 2022 02:17

KelvinYang0320 mentioned this pull request Jul 4, 2022

KHR-3HV - fix coordinate system #76

Merged

10 tasks

KelvinYang0320 marked this pull request as ready for review November 16, 2022 10:42

KelvinYang0320 marked this pull request as draft November 16, 2022 14:07

tsampazk closed this Nov 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finalize the khr-3hv environment. #49

Finalize the khr-3hv environment. #49

NikosKokkinis commented Aug 20, 2021

KelvinYang0320 commented Aug 30, 2021

KelvinYang0320 left a comment •

edited

Loading

KelvinYang0320 Aug 30, 2021 •

edited

Loading

KelvinYang0320 Aug 30, 2021

NikosKokkinis Sep 6, 2021

KelvinYang0320 Sep 9, 2021

KelvinYang0320 Aug 30, 2021

NikosKokkinis Sep 6, 2021

KelvinYang0320 Sep 9, 2021

KelvinYang0320 Aug 30, 2021 •

edited

Loading

KelvinYang0320 Aug 30, 2021

KelvinYang0320 Aug 31, 2021

NikosKokkinis Sep 6, 2021

KelvinYang0320 Sep 9, 2021

KelvinYang0320 Aug 31, 2021

NikosKokkinis Sep 6, 2021

KelvinYang0320 Sep 9, 2021 •

edited

Loading

KelvinYang0320 Aug 31, 2021 •

edited

Loading

KelvinYang0320 Aug 31, 2021 •

edited

Loading

KelvinYang0320 Aug 31, 2021 •

edited

Loading

KelvinYang0320 commented Aug 31, 2021

NikosKokkinis commented Sep 6, 2021

KelvinYang0320 commented Sep 6, 2021

tsampazk commented Jun 29, 2022

NikosKokkinis commented Jun 29, 2022

tsampazk commented Jun 29, 2022

tsampazk commented Nov 16, 2022

KelvinYang0320 commented Nov 16, 2022

tsampazk commented Nov 16, 2022

KelvinYang0320 commented Nov 16, 2022

tsampazk commented Nov 16, 2022 •

edited

Loading

		@@ -0,0 +1,154 @@
		import numpy as np
		from scipy.ndimage.interpolation import shift

	from gym.spaces import Box, Discrete
	from gym.spaces import Box

Finalize the khr-3hv environment. #49

Finalize the khr-3hv environment. #49

Conversation

NikosKokkinis commented Aug 20, 2021

KelvinYang0320 commented Aug 30, 2021

KelvinYang0320 left a comment • edited Loading

Choose a reason for hiding this comment

KelvinYang0320 Aug 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KelvinYang0320 Aug 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KelvinYang0320 Sep 9, 2021 • edited Loading

Choose a reason for hiding this comment

KelvinYang0320 Aug 31, 2021 • edited Loading

Choose a reason for hiding this comment

KelvinYang0320 Aug 31, 2021 • edited Loading

Choose a reason for hiding this comment

KelvinYang0320 Aug 31, 2021 • edited Loading

Choose a reason for hiding this comment

KelvinYang0320 commented Aug 31, 2021

NikosKokkinis commented Sep 6, 2021

KelvinYang0320 commented Sep 6, 2021

tsampazk commented Jun 29, 2022

NikosKokkinis commented Jun 29, 2022

tsampazk commented Jun 29, 2022

tsampazk commented Nov 16, 2022

KelvinYang0320 commented Nov 16, 2022

tsampazk commented Nov 16, 2022

KelvinYang0320 commented Nov 16, 2022

tsampazk commented Nov 16, 2022 • edited Loading

KelvinYang0320 left a comment •

edited

Loading

KelvinYang0320 Aug 30, 2021 •

edited

Loading

KelvinYang0320 Aug 30, 2021 •

edited

Loading

KelvinYang0320 Sep 9, 2021 •

edited

Loading

KelvinYang0320 Aug 31, 2021 •

edited

Loading

KelvinYang0320 Aug 31, 2021 •

edited

Loading

KelvinYang0320 Aug 31, 2021 •

edited

Loading

tsampazk commented Nov 16, 2022 •

edited

Loading