Continuous RL example

[Vittorio-Caggiano]

Hi,

very impressive project. Do you have any example for a continuous RL case e.g. MountainCarContinuous?
I noticed that the environment_pipeline is configured only for a discrete action space e.g.

bindsnet/bindsnet/pipeline/environment_pipeline.py

Line 121 in 2eb3aa5

self.output: torch.zeros((self.time, self.env.action_space.n)).to(

Is there a continuous_environment_pipeline?
Also, is there any solution to encode non-strictly positive observations (from the environment) into a train of spikes?

[Hananel-Hazan]

Can you please elaborate on what would be considered a continuous example for you? and why the breakout example is not continuous in that sense.

About the non-positive observations, their many approaches for doing so in the literature some can be implemented in BindsNET using existing functionality in the library, and some you need to write some yourself.

[Vittorio-Caggiano]

Hi Hananel, thanks for your reply and sorry for the short description ... In the breakout example the actions to interact with the environment are categorical/integer and the observation is an image (discrete values between 0 ... 255). The environment_pipeline.py seems to need those actions/observation as discrete values from/to the environment. For example, the attribute n in self.env.action_space.n doesn't exist in environments requiring continuous control e.g. MountainCarContinuous-v0

In an environment requiring continuous control e.g. MountainCarContinuous, actions have continuous (negative and positive) values and the observation vector is as well composed of non-strictly positive float values.

[Hananel-Hazan]

You are right, environment_pipeline.py does not fit directly to continuous control. But one can think of many ways to discretize bounded continues signal to spike train (analog to digital). I am not sure there is one discretize method that fits all types of inputs, so you need to try more then one.

In the case of MountainCarContinuous-v0, it seems that you need to adapt the pipeline method to work with the continues function.

In any case, let me know what do you try and if its works.