new _run()

[baedan]

hello! i'm currently working on implementing trpo (#134) from the master branch; however, i'm having trouble with the new pipeline.

right now, optimise! is called after supplying the environment with an action. this means algorithms that optimise after each episode won't be able to use optimise!; they need to implement (agent::Agent{PolicyType})(::PostEpisodeState, env) instead, while leaving optimise! as a no-op.

additionally, i'm not sure how to use RLTrajectories.jl to get an entire episode. also, can Agent push episode info to Trajectory correctly? i think push!(agent.trajectory, (agent.cache..., action=action)) would only work if agent.trajectory is a Traces, not Episodes, right? in general, i don't know how to work with the for batch in trajectory paradigm for an entire episode.

if vpg.jl could be updated to the new pipeline it would be super helpful as an example!

[findmyway]

if vpg.jl could be updated to the new pipeline it would be super helpful as an example!

Will do!

[HenriDeh]

additionally, i'm not sure how to use RLTrajectories.jl to get an entire episode.

I think this is not well supported by RLTrajectories yet. This would require the Episode struct to be fully functional and I would say that a new EpisodeSampler would be required.

[findmyway]

additionally, i'm not sure how to use RLTrajectories.jl to get an entire episode.

I think this is not well supported by RLTrajectories yet. This would require the Episode struct to be fully functional and I would say that a new EpisodeSampler would be required.

Indeed. However, VPG is quite simple and I do not want to make things too complicated. So I simply extract the whole Episode and optimize VPG on it directly.

I'll add the EpisodeSampler later when adding more policy gradient algorithms.