Question about turning agents on/off.

[liamFerris]

Hi there,

This is a really exciting project, really appreciate the work that has went in thus far - great job all :)

I am trying to create agents that where their steps are triggered by me, through code (rather than every frame or some fixed frame skip). I'm not sure if maybe this is something that is obvious and I've just missed it or if there is no best practice for what I am trying to do.

Specifically I am trying to train a brain using two agents playing against each other, but in a turn based environment (player one -> player two -> player one etc).

Should I be disabling the Agent scripts? Should I play with the Wait Time during runtime? Neither of these sound like a perfect solution, nor have my attempts at them been successful so far.

Thanks in advance for the help,
Liam

[trulyspinach]

Hi there,
If I were you, I will trying to make a flag in environment indicating who is taking the turn.
And in agent's AgentStep function, I will check if this is "my" turn before taken an action. If this is not "my" turn, the agent will simply do nothing.

I never tried it yet so I am not sure if it will work.

[liamFerris]

Hey,
Yeah this is the approach I am currently taking - my worry is that allowing each agent to run through steps where nothing happens would throw off the training process.

I suppose a followup question about reinforced learning would be during the training process, does a step with a reward of 0 influence the model whatsoever?

[trulyspinach]

The model doesn't care about a specific value that much. It only cares about maximizing the reward.
I haven't read through any PPO paper yet but in the context of reinforcement learning it is "maximizing".

I just got another little idea is that you create a state in agent, and make an "do nothing" action .
So if it is the agent's turn the state become 1 and otherwise 0.
If agent taking actions other than "do nothing" when "0" it get -1 reward.

This is trying to let the agent learn only to play on its turn.

[awjuliani]

Hi @liamFerris,

This is an important use-case which we'd like to think more about how to officially support. Right now I would recommend doing as @SPINACHCEO suggested, and check whether or not it is the agent's turn. When it isn't the agent's turn, the action provided by the model can simply be ignored.

Very interested to see if other ideas come up around this though.

[surferau]

In case of turn based game maybe you should try use only one agent and make it switch sides then add the current side into state list.

[argylearson]

I'm running into a similar issue. I read here to just return an empty list, but that throws an error from Brain.CollectStates(). I considered a "0" state, but I may try surferau's suggestion.

[mmattar]

Folks, closing this out due to inactivity. We've also since released v0.3 which includes several new features. Please create a new issue if you have additional questions.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.