Problem while using the code

[originholic]

Hello @miyosuda,

Thanks for sharing the code, please ignore the title, I tried out your code with the control problem of cartpole balance experiment instead of Atari game, it works well. But few questions want to ask.

I am curious, in the asynchronous paper, they also used another model implementation with 1 linear layer, 1 LSTM, layer, and softmax output, I am thinking of using this model to see whether improve the result, can you suggest how the LSTM can be implemented using tensorflow in the case of playing atari game?

Also wondering that the accumulated states and reward were reversed, do you need to reverse the actions and values as well? Although it did not make any different when I tried out, just wondering why.

states.reverse()
rewards.reverse()

Last, do you really need to accumulate the gradient and then apply the update, since tensorflow can handle the 'batch' for update.

[miyosuda]

I'm really glad to know that you've succeeded to reproduce continuous model.

About the reverse(), as you say, I've forgot to add

actions.reverse()
values.reverse()

in addition to

states.reverse()
rewards.reverse()

and I've pushed the fix just now. Let me explain why there are reverse() of lists.
In pseudo code of A3C algorithm in deep mind paper, there is

for i in {t-1, ...., tstart} do

This means that "i" will decrease like

t-1, t-2, t-3 ... tstart

This is why I put these reverse() for lists collected in the loop.

As far as I tried with Atari model, I still couldn't reproduce good learning result. I was planning to implement LSTM after being able to reproduce good Atari result, but as you say that you've succeeded with continuous model, I should try LSTM now.
Please wait.

About the batch, let me think about whether it is possible to replace it with batch or not.
Just a moment please.

Thank you for suggestions!!

[originholic]

Many thanks for your reply, and glad to hear that you also plan to work on the LSTM model.

I just uploaded the testing codes (based on this repo) for the "batch" update that mentioned.
https://github.com/originholic/a3c_vrep.git

I only tested it with the cartpole balance domain, but somehow I found it actually take longer time to reach the desired score than your implementation. I will try to investigate this later, as now I will continue to work with your implementation to study the LSTM model, which I am not familiar with.

Also Instead of constant learning rate:

math.exp( log_lo * ( 1-rate ) + log_hi * rate)

Don't know whether the random initialization of learning rate that mentioned in the paper can help to improve the results:

math.exp( random.uniform( log_lo, log_hi ) )

[miyosuda]

I just uploaded the testing codes (based on this repo) for the "batch" update that mentioned.
https://github.com/originholic/a3c_vrep.git

Thanks! I'll try.

About the LSTM, I'm also new to LSTM and just started studying it recently, so please don't expect too much! However I'm really interested in 3D labyrinth model with LSTM, so I would like to try it.

About the randomizing learning rate with log_uniform, I also used to randomize initial learning rate for each threads with log_uniform before. However when I looked at the figure on page 23, I found that learning rate varies from 10^-4 to 10^-2, uniformly distributed with log scale sampling.

So my understanding of log_uniform function is that they are using log_uniform for finding best hyper parameter when they use grid search for it.

(In the graph on page 14 and page 22, they are also using log scale for grid searching parameters.)

However, I'm not sure my understanding is correct.

[originholic]

Thanks for pointing out.

After re-think about the random initialization, I think you are right about it, the initial learning rates sampled from the LogUniform range were used to demonstrate the sensitivity of their methods. And it makes sense that a constant (or best choice) of learning rate is apply for RMSPROP and decay to zero over time.

Sorry, it is my bad, just got confused when they use the phrase in the paper.

"each using a different random initialization and initial learning rate"

[miyosuda]

No problem. Any suggestion and discussions are always welcomed. Thanks!

[originholic]

Hi, tuning in again.
May I ask that in the case of continuous action domain of the asynchronous paper, they used two policy outputs of a linear layer and a Softplus activation + linear layer to represent the mean and variance. I am wondering how policy loss can be calculated with two outputs?

self.policy_loss = -( tf.reduce_sum( tf.mul( tf.log(self.pi), self.a ) ) * self.td + entropy * entropy_beta )

I am thinking of calculating the loss separately, by making 2 of the above policy loss function for the outputs. Does this make sense to you?
Sorry this might be out the scope of your interest, as 3D labyrinth doesn't require continuous actions, but any suggestion is highly appreciated. Many thanks!

[miyosuda]

I have never tried continues model, so today I looked into other Simple Cart-Pole Actor-Critic sample without NN to learn about it.
How to define loss for policy for continues action is still difficult for me, so I'll try continues model with creating branch.
(I'm also interested in continuous model too)
Maybe it will be natural to make two different loss function for mean and variance, but I'm not sure now.
I'll try to figure out.

By the way, even with the discrete action model I'm implementing now, the policy loss function is the most difficult part for me and still I'm not certain this is 100% correct.

However, when I tried simple 2D grid maze model (which I implemented in debug_maze branch), this program succeeded to find shortest path with this policy loss function. So the loss function for discrete action seems fine.

Anyway, I'll report here if I found any result with continuous model.

[originholic]

Thanks for the reply.
As far as I know from your codes, the policy loss function for discrete domain is calculated using the negative log-likelihood of the softmax function.

After doing some searches, may be I can apply the same loss function eg. negative log-likelihood, but instead of softmax function, a Gaussian (Normal) distribution function can be used instead since the outputs have mean and variance. So I think the loss function looks like by following the formula, where sigma2: variance, and mu: mean,

D = tf.to_float(tf.size(self.a))
x_prec = tf.exp(-tf.log(self.sigma2))
x_diff = tf.sub(self.a, self.mu)
x_power = tf.square(x_diff) * x_prec * -0.5
gaussian_nll = (tf.reduce_sum(tf.log(self.sigma2)) + D * tf.log(2 * np.pi)) / 2 - tf.reduce_sum(x_power)
self.policy_loss = gaussian_nll * self.td + entropy_beta * entropy

Sorry for the messy typing, I will try this out to see whether it works for the continuous cartpole domain, and let you know how this goes.
Thanks

[miyosuda]

Is this the explanation of this loss function?

http://docs.chainer.org/en/stable/reference/functions.html#chainer.functions.gaussian_nll

I really want to know the result. There is a lot to learn from this thread for me. Super thanks!!

[originholic]

Yes, that's right, the negative log-likelihood of normal distribution is from the chainer site, but I also found another called maximum log-likelihood, I think they are the same thing by looking into the formula alone. Same here, there are lots methods out there waiting to be learned and get confused.

Tried the loss function based on your code, it works moderately well with cart-pole balance task of continuous action domain, at least it is able to converge (or said reach the desired score). But possibly need some more examples to study the codes in order to draw conclusion that the loss function actually works for continuous action. So keep working on it!! Thanks.

However when I turned back to try it with the "batch" method, it reached a score of around 2000(desired score was 3000), the network somehow diverged immediately(I am not pretty sure it was diverged or not, or explode, the network just gave "NaN" for its output all the time).

[miyosuda]

Thank you for reporting.
I was trying batch with my discrete action code in "batch" and "debug_maze_batch" branch.
I'm checking whether gradient accumulation is working correctly when batched.

@miyosuda:
Hey, I had been trying to implement the same, on Theano. Implemented an A2C version (single thread), which obviously never converges inspite of training on GPU, for even a week or so... Came across your git source. Could you please let me know what exactly are the issues that you are facing right now, that makes your learning still not as good as required? Is it NaNs and stability; or no-convergence of the network? We can try to catch up on this, as I am also in urgent need to have an Actor Critic learner on Pong.

[miyosuda]

@aravindsrinivas
Thank you for joining the discussion.
Let me explain what I tried, what I succeeded and what I have not succeeded yet.

I have beed trying pong with A3C with CPU 8 threads.
The problem is that the score of the game does not increase even with one or two days learning.
The AI can hit back tree or four times in one game, but the score does not increase like the deep mind paper shows.

(As far as I'm trying with pong, the network does not diverges like NaN)

To confirm whether my implementation has a problem or not, I tried easier task.
I implemented 10x10 grid 2D maze, and let this A3C algorithm find the shortest path.
After running two or thee minutes, the AI converged to optimal result. (It succeeded to find the shortest path)

I tried this in "debug_maze" branch.

After confirming that this algorithm can solve easy RL task, I'm changing hyper parameter little by little to check whether the game score will increase like paper shows.
But the result is still same.

I once heard that the DQN is very sensitive with hyper parameters, and as far as I see the paper, hyper parameters of this method seems sensitive.

Along with tuning hyper parameters, I'm also planning to try another task, task that doesn't use CNN.

By the way, the key concept of this method is to get the stability of the network by running multiple threads at the same time, not to diverge or oscillate.
So if you have problem with single thread, how about trying multiple threads?

I have never tried Theano, but if you would like to run it with TensorFlow, I can help you.

@miyosuda
I mailed the authors (from DeepMind). These are some hyper parameters that they explicitly told me in the mail:

The decay parameter (called alpha in the paper) for RMSProp was 0.99 and the regularization constant (called epsilon in the paper) was 0.1. The maximum allowed gradient norm was 40. The best learning rates were around 7*10^-4. Backups of length 20 were used which corresponds to setting the t_max parameter to 20.

Also, I am not sure if you used the frame skip in your implementation. From what I saw in the game_state.py, you just have a reward = ale.act(action)? Shouldn't it be in a for loop like
for _ in range(frame_skip):
reward += ale.act(action)

Also, are you clipping the reward to lie between -1 and 1? In DQN, rewards were clipped between -1 and 1. I am not sure what the rewards are for Pong from the ALE src.

[miyosuda]

Wowwowow! They are the parameters that I really wanted!!! Super thanks!!

I was always using t_max with 5 and didn't use gradient norm clipping.
(In the paper there was only one line just referring gradient norm clipping, so I didn't tried it)

also, I am not sure if you used the frame skip in your implementation

I've set the frame skipping with every 4 frames at "ale.cfg" file, but as you say it might better to put loop as you suggested.

I'm not clipping the score, but ALE pong gives reward 1 or -1, so it will be ok.

Anyway, super thanks for giving me such a valuable information!!! I'll try these parameters.