Mean Actor-Critic

[findmyway]

https://arxiv.org/pdf/1709.00503.pdf

[RajGhugare19]

@findmyway Can I take up this issue as a starting point?

[RajGhugare19]

@findmyway I have some doubts

1. I don't understand how gains are calculated in A2C. Are they just the discounted reward to go? let's say the rewards is a matrix of (nthreads, m). Where m is the update freq. So, the gains would be an array of shape (nthreads*m,).
2. In A2C, shouldn't the updates lengths be equal to the length of the most recent trajectory rather than a fixed update freq?

[findmyway]

1. Yes

The related code is here:

ReinforcementLearningZoo.jl/src/algorithms/policy_gradient/A2C.jl

Lines 67 to 74 in 2e5a43f

	gains = discount_rewards(
	rewards,
	γ;
	dims = 2,
	init = send_to_host(next_state_values),
	terminal = terminals,
	)
	gains = send_to_device(D, gains)

But note that the size is (m, nthreads), not (nthreads, m) here.

2. In theory, the update frequency can be a different value of the length of the trajectory. But here for simplicity, the update frequency is set to be the same with the length of the trajectory. So yes you're right here.

ReinforcementLearningZoo.jl/src/algorithms/policy_gradient/A2C.jl

Line 45 in 2e5a43f

isfull(t) || return

ReinforcementLearningZoo.jl/src/algorithms/policy_gradient/A2C.jl

Lines 112 to 115 in 2e5a43f

	if isfull(agent.trajectory)
	empty!(agent.trajectory)
	push!(agent.trajectory; state = state, action = action)
	end

[RajGhugare19]

I have implemented MAC, The agent is learning but isn't getting a constant reward of 200, rather it is getting noisy rewards. I tried hyperparam tuning but the performance isn't like A2C. The agent is getting an average return of 150

[findmyway]

Average reward of 150 is not so bad, though slightly lower the the reported value. Could you create a pr and see if there's any bug?

[RajGhugare19]

Yeah I am checking for bugs, I am positive that there aren't any (because the agent is learning). I'll check once more and create a pr by tonight

[RajGhugare19]

I have made the pr (sorry for the delay :p)