[RLlib] Add SUPER algorithm.

[mgerstgrasser]

Why are these changes needed?

In our recent NeurIPS paper "Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning" we develop a new multi-agent learning approach based on sharing a small number of experiences between agents. This PR adds a reference implementation of the algorithm to RLlib.

NeurIPS paper link
arxiv paper link

The algorithm is built on top of DQN, adding an experience sharing step between sampling experiences and learning on them. Since code-wise this is a relatively small and clean addition to the existing DQN trainer, we have for now submitted our code as changes to that trainer. We could also create a separate trainer class instead - we can see pros and cons to either, and happy to rework the PR if the RLlib team feels the tradeoffs make a separate class preferable.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

• If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[mgerstgrasser]

Just a gentle ping to make this unstale.
@ArturNiederfahrenhorst @sven1977 @avnishn Anything we could do to help facilitate things?