[Algorithm] SOTA discrete offline CQL

[Ibinarriaga8]

Description

This PR solves issue #3097 by providing a state-of-the-art (SOTA) implementation for discrete offline Conservative Q-Learning (CQL) within the repository.

• Implemented a SOTA discrete offline CQL algorithm fully integrated into the codebase.
• As with the SOTA implementation of discrete online CQL (already present in the repo), the CartPole environment was used for offline discrete CQL training.
• Leveraged the changes from commit 0627e85c78756b4e4dbde726a9d5e85300e239a0 to utilize the updated MinariExperienceReplay class for loading experiences from a custom CartPole dataset into the offline replay buffer.
• Updated the discrete loss function according to the latest torchrl documentation, explicitly handling the categorical action space as required for discrete environments.
• Added a SOTA-check test for this new algorithm to ensure correctness and performance.

Motivation and Context

This change was required to complete the set of SOTA CQL implementations in the repository, specifically by adding support for discrete offline CQL. Previously, only the online discrete CQL and both online/offline continuous CQL were available. By implementing discrete offline CQL, the repository now covers all major CQL benchmarks, enabling users to run and benchmark offline RL in discrete action spaces natively within torchrl.

The change also brings the repository in line with other top RL codebases and helps the community reproducibly benchmark and compare discrete offline RL algorithms.

This PR closes #3097.

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds core functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)
• Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3098

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 2 Cancelled Jobs, 3 Unrelated Failures

As of commit 27a065a with merge base 3f10cb1 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Process completed with exit code 2.
• Continuous Benchmark / CPU Pytest benchmark (gh)
  Process completed with exit code 2.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t c6b61ea8e49e75ef68bf1bd58d18ab157e657aca1d5e33031c2b40b9f6472254 /exec failed with exit code 1
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 4487ce55b2b1778e370a789ebb2bc4c8a4903c12e366dba97324464217a5ab94 /exec failed with exit code 1

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 2.
• Continuous Benchmark / GPU Pytest benchmark (gh)
  Process completed with exit code 2.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh) (trunk failure)
  AttributeError: module 'torch.utils._pytree' has no attribute 'register_pytree_node'
• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh) (trunk failure)
  AttributeError: module 'torch.utils._pytree' has no attribute 'register_pytree_node'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Ibinarriaga8]

Added sota-check

[Ibinarriaga8]

Replay buffer from a custom minari dataset

[Ibinarriaga8]

Updated make_discrete_loss to follow torchrl documentation

[Ibinarriaga8]

Log runs:

https://api.wandb.ai/links/202206789-universidad-pontificia-comillas/ir36pc30

[vmoens]

I think we're good to go when the linter is fixed!

[vmoens]

Do we want this?