[RLlib][Windows] Windows Invalid Directory Name Error in Ray RLlib

[ufukyilmaz]

What happened + What you expected to happen

I encountered a NotADirectoryError on creating the temp logging directory while experimenting with the Pong environment "ale_py:ALE/Pong-v5" in Ray RLlib. The error is caused by an unsanitized reserved character (:) in the environment name, which is invalid for directory names on Windows systems.
The env name should be sanitized to remove or replace reserved characters, ensuring that the directory name is valid on all operating systems.
Here is the exception that I encountered:

Traceback (most recent call last):
  File "<redacted_path>\atari.py", line 18, in <module>
    algo = config.build()
           ^^^^^^^^^^^^^^
  File "<redacted_path>\\.venv\Lib\site-packages\ray\rllib\algorithms\algorithm_config.py", line 949, in build
    return algo_class(
           ^^^^^^^^^^^
  File "<redacted_path>\.venv\Lib\site-packages\ray\rllib\algorithms\algorithm.py", line 549, in __init__
    logdir = tempfile.mkdtemp(prefix=logdir_prefix, dir=DEFAULT_STORAGE_PATH)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted_path>\\Python311\Lib\tempfile.py", line 385, in mkdtemp
    _os.mkdir(file, 0o700)
NotADirectoryError: [WinError 267] The directory name is invalid: '<redacted_path>\/ray_results\\PPO_ale_py:ALE-Pong-v5_2024-12-28_16-20-57vy4q_uxa'

The directory name sanitization performed here is not enough to avoid having invalid directory names for the windows systems.

See the reference for the reserved characters in Windows here

...
< (less than)
> (greater than)
: (colon)
" (double quote)
/ (forward slash)
\ (backslash)
| (vertical bar or pipe)
? (question mark)
* (asterisk)

I can create a simple pull request with the following change to sanitize the environment name:

env_descr_for_dir = re.sub(r'[<>:"/\\|?*]', "-", str(env_descr))

However, I’d appreciate some guidance before proceeding. I wonder if you’d be happy with this solution or prefer a more comprehensive sanitization approach that also addresses other restrictions, such as name length. We can also consider using pathvalidate#sanitize or porting this function and tests.

Versions / Dependencies

OS: Windows

dependencies = [
   "gymnasium[atari]>=1.0.0",
   "ray[rllib]>=2.40.0",
   "torch>=2.5.1"
]

Reproduction script

from ray.rllib.algorithms import PPOConfig
from ray.rllib.connectors.env_to_module import FlattenObservations

config = (
    PPOConfig()
    .environment(
        "ale_py:ALE/Pong-v5"
    )  # Atari pong game
    .env_runners(
        num_env_runners=2,
        env_to_module_connector=lambda _: FlattenObservations(),
    )
    .evaluation(evaluation_num_env_runners=1)
)
algo = config.build()
for _ in range(5):
    print(algo.train())
algo.evaluate()

Issue Severity

Medium: It is a significant difficulty but I can work around it.