Lightning CLI argparse error with specific python versions on 3.11 and 3.12

[nilsleh]

🐛 Bug report

We using the lightning CLI for a package we are trying to develop called lightning-uq-box. For testing we also utilize the CLI and recently encountered CI pytests failing on python 3.11. Tests are passing on main, where the CI uses python 3.11.8, but on more recent PRs, python 3.11.9 is picked and we encounter the following error:

 lightning_uq_box/main.py:11: in get_uq_box_cli
    return LightningCLI(
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lightning/pytorch/cli.py:378: in __init__
    self.parse_arguments(self.parser, args)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/lightning/pytorch/cli.py:528: in parse_arguments
    self.config = parser.parse_args(args)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/jsonargparse/_deprecated.py:124: in patched_parse
    cfg = parse_method(*args, _skip_check=_skip_check, **kwargs)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/jsonargparse/_core.py:390: in parse_args
    cfg, unk = self.parse_known_args(args=args, namespace=cfg)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/jsonargparse/_core.py:261: in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/argparse.py:2128: in _parse_known_args
    start_index = consume_optional(start_index)


    def consume_optional(start_index):
    
        # get the optional identified at this index
        option_tuple = option_string_indices[start_index]
>       action, option_string, sep, explicit_arg = option_tuple
E       ValueError: not enough values to unpack (expected 4, got 3)

To reproduce

I have made a separate PR branch, where I have just made a random change to the README.md where one can see that the python version on the PR used is 3.11.9 while on the main branch it is still 3.11.8 (where the tests pass). The same thing happens for python 3.12 now, where 3.12.3 breaks, but 3.12.2 (on main) still passes.

Environment

Specified also in requirements/tests.yaml

• jsonargparse version: jsonargparse[signatures]==4.27.5
• Python version: 3.11, 3.12
• How jsonargparse was installed : pip install
• OS: Ubuntu, Mac, Linux

@adamjstewart

[adamjstewart]

https://docs.python.org/3.12/whatsnew/changelog.html#python-3-12-3-final contains a few argparse-related changes. I'm guessing jsonargparse relies on undocumented or untested argparse internals and something changed.

[adamjstewart]

It looks like this specific commit broke things for us: python/cpython@c02b7ae, python/cpython#114180

[mauvilsa]

Thank you for reporting! This issue has already been fixed in #486. The fix will be included in a release tomorrow or the day after.