ufmt check fails on large directories

[calvinytong]

We have ufmt set up on a fairly large src tree and are in the process of migration. It works great for smaller folders, but on our larger folders we get the following error when running ufmt check <large_dir>. Have not had time to quantify "large", but can do so if needed

concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 151, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 156, in _add_token
    self._pop()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 221, in _pop
    new_node = self.convert_nonterminal(tos.dfa.from_rule, tos.nodes)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/python_parser.py", line 44, in convert_nonterminal
    return self.nonterminal_conversions[nonterminal](self.config, children)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/conversions/statement.py", line 532, in convert_import_as_name
    asname=AsName(
  File "<string>", line 6, in __init__
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_nodes/base.py", line 115, in __post_init__
    self._validate()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_nodes/statement.py", line 728, in _validate
    raise CSTValidationError(
  File "<string>", line None
libcst._nodes.base.CSTValidationError: There must be at least one space between 'as' and name.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/process.py", line 198, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/process.py", line 198, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/ufmt/core.py", line 84, in ufmt_file
    dst_contents = ufmt_string(path, src_contents, usort_config, black_config)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/ufmt/core.py", line 43, in ufmt_string
    content = usort_string(content, usort_config, path)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/usort/sorting.py", line 257, in usort_string
    return usort_bytes(data=data.encode(), config=config, path=path)[0].decode()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/usort/sorting.py", line 269, in usort_bytes
    mod = try_parse(data=data, path=path)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/usort/util.py", line 77, in try_parse
    raise parse_error or Exception("unknown parse failure")
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/usort/util.py", line 66, in try_parse
    mod = cst.parse_module(
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/entrypoints.py", line 70, in parse_module
    result = _parse(
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/entrypoints.py", line 50, in _parse
    result = parser.parse()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 110, in parse
    self._add_token(token)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 170, in _add_token
    raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 13:48.
Internal error: There must be at least one space between 'as' and name.

                                   conda_deploy)
                                               ^
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/calvinytong/miniconda3/envs/ctrldev/bin/ufmt", line 8, in <module>
    sys.exit(main())
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/ufmt/cli.py", line 54, in check
    results = ufmt_paths(paths, dry_run=True)
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/ufmt/core.py", line 113, in ufmt_paths
    results = list(runner.run(all_paths, fn).values())
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/site-packages/trailrunner/core.py", line 183, in run
    results = list(exe.map(func, paths))
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/Users/calvinytong/miniconda3/envs/ctrldev/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
libcst._exceptions.ParserSyntaxError: Syntax Error @ 13:48.
Internal error: There must be at least one space between 'as' and name.

                                   conda_deploy)

[JelleZijlstra]

This sounds like a bug in libcst instead. Most likely it's crashing on some specific code pattern, so I'd try to narrow it to the file it's failing on and then report the bug to libcst.

[calvinytong]

Gotcha, this turned out to be it: Instagram/LibCST#530, so closing