Formatting isn't applied across the whole file

[deepyaman]

Python Version

3.10.11

Package Version

1.16.0

Description

When I format https://github.com/deepyaman/ibis/blob/build/blacken-docs-issue/ibis/expr/operations/udf.py using blacken-docs, only one code block seems to be getting parsed:

diff --git a/ibis/expr/operations/udf.py b/ibis/expr/operations/udf.py
index 09f6e41ee..bac4033ae 100644
--- a/ibis/expr/operations/udf.py
+++ b/ibis/expr/operations/udf.py
@@ -191,7 +191,8 @@ class scalar(_UDF):
         >>> import ibis
         >>> @ibis.udf.scalar.builtin
         ... def hamming(a: str, b: str) -> int:
-        ...     '''Compute the Hamming distance between two strings.'''
+        ...     """Compute the Hamming distance between two strings."""
+        ...
         >>> expr = hamming("duck", "luck")
         >>> con = ibis.connect("duckdb://")
         >>> con.execute(expr)

I would expect all of the remaining code blocks to also get formatted (similar issues with missing blank line and docstring quoting).

[adamchainz]

Yeah, I can't really spot why the final block isn't being picked up. Could you minimize the file down to the smallest failing example? That would probably reveal the problem.

[deepyaman]

Yeah, I can't really spot why the final block isn't being picked up. Could you minimize the file down to the smallest failing example? That would probably reveal the problem.

@adamchainz First off, thanks for getting back to me so quickly, and sorry for taking so long to reply! Kept slipping my mind.

Second, it looks like the issue is actually more significant. Here's a fairly minimal file, foo.py:

import operator


def accumulate(iterable, func=operator.add, *, initial=None):
    """Returns running totals.

    ```pycon
    >>> def mul(a, b):
    ...     '''Return ``a * b``, for *a* and *b* numbers.'''
    ...     return a * b
    ...
    >>> list(accumulate([1, 2, 3, 4, 5], mul))
    [1, 2, 6, 24, 120]

    ```
    """
    it = iter(iterable)
    total = initial
    if initial is None:
        try:
            total = next(it)
        except StopIteration:
            return
    yield total
    for element in it:
        total = func(total, element)
        yield total


def accumulate2(iterable, func=operator.add, *, initial=None):
    """Returns running totals.

    ```pycon
    >>> def mul(a, b):
    ...     '''Return ``a * b``, for *a* and *b* numbers.'''
    ...     return a * b
    ...
    >>> list(accumulate([1, 2, 3, 4, 5], mul))
    [1, 2, 6, 24, 120]

    ```
    """
    return accumulate(iterable, func, initial=initial)

(It works; python -m doctest -v foo.py will tell you "Test passed.")

Once again, blacken-docs foo.py only formats the first docstring:

diff --git a/foo.py b/foo.py
index 4434688d7..2bf750329 100644
--- a/foo.py
+++ b/foo.py
@@ -6,7 +6,7 @@ def accumulate(iterable, func=operator.add, *, initial=None):
 
     ```pycon
     >>> def mul(a, b):
-    ...     '''Return ``a * b``, for *a* and *b* numbers.'''
+    ...     """Return ``a * b``, for *a* and *b* numbers."""
     ...     return a * b
     ...
     >>> list(accumulate([1, 2, 3, 4, 5], mul))

Without having debugged it, or having bothered to look into how blacken-docs works, my guess is (correct me if I'm wrong) that it iterates over docstrings, and it doesn't do anything past the first docstring because the first change results in invalid code. The above change actually shouldn't be applied, because the leading """ in mul closes the accumulate docstring.

[adamchainz]

I think I just fixed the “rest of the file” bug in #347.

But the docstring is still being left in a broken state, and that isn't something we can easily prevent. The snippet passed to Black has no context that it's in a docstring, and I don't fancy trying to fix it up after the fact. Perhaps we should have a guard for ''' - I will give that a try.