Standalone comments still partially broken

[sumezulike]

Describe the bug

Specific placements of standalone comments, primarily in list comprehensions, will cause parsing errors.
These are actually formatting errors; black generates invalid code on the first run of _format_str_once, which then can't be parsed.

To Reproduce

I stumbled over two cases while working on #4026:

[
    x for
    # comment
    x, y in ["AB",]
]

causes

Cannot parse: 9:0: ]

This is because the code is first split at # comment, leaving

    x, y in ["AB",]

to delimiter_split, which doesn't know about the comprehension and adds a trailing comma after the ].

[
    [
        x for x
        # comment
        in [
            # comment
            "ABC"
        ]
    ]
]

This looks similar, but leads to a different error:

Cannot parse: 7:0: EOF in multi-line statement

This is the code produced in the first formatting run:

[
    [x for x
    # comment
    in [# comment"ABC"]]
]

Besides the broken comment, the style also looks pretty weird to me.
This is somehow related to the way append_safe doesn't allow comments if there is a comprehension or lambda in the line and can also be triggered with a lambda:

[(lambda 
    # comment
    x: [
        # comment
        1
    ]
)]

There's other weirdness here. I haven't fully understood why this happens yet, but couldn't reduce the input further.

Expected behavior
The code should be formatted successfully and correctly.

Suggestions

I feel like there are probably more edge cases in which black expects a line to have a certain structure and crashes if standalone comments chop it up arbitrarily. It might be better to mostly ignore standalone comments and fix them as a last step.

Any other solution would have to give lines more context about their origin.
For example, the input

[
    x, y in ["AB",]
]

causes the exact same call to delimiter_split as the first example, but here the trailing comma is valid and correct.

Environment

$ black --version
black, 24.3.0 (compiled: yes)
Python (CPython) 3.12.1

Additional context