fix(markdown): preserve hard line breaks

[aaarif796]

Description

This PR fixes the Markdown parsing side of the hard line break round-trip
described in #4011.

A hard line break in GitHub Flavored Markdown (GFM) can be represented using
two trailing spaces before a newline:

Author 1  
Affiliation 1

The expected Docling representation is a single TextItem containing an
embedded newline:

TextItem.text == "Author 1\nAffiliation 1"

Previously, the Markdown backend could interpret the content after the hard
line break as a separate TextItem, resulting in a loss of the distinction
between an intra-block hard line break and a paragraph/document-item
boundary.

This PR preserves the hard line break as \n inside the appropriate
TextItem.

---

Related Issue

Resolves #4011

---

Problem

The Markdown round-trip consists of two directions:

DoclingDocument
      │
      │ serialize
      ▼
   Markdown
      │
      │ parse
      ▼
DoclingDocument

For this round-trip to be lossless, an embedded newline in a TextItem must
have the same semantics in both directions.

For example, a TextItem containing:

"Author 1\nAffiliation 1"

represents one text block with an intra-block line break.

It is different from two separate text items:

doc.add_text(
    label=DocItemLabel.TEXT,
    text="Author 1",
)

doc.add_text(
    label=DocItemLabel.TEXT,
    text="Affiliation 1",
)

The Markdown representation of an embedded hard line break is:

Author 1  
Affiliation 1

where the two spaces before the newline are significant.

---

Investigation

I traced the problem through the Markdown parser and Docling document model
to understand where the hard line break was being lost.

1. Docling text-item representation

I first inspected DoclingDocument.add_text() and the TextItem structure.

Every call to add_text() creates a separate TextItem.

Therefore:

doc.add_text(
    label=DocItemLabel.TEXT,
    text="Author 1",
)

doc.add_text(
    label=DocItemLabel.TEXT,
    text="Affiliation 1",
)

produces two separate document items.

Whereas:

doc.add_text(
    label=DocItemLabel.TEXT,
    text="Author 1\nAffiliation 1",
)

keeps the newline inside one text item.

This confirmed that the parser must preserve a hard line break inside the same
TextItem.

2. Marko AST investigation

I then inspected how Marko parses Markdown line breaks.

For:

Author 1  
Affiliation 1

Marko produces:

marko.inline.LineBreak

with:

soft=False

For a normal soft Markdown line break, Marko produces the same node type with:

soft=True

This provides a semantic distinction between hard and soft line breaks without
having to manually inspect the original Markdown source.

The implementation therefore uses:

if isinstance(element, marko.inline.LineBreak):
    if not element.soft:
        # hard line break

3. Inline formatting investigation

I also tested:

Author **John**  
University XYZ

Marko represents formatted content as separate inline nodes.

Therefore, reconstructing the complete paragraph as plain text would risk losing
formatting information.

The implementation keeps the existing inline-node processing and introduces
only small state to represent a pending hard line break.

---

Root Cause

The Markdown backend already processed:

marko.inline.LineBreak

but did not preserve the semantic difference between:

LineBreak(soft=True)

and:

LineBreak(soft=False)

As a result, an explicit hard line break was not carried forward when the
following text was processed.

The parser could consequently create a new TextItem instead of keeping the
newline within the existing text item.

---

Implementation

A pending hard-line-break state was added to the Markdown backend:

self._pending_hard_line_break = False

When Marko reports an explicit hard line break:

elif isinstance(element, marko.inline.LineBreak):
    if self.in_table:
        _log.debug("Line break in a table")
        self.md_table_buffer.append("")
    elif not element.soft:
        _log.debug("Hard line break")
        self._pending_hard_line_break = True

The newline is not immediately added as a separate document item.

Instead, the backend waits for the following text content.

When the next text node is processed, the pending state is consumed. If the
previous text item belongs to the same parent, the following text is appended
using \n:

previous.text += "\n" + snippet_text
previous.orig += "\n" + snippet_text

Otherwise, normal add_text() behavior is retained.

The pending state is then reset:

self._pending_hard_line_break = False

This preserves the hard line break without introducing an unnecessary
paragraph-level TextItem.

---

Example

Before

Input:

Author 1  
Affiliation 1

The parser could produce:

TextItem("Author 1")
TextItem("Affiliation 1")

which loses the distinction between a hard line break and separate document
items.

After

The parser produces:

TextItem("Author 1\nAffiliation 1")

This matches the expected Docling representation.

---

Formatted Example

Input:

Author **John**  
University XYZ

The parser continues to preserve the formatting of John while also
preserving the hard line break.

The implementation operates on Marko's inline AST rather than flattening the
paragraph into plain text.

---

Tables

Table handling is kept separate.

The existing table-specific behavior remains unchanged:

if self.in_table:
    ...

Hard-line-break handling is therefore not applied as normal paragraph text
while processing table content.

---

Soft Line Breaks

Normal soft line breaks are intentionally not treated as hard line breaks.

Marko distinguishes them using:

element.soft

Therefore:

element.soft is True

continues through the existing behavior, while:

element.soft is False

is treated as an explicit hard line break.

This avoids changing existing semantics for ordinary Markdown line wrapping.

---

Code and Formula Content

The implementation is limited to normal inline text processing.

Existing handling for code blocks, code spans, formulas, and other specialized
document elements remains unchanged.

---

Tests

Regression tests were added to ensure the behavior remains stable.

Plain hard line break

def test_convert_hard_line_break():
    markdown = "Author 1  \nAffiliation 1"

    doc = _convert_markdown(markdown, MarkdownBackendOptions())

    assert len(doc.texts) == 1
    assert doc.texts[0].text == "Author 1\nAffiliation 1"

This verifies that the two Markdown lines remain inside a single TextItem.

Hard line break with formatting

def test_convert_hard_line_break_preserves_formatting():
    markdown = "Author **John**  \nUniversity XYZ"

    doc = _convert_markdown(markdown, MarkdownBackendOptions())

This verifies that hard-line-break handling does not break existing inline
formatting behavior.

---

Verification

Targeted hard-line-break test

uv run pytest tests/test_backend_markdown.py::test_convert_hard_line_break -q

Result:

1 passed

Formatted hard-line-break test

uv run pytest tests/test_backend_markdown.py::test_convert_hard_line_break_preserves_formatting -q

Result:

1 passed

Full Markdown backend test suite

uv run pytest tests/test_backend_markdown.py -q

Result:

17 passed, 1 skipped, 1 warning

The warning is from the existing
test_convert_leading_dash_sequences test and is unrelated to this change.

---

Code Quality Checks

The repository checks were run with:

uv run prek run --all-files

The following checks passed:

trim trailing whitespace
fix end of files
check for merge conflicts
check for added large files
Ruff linter
Ruff formatter
ty
Tach

Two repository hooks invoke python3 directly:

tach-module-coverage
max-lines

On the Windows development environment, python3 is not available as a
command.

The underlying scripts were therefore verified directly using the project's
Python environment:

uv run python scripts/check_tach_module_coverage.py

and:

uv run python scripts/check_max_lines.py --max-lines=1500

Both completed successfully.

No repository hook configuration was changed for this platform-specific issue.

---

Files Changed

docling/backend/md_backend.py

Updated Markdown AST processing to recognize explicit hard line breaks and
preserve them as embedded newlines in the appropriate TextItem.

tests/test_backend_markdown.py

Added regression coverage for:

• Plain hard line breaks.
• Hard line breaks combined with inline formatting.

---

Scope

This PR focuses specifically on the Markdown parsing side of the round-trip
described in #4011.

Markdown serialization is handled separately by docling-core.

No changes were made to the Markdown serializer in this repository.

---

Acceptance Criteria

• Markdown hard line breaks represented by
  LineBreak(soft=False) are recognized.
• Hard line breaks are preserved as \n.
• The newline remains inside the appropriate TextItem.
• Unnecessary separate TextItems are avoided.
• Inline formatting is preserved.
• Existing table handling remains unchanged.
• Existing Markdown tests continue to pass.
• Regression test added for a plain hard line break.
• Regression test added for a formatted hard line break.
• Ruff linting and formatting pass.
• ty passes.
• Tach passes.
• Underlying tach-module-coverage check passes.
• Underlying max-lines check passes.

---

Checklist

• Documentation has been updated, if necessary.
• Examples have been added, if necessary.
• Tests have been added, if necessary.

[github-actions]

✅ DCO Check Passed

Thanks @aaarif796, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[ceberam]

Thanks @aaarif796 for suggesting this PR. This is part of a broader issue as described in #4011
The PR on docling-core should be first accepted and merged, then docling-core should make a new release. We would then have a clear picture of how docling's markdown backend should be refactored.
The link to #4011 resolution should be removed since this PR should not automatically close the issue (the issue resolution depends on another PR).
Therefore, I will set this PR as Draft until we merge docling-project/docling-core#721

[aaarif796]

Thanks @aaarif796 for suggesting this PR. This is part of a broader issue as described in #4011 The PR on docling-core should be first accepted and merged, then docling-core should make a new release. We would then have a clear picture of how docling's markdown backend should be refactored. The link to #4011 resolution should be removed since this PR should not automatically close the issue (the issue resolution depends on another PR). Therefore, I will set this PR as Draft until we merge docling-project/docling-core#721

Thanks @ceberam for the detailed clarification