Skip to content

fix(normalization): preserve blank-line runs and trailing spaces inside fenced code blocks (#2528) - #2633

Open
İlker Oğulcan SAYIN (iosayin) wants to merge 1 commit into
microsoft:mainfrom
iosayin:fix/2528-fenced-code-whitespace
Open

İlker Oğulcan SAYIN (iosayin) wants to merge 1 commit into
microsoft:mainfrom
iosayin:fix/2528-fenced-code-whitespace

Conversation

@iosayin

Copy link
Copy Markdown

Closes #2528

Summary

During final document normalization in MarkItDown, the text processing pipeline previously applied a blanket line.rstrip() and collapsed 3+ newline runs (re.sub(r"\n{3,}", "\n\n", ...)) uniformly across the entire document without distinguishing between prose and code fences. This resulted in data loss inside fenced code blocks (````` or ~~~), collapsing consecutive blank lines and stripping semantically significant trailing spaces.

Solution

  • Make final markdown normalization fence-aware: text outside code fences undergoes standard line-trimming and newline collapse, while content inside fenced code blocks preserves exact trailing whitespace and consecutive blank lines.
  • Added regression unit tests in tests/test_module_misc.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fenced code blocks lose blank-line runs and trailing spaces during conversion

1 participant