Skip to content

[Bug]: md-fit strips meaningful content metadata (usernames, attribution) not just boilerplate #1900

Closed
Closed
[Bug]: md-fit strips meaningful content metadata (usernames, attribution) not just boilerplate#1900
Labels
⚙ DoneBug fix, enhancement, FR that's completed pending release✨ EnhancementImprovement on an existing feature

Description

@mattheworiordan
mattheworiordan
opened on Apr 6, 2026

crawl4ai version

0.8.6

Expected Behavior

md-fit should strip page chrome (navigation, footers, sidebars, cookie banners) but preserve meaningful content metadata like author names and attribution on user-generated content. Comment attribution (who said what) is semantic content, not boilerplate.

Current Behavior

md-fit strips comment author usernames and profile links from GitHub pages while keeping the comment body and date. The output shows commented Apr 6, 2026 with no indication of who wrote it.

With -o md, you get:

**[mattheworiordan](https://gist.github.com/mattheworiordan)** commented Apr 6, 2026

With -o md-fit, you get:

commented Apr 6, 2026

The username and profile link are gone. On pages with multiple commenters, it's impossible to tell who said what.

Is this reproducible?

Yes

Inputs Causing the Bug

- URL: https://gist.github.com/mattheworiordan/99ba717f7a8ca5a7f838913722cfe7ac
- Settings: `-o md-fit` (default fit mode, no other flags)

Steps to Reproduce

# 1. Crawl the gist with full markdown - username preserved
crwl crawl https://gist.github.com/mattheworiordan/99ba717f7a8ca5a7f838913722cfe7ac -o md

# Look for: **[mattheworiordan](...)** commented Apr 6, 2026

# 2. Same URL with md-fit - username stripped
crwl crawl https://gist.github.com/mattheworiordan/99ba717f7a8ca5a7f838913722cfe7ac -o md-fit

# Look for: commented Apr 6, 2026  (no username)

Code snippets

# Not a code bug - this is CLI / markdown generation behaviour.
# Reproduces purely via crwl CLI as shown above.

OS

macOS (Darwin 25.2.0)

Python version

3.14.3

Browser

(default Chromium managed by crawl4ai)

Browser version

(crawl4ai default)

Error logs & Screenshots (if applicable)

n/a

Activity

  1. mattheworiordan
    added
    🐞 BugSomething isn't working
    🩺 Needs TriageNeeds attention of maintainers
    on Apr 6, 2026
  2. hafezparast
    added a commit that references this issue on Apr 7, 2026

    feat: add preserve_classes/preserve_tags to PruningContentFilter (unc…

    8ff2fcb

  3. hafezparast commented on Apr 7, 2026

    @hafezparast
    hafezparast
    on Apr 7, 2026
    SponsorContributor

    This is expected behavior of the PruningContentFilter — it scores each HTML block on text density, link ratio, tag weight, and class patterns. Short metadata elements (usernames, timestamps) score low on density and get pruned alongside actual boilerplate.

    On your GitHub gist page, the author <span class="author"> is short text, heavily linked, and the class matches the "comment" negative pattern — all of which push the score below the 0.48 threshold.

    PR #1904 adds preserve_classes and preserve_tags params to PruningContentFilter so you can whitelist elements that should always be kept:

    from crawl4ai import CrawlerRunConfig, DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import PruningContentFilter

config = CrawlerRunConfig(
    markdown_generator=DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(
            preserve_classes=["author", "comment-header"],
            preserve_tags=["time"],
        )
    )
)

    Immediate workarounds (before the PR merges):

    1. Lower the threshold: PruningContentFilter(threshold=0.35) — keeps more content but also more noise
    2. Use BM25ContentFilter(user_query="author comments") instead — relevance-based, not density-based

    To test the whitelist before merge:

    pip install git+https://github.com/hafezparast/crawl4ai.git@fix/maysam-pruning-preserve-whitelist-1900
  4. ntohidi
    added
    ⚙️ In-progressIssues, Features requests that are in Progress
    and removed
    🩺 Needs TriageNeeds attention of maintainers
    on Apr 7, 2026
  5. ntohidi
    added
    ✨ EnhancementImprovement on an existing feature
    and removed
    🐞 BugSomething isn't working
    on Apr 24, 2026
  6. ntohidi
    added a commit that references this issue on Jul 7, 2026

    Merge pull request #1904 from hafezparast/fix/maysam-pruning-preserve…

    1c6370b
  7. ntohidi
    added
    ⚙ DoneBug fix, enhancement, FR that's completed pending release
    and removed
    ⚙️ In-progressIssues, Features requests that are in Progress
    on Jul 7, 2026

  8. ntohidi commented on Jul 7, 2026

    @ntohidi
    ntohidi
    on Jul 7, 2026
    SponsorCollaborator

    This is now on develop via PR #1904 and will be included in the next release. Thanks for the report, @mattheworiordan.

  9. ntohidi commented on Jul 8, 2026

    @ntohidi
    ntohidi
    on Jul 8, 2026
    SponsorCollaborator

    This has been fixed and is available in the latest release, v0.9.1.

  10. ntohidi
    closed this as completedon Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ⚙ DoneBug fix, enhancement, FR that's completed pending release✨ EnhancementImprovement on an existing feature

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions