Repository navigation
[Bug]: preserve_tags/preserve_classes are a no-op for excluded tags (aside, nav, footer, header, form) #2125
Description
crawl4ai version
0.9.2
Expected Behavior
PruningContentFilter(preserve_tags=["aside"]) should keep <aside> content in fit_markdown.
#1900 added these options as an escape hatch for content the fit pipeline treats as boilerplate but the user knows is real content.
Same for preserve_classes=["related"] on an <aside class="related">.
Current Behavior
Both options are silently ignored for every tag in excluded_tags (content_filter_strategy.py:101-110): nav, footer, header, aside, script, style, form, iframe, noscript.
filter_content() removes those tags before pruning ever runs:
self._remove_unwanted_tags(soup) # content_filter_strategy.py:664, decomposes aside
body = soup.find("body")
self._prune_tree(body) # :668, the only place _is_preserved() is consulted_remove_unwanted_tags decomposes by tag name and never consults the whitelist:
def _remove_unwanted_tags(self, soup): # :685
"""Removes unwanted tags"""
for tag in self.excluded_tags:
for element in soup.find_all(tag):
element.decompose()By the time _is_preserved() (:691) is reached, the node is gone from the tree.
The filter raises nothing and logs nothing.
excluded_tags is hardcoded in RelevantContentFilter.__init__, not a constructor argument, so the only workaround is mutating the instance attribute (f.excluded_tags.discard("aside")).
Is this reproducible?
Yes
Inputs Causing the Bug
- URL(s): none needed, filter_content() takes an HTML string directly
- Settings used: PruningContentFilter(preserve_tags=["aside"])
- Input data: any HTML with an <aside> holding real contentSteps to Reproduce
1. Run the snippet below (no browser, no network).
2. All three cases print "aside kept: False".
3. Expected: True for the preserve_tags and preserve_classes cases.Code snippets
from crawl4ai.content_filter_strategy import PruningContentFilter
html = (
"<html><body>"
"<article><p>" + "Main article body with plenty of words to pass the pruning threshold easily. " * 8 + "</p></article>"
'<aside class="related"><h2>Related reading</h2><p>'
+ "This sidebar note is real content the author wrote and wants kept. " * 8
+ "</p></aside>"
"</body></html>"
)
for kwargs in ({}, {"preserve_tags": ["aside"]}, {"preserve_classes": ["related"]}):
out = " ".join(PruningContentFilter(**kwargs).filter_content(html))
print(kwargs, "-> aside kept:", "Related reading" in out)
# 0.9.2 / develop @ 2d8f673:
# {} -> aside kept: False # correct, that's the default
# {'preserve_tags': ['aside']} -> aside kept: False # BUG
# {'preserve_classes': ['related']} -> aside kept: False # BUGOS
macOS
Python version
3.12
Browser
N/A
Browser version
No response
Error logs & Screenshots (if applicable)
No response
Activity
NG-PR0JECT commentedon Aug 13, 2026
Confirmed still present on
develop(checked content_filter_strategy.py today):_remove_unwanted_tags(:685) decomposes by tag name before_prune_tree(:668) ever runs, so the preserve whitelist is never consulted for excluded tags. The report is accurate — and the silent no-op is the worst part: content vanishes without a warning, which for RAG ingestion is strictly worse than a wrong extraction because it is invisible in the embedded corpus.Minimal fix: consult the same whitelist
_prune_treealready respects (:712-714) inside the exclusion loop:_is_preserved(:691) is defined just below and is signature-compatible withTag, so this is a 2-line change with no new state. One caveat:preserve_tagscurrently also gates pruning elsewhere, so reusing it here keeps behavior consistent — a<header>or<aside class="related">that is preserved will be skipped in both passes, which matches the escape-hatch intent of #1900.For anyone hitting this while building RAG pipelines: the practical workaround until fixed is the one in the report (
f.excluded_tags.discard("aside")), but note thatnegative_patterns(:113) can still prune preserved content by class/id heuristics later in the pipeline, so whitelisting by tag only is not a full guarantee. This is the exact boilerplate-vs-content tradeoff Scrapiq (https://github.com/NG-PR0JECT/scrapiq) handles with a deterministic whitelist that keeps links/metadata separate from main content — a lighter alternative when you want extraction semantics you can rely on without a pruning heuristic.bong-u commentedon Aug 13, 2026
@NG-PR0JECT
Thanks for confirming.
PR #2126 has been open since the day I filed this and does the same thing, just with one traversal instead of the nested loop:
It also adds a regression test for preserve_tags, preserve_classes, and the default case where nothing is preserved.
The negative_patterns caveat doesn't hold, though. _prune_tree returns at :714 as soon as _is_preserved(node) is true, so that node and everything under it is never scored.
And negative_patterns isn't a prune at all, it's a -0.5 class/id weight inside _compute_class_id_weight (:801-805).
After #2126, whitelisting by tag or class keeps the element through both passes.
fix(filter): respect preserve_tags in excluded_tags removal (unclecod…