[Bug]: CrawlRunConfig doesn't work

[TejaCherukuri]

crawl4ai version

0.4.248

Expected Behavior

I am trying to exclude the html tags and external links while scraping the web page. Following is how I defined my CrawlerRunConfig()

run_config = CrawlerRunConfig(
            excluded_tags=["header", "footer", "nav", "aside", "script", "style"], # Remove entire tag blocks
            exclude_external_links=True
        )

I am using it as

async with AsyncWebCrawler() as crawler:
            # Run the crawler on a URL
            result = await crawler.arun(
                url="https://www.example.com",
                run_config=run_config
            )

I expect to not see any header, footer, or links inside my markdown.

Current Behavior

I could see everything on the webpage, it just doesn't filter out. Using below

run_config = CrawlerRunConfig(
            excluded_tags=["header", "footer", "nav", "aside", "script", "style"], # Remove entire tag blocks
            exclude_external_links=True
        )
async with AsyncWebCrawler() as crawler:
            # Run the crawler on a URL
            result = await crawler.arun(
                url="https://www.example.com",
                run_config=run_config
            )

is as good as using

async with AsyncWebCrawler() as crawler:
            # Run the crawler on a URL
            result = await crawler.arun(
                url="https://www.example.com"
            )

What might be reason?

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

I am running inside jupyter notebook, hence using nest_asyncio. See entire code snippet below

Code snippets

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
import nest_asyncio

nest_asyncio.apply()

async def main():
    try:
        run_config = CrawlerRunConfig(
            excluded_tags=["header", "footer", "nav", "aside", "script", "style"], # Remove entire tag blocks
            exclude_external_links=True
        )
        # Create an instance of AsyncWebCrawler
        async with AsyncWebCrawler() as crawler:
            # Run the crawler on a URL
            result = await crawler.arun(
                url="https://www.example.com",
                run_config=run_config
            )

            # Extracted markdown content
            markdown_content = result.markdown

            # Define output file path
            output_file = "output.md"

            # Write the markdown content to a file
            with open(output_file, "w", encoding="utf-8") as file:
                file.write(markdown_content)

            print(f"Markdown content successfully saved to {output_file}")

    except Exception as e:
        print(f"Error occurred: {e}")

# Run the async main function
await main()

OS

macOS

Python version

3.10.9

Browser

Chrome

Browser version

132.0.6834.112

Error logs & Screenshots (if applicable)

No response

[aravindkarnam]

@TejaCherukuri

1. There is an issue in your code, the CrawlerRunConfig has to be passed to the config keyword argument, but you are passing it to some run_config argument(referring to code snippet you shared). In this case you would not have received any complaint from Crawler because it's considering None as value of CrawlerRunConfig.
2. Still I verified the issue you shared. So if you do not disable cache, but change the excluded_tags for same URL, then previously cached results(with previous run's excluded_tags) will be returned in the result. So make sure to clear cache when you change anything in CrawlerRunConfig. Other than this excluded_tags is working exactly as expected.

I can see this is a problem, we'll figure out something for this.

For now closing the issue, since this is not a valid bug.

[TejaCherukuri]

@aravindkarnam

1. Thanks for pointing that out! My bad, I wonder how I missed it. However, even after testing now, I still face same issue. I ensured I disabled cache after each subsequent call. I am crawling link and trying to exclude nav tag. However, I am getting same output with/without the config, I mean full output. Attaching my output file too.

crawl4ai_2.md

Below is my code:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def crawl():
    try:
        run_config = CrawlerRunConfig(
            excluded_tags=["nav"],  # Remove entire tag blocks
            disable_cache=True
        )
        # Create an instance of AsyncWebCrawler
        async with AsyncWebCrawler() as crawler:
            # Run the crawler on a URL
            result = await crawler.arun(
                url="https://crawl4ai.com/mkdocs/api/arun/",
                config=run_config
            )

            # Extracted markdown content
            markdown_content = result.markdown

            # Define output file path
            output_file = "crawl4ai_2.md"

            # Write the markdown content to a file
            with open(output_file, "w", encoding="utf-8") as file:
                file.write(markdown_content)

            print(f"Markdown content successfully saved to {output_file}")

    except Exception as e:
        print(f"Error occurred: {e}")

def main():
    asyncio.run(crawl())

if __name__ == "__main__":
    main()

[aravindkarnam]

@TejaCherukuri It's working as expected for me! I've even checked result.cleaned_html along with markdown, none of the nav tags got picked up. Can you quickly bump your version to v0.4.3bx and try again. Reopen this issue if you still the problem.