[Bug]: Docker server is not decoding or applying filter_chain

[Sjoeborg]

crawl4ai version

0.7.4

Expected Behavior

The crawl4ai server can decode filter_chain and executes the deep crawl accordingly

Current Behavior

• The docker server returns 500 if a filter_chain is specified in the deep_crawl_strategy

• Using the RESI API, the server performs the crawl, but it does not take the filter_chain into account (the crawler crawls pages even though they are matched by the filter)

Is this reproducible?

Yes

Inputs Causing the Bug

filter_chain = [
            URLPatternFilter(
                patterns=["*about-us*"],
                reverse=True,
            ),
        ]
        crawler_config = CrawlerRunConfig(
            deep_crawl_strategy=BFSDeepCrawlStrategy(
                max_depth=2, filter_chain=FilterChain(filter_chain)
            ),
            cache_mode=CacheMode.BYPASS,
        )

Steps to Reproduce

1. Set up Docker server
2. Use the Crawl4aiDockerClient and specify a BFSDeepCrawlStrategy with filter_chain

Code snippets

# CODE FOR DOCKER CLIENT ISSUE
import asyncio

from crawl4ai import (  # Assuming you have crawl4ai installed
    BrowserConfig,
    CacheMode,
    CrawlerRunConfig,
)
from crawl4ai.deep_crawling import (
    BFSDeepCrawlStrategy,
    FilterChain,
    URLPatternFilter,
)
from crawl4ai.docker_client import Crawl4aiDockerClient


async def main():
    # Point to the correct server port
    async with Crawl4aiDockerClient(
        base_url="http://xyz:11235/",
        verbose=True,
    ) as client:
        # If JWT is enabled on the server, authenticate first:
        await client.authenticate(
            "user@example.com"
        )  # See Server Configuration section

        filter_chain = [
            URLPatternFilter(
                patterns=["*about-us*"],
                reverse=True,
            ),
        ]
        crawler_config = CrawlerRunConfig(
            deep_crawl_strategy=BFSDeepCrawlStrategy(
                max_depth=2, filter_chain=FilterChain(filter_chain)
            ),
            cache_mode=CacheMode.BYPASS,
        )

        results = await client.crawl(
            ["https://httpbin.org/html"],
            browser_config=BrowserConfig(
                headless=True
            ),  # Use library classes for config aid
            crawler_config=crawler_config,
        )
        if results:  # client.crawl returns None on failure
            print(type(results))
            print(f"Non-streaming results success: {results.success}")
            if results.success:
                for result in results:  # Iterate through the CrawlResultContainer
                    print(result)
        else:
            print("Non-streaming crawl failed.")

        # Example Streaming crawl
        print("\n--- Running Streaming Crawl ---")


if __name__ == "__main__":
    asyncio.run(main())


# CODE FOR REST API ISSUE
URL_FILTERS: list = [
    "*privacy*",
    "*terms*",
    "*cookie*",
    "*contact*",
    "*support*",
    "*board*",
    "*blog*",
    "*press*",
    "*career*",
    "*about*",
    "*sustainability*",
    "*governance*",
    "*youtube*",
]
browser_config_payload = {"type": "BrowserConfig", "params": {"headless": True}}
deep_crawl_strategy_payload = {
    "type": "BFSDeepCrawlStrategy",
    "params": {
        "max_depth": 2,
        "max_pages": 10,
        "include_external": True,
        "filter_chain": {
            "type": "FilterChain",
            "params": {
                "filters": [
                    {
                        "type": "URLPatternFilter",
                        "params": {"patterns": URL_FILTERS},
                    }
                ],
            },
        },
    },
}
crawler_config_payload = {
    "type": "CrawlerRunConfig",
    "params": {
        "stream": False,
        "cache_mode": "bypass",
        "deep_crawl_strategy": deep_crawl_strategy_payload,
    },  # Use string value of enum
}


crawl_payload = {
    "urls": ["https://www.drdgold.com/investors/sens-news/2023"],
    "browser_config": browser_config_payload,
    "crawler_config": crawler_config_payload,
}
response = httpx.post(
    "http://xyz:11235/crawl",  # Updated port
    json=crawl_payload,
    timeout=90,
)
print(f"Status Code: {response.status_code}")

OS

Linux

Python version

3.12

Browser

chromium

Browser version

No response

Error logs & Screenshots (if applicable)

Traceback (most recent call last):
  File "/app/api.py", line 433, in handle_crawl_request
    crawler_config = CrawlerRunConfig.load(crawler_config)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_configs.py", line 1553, in load
    config = from_serializable_dict(data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_configs.py", line 148, in from_serializable_dict
    k: from_serializable_dict(v) for k, v in data["params"].items()
       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_configs.py", line 148, in from_serializable_dict
    k: from_serializable_dict(v) for k, v in data["params"].items()
       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_configs.py", line 148, in from_serializable_dict
    k: from_serializable_dict(v) for k, v in data["params"].items()
       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_configs.py", line 154, in from_serializable_dict
    return [from_serializable_dict(item) for item in data]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_configs.py", line 150, in from_serializable_dict
    return cls(**constructor_args)
           ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: URLPatternFilter.__init__() got an unexpected keyword argument 'simple_suffixes'

[ntohidi]

Root Cause

After investigating the code, here's the complete root cause analysis of the issue:

The Core Problem

The serialization/deserialization system has a fundamental design flaw where it treats internal implementation details the same as constructor parameters.

How the Bug Occurs

1. Client Side - Serialization Process:

• When URLPatternFilter is created, it takes patterns, use_glob, and reverse as constructor params
• Internally, it creates private attributes like _simple_suffixes, _simple_prefixes, _domain_patterns, _path_patterns for optimization (filters.py:148-151)
• These are defined in slots (filters.py:121-128)
• When .dump() is called, it uses to_serializable_dict() (async_configs.py:611-613)

2. The Serialization Bug (async_configs.py:100-106):

if hasattr(obj, '__slots__'):
    for slot in obj.__slots__:
        if slot.startswith('_'):  # Handle private slots
            attr_name = slot[1:]  # Remove leading '_'
            value = getattr(obj, slot, None)
            if value is not None:
                current_values[attr_name] = to_serializable_dict(value)

2. This code:

• Iterates through slots
• Finds private attributes (starting with _)
• Removes the underscore and adds them to serialized data
• So _simple_suffixes becomes simple_suffixes in the JSON

3. Server Side - Deserialization Process:

• Server receives the serialized data
• from_serializable_dict() attempts to recreate the object (async_configs.py:147-150):

constructor_args = {
    k: from_serializable_dict(v) for k, v in data["params"].items()
}

return cls(**constructor_args)

• It passes ALL serialized params as constructor arguments
• Including simple_suffixes, simple_prefixes, etc.

4. The Crash:

• URLPatternFilter.init() only accepts patterns, use_glob, reverse
• It doesn't recognize simple_suffixes as a valid parameter
• Results in: TypeError: URLPatternFilter.init() got an unexpected keyword argument 'simple_suffixes'

Why This Design is Flawed

The serialization system incorrectly assumes that:

1. All object attributes should be serialized
2. All serialized attributes can be passed as constructor parameters
3. Private implementation details (_simple_suffixes) are equivalent to public API parameters

This violates the principle of encapsulation - internal implementation details shouldn't be exposed in serialization.

Impact

1. Docker Client Issue: Complete failure (500 error) when using filters
2. REST API Issue: Filters might be ignored because deserialization fails silently or uses defaults

The Second Issue - Filter Not Being Applied

Even if the filter deserializes correctly, there might be a second issue where the filter chain isn't being properly applied during deep crawling. This could be because:

• The filter chain reference is lost during serialization/deserialization
• The BFS strategy isn't correctly receiving or using the filter chain
• The async filter application isn't awaited properly

Summary

The root cause is that to_serializable_dict() in async_configs.py serializes internal implementation details (from slots) that aren't constructor parameters, and from_serializable_dict() tries to use all
of them to reconstruct the object, causing a TypeError.

The fix requires either:

1. Modifying serialization to only include constructor parameters
2. Adding custom serialization methods to filter classes
3. Storing original constructor params separately for serialization

[ntohidi]

@Sjoeborg I will reopen this issue since it's still not merged with the main branch. Once we release a new version, I will close it. Thanks :)