[Bug]: remove_overlay_elements not working (function never called)

[rubenamran-arch]

crawl4ai version

0.7.2

Expected Behavior

In async_crawler_strategy.py, the remove_overlay_elements method loads the remove_overlay_elements.js script and evaluates it, but the injected source code is only wrapped in a try/catch block and never actually called.

Current (non-working) code:
try: await page.evaluate( f""" (() => {{ try {{ {remove_overlays_js} // only injected but not called return {{ success: true }}; }} catch (error) {{ return {{ success: false, error: error.toString(), stack: error.stack }}; }} }})() """ ) await page.wait_for_timeout(500) # Wait for any animations to complete

Working fix:
try: await page.evaluate( f""" (async () => {{ try {{ const removeOverlays = {remove_overlays_js}; // Injected await removeOverlays(); // called here return {{ success: true }}; }} catch (error) {{ return {{ success: false, error: error.toString(), stack: error.stack }}; }} }})() """ ) await page.wait_for_timeout(500) # Wait for any animations to complete

Current Behavior

The remove_overlay_elements flag in CrawlerConfig is not working because of this. After inspecting the source code, I found the issue above.
I’m surprised this hasn’t been reported before.

Is this reproducible?

Yes

Inputs Causing the Bug

config = CrawlerRunConfig(
            remove_overlay_elements=True,
           
        )
        b_config = BrowserConfig(
            headless=True)


        async with AsyncWebCrawler(config=b_config) as crawler:
            results = await crawler.arun_many(urls, config=config)

Steps to Reproduce

Code snippets

OS

macOs

Python version

3.12

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[SohamKukreti]

Hey, thanks for raising this issue and also providing the exact root cause of the bug!

The remove_overlay_elements functionality is indeed broken due to a critical implementation flaw.

Root Cause

The issue is in crawl4ai/async_crawler_strategy.py at lines 1374-1400:

async def remove_overlay_elements(self, page: Page) -> None:
    """
    Removes popup overlays, modals, cookie notices, and other intrusive elements from the page.

    Args:
        page (Page): The Playwright page instance
    """
    remove_overlays_js = load_js_script("remove_overlay_elements")

    try:
        await self.adapter.evaluate(page,
            f"""
            (() => {{
                try {{
                    {remove_overlays_js}  # ❌ INJECTED BUT NEVER CALLED
                    return {{ success: true }};
                }} catch (error) {{
                    return {{
                        success: false,
                        error: error.toString(),
                        stack: error.stack
                    }};
                }}
            }})()
        """
        )
        await page.wait_for_timeout(500)  # Wait for any animations to complete

Technical Analysis

1. JavaScript Structure: The crawl4ai/js_snippet/remove_overlay_elements.js file contains an async function definition:

   async () => {
       // Function to check if element is visible
       const isVisible = (elem) => { ... };
       // ... overlay removal logic
   };

2. Current Implementation Problem: The code injects the JavaScript function definition into the page but never calls it. The injected code becomes:

   (() => {
       try {
           async () => { /* overlay removal logic */ };  // ❌ Function defined but not called
           return { success: true };
       } catch (error) {
           return { success: false, error: error.toString(), stack: error.stack };
       }
   })()

3. Expected Behavior: The function should be called to actually execute the overlay removal logic.

Proposed Fix

Replace the current implementation in crawl4ai/async_crawler_strategy.py at lines 1383-1398:

# Current broken code (lines 1383-1398):
await self.adapter.evaluate(page,
    f"""
    (() => {{
        try {{
            {remove_overlays_js}
            return {{ success: true }};
        }} catch (error) {{
            return {{
                success: false,
                error: error.toString(),
                stack: error.stack
            }};
        }}
    }})()
"""
)

# Fixed code:
await self.adapter.evaluate(page,
    f"""
    (async () => {{
        try {{
            const removeOverlays = {remove_overlays_js};
            await removeOverlays();  # ✅ Actually call the function
            return {{ success: true }};
        }} catch (error) {{
            return {{
                success: false,
                error: error.toString(),
                stack: error.stack
            }};
        }}
    }})()
"""
)

Complete Fixed Method

Here's the complete corrected method:

async def remove_overlay_elements(self, page: Page) -> None:
    """
    Removes popup overlays, modals, cookie notices, and other intrusive elements from the page.

    Args:
        page (Page): The Playwright page instance
    """
    remove_overlays_js = load_js_script("remove_overlay_elements")

    try:
        await self.adapter.evaluate(page,
            f"""
            (async () => {{
                try {{
                    const removeOverlays = {remove_overlays_js};
                    await removeOverlays();
                    return {{ success: true }};
                }} catch (error) {{
                    return {{
                        success: false,
                        error: error.toString(),
                        stack: error.stack
                    }};
                }}
            }})()
        """
        )
        await page.wait_for_timeout(500)  # Wait for any animations to complete

Verification Steps

1. Current State: Function is injected but never executed
2. Expected State: Function is injected, called, and overlay elements are removed
3. Test Case: Use remove_overlay_elements=True on a page with cookie banners/popups

Why This Wasn't Caught Earlier

1. Silent Failure: No errors are thrown, making it hard to detect
2. Limited Testing: The feature may not have been thoroughly tested with actual overlay-heavy sites
3. Code Review Gap: The JavaScript injection pattern wasn't validated for proper function execution

Conclusion

This is a legitimate, high-priority bug that completely breaks the remove_overlay_elements feature. The fix is straightforward but critical for users relying on this functionality to clean pages before content extraction.

Priority: High - Should be fixed in the next patch release
Effort: Low - Simple change