Error with Crawler closing the webrowser suddenly on `arun()` and `BrowserContext.new_page`

[arousi]

[ERROR] 🚫 arun(): Failed to crawl https://docs.djangoproject.com/en/5.1, error: BrowserContext.new_page: Target page, context or browser has been closed

this also appeared in another query:

URL: https://docs.djangoproject.com/en/5.1/intro/tutorial01

[ERROR] 🚫 arun(): Failed to crawl https://docs.djangoproject.com/en/5.1/intro/tutorial01, error: BrowserContext.new_page: Target page, context or browser has been closed
Browser logs:

<launching> /root/.cache/ms-playwright/chromium-1134/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate,HttpsUpgrades,PaintHolding,PlzDedicatedWorker --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --disable-search-engine-choice-screen --unsafely-disable-devtools-self-xss-warnings --headless=old --hide-scrollbars --mute-audio --blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4 --no-sandbox --no-sandbox --disable-dev-shm-usage --no-first-run --no-default-browser-check --disable-infobars --window-position=0,0 --ignore-certificate-errors --ignore-certificate-errors-spki-list --user-data-dir=/tmp/playwright_chromiumdev_profile-envk2i --remote-debugging-pipe --no-startup-window
<launched> pid=406
[pid=406][err] Old Headless mode will be removed from the Chrome binary soon. Please use the new Headless mode (https://developer.chrome.com/docs/chromium/new-headless) or the chrome-headless-shell which is a standalone implementation of the old Headless mode (https://developer.chrome.com/blog/chrome-headless-shell).
[pid=406][err] 
[pid=406][err] [0327/225803.239038:FATAL:zygote_main_linux.cc(145)] Check failed: sandbox::ThreadHelpers::IsSingleThreaded(). 
[pid=406][err] #0 0x7f3f47732990 base::debug::CollectStackTrace()
[pid=406][err] #1 0x7f3f477214d4 base::debug::StackTrace::StackTrace()
[pid=406][err] #2 0x7f3f4767d268 logging::LogMessage::Flush()
[pid=406][err] #3 0x7f3f4767d16c logging::LogMessage::~LogMessage()
[pid=406][err] #4 0x7f3f47666dec logging::(anonymous namespace)::CheckLogMessage::~CheckLogMessage()
[pid=406][err] #5 0x7f3f47666e4c logging::(anonymous namespace)::CheckLogMessage::~CheckLogMessage()
[pid=406][err] #6 0x7f3f47666be0 logging::CheckError::~CheckError()
[pid=406][err] #7 0x7f3f46bec79c content::ZygoteMain()
[pid=406][err] #8 0x7f3f46be5eac content::RunZygote()
[pid=406][err] #9 0x7f3f46be6bd0 content::RunOtherNamedProcessTypeMain()
[pid=406][err] #10 0x7f3f46be7a38 content::ContentMainRunnerImpl::Run()
[pid=406][err] #11 0x7f3f46be5580 content::RunContentProcess()
[pid=406][err] #12 0x7f3f46be5a3c content::ContentMain()
[pid=406][err] #13 0x7f3f47181b48 headless::HeadlessShellMain()
[pid=406][err] #14 0x7f3f42e6947c ChromeMain
[pid=406][err] #15 0x400001987740 (/usr/lib/aarch64-linux-gnu/libc.so.6+0x2773f)
[pid=406][err] #16 0x400001987818 __libc_start_main
[pid=406][err] #17 0x7f3f42e69034 (/root/.cache/ms-playwright/chromium-1134/chrome-linux/chrome+0x463c033)
[pid=406][err] 
[pid=406][err] [0327/225803.235080:FATAL:zygote_main_linux.cc(145)] Check failed: sandbox::ThreadHelpers::IsSingleThreaded(). 
[pid=406][err] #0 0x7f24d4a42990 base::debug::CollectStackTrace()
[pid=406][err] #1 0x7f24d4a314d4 base::debug::StackTrace::StackTrace()
[pid=406][err] #2 0x7f24d498d268 logging::LogMessage::Flush()
[pid=406][err] #3 0x7f24d498d16c logging::LogMessage::~LogMessage()
[pid=406][err] #4 0x7f24d4976dec logging::(anonymous namespace)::CheckLogMessage::~CheckLogMessage()
[pid=406][err] #5 0x7f24d4976e4c logging::(anonymous namespace)::CheckLogMessage::~CheckLogMessage()
[pid=406][err] #6 0x7f24d4976be0 logging::CheckError::~CheckError()
[pid=406][err] #7 0x7f24d3efc79c content::ZygoteMain()
[pid=406][err] #8 0x7f24d3ef5eac content::RunZygote()
[pid=406][err] #9 0x7f24d3ef6bd0 content::RunOtherNamedProcessTypeMain()
[pid=406][err] #10 0x7f24d3ef7a38 content::ContentMainRunnerImpl::Run()
[pid=406][err] #11 0x7f24d3ef5580 content::RunContentProcess()
[pid=406][err] #12 0x7f24d3ef5a3c content::ContentMain()
[pid=406][err] #13 0x7f24d4491b48 headless::HeadlessShellMain()
[pid=406][err] #14 0x7f24d017947c ChromeMain
[pid=406][err] #15 0x400001987740 (/usr/lib/aarch64-linux-gnu/libc.so.6+0x2773f)
[pid=406][err] #16 0x400001987818 __libc_start_main
[pid=406][err] #17 0x7f24d0179034 (/root/.cache/ms-playwright/chromium-1134/chrome-linux/chrome+0x463c033)
[pid=406][err] 
[pid=406][err] qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
[pid=406][err] qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
[pid=406][err] [0327/225810.874810:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[pid=406][err] [0327/225810.937846:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[pid=406][err] [0327/225810.942531:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[pid=406][err] [0327/225811.551743:WARNING:bluez_dbus_manager.cc(234)] Floss manager service not available, cannot set Floss enable/disable.
[pid=406][err] [0327/225811.565648:ERROR:gpu_process_host.cc(974)] GPU process launch failed: error_code=1002
[pid=406][err] [0327/225811.566228:WARNING:gpu_process_host.cc(1414)] The GPU process has crashed 1 time(s)
[pid=406][err] [0327/225811.738134:ERROR:gpu_process_host.cc(974)] GPU process launch failed: error_code=1002
[pid=406][err] [0327/225811.738364:WARNING:gpu_process_host.cc(1414)] The GPU process has crashed 2 time(s)
[pid=406][err] [0327/225811.845772:ERROR:gpu_process_host.cc(974)] GPU process launch failed: error_code=1002
[pid=406][err] [0327/225811.846013:WARNING:gpu_process_host.cc(1414)] The GPU process has crashed 3 time(s)
[pid=406][err] [0327/225811.863081:ERROR:gpu_process_host.cc(974)] GPU process launch failed: error_code=1002
[pid=406][err] [0327/225811.863357:WARNING:gpu_process_host.cc(1414)] The GPU process has crashed 4 time(s)
[pid=406][err] [0327/225811.880458:ERROR:gpu_process_host.cc(974)] GPU process launch failed: error_code=1002
[pid=406][err] [0327/225811.880671:WARNING:gpu_process_host.cc(1414)] The GPU process has crashed 5 time(s)
[pid=406][err] [0327/225811.914149:ERROR:gpu_process_host.cc(974)] GPU process launch failed: error_code=1002
[pid=406][err] [0327/225811.914352:WARNING:gpu_process_host.cc(1414)] The GPU process has crashed 6 time(s)
[pid=406][err] [0327/225812.000510:ERROR:gpu_process_host.cc(974)] GPU process launch failed: error_code=1002
[pid=406][err] [0327/225812.000767:WARNING:gpu_process_host.cc(1414)] The GPU process has crashed 7 time(s)
[pid=406][err] [0327/225812.015766:ERROR:gpu_process_host.cc(974)] GPU process launch failed: error_code=1002
[pid=406][err] [0327/225812.015979:WARNING:gpu_process_host.cc(1414)] The GPU process has crashed 8 time(s)
[pid=406][err] [0327/225812.030257:ERROR:gpu_process_host.cc(974)] GPU process launch failed: error_code=1002
[pid=406][err] [0327/225812.030560:WARNING:gpu_process_host.cc(1414)] The GPU process has crashed 9 time(s)
[pid=406][err] [0327/225812.030820:FATAL:gpu_data_manager_impl_private.cc(436)] GPU process isn't usable. Goodbye.
[pid=406][err] #0 0x7fba02241990 base::debug::CollectStackTrace()
[pid=406][err] #1 0x7fba022304d4 base::debug::StackTrace::StackTrace()
[pid=406][err] #2 0x7fba0218c268 logging::LogMessage::Flush()
[pid=406][err] #3 0x7fba0218ccdc logging::LogMessageFatal::~LogMessageFatal()
[pid=406][err] #4 0x7fba000f9764 content::(anonymous namespace)::IntentionallyCrashBrowserForUnusableGpuProcess()
[pid=406][err] #5 0x7fba000f6a64 content::GpuDataManagerImplPrivate::FallBackToNextGpuMode()
[pid=406][err] #6 0x7fba000f568c content::GpuDataManagerImpl::FallBackToNextGpuMode()
[pid=406][err] #7 0x7fba00100b60 content::GpuProcessHost::RecordProcessCrash()
[pid=406][err] #8 0x7fba00102060 content::GpuProcessHost::OnProcessLaunchFailed()
[pid=406][err] #9 0x7fb9ffef1f80 content::BrowserChildProcessHostImpl::OnProcessLaunchFailed()
[pid=406][err] #10 0x7fb9fff678e4 content::internal::ChildProcessLauncherHelper::PostLaunchOnClientThread()
[pid=406][err] #11 0x7fb9fff67c68 base::internal::DecayedFunctorTraits<>::Invoke<>()
[pid=406][err] #12 0x7fba021dac0c base::TaskAnnotator::RunTaskImpl()
[pid=406][err] #13 0x7fba021f486c base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl()
[pid=406][err] #14 0x7fba021f4410 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
[pid=406][err] #15 0x7fba022569dc base::MessagePumpGlib::Run()
[pid=406][err] #16 0x7fba021f4f08 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run()
[pid=406][err] #17 0x7fba021bcd38 base::RunLoop::Run()
[pid=406][err] #18 0x7fb9fff13218 content::BrowserMainLoop::RunMainMessageLoop()
[pid=406][err] #19 0x7fb9fff14d2c content::BrowserMainRunnerImpl::Run()
[pid=406][err] #20 0x7fba082cd0a4 headless::HeadlessContentMainDelegate::RunProcess()
[pid=406][err] #21 0x7fba016f583c content::RunBrowserProcessMain()
[pid=406][err] #22 0x7fba016f6e40 content::ContentMainRunnerImpl::RunBrowser()
[pid=406][err] #23 0x7fba016f6a5c content::ContentMainRunnerImpl::Run()
[pid=406][err] #24 0x7fba016f4580 content::RunContentProcess()
[pid=406][err] #25 0x7fba016f4a3c content::ContentMain()
[pid=406][err] #26 0x7fba01c90a64 headless::HeadlessShellMain()
[pid=406][err] #27 0x7fb9fd97847c ChromeMain
[pid=406][err] #28 0x400001987740 (/usr/lib/aarch64-linux-gnu/libc.so.6+0x2773f)
[pid=406][err] #29 0x400001987818 __libc_start_main
[pid=406][err] #30 0x7fb9fd978034 (/root/.cache/ms-playwright/chromium-1134/chrome-linux/chrome+0x463c033)
[pid=406][err] Task trace:
[pid=406][err] #0 0x7fb9fff6772c content::internal::ChildProcessLauncherHelper::PostLaunchOnLauncherThread()
[pid=406][err] #1 0x7fb9fff66b68 content::internal::ChildProcessLauncherHelper::StartLaunchOnClientThread()
[pid=406][err] #2 0x7fba0252a38c mojo::SimpleWatcher::Context::Notify()
[pid=406][err] #3 0x7fba0252a38c mojo::SimpleWatcher::Context::Notify()
[pid=406][err] #4 0x7fba0252a38c mojo::SimpleWatcher::Context::Notify()
[pid=406][err] Task trace buffer limit hit, update PendingTask::kTaskBacktraceLength to increase.
[pid=406][err] 
[pid=406][err] qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped

We also have dependency issues for some libraries between the crawler and the frontend:

crawl4ai 0.5.0.post8 requires aiofiles>=24.1.0, but you have aiofiles 23.2.1 which is incompatible.
pyopenssl 25.0.0 requires cryptography<45,>=41.0.5, but you have cryptography 3.2.1 which is incompatible

and if i try to resolve the dependency conflict manually, it turns into a dependency circular conflict:

crawl4ai 0.5.0.post8 requires aiofiles>=24.1.0, but you have aiofiles 23.1.0 which is incompatible.
gradio 4.39.0 requires markupsafe~=2.0, but you have markupsafe 3.0.2 which is incompatible.

tried this with 2 links, one in devdocs.io the other on django website docs, both failed.

so Crawl4Ai and Gradio might be crashing either each other for this conflict. Hope this helps the team

P.S:
issues with the lora after make run make-run.txt , manually checked it in the directory and it is still there intact,

[Shubham-Khichi]

Hi @arousi are you running this through docker ?

[arousi]

Yes, i specifically have elevated cmd for docker, another elevated for venv. Both running, only docker cmd gave me 1 tiny warning bc of ARM, Nothing else in the Docker CMD

Elevated docker-cmd.md

@Shubham-Khichi

[RimoVR]

I just did a fresh install using docker on Windows 11 and get the same error on my first tries:
[ERROR] 🚫 arun(): Failed to crawl https://bevyengine.org/learn/migration-guides/introduction, error: BrowserContext.new_page: Target page, context or browser has been closed

[Shubham-Khichi]

Hi @arousi and @RimoVR I found out the reason for your error. Seems like GPU is crashing

I want you to try a fix:

1. Go to docker-compose.yml file
2. Line In the latest Docker release of Craw4AI, two specific challenges have been identified #62 after Just installed, using crawl producing errors in created .md files. #61 which is called MAX_CONCURRENT_TASKS=5 like the image

3. Then punch in this one liner : - PLAYWRIGHT_CHROMIUM_LAUNCH_ARGS=--disable-gpu --no-sandbox

Then run the docker-compose.yml and let me know what errors you are seeing.

Explanation: Fixing the Crawl4ai Browser Crash

Problem Summary

The crawl4ai service, responsible for web crawling, is failing intermittently or consistently during operation. The primary error message observed in the backend service logs is:

[ERROR] 🚫 arun(): Failed to crawl [URL], error: BrowserContext.new_page: Target page, context or browser has been closed

This indicates that the web browser instance managed by crawl4ai is terminating unexpectedly before or during the crawling process.

Root Cause Analysis

Detailed logs from the crawl4ai service container (running the unclecode/crawl4ai:all image) reveal underlying crashes within the headless Chromium browser used by Playwright:

1. GPU Process Failures: The logs show repeated errors like GPU process launch failed: error_code=1002 and The GPU process has crashed X time(s), ultimately leading to FATAL:gpu_data_manager_impl_private.cc(436)] GPU process isn't usable. Goodbye.. This is a strong indicator that Chromium is attempting to use hardware GPU acceleration for rendering, but failing within the Docker container environment, which typically lacks direct and stable access to the host's GPU.
2. Sandbox/Zygote Issues: Errors like FATAL:zygote_main_linux.cc(145)] Check failed: sandbox::ThreadHelpers::IsSingleThreaded() suggest potential problems with initializing Chromium's security sandbox process (Zygote) within the constraints of the container environment (possibly exacerbated by running under QEMU emulation, as hinted by other log lines).

The combination of these issues leads to the browser process crashing, causing the Target page... closed error seen by the backend service trying to use it.

Proposed Fix

The proposed solution is to modify the crawl4ai service definition in the main docker-compose.yml file by adding an environment variable:

services:
  crawl4ai:
    image: unclecode/crawl4ai:all
    container_name: devdocs-crawl4ai
    ports:
      - "11235:11235"
    environment:
      - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-devdocs-demo-key}
      - MAX_CONCURRENT_TASKS=5
      # Add this line: Attempt to pass browser arguments via environment variables
      - PLAYWRIGHT_CHROMIUM_LAUNCH_ARGS=--disable-gpu --no-sandbox
      - DISABLE_AUTH=false
    # ... rest of the definition

Mechanism of Fix (How it's Expected to Work)

This environment variable aims to pass specific command-line flags to the Chromium browser when Playwright launches it inside the crawl4ai container:

• --disable-gpu: This flag instructs Chromium not to use hardware GPU acceleration. It forces the browser to use software rendering (CPU-based). This directly prevents the GPU process initialization failures and subsequent crashes observed in the logs. While potentially slower, it significantly increases stability in environments without reliable GPU access.
• --no-sandbox: This flag disables Chromium's security sandbox. This is often required for stability when running in restricted container environments where the sandbox cannot initialize properly, potentially addressing the Zygote-related errors. (Note: The logs indicated this flag might already be in use internally, but explicitly setting it via the environment variable ensures it's applied if the variable takes precedence).
• PLAYWRIGHT_CHROMIUM_LAUNCH_ARGS: This specific environment variable name is a convention sometimes used by applications built with Playwright to allow users to inject browser launch arguments. We are assuming the unclecode/crawl4ai:all image respects this variable. If it doesn't, this fix will have no effect.

Scope and Limitations

• Global Change: This modification is being made to the main docker-compose.yml file. Therefore, this configuration change will apply to all users who use this file to run the application stack.
• Browser Crash Only: This fix specifically targets the browser crashing issue (GPU and potentially sandbox related).
• Dependency Conflicts Unresolved: This fix does not address the separate Python dependency conflicts (e.g., aiofiles, cryptography, gradio, markupsafe) that were also noted. These conflicts exist within the pre-built unclecode/crawl4ai:all image's Python environment and would require changes to the image itself (or reporting the issue to the image maintainer) to be resolved. They might still cause other runtime issues.

[C-Digi]

I have the same issue. Applied your fix using PLAYWRIGHT_CHROMIUM_LAUNCH_ARGS=--disable-gpu --no-sandbox but nothing changed, I still get the same error.

[Shubham-Khichi]

Hi @C-Digi can you share your GPU allocation from docker? IT should be just like this image

[C-Digi]

I am in Windows with default settings and I don't have those options. I don't have a .wslconfig file made.

[arousi]

+1 as @C-Digi , none worked here @Shubham-Khichi

[Shubham-Khichi]

+1 as @C-Digi , none worked here @Shubham-Khichi

I'm looking into it, windows installation is experimental as I am on Mac only. I'm learning about these issues from you guys. Thank you for being patient with this.