Timeout set to 10 minutes, but timing out after 3 minutes

[githubuser100007]

🐛 Bug Report

I am setting the timeout to 10 minutes because I am using https://github.com/scrapinghub/crawlera-headless-proxy

I am seeing it timeout after 3 minutes (still downloading all of the JS/CSS files)

To Reproduce

Set page load timeout to 10 minutes

Expected behavior

Expecting it to timeout after 10 minutes, not 3.

Test script or set of commands reproducing this issue

driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));

try {
driver.get(uRL.toExternalForm());
} catch (TimeoutException te2) {
log.warn("Chrome Driver thread timed out: " + uRL.toExternalForm());
}

Environment

OS: Windows
Browser: Chrome
Browser version: Version 89.0.4389.90 (Official Build) (64-bit)
Browser Driver version: ChromeDriver 89.0.4389.23
Language Bindings version: Java 8
Selenium Grid version (if applicable): Beta 2

[githubuser100007]

Updated to reflect OS was Windows.

[githubuser100007]

This is what is being printed from Selenium Beta-2 (note the pageLoad and script timouts disregard my setting, and what is weird is that the outcome is timing out at 180000 ms, not the 600000 that was set, or the 300000 that is shown in these logs)

Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [472f80a2d4250c338c8fb9b8ade2c9be, getPageSource {}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 89.0.4389.90, chrome: {chromedriverVersion: 88.0.4324.96 (68dba2d8a0b14..., userDataDir: /tmp/.com.google.Chrome.374gW6}, goog:chromeOptions: {debuggerAddress:
localhost:39145}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: eager, platform: LINUX, platformName: LINUX, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, **timeouts: {implicit: 0, pageL
oad: 300000, script: 30000},** unhandledPromptBehavior: dismiss, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 472f80a2d4250c338c8fb9b8ade2c9be

[githubuser100007]

Here is more stack dump, again happens at 3 minutes instead of the 10 minute timeout I set:

org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException
Build info: version: '4.0.0-beta-2', revision: 'Unknown'
System info: host: 'DEV01', ip: '172.30.0.127', os.name: 'Windows Server 2016', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_231'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [12c302ddd6915400e72f0497133acb7c, get {url=https://www.walmart.com/}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 89.0.4389.90, chrome: {chromedriverVersion: 89.0.4389.23 (61b08ee2c5002..., userDataDir: C:\Users\user\AppData\Loc...}, goog:chromeOptions: {debuggerAddress: localhost:55683}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: eager, platform: WINDOWS, platformName: WINDOWS, proxy: Proxy(manual, http=tst05.de..., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 12c302ddd6915400e72f0497133acb7c
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:71)
	at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
	at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:51)
	at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
	at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
	at org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:103)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:153)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:128)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:620)
	at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:334)
	at com.util.SeleniumBrowser.browse(SeleniumBrowser.java:169)
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
	at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:206)
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:65)
	... 13 more

[diemol]

Could you please provide a complete code example to understand better what is happening? Code snippets are not enough in this case. Would be great to see how the driver is created and the url being loaded.

[githubuser100007]

Note that I use the webVitalsSnippet script injection on a driver without proxy and it works fine. So I don't think that has anything to do with it, but I included it here anyways, since that is what is happening.

    private RemoteWebDriver createDriver() throws Exception {
    	
		ChromiumDriver driver = null;

    	ChromeOptions options = new ChromeOptions();
		options.setBinary("/usr/bin/chrome"); 

		//remove unexepectedalertexceptions
		options.addArguments("--silent", "--headless", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");  
		options.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.DISMISS);

		// no downloads
		Map<String, Object> chromePrefs = new HashMap<String, Object>();
		chromePrefs.put("download_restrictions", 3);
		options.setExperimentalOption("prefs", chromePrefs);

        LoggingPreferences logPrefs = new LoggingPreferences();
        logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
        options.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
        options.setCapability( "goog:loggingPrefs", logPrefs );
        options.setPageLoadStrategy(PageLoadStrategy.EAGER);

		DriverService.Builder serviceBuilder = new ChromeDriverService.Builder().withSilent(true);
		ChromeDriverService chromeDriverService = (ChromeDriverService)serviceBuilder.build(); 
		chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));

		//running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
		Proxy proxy = new Proxy(); 
        proxy.setHttpProxy("127.0.0.1:3128"); 
        proxy.setSslProxy("127.0.0.1:3128"); 
        options.setCapability(CapabilityType.PROXY, proxy); 

		driver = new ChromeDriver(chromeDriverService, options);

		// Inject Core Web Vitals scripts
		Map<String, Object> parameters = new HashMap<String, Object>();
		parameters.put("source",webVitalsSnippet);
		driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);

		// Stop any unwanted downloading of *.pdf *.crdownload et al. files
		parameters = new HashMap<String, Object>();
		parameters.put("behavior","deny");
		driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);

    	driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
    	driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(5000));

		return driver;

    }

	public static void main(String[] args) throws Exception {

		RemoteWebDriver driver = createDriver();

		long timer = System.currentTimeMillis();
		try {
			driver.get("https://www.walmart.com");
		} catch (TimeoutException te2) {
			log.warn("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer) ); // prints approximately 3 minutes, no matter what
		} finally {
			driver.quit();
		}
		
	}

webVitalsSnippet:

var vitalsCLS = 'unset';
function logCLS({value}) {
  	vitalsCLS = value.toString();
  	console.log('CLS: ' + vitalsCLS);
}

var vitalsFID = 'unset';
function logFID({value}) {
  	vitalsFID = value.toString();
  	console.log('FID: ' + vitalsFID);
}

var vitalsLCP = 'unset';
function logLCP({value}) {
  	vitalsLCP = value.toString();
  	console.log('LCP: ' + vitalsLCP);
}

window.onload = function() {
	
	var script = document.createElement('script');
	script.src = 'https://unpkg.com/web-vitals';
  	script.onload = function() {
  	
    // When loading `web-vitals` using a classic script, all the public
    // methods can be found on the `webVitals` global namespace.
	webVitals.getCLS(logCLS, true); 
    webVitals.getFID(logFID, true); 
    webVitals.getLCP(logLCP, true); 
  
  }

  document.body.appendChild(script);

} 

[pujagani]

Thank you for sharing the script and the stack trace. In my attempt to recreate the issue, I have been running the same example that is shared with a few tweaks (commenting out the proxy bit since it was mentioned in the follow-up comment that it is not a concern here).

public static void main(String[] args) throws Exception {

    SetTimeout setTimeout = new SetTimeout();

    RemoteWebDriver driver = setTimeout.createDriver();

    long timer = System.currentTimeMillis();
    try {
        driver.get("https://www.walmart.com/");
        System.out.println(driver.getTitle());
    } catch (TimeoutException te2) {
        System.out.println("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer) ); // prints approximately 3 minutes, no matter what
    } finally {
        driver.quit();
    }

}

private RemoteWebDriver createDriver() throws Exception {

    ChromiumDriver driver = null;

    ChromeOptions options = new ChromeOptions();
   // options.setBinary("/usr/bin/chrome");

    //remove unexepectedalertexceptions
    options.addArguments("--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");
    options.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.DISMISS);

    // no downloads
    Map<String, Object> chromePrefs = new HashMap<String, Object>();
    chromePrefs.put("download_restrictions", 3);
    options.setExperimentalOption("prefs", chromePrefs);

    LoggingPreferences logPrefs = new LoggingPreferences();
    logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
    options.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
    options.setCapability( "goog:loggingPrefs", logPrefs );
    options.setPageLoadStrategy(PageLoadStrategy.EAGER);

    DriverService.Builder serviceBuilder = new ChromeDriverService.Builder().withLogLevel(ChromeDriverLogLevel.INFO);
    ChromeDriverService chromeDriverService = (ChromeDriverService)serviceBuilder.build();
    chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));

    //running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
//        Proxy proxy = new Proxy();
//        proxy.setHttpProxy("127.0.0.1:3128");
//        proxy.setSslProxy("127.0.0.1:3128");
//        options.setCapability(CapabilityType.PROXY, proxy);

    driver = new ChromeDriver(chromeDriverService, options);

     //Inject Core Web Vitals scripts
//        Map<String, Object> parameters = new HashMap<String, Object>();
//        parameters.put("source", "var vitalsCLS = 'unset';\n" +
//                "function logCLS({value}) {\n" +
//                "  \tvitalsCLS = value.toString();\n" +
//                "  \tconsole.log('CLS: ' + vitalsCLS);\n" +
//                "}\n" +
//                "\n" +
//                "var vitalsFID = 'unset';\n" +
//                "function logFID({value}) {\n" +
//                "  \tvitalsFID = value.toString();\n" +
//                "  \tconsole.log('FID: ' + vitalsFID);\n" +
//                "}\n" +
//                "\n" +
//                "var vitalsLCP = 'unset';\n" +
//                "function logLCP({value}) {\n" +
//                "  \tvitalsLCP = value.toString();\n" +
//                "  \tconsole.log('LCP: ' + vitalsLCP);\n" +
//                "}\n" +
//                "\n" +
//                "window.onload = function() {\n" +
//                "\t\n" +
//                "\tvar script = document.createElement('script');\n" +
//                "\tscript.src = 'https://unpkg.com/web-vitals';\n" +
//                "  \tscript.onload = function() {\n" +
//                "  \t\n" +
//                "    // When loading `web-vitals` using a classic script, all the public\n" +
//                "    // methods can be found on the `webVitals` global namespace.\n" +
//                "\twebVitals.getCLS(logCLS, true); \n" +
//                "    webVitals.getFID(logFID, true); \n" +
//                "    webVitals.getLCP(logLCP, true); \n" +
//                "  \n" +
//                "  }\n" +
//                "\n" +
//                "  document.body.appendChild(script);\n" +
//                "\n" +
//                "} ");
//        driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);

     //Stop any unwanted downloading of *.pdf *.crdownload et al. files
//        parameters = new HashMap<String, Object>();
//        parameters.put("behavior","deny");
//        driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);

    WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
    System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
    WebDriver.Timeouts scriptTimeout  = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
    System.out.println(scriptTimeout.getScriptTimeout().toMillis());

    return driver;

} 

Logs:

Mar 30, 2021 7:39:27 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Mar 30, 2021 7:39:28 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
INFO: Found exact CDP implementation for version 89
600000
600000
Verify your identity

When the page load timeout is set to 2 milliseconds as shown below :

WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(2));
System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
WebDriver.Timeouts scriptTimeout  = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
System.out.println(scriptTimeout.getScriptTimeout().toMillis());

Logs:

Mar 30, 2021 7:46:25 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Mar 30, 2021 7:46:25 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
INFO: Found exact CDP implementation for version 89
2
600000
Chrome Driver thread timed out: 36

I have run the same two code samples and checked the debug logs and the timeout values are set as expected with the Chrome driver.

[githubuser100007]

An early timeout is not the problem. It's when you set it to 10 minutes that is the problem.

I'm wondering if the chromedriver doesn't support changing its timeouts, so anything earlier than it's default of 3 minutes is okay but anything after 3 minutes the chrome driver times out no matter what at 3 minutes.

[pujagani]

As Diego rightly pointed out, the issue seems to be due to the default values of HTTP client config. The read timeout defaults to 3 mins. Hence as you mentioned, the issue springs up when the timeout is set higher than 3 mins.
The issue is not with the Chrome driver since Chrome driver debug logs show the timeout being set correctly and when the value is set less than 3 mins the value is considered.
The solution for this is to configure the client timeout when creating the driver using the RemoteWebDriverBuilder.

private RemoteWebDriver createDriver() throws Exception {

  RemoteWebDriver driver = null;

  ChromeOptions options = new ChromeOptions();
  // options.setBinary("/usr/bin/chrome");

  //remove unexepectedalertexceptions
  options.addArguments("--unexpectedAlertBehaviour=dismiss","--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");

  // no downloads
  Map<String, Object> chromePrefs = new HashMap<String, Object>();
  chromePrefs.put("download_restrictions", 3);
  options.setExperimentalOption("prefs", chromePrefs);

  LoggingPreferences logPrefs = new LoggingPreferences();
  logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
  //options.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
  options.setCapability("goog:loggingPrefs", logPrefs );
  options.setPageLoadStrategy(PageLoadStrategy.EAGER);

  DriverService.Builder serviceBuilder = new ChromeDriverService.Builder();
  ChromeDriverService chromeDriverService = (ChromeDriverService) serviceBuilder.build();
  chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));

  //running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
  Proxy proxy = new Proxy();
  proxy.setHttpProxy("127.0.0.1:3128");
  proxy.setSslProxy("127.0.0.1:3128");
  //options.setProxy(proxy);

  //driver = new ChromeDriver(chromeDriverService, options);

  driver = (RemoteWebDriver) RemoteWebDriver
          .builder()
          .config(ClientConfig.defaultConfig()
                  .connectionTimeout(Duration.ofMinutes(10))
                  .readTimeout(Duration.ofMinutes(10)))
          .withDriverService(chromeDriverService)
          .addAlternative(options)
          .setCapability("proxy", proxy)
          .build();

   //Inject Core Web Vitals scripts
//        Map<String, Object> parameters = new HashMap<String, Object>();
//        parameters.put("source", "var vitalsCLS = 'unset';\n" +
//                "function logCLS({value}) {\n" +
//                "  \tvitalsCLS = value.toString();\n" +
//                "  \tconsole.log('CLS: ' + vitalsCLS);\n" +
//                "}\n" +
//                "\n" +
//                "var vitalsFID = 'unset';\n" +
//                "function logFID({value}) {\n" +
//                "  \tvitalsFID = value.toString();\n" +
//                "  \tconsole.log('FID: ' + vitalsFID);\n" +
//                "}\n" +
//                "\n" +
//                "var vitalsLCP = 'unset';\n" +
//                "function logLCP({value}) {\n" +
//                "  \tvitalsLCP = value.toString();\n" +
//                "  \tconsole.log('LCP: ' + vitalsLCP);\n" +
//                "}\n" +
//                "\n" +
//                "window.onload = function() {\n" +
//                "\t\n" +
//                "\tvar script = document.createElement('script');\n" +
//                "\tscript.src = 'https://unpkg.com/web-vitals';\n" +
//                "  \tscript.onload = function() {\n" +
//                "  \t\n" +
//                "    // When loading `web-vitals` using a classic script, all the public\n" +
//                "    // methods can be found on the `webVitals` global namespace.\n" +
//                "\twebVitals.getCLS(logCLS, true); \n" +
//                "    webVitals.getFID(logFID, true); \n" +
//                "    webVitals.getLCP(logLCP, true); \n" +
//                "  \n" +
//                "  }\n" +
//                "\n" +
//                "  document.body.appendChild(script);\n" +
//                "\n" +
//                "} ");
//        driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);

   //Stop any unwanted downloading of *.pdf *.crdownload et al. files
//        parameters = new HashMap<String, Object>();
//        parameters.put("behavior","deny");
//        driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);

  WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
  System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
  WebDriver.Timeouts scriptTimeout  = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
  System.out.println(scriptTimeout.getScriptTimeout().toMillis());

  return driver;

}

Can you please try as shown above and see if changing the client config values helps solve the issue you are facing? Thank you! Please provide feedback on the same.

[githubuser100007]

I'm getting the following error when building the driver the new way:

java.lang.IllegalArgumentException: Illegal key values seen in w3c capabilities: [loggingPrefs, unexpectedAlertBehaviour]
at org.openqa.selenium.remote.NewSessionPayload.lambda$validate$5(NewSessionPayload.java:203)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:372)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.openqa.selenium.remote.NewSessionPayload.validate(NewSessionPayload.java:207)
at org.openqa.selenium.remote.NewSessionPayload.(NewSessionPayload.java:152)
at org.openqa.selenium.remote.NewSessionPayload.create(NewSessionPayload.java:99)
at org.openqa.selenium.remote.RemoteWebDriverBuilder.getPayload(RemoteWebDriverBuilder.java:465)
at org.openqa.selenium.remote.RemoteWebDriverBuilder.getRemoteDriver(RemoteWebDriverBuilder.java:362)
at org.openqa.selenium.remote.RemoteWebDriverBuilder.build(RemoteWebDriverBuilder.java:339)