[🐛 Bug]: Selenium Manager - SE_BROWSER_MIRROR_URL/SE_DRIVER_MIRROR_URL don't suffice for downloading chrome binaries >= 115 from mirror

[tophexbit]

What happened?

DEBUG	Found chromedriver 119.0.6045.105 in PATH: /usr/bin/chromedriver
DEBUG	chrome detected at /usr/bin/google-chrome
DEBUG	Running command: /usr/bin/google-chrome --version
DEBUG	Output: "Google Chrome 119.0.6045.199 "
DEBUG	Detected browser: chrome 119.0.6045.199
DEBUG	Discovered chrome version (119) different to specified browser version (120)
DEBUG	Discovering versions from https://our.mirror/artifactory/chromedriver-meta-remote/chrome-for-testing/known-good-versions-with-downloads.json
DEBUG	Required browser: chrome 120.0.6082.0
DEBUG	Downloading chrome 120.0.6082.0 from https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/120.0.6082.0/linux64/chrome-linux64.zip

The mirrored JSON files still include the original download URLs, which results in chrome-linux64.zip still being downloaded from the original URL.

Of course this is a general issue with chromelabs/chrome-for-testing mirrors, but there are solutions for it. For example node-chromedriver uses an additional ENV variable to handle this issue:

• CHROMEDRIVER_CDNURL is analog to SE_CHROMEDRIVER_MIRROR_URL
• CHROMEDRIVER_CDNBINARIESURL has no analog in Selenium Manager

node-chromedriver uses CHROMEDRIVER_CDNBINARIESURL to put together actual mirror download URLs: https://github.com/giggio/node-chromedriver/blob/main/install.js#L131

Am I wrong to assume that without such a variable in Selenium Manager we could only solve it by providing a mirror for known-good-versions-with-downloads.json (and the other -with-downloads.json files) which supplies the files with replaced download URLs?

Would be great if Selenium Manager would handle this issue like node-chromedriver does 🙂

How can we reproduce the issue?

Use SE_CHROME_MIRROR_URL with a mirror for the JSON files of https://github.com/GoogleChromeLabs/chrome-for-testing#json-api-endpoints

Relevant log output

DEBUG	Found chromedriver 119.0.6045.105 in PATH: /usr/bin/chromedriver
DEBUG	chrome detected at /usr/bin/google-chrome
DEBUG	Running command: /usr/bin/google-chrome --version
DEBUG	Output: "Google Chrome 119.0.6045.199 "
DEBUG	Detected browser: chrome 119.0.6045.199
DEBUG	Discovered chrome version (119) different to specified browser version (120)
DEBUG	Discovering versions from https://our.mirror/artifactory/chromedriver-meta-remote/chrome-for-testing/known-good-versions-with-downloads.json
DEBUG	Required browser: chrome 120.0.6082.0
DEBUG	Downloading chrome 120.0.6082.0 from https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/120.0.6082.0/linux64/chrome-linux64.zip

Operating System

Ubuntu

Selenium version

Python webdriver-manager-4.0.1

What are the browser(s) and version(s) where you see this issue?

At least all > 114

What are the browser driver(s) and version(s) where you see this issue?

At least all > 114

Are you using Selenium Grid?

No response

[github-actions]

@daniel-stockhausen, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[titusfortner]

Let me make sure we're on the same page since some of the things here seem confusing.

1. What version of Selenium are you using?
2. Python webdriver-manager-4.0.1 is not part of Selenium and will get in the way of what Selenium is doing, please remove this for any code / logging you share
3. Selenium supports SE_DRIVER_MIRROR_URL and SE_BROWSER_MIRROR_URL environment variables (see details here: https://www.selenium.dev/documentation/selenium_manager/)
4. I raised as an issue a request to document a basic use case for what needs to be done to use it because I don't actually understand minimum requirements — [🚀 Feature]: Document custom Selenium Manager seleniumhq.github.io#1492 maybe you can help us make sure we have the right steps documented for this?

[tophexbit]

Thanks for the quick reply 🙂

1. What version of Selenium are you using?

python selenium 4.16.0

2. Python webdriver-manager-4.0.1 is not part of Selenium and will get in the way of what Selenium is doing, please remove this for any code / logging you share

You're correct, I removed it.

3. Selenium supports SE_DRIVER_MIRROR_URL and SE_BROWSER_MIRROR_URL environment variables (see details here: https://www.selenium.dev/documentation/selenium_manager/)

Yes, my original post was confusing: I wrote about chromedriver ENV vars but then posted log output of a browser download.

Explanation

• The issue is the same for chrome browser and chrome driver downloads.
• The issue is the same when using:
  • general mirror ENV vars SE_BROWSER_MIRROR_URL/SE_DRIVER_MIRROR_URL
  • chrome specific mirror ENV vars SE_CHROME_MIRROR_URL/SE_CHROMEDRIVER_MIRROR_URL

"Just" mirroring https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json generally can't work for downloading browsers/drivers from a mirror, because all these JSON files ending with with-downloads.json still contain the original download URLs:

https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/122.0.6259.0/linux64/chrome-linux64.zip
https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/122.0.6259.0/linux64/chromedriver-linux64.zip

Leading to Selenium Manager correctly using the mirror for fetching last-known-good-versions-with-downloads.json but then still downloading from edgedl.me.gvt1.com:

DEBUG   Discovering versions from https://our.mirror.org/chrome-for-testing/last-known-good-versions-with-downloads.json
DEBUG   Required browser: chrome 120.0.6099.109
DEBUG   Downloading chrome 120.0.6099.109 from https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/120.0.6099.109/linux64/chrome-linux64.zip

As an example for a possible solution: node-chromedriver uses an additional ENV var CHROMEDRIVER_CDNBINARIESURL to put together download URLs that actually use a specified mirror: https://github.com/giggio/node-chromedriver/blob/120.0.2/install.js#L131

[titusfortner]

The idea is that you create your own "last-known-good-versions-with-downloads.json" file with the download locations you want to use, and put its location in the environment variable.

That approach does seem like a lot of work but it is doable, it is much more flexible than requiring a specific "version/platform/name" format, and it is much easier for us to maintain toggling a URL.

[dongfangtianyu]

Selenium Manager 会正确地从镜像服务器获取last-known-good-versions-with-downloads.json，然后根据json文件中的下载地址，下载具体的文件。

The Selenium Manager will correctly retrieve last-known-good-versions-with-downloads.json from the mirror server and then download specific files based on the download URLs in the JSON file.

---

所以，如果想要 Selenium Manager从指定的服务器下载文件，需要完成2个步骤：
Therefore, if you want Selenium Manager to download files from a specified server, you need to complete two steps:

1，在一个镜像服务器上提供last-known-good-versions-with-downloads.json
1， Provide last-known-good-versions-with-downloads.json on a mirror server.

2，修改last-known-good-versions-with-downloads.json中的下载地址，下载地址也指向镜像服务器
2，Modify the download URLs in last-known-good-versions-with-downloads.json; the download URLs should also point to the mirror server.

我已经这么做了，目前工作得很好。
I have already done this, and it is currently working well.

另外，根据实践经验，还应该同时镜像known-good-versions-with-downloads.json
Additionally, based on practical experience, it is advisable to mirror known-good-versions-with-downloads.json simultaneously.

[tophexbit]

Thank you both for the replies. We're gonna implement it like that on our end and it'll be fine - it just still feels like the implementation of this Selenium Manager feature is incomplete and the required search-and-replace of the binary URLs should just happen on-the-fly as part of Selenium Manager. Seems unnecessary to require everyone using this feature to setup such a "special" mirror - even if not complicated in and of itself it's just another thing to maintain on the user end. But oh well 🤷‍♂️

[titusfortner]

I agree it could be easier, but we're prioritizing other features.
Pull requests are always welcome!

I opened a new ticket to track the feature request.