#1181: UrlUpdater fetches every defined version for Docker

[mbilda]

Closes: #1181

• deleted the new line patterns out of the regex in DockerDesktopUrlUpdater because Pattern.DOTALL already handles this
• TBH I do not know why this pattern now works, because it also should have worked with the (\n\r...) regex. Maybe both, regex and DOTALL don't work together
• changed the group to group1 because there is only one group left

With this implementation i was able to create versionized folders in my custom repository for the defined docker versions:

This still needs to be tested with workflow because test <-> workflow isn't the same environment and so on.

We were also able to test the "real" UrlUpdater via github actions. We created a custom github action for testing the url updater. We checked out the url repository to fetch all existing versions and were able to upload/download the temporarily produced new repo.
The results can be seen here:
https://github.com/mbilda/IDEasy/actions/runs/13834518742

[coveralls]

Pull Request Test Coverage Report for Build 18510598572

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 5 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.2%) to 68.597%

---

Files with Coverage Reduction	New Missed Lines	%
com/devonfw/tools/ide/url/tool/docker/DockerDesktopUrlUpdater.java	5	81.82%

Totals
Change from base Build 18493975671:	0.2%
Covered Lines:	9040
Relevant Lines:	12687

---

💛 - Coveralls

[hohwille]

@mbilda thank you very much for your PR. You did a great job and very well analysed the problem. Even you tested the URL Updater via GHA in your fork. Excellent 🚀

However, I checked the logs of your test run and found e.g. this line:

12:28:42.793 [com.devonfw.tools.ide.url.UpdateInitiator.main()] - WARN  - c.d.t.i.u.updater.AbstractUrlUpdater - For tool docker and version 4.33.2 the download verification failed with status code 403 for URL https://desktop.docker.com/win/main/amd64/179689/Docker%20Desktop%20Installer.exe.

It seems that there are a lot of such errors and I can also reproduce this in my browser:

My assumption is that after merging this we get dedicated versions of docker desktop but no URL for Windows and therefore we will entirely break it for Windows users.
Could you confirm my observations and maybe find a solution to improve your PR?

[hohwille]

See also this final summary:

docker/docker versions added: 0 failed, 98 succeeded, 98 total, 0.00% error - versions verified: 17 failed, 56 succeeded, 73 total, 23.29% error

[mbilda]

@hohwille good point.
So the error occurs because there is no version to download for windows at all. We still find the version because it is listed in the release notes, but there is no download link provided for windows - only for mac.
See here (example for 179689 - 4.33.2):

That means there is no download link at all and thats why we receive this error, even when we directly pasting the download url into the browser.

Why do we even try to download this version?
The regex:

String regex = "href=#" + version
        // .....................................................(Group.1.)..........
        + ".{8,12}.{0,350}href=https://desktop\\.docker\\.com.*?(\\d{5,6}).*\\.exe";

The crucial part of this is the following:
.*\\.exe
It matches as many characters as needed until it finds .exe
The next match which will be found is actually the next version where a windows version exists.
See here (regex101 example matching two versions)
https://regex101.com/r/hEmRX3/2
At this point we do not know the "actual docker version" (179689) - thats why we use the regex above to extract it.
Even though it is working, we receive lots of errors with 403, because we try to download a non-existing file.

TODO:

• We need a more specific regex to match exactly the stuff we need - not more, not less.
  • How could we do that?
    • We could delete the .exe of the regex to match the first finding of the docker-version-number. As we see in the above example, it doesn't matter if we have the .exe in the regex. It just searches for anything until it finds .exe - which will definitely be the case at some time. But at this point we already have what we want - the docker-version-number
    • We then could use the found group-match to execute a new regex but with specific windows urls or mac urls and so on (we actually do this in a different kind of way - we just try to download it - which is kinda finding it - but it fails)
    • After another regex, we can then download it, if it exists or not

Furthermore, if the download of the non-existing windows version fails, we return out of the addVersions()-function and skipping the "maybe existing" other versions for mac.

Questions:
Should we provide versions only for one OS at all?

• Like in the example above, if we wouldn't return out of the function we would just download the 4.33.2 versions for mac and failing with windows. But then only mac-versions would exist in ide - good or bad?

[hohwille]

so far we do not have dedicated versions of docker-desktop:
https://github.com/devonfw/ide-urls/tree/master/docker/docker/

That means this PR is adding this as a new feature.
I found that the web-site you are crawling (https://docs.docker.com/desktop/release-notes/) also contains checksum links pointing to SHA256 checksums.
You should also use these checksums since otherwise our fallback applies to compute them ourselves what requires that we download every entire release and compute the checksum ourselves what causes gigabytes of wasted bandwidth and makes the process also much slower.
See here for an existing example:

IDEasy/url-updater/src/main/java/com/devonfw/tools/ide/url/tool/intellij/IntellijUrlUpdater.java

Lines 83 to 84 in a44dc22

	String cs = getCheckSum(checkSumLink);
	doAddVersion(url, link, os, systemArchitecture, cs);

[mbilda]

This PR extends on the fly because there are several open points to discuss to handle the new url updater for docker desktop.
Open points:

• the more regexes and compiles of the regexes, the more ide loses performance
• We need the checksum check as well
• Do we really need docker desktop? Isn't it enough to install docker (cli)
• Should we really pull all versions? (There are versions which only fixes bugs for mac -> only download links for mac)

This PR handles two individual things. The first one is the fix for the actual bug. IDE could not handle latest version. Because it is resolved to " * ". The second one is the improvement of the url updater which throws a lot of open questions.
I will split this PR into two parts.
First part: Fix the actual bug
Second part: Url updater improvement

To get things done, i will exclude the first part of this PR and create a new PR for this fix only. (#1182)
I also will detach the issue from this PR and create a new issue for all open points and improvement of the url updater and link it to this PR. (#1181)

[mbilda]

You should also use these checksums since otherwise our fallback applies to compute them ourselves what requires that we download every entire release and compute the checksum ourselves what causes gigabytes of wasted bandwidth and makes the process also much slower.

I added the checksum to the addVersion method.
For each checksum we need to curl the checksum url and get the first part out of it.
We only do this if the version exists, a check that we are already doing.

[jan-vcapgemini]

Thanks for adding the checksum check too. I've added a small suggestion, please check.

[jan-vcapgemini]

LGTM.