New approach for ide-mirrors

[hohwille]

This epic is about replacing ide-mirrors with ide-urls. The latter has been created from scratch for this epic.
The rationale is that ide-mirrors had several design flaws in its structure that prevented us from automatic updating and required manual intervention. The idea of both mirrors and urls is to provide the meta-data of all the various tools we support with their available versions and URLs where to download which version for what operating system and architecture.
With the new approach of ide-urls we want to make a clean cut and redesign the entire structure of the metadata such that we get rid of manual maintenance and our end-users benefit from getting always the latest meta-data what especially means the latest available version information. The benefit of creating a new ide-urls repository is that we can keep the existing ide-mirrors untouched so existing releases of devonfw-ide out there that still rely on it do not get broken by our changes. Starting with release 2023.07.001 the new ide-urls will be used and ide-mirrors becomes legacy. For downward compatibility we will leave ide-mirrors online even for a long time after the new release is published. However, after a year or so it could also be deleted.

So far we have implemented an approach in get-versions script that computes the available-versions from scratch every time via curl download and pipe magic to extract the versions from formats such as JSON, do specific replacements, filter unstable/undesired versions, sort versions, etc.

The actual details can be found in the comments below.

I marked this complex issue as an Epic and therefore created sub-tasks for the actual stories to break down the epic:

• create domain model for ide-urls #1007
• create URL checker #1008
• Create crawler to determine available versions and according download URLs #1009
• Github action to update ide-urls #1035
• Adapt scripts to use ide-urls #1084
• Align tool names in scripts #1085
• Align tool names in ide-urls #1086
• Create migration script to switch from mirrors to urls #1087
• Align editions in ide-urls #1088
• Add updater for GCloud #1176
• wrong checksum in ide-urls #1142
• add yarn to ide-urls #1180
• rename azure to az in ide-urls #1181
• verify checksum in doDownload #1182
• Not required anymore: Add version handling to commandlet pip #1170 (the blocker part has been resolved, the rest can also be fixed later)
• Rancher Desktop download file changes from exe to msi #1200
• Use reliable ways to determine tool versions for intellij #1132
• eclipse milestone version not working #1210
• purge git history of ide-urls #1213
• gradle download broken #1218
• ide-urls not updated before being used #1229
• Edition backward compatibility #1242

[hohwille]

For the record we could even change more radically:
Create a new repo ide-urls (as replacement for ide-mirrors) that is used by newer versions of devonfw-ide and has a new structure.

urls/«tool»/«edition»/«version»/[«os»[_«arch»].]urls

So each version would be a folder containing config files. That would directly contain the complete download URL(s) for the exact version, OS, architecture. If no matching «os»_«arch».urls file is found, «os».urls is used, if again not found urls is used.
This way no variable substitution, code hacks, etc. would be required and we do not have problems when the URL pattern differs per (major) version. The generation of that new mirrors would be more complex and its size would be more greedy (lots of redudant URL-prefixes) but doDownload would be much simpler and future problems could be solved easily that we might not even be aware of as of today.

[hohwille]

Actually only adding the "latest" version does not work or solve our issue.
As we can see e.g. in devonfw/ide-mirrors@2dca8e4 there are hot- or security-fixes that get published for multiple "main" versions as we can see in the commit for python.
Still we should consider the main idea of this story and modify it so that we can come up with an improvement.

In that linked commit we can also see that the eclipse versions 2019-06, 2019-03, 2018-12, and 2018-09 have been removed. We should analyze if those can still be downloaded or really disappeared...

[hohwille]

After we continue to have various bugs and complex problems computing the download URLs for all combinations of OS, Architecture, Version, Edition and we face that providers do not follow conventions or break their own conventions for single releases or from one version to the next, etc. I come to the conclusion that we really should create a new mirrors git repository from scratch with a new layout as suggested above.

We should consider using editions as sub-folders (intellij/community and intellij/ultimate) vs. top-folders (intellij-community and intellij-ultimate). The sub-folders have the advantage that we structurally express that there is a tool (intellij) that has mutliple editions (community and ultimate). Whereas with top-folders we can not derive this in a generic way.

We could even think if we want to express that for docker we have two editions docker and rancher resulting in docker/docker and docker/rancher.
If we want to make it even more generic, we start every «tool» with a sub-folder named after the tool itself what is the "default edition". Then we can later on add new editions without breaking anything. For docker, we can still choose in the commandlet, that rancher is the default edition.
Nice thing is that we already have the variable DOCKER_EDITION for this. We could also consider the value DOCKERDESKTOP as "deprecated" but still supported if we want to choose the subfolder as edition and re-map DOCKERDESKTOP to docker then and use rancher as default if undefined.
In the same sense we could consider INTELLIJ_EDITION_TYPE as deprecated in favor of a new INTELLIJ_EDITION variable that will point to the sub-folders community (default) or ultimate. But for compatibility if INTELLIJ_EDITION is undefined but INTELLIJ_EDITION_TYPE is set, we can remap it (C=community and U=ultimate).

This leads us to the schema:

urls/«tool»/«edition»/«version»/[«os»[_«arch»]].urls

We could even automatically find the URLs file in the «version» folder via the following approach:

• «os»_«arch».urls
• «os».urls (fallback if above not found)
• urls (fallback if above not found, typically then only this file exists and OS also does not matter)

Values for «os» are always:

• windows for Windows
• mac for MacOS
• linux for Linux

Values for «arch» would be:

• x32 for 32-bit x86 architecture (i386, i686)
• x64 for 64-bit x86 architecture (amd64, x68_64)
• arm64 for 64-bit ARM architecture (A1/A2 on MacOS, aarch64)

In devonfw-ide itself the new approach would have the following implications per use-case:

• list versions: instead of getting from available-versions file, we list the «version» folders, we have to sort them semantically every time ourselves if we do not want to keep a redundant available-versions file (what I would suggest as it is redundant).
• get latest version: same as above but get the first/latest one after sorting
• doInstall/doDownload: significant simplification. We do not even have to do any os-mapping and arch-mapping in devonfw-ide itself as we would always strictly apply to the defaults. We do not have to pass any information from the commandlet anymore like - for OS in order to tell that something is OS indenpendent as all this will be figured out automatically via the above specified approach. Even better, we stop having 404 errors for mirror URLs that never exist but by random we sometimes try as they only ship specific release versions (applies for eclipse or java).

On the other hand the github action workflow for this new mirrors repo crawling for new versions and computing the URLs, etc. would get somewhat more complex. Here we still have to deal with things like Adoptium vs. Temurin, etc.
However, we can use APIs that already provide the full download URLs for a specific release versions wherever possible.
Also my suggestion would be the following when crawling for all versions of a tool:

• if the version already exists and the URLs do not change, everything keeps as is (delete missing file if exist, see below)
• if a new version was detected that we did not know before, we test all download URLs and only if we successfully downloaded, we will add the according URL (per OS, arch, mirror, edition, ...).
• if a version got removed, we simply add a missing file in the «version» folder if it does not exist and write the current date (YYYY-MM-DD) to it. However, if we get the version again in a new workflow run, we delete the file. So if the file exists, we can easily see for how long the version is gone. We can then even implement to delete the entire «version» folder if that happened for e.g. 30 days or more. In that process we could still test the download URL(s) again. If they still work, we only log a WARNING and do not delete the version. We can then manually react on such WARNING and decide if we want to delete manually or not.
• in order to somehow allow manual interaction, we could have a manual file in the «version» folder that indicates that this version is maintained manually and we only add this if we manually add this via git commit. Then the version folder gets ignored and no missing file is created and nothing gets deleted automatically.
• this is still in discussion, we could also have a status file per *.urls file that can handle the manual and missing case together and even distinguish per *.urls file. So we can notice if a version only disappears for a specific OS or architecture. Further we could have some check that tests an URL after ~30 days if it still works and tracks success or error information together with date-information

Example for the last bullet if we decide to do so e.g. in file windows_x64.urls.status:
UPDATE: As we have already decided to implement this with Java instead of bash and the status is only relevant for the Java part but not for devonfw-ide bash scripts, we can use a proper JSON structure for a simple and single status.json file in the version folder:

{
  "manual": false,
  "«url-hash»": {
    "success": {
      "timestamp": "2022-10-01T01:23:59Z"
    },
    "error": {
      "timestamp": "2022-10-01T01:23:59Z"
      "message": "404 Not Found"
    }
  },
  "«url-hash2»": {
    ...
  }
  ...
}

~

[hohwille]

For the record: In 2022 we can even decide to drop support for 64 vs. 32 bit architecture.
However, we still need to consider arch as on MacOS we can have x86 (64Bit) or arm (64Bit) what requires different tool binaries.

Therefore relevant combinations are currently:

• windows_x64
• linux_x64
• mac_x64
• mac_arm64

[hohwille]

I created the new repository here:
https://github.com/devonfw/ide-urls

[hohwille]

Currently for mirrors workflow we have good support for github API.
Another important source for tool downloads is maven central:
https://repo1.maven.org/maven2/

It also has several mirrors:

• https://repo.maven.apache.org/maven2

The great thing is that is follows very strict conventions.
So devonfw-ide scripts can be found here:
https://repo1.maven.org/maven2/com/devonfw/tools/ide/devonfw-ide-scripts/

And we can simply get the versions here:
https://repo1.maven.org/maven2/com/devonfw/tools/ide/devonfw-ide-scripts/maven-metadata.xml

So urls/ide/ide/2022.08.003/urls could contain:

https://repo1.maven.org/maven2/com/devonfw/tools/ide/devonfw-ide-scripts/2022.08.003/devonfw-ide-scripts-2022.08.003.tar.gz

I am not saying that we have to add devonfw-ide itself to ide-urls repository, but we should support maven central in the new approach alongside to github API and others.
Therefore we should add cobigen:
https://repo1.maven.org/maven2/com/devonfw/cobigen/cli/