#941: #1009: create crawler #1042

Amueller36 · 2023-01-23T13:59:14Z

Because the status json handling that was done before seemed wrong I refactored the corresponding methods/classes.

I am sure there is still room for improvement in terms of java coding conventions.
Waiting for the review for now.

Because the status json handling that was done before seemed wrong I refactored the corresponding methods/classes.

Amueller36 · 2023-01-23T15:39:45Z

Github doesn't let me connect the other 2 issues (1008,1009)

maybeec · 2023-01-30T06:40:09Z

@Amueller36 you can simply name the tickets in the comment starting with a hash like this: #1008, #1009

Amueller36 · 2023-01-30T11:17:03Z

@Amueller36 you can simply name the tickets in the comment starting with a hash like this: #1008, #1009

Thank you for the tip!

…009-Create-crawler-Create-1008-create-URL-checker

Updated Pom.xml

…irrors-1009-Create-crawler-Create-1008-create-URL-checker

…awler-Create-1008-create-URL-checker' of https://github.com/Amueller36/ide into feature-941-New-approach-for-ide-mirrors-1009-Create-crawler-Create-1008-create-URL-checker

hohwille · 2023-02-13T15:46:45Z

As PR #1040 has been merged some conflicts now need to be resolved.

…009-Create-crawler-Create-1008-create-URL-checker

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/jsonfile/Error.java

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/UrlVersion.java

url-updater/src/main/java/com/devonfw/tools/ide/url/updater/AbstractCrawler.java

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/UrlVersion.java

sonatype-lift · 2023-02-13T18:42:05Z

🛠 Lift Auto-fix

Some of the Lift findings in this PR can be automatically fixed. You can download and apply these changes in your local project directory of your branch to review the suggestions before committing.¹

# Download the patch
curl https://lift.sonatype.com/api/patch/github.com/devonfw/ide/1042.diff -o lift-autofixes.diff

# Apply the patch with git
git apply lift-autofixes.diff

# Review the changes
git diff

Want it all in a single command? Open a terminal in your project's directory and copy and paste the following command:

curl https://lift.sonatype.com/api/patch/github.com/devonfw/ide/1042.diff | git apply

Once you're satisfied, commit and push your changes in your project.

You can preview the patch by opening the patch URL in the browser. ↩

…ling/UrlVersion.java Co-authored-by: sonatype-lift[bot] <37194012+sonatype-lift[bot]@users.noreply.github.com>

Refactored UpdateManager Fixxed problem with HelmCrawler, which caused no versions to be found. Fixxed MavenCrawler not creating status.json files. Rancher now is part of the folder docker. Added support For Kotlin due to ticket devonfw#1039

…009-Create-crawler-Create-1008-create-URL-checker

Aligned Kotlin Native Convention with commandlet

…009-Create-crawler-Create-1008-create-URL-checker

hohwille

@Amueller36 thank you very much for your great PR. This is a really complex story and a rather big PR. You solved many problems and surely did a lot of analysis, implementation and testing 👍

As always with big PRs comes a long list of review feedback.
First of all, I have to say sorry because I did a bad mistake: I thought I had completed the review of this PR during your holiday long time ago. However, I noticed that I had started it and many of my review comments have been pending in my personal github space but invisible to you.
Therefore, I now come with a huge list of review feedback to rework that you most probably can not complete till Friday when you leave the team.
My suggestion is that you have a look and rework all the comments that you can address easily. On Thursday or Friday we can schedule a meeting together and talk about the remaining ones. For some of them I already commented, that we can address them after the merge by someone else (probably me). Sorry for the mess and thanks for your patience.

.lift.toml

url-updater/src/main/java/com/devonfw/tools/ide/url/UpdateInitiator.java

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/UrlDownloadFile.java

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/UrlStatusFile.java

hohwille · 2023-03-28T16:42:49Z

url-updater/src/main/java/com/devonfw/tools/ide/url/updater/eclipse/EclipseCrawler.java

+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/R/eclipse-${edition}-${version}-R-win32-x86_64.zip", OSType.WINDOWS,"x64",getEdition());
+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/M1/eclipse-${edition}-${version}-M1-win32-x86_64.zip", OSType.WINDOWS,"x64",getEdition());
+
+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/R/eclipse-${edition}-${version}-R-linux-gtk-x86_64.tar.gz", OSType.LINUX,"x64",getEdition());
+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/M1/eclipse-${edition}-${version}-M1-linux-gtk-x86_64.tar.gz", OSType.LINUX,"x64",getEdition());
+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/R/eclipse-${edition}-${version}-R-linux-gtk-aarch64.tar.gz", OSType.LINUX,"arm64",getEdition());
+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/M1/eclipse-${edition}-${version}-M1-linux-gtk-aarch64.tar.gz", OSType.LINUX,"arm64",getEdition());
+
+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/R/eclipse-${edition}-${version}-R-macosx-cocoa-x86_64.tar.gz", OSType.LINUX,"x64",getEdition());
+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/M1/eclipse-${edition}-${version}-M1-macosx-cocoa-x86_64.tar.gz", OSType.LINUX,"x64",getEdition());
+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/R/eclipse-${edition}-${version}-R-macosx-cocoa-aarch64.tar.gz", OSType.LINUX,"arm64",getEdition());
+        doUpdateVersion(urlVersion, "https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${version}/M1/eclipse-${edition}-${version}-M1-macosx-cocoa-aarch64.tar.gz", OSType.LINUX,"arm64",getEdition());


Indeed eclipse ships milestones (M) and release (R) versions.
However, how can this work. If we provide a version 2023-03 there may be

2023-03-M1

2023-03-M2

2023-03-M3

...

2023-03-R

So far we did not seem to support milestone releases in devonfw-ide:
https://github.com/devonfw/ide-mirrors/blob/master/eclipse/urls

If we want to support milestones in devonfw-ide (what is surely nice for early adopter geek users), the user has to provide this as part of the ECLIPSE_VERSION. Then this code would instead need to have some if condition to detect if the version does end with -M\d+ or not.
To make use of the ${version} pattern in the URLs according to the if condition you would either add -R in case of a release version in the URL or otherwise omit such suffix (neither -M1 nor -R).

Also our version detection or whatever seems broken as there are many eclipse versions missing:
https://github.com/alfeilex/ide-urls/tree/master/eclipse/java

Compare with this:
https://github.com/devonfw/ide-mirrors/blob/master/eclipse/available-versions

I found the issue with the missing versions, but could you please explain further what you meant with your first comment?
I don't understand which functionality I should add to the eclipse crawler specifically.

For KISS we simply drop the M1 lines from the code and do not yet support milestones. I will create a separate story for that to be implemented in the future.

url-updater/src/main/java/com/devonfw/tools/ide/url/updater/java/JavaJsonVersion.java

url-updater/src/main/java/com/devonfw/tools/ide/url/updater/kotlin/KotlinNativeCrawler.java

hohwille · 2023-03-28T16:53:41Z

url-updater/src/main/java/com/devonfw/tools/ide/url/updater/pip/PipCrawler.java

+        if(Objects.equals(urlVersion.getName(), "22.3.1")){
+            doUpdateVersion(urlVersion, "https://bootstrap.pypa.io/pip/get-pip.py");
+        }


very interesting workaround. Do you happen to remember why we do this and can leave a comment e.g. to an issue, stackoverflow, etc. or describe in one line why we do this?

Tbh that was the best workaround I could find at that time. For whatever reason pip decided to only have a few standard versions listed under:
https://bootstrap.pypa.io/pip/

but the latest version (currently 23.0.1, before when I wrote the code it was 22.3.1)
will always be located under this url https://bootstrap.pypa.io/pip/get-pip.py , as you can see this url does not contain any version in it unlike the other old pip versions https://bootstrap.pypa.io/pip/3.3/get-pip.py
So there is not really a way for me to figure out the latest version which is kind of an issue itself.
While Migrating our past solutions of ide-mirrors to java I noticed that the fetching of versions for a few tools did not work at all so I had to come up with some own solutions. I didn't find a better solution for pip yet.

Here we can simply remove the commented if block.
I will additionally create a story around pip to:

manually create a version folder latest in ide-urls for pip pointing to https://bootstrap.pypa.io/pip/get-pip.py

adapt pip commandlet and documentation to actually use PYTHON_VERSION instead of PIP_VERSION and if version is unset, use latest.

url-updater/src/test/java/com/devonfw/tools/ide/url/folderhandling/UrlFileTest.java

…awler-Create-1008-create-URL-checker' of https://github.com/Amueller36/ide into feature-941-New-approach-for-ide-mirrors-1009-Create-crawler-Create-1008-create-URL-checker

sonatype-lift · 2023-03-28T23:13:46Z

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/jsonfile/URLStatus.java

+		return URLStatusSuccess;
+	}
+
+	public void setSuccess(URLStatusSuccess URLStatusSuccess) {


VariableNameSameAsType: Variable named URLStatusSuccess has the type URLStatusSuccess. Calling methods using "URLStatusSuccess.something" are difficult to distinguish between static and instance methods.

❗❗ 5 similar findings have been found in this PR

🔎 Expand here to view all instances of this finding

File Path Line Number

url-updater/src/main/java/com/devonfw/tools/ide/url/updater/AbstractCrawler.java 159

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/jsonfile/URLStatus.java 22

url-updater/src/main/java/com/devonfw/tools/ide/url/updater/AbstractCrawler.java 132

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/jsonfile/URLStatus.java 4

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/jsonfile/URLStatus.java 5

Visit the Lift Web Console to find more details in your report.

ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Help us improve LIFT! (Sonatype LiftBot external survey)

Was this a good recommendation for you? _{Answering this survey will not impact your Lift settings.}

[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2023-03-30T14:22:34Z

url-updater/src/main/java/com/devonfw/tools/ide/url/folderhandling/jsonfile/URLStatus.java

+		return urlStatusError;
+	}
+
+	public void setError(URLStatusError URLStatusError) {


InconsistentCapitalization: Found the field 'urlStatusError' with the same name as the parameter 'URLStatusError' but with different capitalization.

Suggested change

public void setError(URLStatusError URLStatusError) {

public void setError(URLStatusError urlStatusError) {

ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Help us improve LIFT! (Sonatype LiftBot external survey)

Was this a good recommendation for you? _{Answering this survey will not impact your Lift settings.}

[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

hohwille · 2023-03-31T18:07:32Z

The build is broken due to accidental submodule changes. I am do not see an easy way to change this in the open PR, I will simply merge this and then fix it after the merge.

hohwille

Some review comments have been resolved without doing the according change but we will simply merge and do that after merging.

Crawler implemented And Refactored Status Json

57726c3

Because the status json handling that was done before seemed wrong I refactored the corresponding methods/classes.

github-actions bot added the bash related to bash shell or scripts label Jan 23, 2023

Amueller36 self-assigned this Jan 23, 2023

Amueller36 changed the title ~~Crawler implemented And Refactored Status Json~~ Feature/941-New-approach-for-ide-mirrors/1009-Create-crawler-Create/1008-create-URL-checker Jan 23, 2023

Amueller36 added 6 commits January 30, 2023 12:17

Merge branch 'master' into feature-941-New-approach-for-ide-mirrors-1…

11d60da

…009-Create-crawler-Create-1008-create-URL-checker

Pom.xml update

6d722f0

Updated Pom.xml

Merge branch 'devonfw:master' into feature-941-New-approach-for-ide-m…

33f0484

…irrors-1009-Create-crawler-Create-1008-create-URL-checker

semicolon missing

43bb7fb

Merge branch 'feature-941-New-approach-for-ide-mirrors-1009-Create-cr…

6d5fc3c

…awler-Create-1008-create-URL-checker' of https://github.com/Amueller36/ide into feature-941-New-approach-for-ide-mirrors-1009-Create-crawler-Create-1008-create-URL-checker

Fixxed bug where files from repo were loaded that shouldnt be loaded

d177c21

hohwille linked an issue Feb 13, 2023 that may be closed by this pull request

Create crawler to determine available versions and according download URLs #1009

Closed

Merge branch 'master' into feature-941-New-approach-for-ide-mirrors-1…

b6deba9

…009-Create-crawler-Create-1008-create-URL-checker