Request builds for amber raw-string-literal branch

[briangoetz]

I'd like to ask for builds on a few branches of the amber repository, starting with the branch "raw-string-literal" of the hg repo http://hg.openjdk.java.net/amber/amber. Builds on 64-bit linux/mac/windows is ideal; minimal or no testing needed since testing resources are strained and the skew from jdk tip is small. No known build differences.

[karianna]

@briangoetz Thanks for the request, I'm personally looking forward to Amber - I'll pick this up.

[karianna]

For those following the task list below will be templated and transformed into formal docs at the TSC. I'll update this comment as I move along.

• Create AdoptOpenJDK amber source repo
• Create AdoptOpenJDK amber nightly binary repo
• Create git-hg job to clone amber with the string-literal branch
• Add Jenkins jobs to build AdoptOpenJDK amber repo on Linux x86
• Add Jenkins jobs to build AdoptOpenJDK amber repo on Windows
• Add Jenkins jobs to build AdoptOpenJDK amber repo on Mac OS X
• Build pipeline to deploy those to AdoptOpenJDK amber nightly binary repo
• Update api code to accept this new repo
• Update website code to accept new repo

[karianna]

@gdams has kindly kicked off the initial git-hg job, we'll come back in the morning (UK time) and check it's status and go from there.

[karianna]

git-hg has completed after some script modifications to support branch pulls.

[karianna]

we're 'manually' testing a build to see what extra deps we might need on the build hosts..

[gdams]

okay a linux x64 build is complete. It can be downloaded from https://ci.adoptopenjdk.net/job/openjdk_amber_build_x86-64_linux/

[briangoetz]

Thanks!

Note that this is just one branch of amber (raw string literals), and there will be other branches we will want to build concurrently in the future. So I suggest naming it something like amber-rsl for clarity.

[karianna]

linux builds OK.

Removing unnecessary files now...
Fetching the first tag from the OpenJDK git repo...
Currently at '/home/jenkins/workspace/openjdk_amber_build_x86-64_linux/openjdk/build/linux-x86_64-normal-server-release/images'
moving jdk to jdk-10+43
Finished removing unnecessary files from jdk-10+43
Archiving the build OpenJDK image...
OpenJDK repo tag is jdk-10+43
Your final .tar.gz was created at /home/jenkins/workspace/openjdk_amber_build_x86-64_linux/openjdk/build/linux-x86_64-normal-server-release/images�
Moving the artifact to /home/jenkins/workspace/openjdk_amber_build_x86-64_linux/OpenJDK_amber_x64_Linux_201809031925.tar.gz�

We'll continue on Monday with Mac and Windows builds and then finally the API and website release

[gdams]

MacOS builds complete https://ci.adoptopenjdk.net/job/openjdk_amber_build_x86-64_macos/

[karianna]

All 3 O/S's are building now. Just scripting up the website and API release part

[smlambert]

Run a subset of openjdk regression tests on x64_linux/x64_mac (no windows test machine available):
https://ci.adoptopenjdk.net/view/work%20in%20progress/job/amber-rsl_hs_openjdktest_x86-64_linux/
https://ci.adoptopenjdk.net/view/work%20in%20progress/job/amber-rsl_hs_openjdktest_x86-64_macos/

Still need to update exclude file, and also there is an openjdk-build issue #248 - where build scripts need to be updated to correctly build the native test libraries before we would enable native tests in Java10+.

[karianna]

Nightly builds now available at: https://adoptopenjdk.net/nightly.html?variant=amber

[karianna]

I'm going to try to apply the rsl naming

[karianna]

We will have to redo Amber in the new build scripts way.

[briangoetz]

Ask on amber-dev to find out which branches are still active?

…

On Mar 7, 2019, at 3:53 PM, Martijn Verburg ***@***.***> wrote: We will have to redo Amber in the new build scripts way. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#251 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKeRPybyfWCCmW07I5dNdEGyE_5b_T9ks5vUTXdgaJpZM4SjTdx>.

[karianna]

Will do!

[adamfarley]

Is this something that others are working on? Happy to lend a hand, so long as I'm not duplicating effort.

[briangoetz]

These branches have been migrated to branches in github. Ask on amber-dev which branches are the ones that would most benefit from a build?

[adamfarley]

First we need to know how many additional builds we can run. Presuming we still want mac/win/lin nightlies, how many builds worth of runtime can we spare on those platforms?

@M-Davies - Do you have the numbers on this?

[M-Davies]

First we need to know how many additional builds we can run. Presuming we still want mac/win/lin nightlies, how many builds worth of runtime can we spare on those platforms?

@M-Davies - Do you have the numbers on this?

No idea I'm afraid. Maybe check the nightlies and see how many machines are free during the night?

[aahlenst]

Running additional builds on Windows/Linux/macOS on x64 is unlikely to be a problem because we dynamically provision build nodes on those platforms (not 100% sure with Windows).

[adamfarley]

@M-Davies - I don't know how to do that. Is there a graph that shows machines occupied/unoccupied, by platform, over time?

@aahlenst - Thanks. Will ask about the builds without mentioning platforms, and we can make those judgements later on as we get a feel for the impact the extra builds are having on the nightlies.

[aahlenst]

@adamfarley From my POV, we need first to figure out what exactly we want to do.

• As Martijn said further up, it's in the interest of the project to build various preview features.
• Do we want to test those? And to what extent?
• Do we need daily builds or a weekly builds sufficient?
• Does it make sense to build on additional platforms besides Windows/Linux/macOS on x64?

Once we know all this, we can exactly decide what to do.

As far as I know, we don't have graphs that show our machine utilization. Would be a nice student project, I guess.

[adamfarley]

@aahlenst These all sound like questions for the Amber project, as they are more likely to know what they need.

Here's a rough sketch of the email I planned to send to amber-dev, plus some extra bits to cover your bullet points.

Subject: Amber Builds Offer

Hi All,

Could the Amber project use some regularly scheduled builds at AdoptOpenJDK?

If so, any suggestions on which branches we should build?

We'd also appreciate any thoughts on testing needed (by branch), frequency of builds, 
and any specific platform requests.

Issue link: https://github.com/AdoptOpenJDK/openjdk-build/issues/251

Best Regards
Adam Farley
Software Developer
Red Hat

[adamfarley]

As for the graph, that sounds like an infra thing. Will raise an infra issue as a "good first issue".

[aahlenst]

These all sound like questions for the Amber project, as they are more likely to know what they need.

Those might also be questions for the Amber project, but we as AdoptOpenJDK have to answer those for ourselves in the first place.

Will raise an infra issue as a "good first issue".

A "good first issue" requires significant effort from our side to document and prepare. Solving the task requires good knowledge of our infra. That's basically the opposite of a "good first issue".

[adamfarley]

These all sound like questions for the Amber project, as they are more likely to know what they need.

Those might also be questions for the Amber project, but we as AdoptOpenJDK have to answer those for ourselves in the first place.

I'm not sure I understand how we would know what sort of testing or platform coverage has the best value for each branch of the Amber project. I mean, we could learn, or ask around inside Adopt to see if anyone has Amber insight, but it still seems simpler to me to ask the project members themselves.

Will raise an infra issue as a "good first issue".

A "good first issue" requires significant effort from our side to document and prepare. Solving the task requires good knowledge of our infra. That's basically the opposite of a "good first issue".

Very well. No "good first issue" tag.

[M-Davies]

@M-Davies - I don't know how to do that. Is there a graph that shows machines occupied/unoccupied, by platform, over time?

When the nightlies are running, you can search Jenkins for a specific node pattern and it will return a list of machines, each containing what they're running at the present time. There's also a statistics tab that would provide you a graph of that specific build label over a longer period

So say if you wanted to see how many test riscv machines are online, you would do https://ci.adoptopenjdk.net/label/riscv&&ci.role.test/ and checkout each machine OR look at the statistics tab.

To get the label in the first place, take a look at the config files OR look at the downstream job log of whatever you want to recreate in amber and search near the top for [NODE SHIFT]. That output will always contain the full label name which you can search for in jenkins above.

03:30:59  [NODE SHIFT] MOVING INTO NODE MATCHING LABELNAME build&&linux&&arm...

[adamfarley]

Thanks Morgan. This definitely seems the place that should contain the information I'm after.

However, either due to data corruption or some peculiarity with data representation, I'm not entirely convinced that this data is accurate.

On the basis that the number of executors, as well as the number of in-use executors (that dark "bump" line near the bottom), never quite seems to equal a whole number.

Will open an infra issue anyway. It'd be good to have analytics on this. :)

EDIT: Maybe a website issue would be better, as it's more of a data representation enhancement than an infra issue.

https://github.com/AdoptOpenJDK/openjdk-website/issues/868

[smlambert]

I presume adoptium/infrastructure#1660 (use Nagios for monitoring machine usage) may be related to the side discussion happening here.

But also, if these builds are of interest to the community and the project, the presumption can be that we find resources if they are needed. That is a somewhat separate issue from putting the changes in place to enable running of these builds. How often they are run (perhaps only weekly to start)? Do we run testing? These are ways to mitigate machine resource issues while providing development builds.

[aahlenst]

Adam volunteered to open a TSC issue to work out the framework for providing preview builds of Amber and other OpenJDK projects.

[adamfarley]

Adam wrote a TSC issue, decided it was too long, and shrunk it down to the essentials: AdoptOpenJDK/TSC#191

There's more than a few larger-scope things in there, so we should probably resolve that issue before continuing this issue.

[sxa]

Is this something that others are working on? Happy to lend a hand, so long as I'm not duplicating effort.

We already run https://ci.adoptopenjdk.net/view/Build%20and%20Test%20Pipeline%20Calendar/job/JavaFutures/ on an unofficial basis - would need to be tweaked/added to in order to meet the OP's requirements of the laternate branch - they're not monitored on a regular basis to ensure they run ok though

[smlambert]

Removing this from Top Priorities project, because it clearly is not, given the lack of movement on this, same for #258. Re-add top project if you feel otherwise and an assignee is named to own it.

[andrew-m-leonard]

@karianna Is this still required?

[karianna]

No, this can be closed.