Add Centos 6 docker build to centos7_docker_image_updater jenkins job

[Haroon-Khel]

Currently our centos6 docker image is built and pushed from a github workflow,

infrastructure/.github/workflows/build.yml

Line 20 in 76d51f1

build-and-push-centos6:

It would make sense to add this to our centos7_docker_image_updater jenkins job as part of the docker images it builds

[sxa]

NOTE: Any updates to this should be covered in a corresponding doc update to https://github.com/adoptium/infrastructure/blob/master/FAQ.md#what-about-the-builds-that-use-the-dockerbuild-tag

[sxa]

@Haroon-Khel @steelhead31 it looks like we have consistent failures on the job after the related PR was merged on the CentOS6 part. Can one of you take a look at this please as we need to be sure that when PRs are merged we verify that they don't cause failures.

[Haroon-Khel]

I'll take a look

[Haroon-Khel]

Error log on latest build https://ci.adoptopenjdk.net/view/Failing%20Temurin%20jobs/job/centos7_docker_image_updater/331/execution/node/212/log/

+ docker push index.docker.io/adoptopenjdk/centos6_build_image:linux-amd64
The push refers to repository [docker.io/adoptopenjdk/centos6_build_image]
bc6a0138fc79: Preparing
513c80085c29: Preparing
54d345198942: Preparing
10506ac33692: Preparing
42efb8220fde: Preparing
14e739122a1b: Preparing
d4f51880a155: Preparing
aaa5621d7c01: Preparing
d4f51880a155: Waiting
aaa5621d7c01: Waiting
14e739122a1b: Waiting
denied: requested access to the resource is denied

Got a replica jenkins job which is just running the centos6 build at the moment https://ci.adoptopenjdk.net/job/centos7_image_update_haroon_test/8/console

[Haroon-Khel]

Might be related to the fact that the first build and push requires the image to be tagged https://stackoverflow.com/questions/41984399/denied-requested-access-to-the-resource-is-denied-docker

Rerunning my replica jenkins job with a tag stage
https://ci.adoptopenjdk.net/job/centos7_image_update_haroon_test/9/console

[Haroon-Khel]

Still fails https://ci.adoptopenjdk.net/job/centos7_image_update_haroon_test/10/console

I'll try to replicate the build in an ssh session on the machine itself

[Haroon-Khel]

While I am in the middle of finding the dockerhub credentials to be able to recreate the pushing of the image to dockerhub via the command line, I notice that without the credentials, I get the same error as above

jenkins@dockerhost-equinix-ubuntu2204-x64-1:~$ docker push index.docker.io/adoptopenjdk/centos6_build_image:linux-amd64
The push refers to repository [docker.io/adoptopenjdk/centos6_build_image]
a1c6beca1891: Preparing 
c70d0fcd04cc: Preparing 
ebb00c9d1a8e: Preparing 
75f452af8a19: Preparing 
bc0bb3a89452: Preparing 
83c9603dc659: Waiting 
d6d9e9e952b8: Waiting 
aaa5621d7c01: Waiting 
denied: requested access to the resource is denied

[Haroon-Khel]

Even after logging into dockerhub I still get the same error

jenkins@dockerhost-equinix-ubuntu2204-x64-1:~$ dockerpush index.docker.io/adoptopenjdk/centos6_build_image:linux-amd64
The push refers to repository [docker.io/adoptopenjdk/centos6_build_image]
6d62c608a741: Preparing 
b1280007a593: Preparing 
a743d9d92074: Preparing 
dbd761fa66a3: Preparing 
aa6a55730597: Waiting 
bec408233208: Waiting 
e3d1bdc2e37f: Waiting 
aaa5621d7c01: Waiting 
denied: requested access to the resource is denied

[Haroon-Khel]

Retagging the image does not solve the error, neither does building and pushing the image from the other x64 dockerhost machine dockerhost-equinix-ubuntu2004-x64-1

[steelhead31]

@Haroon-Khel , as mentioned on Friday :) , for both this one and the Risc-V push, I've ready a few articles on this issue, and it appears that this process sequence may work ... 

1. Build Image
2. Docker Login
3. Docker Push ( Expected to fail as above )
4. Docker Logout
5. Docker Login
6. Docker Tag
7. Docker Push

See this: https://stackoverflow.com/questions/41984399/denied-requested-access-to-the-resource-is-denied-docker

[Haroon-Khel]

Update:

Scott's solution unfortunately did not work. Same denied: requested access to the resource is denied.

I suspect the error is to do with how the adoptopenjdk/centos6_build_image repository is setup. I have no problems pushing to the adoptopenjdk/centos7_build_image repo

[sxa]

Noting that while centos7_build_image was pushed by the adoptopenjdk user the last centos6_build_image (a year ago) was pushed by @gdams own user ID by the look of it, so I wonde if that's why adoptopenjdk can't override it.
Although that wouldn't explain any block on creating the ubuntu2004 one, unless wed prevented pushing of newly named images.

George, do you know the history of what's happened to this repository that might be causing these failures? And do you have an account that can log into it? Doesn't look like adoptopenjdk (as an organisation account) can log into the dockerhub web interface.

[Haroon-Khel]

@sxa Do you think removing the latest centos6_build_image image in dockerhub and then re-pushing an image would help? The dockerhub credentials I have can login to docker in the cli but not into hub.docker.com else I would have tested this theory

[Haroon-Khel]

@sxa Do you think removing the latest centos6_build_image image in dockerhub and then re-pushing an image would help? The dockerhub credentials I have can login to docker in the cli but not into hub.docker.com else I would have tested this theory

Adding to this, it might be possible to view a permissions config for each repo in the adoptopenjdk dockerhub. Perhaps something changed a year ago without notifying us? Again I cannot login to docker hub using the cli credentials so if someone who can login to docker could check?

[Haroon-Khel]

The username and password worked for me this time, that's strange. I'll do some digging

[Haroon-Khel]

Noting that while centos7_build_image was pushed by the adoptopenjdk user the last centos6_build_image (a year ago) was pushed by @gdams own user ID by the look of it, so I wonde if that's why adoptopenjdk can't override it.

The adoptopenjdkuser account, used for automated pushing, is of the automated team not the owners team. I assume members of the owners team have greater permissions, @gdams's account is of the owners team so @sxa might be right in thinking the adoptopenjdkuser user might not be able to override an image published by a user of the owners account

[Haroon-Khel]

@sxa Do you think removing the latest centos6_build_image image in dockerhub and then re-pushing an image would help? The dockerhub credentials I have can login to docker in the cli but not into hub.docker.com else I would have tested this theory

The adoptopenjdk user looks like it does not have permissions to delete an image/tag. I think only owners can do that, so @gdams @tellison and @karianna .

I recommend one of the owners delete the latest tag of centos6_build_image and then we can see if the adoptopenjdkuser account is able to push a new centos6 image

[Haroon-Khel]

Though we should first confirm if an owner account can push an image to centos6_build_image before deleting the latest image

[tellison]

Will wait for further info

[sxa]

@sxa Do you think removing the latest centos6_build_image image in dockerhub and then re-pushing an image would help?

The issue with that is because we are storing SHAs of docker files in the SBoMs, deleting the SHAs would invalidate the values in there, so we should avoid deleting existing ones if at all possible.

[tellison]

Image rebuilt and updated on DockerHub

[Haroon-Khel]

@tellison added a permission to the centos6 repo to allow the adoptopenjdkuser user to push images to the centos6 repo. It seems that the automated user needs a specific rule to allow it to push to the repo. Tim also created the adoptopenjdk/ubuntu2004_build_image:linux-riscv64 repo and then gave the automated user write permissions to solve the same error we were seeing with this image

[Haroon-Khel]

WIth this solved I think its good to put the centos6 build back into https://ci.adoptium.net/job/centos7_docker_image_updater/
#3296

[Haroon-Khel]

Closing this now that #3296 is merged and the centos6 part of https://ci.adoptium.net/job/centos7_docker_image_updater/ passes