Build shibboleth-idp for testing purposes #91216

jakelandis · 2022-10-31T17:38:10Z

We currently use unicon/shibboleth-idp:3.4.2 to help test our SAML integration.
That container is no longer actively supported and does not support
ARM architectures.

This commit is a partial clone from https://github.com/Unicon/shibboleth-idp-dockerized 3.4.3.

Changes from upstream include:

Use openjdk:11.0.16-jre as the base image for support for ARM architectures
Handle missing keystore download from Jetty
Fix URL paths for artifacts to download

Changes to this repository include:

Copied required Jetty configuration files from upstream project
Updates to docker compose
Placed the missing keystore Jetty downloads in a separate location (jetty-custom)

The final result is a bit messy. Mixing cloned files with custom files and mixing
Jetty and IDP concerns. However, it is not much messier than prior and now
that we control building the image we can more easily upgrade shibboleth IDP
The upgrade to the latest version is fairly involved and as such we will need to
deviate more from the clone which should allow some additional clean up.

part of: #71378
related: #91144
supersedes: #89674

===
Below is a comparison report from this PR to the upstream Dockerfile.
compare.pdf

All files except build.gradle, docker-compose.yml, and Dockerfile are copy/pasted.

jakelandis · 2022-10-31T17:41:43Z

slightly more readable diff: compare.pdf

jakelandis · 2022-10-31T17:49:18Z

to test manually : ./gradlew :x-pack:qa:saml-idp-tests:javaRestTest --info --stacktrace , note since c2id is part of the same test fixture you may need to comment that container out from the build (at least until #91144 is merged).

elasticsearchmachine · 2022-10-31T17:49:48Z

Pinging @elastic/es-security (Team:Security)

jakelandis · 2022-10-31T17:51:55Z

x-pack/test/idp-fixture/docker-compose.yml

-      - ./idp/shibboleth-idp/conf:/opt/shibboleth-idp/conf
-      - ./idp/shibboleth-idp/credentials:/opt/shibboleth-idp/credentials
-      - ./idp/shibboleth-idp/metadata:/opt/shibboleth-idp/metadata
-      - ./idp/shib-jetty-base/start.d/ssl.ini:/opt/shib-jetty-base/start.d/ssl.ini


these are now copied via the Dockerfile

jakelandis · 2022-10-31T17:52:49Z

x-pack/test/idp-fixture/docker-compose.yml

    depends_on:
      - openldap
    environment:
      - JETTY_MAX_HEAP=64m
-      - JETTY_BROWSER_SSL_KEYSTORE_PASSWORD=secret
-      - JETTY_BACKCHANNEL_SSL_KEYSTORE_PASSWORD=secret


this is now baked into the image

jakelandis · 2022-10-31T17:58:09Z

x-pack/test/idp-fixture/idp/jetty-custom/ssl.mod

+
+[files]
+# keystore originally sourced from https://github.com/eclipse/jetty.project/raw/jetty-9.3.x/jetty-server/src/main/config/etc/keystore
+/opt/shib-jetty-base/etc/keystore


this is the only change in this file.

jakelandis · 2022-10-31T18:32:26Z

test failure : https://gradle-enterprise.elastic.co/s/donalxonp3lge might be related or could be a latent issue... looking now.

Cannot get property 'test.fixtures.shibboleth-idp.tcp.4443' on extra properties extension as it does not exist

jakelandis · 2022-10-31T21:45:24Z

Cannot get property 'test.fixtures.shibboleth-idp.tcp.4443' on extra properties extension as it does not exist

This is only an issue for a brand new image build. i.e. if the image is already built then there are no issue.

Further up in the log "Gateway cannot be read from container inspection - trying to read from network inspection" is unique for this container. When run in debug mode you can see an empty port mapping

"PortBindings": {
                "4443/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ]
            },

after a bit googling... I think this might be due to the docker plugin trying reading the build container, not the final container, so I added a "restart: always" which fixes the issue but now there is another issue. The other issue I believe is that the port from $outputDir/idp-metadata.xml is not always updated. To fix that I switched from a Copy to Sync task and set outputs.upToDateWhen { false } for that task. Pretty sure the last 1/2 of that is a latent bug with running these tests.

Fixed on 82b2b26 (#91216)

jakelandis · 2022-10-31T22:22:50Z

x-pack/qa/saml-idp-tests/build.gradle

  from idpFixtureProject.files('idp/shibboleth-idp/credentials/idp-browser.pem', 'idp/shibboleth-idp/metadata/idp-metadata.xml',
    'idp/shibboleth-idp/credentials/sp-signing.key', 'idp/shibboleth-idp/credentials/sp-signing.crt');
  into outputDir
+  outputs.upToDateWhen { false } //ensure this copy is always done to prevent stale configuration


This is kinda hacky ... I think I can fix this, but would prefer to do so on a different PR.

I saw the comments in slack. I dont' think this is necessary. A sync task will also do a delete, followed by a copy so in what scenario would there be stale content here?

I think what is happening is

First run: this task not upto date, copies the files, next task creates new file (with the same name) in the output directory

Second run: this task reports up to date , even with sync because I ~~think~~ guess that the Sync task is only tracking the files it touches and did not touch the file was replaced by the other task (it has same name but it is a different file/inode), since it is up to date it does nothing and the elder file is still there and is used.

I didn't spent too much time with this, so I could be wrong. It is pretty to reproduce by echoing out the contents of $outputDir/idp-metadata.xml in a doFirst of the setupPorts task.

Yeah, that's quite odd. If one of the output files is modified it should trigger the task to be out-of-date, in which case, it would be the same behavior as just being never up to date as we have here. Kinda gross but yeah, I have plans to ditch all this build logic around test fixtures. It's positively awful.

elasticsearchmachine · 2022-11-03T19:00:03Z

💔 Backport failed

Status	Branch	Result
❌	7.17	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 91216

We currently use unicon/shibboleth-idp:3.4.2 to help test our SAML integration. That container is no longer actively supported and does not support ARM architectures. This commit is a partial clone from Unicon/shibboleth-idp-dockerized 3.4.3. Changes from upstream include: Use openjdk:11.0.16-jre as the base image for support for ARM architectures Handle missing keystore download from Jetty Fix URL paths for artifacts to download Changes to this repository include: Copied required Jetty configuration files from upstream project Updates to docker compose Placed the missing keystore Jetty downloads in a separate location (jetty-custom) The final result is a bit messy. Mixing cloned files with custom files and mixing Jetty and IDP concerns. However, it is not much messier than prior and now that we control building the image we can more easily upgrade shibboleth IDP The upgrade to the latest version is fairly involved and as such we will need to deviate more from the clone which should allow some additional clean up. part of: elastic#71378 related: elastic#91144 supersedes: elastic#89674

We currently use unicon/shibboleth-idp:3.4.2 to help test our SAML integration. That container is no longer actively supported and does not support ARM architectures. This commit is a partial clone from Unicon/shibboleth-idp-dockerized 3.4.3. Changes from upstream include: Use openjdk:11.0.16-jre as the base image for support for ARM architectures Handle missing keystore download from Jetty Fix URL paths for artifacts to download Changes to this repository include: Copied required Jetty configuration files from upstream project Updates to docker compose Placed the missing keystore Jetty downloads in a separate location (jetty-custom) The final result is a bit messy. Mixing cloned files with custom files and mixing Jetty and IDP concerns. However, it is not much messier than prior and now that we control building the image we can more easily upgrade shibboleth IDP The upgrade to the latest version is fairly involved and as such we will need to deviate more from the clone which should allow some additional clean up. part of: #71378 related: #91144 supersedes: #89674

jakelandis added 5 commits October 28, 2022 17:33

intial

933a9c4

compose up works, but tests fail

ba422b0

tests pass

1198a18

move files around

05add03

nits

c7f6dd3

elasticsearchmachine added the v8.6.0 label Oct 31, 2022

This was referenced Oct 31, 2022

update c2id/c2id-server-demo docker image to support ARM #91144

Merged

support arm for shibboleth-idp #89674

Closed

jakelandis added v7.17.8 :Security/Security Security issues without another label >non-issue labels Oct 31, 2022

jakelandis requested a review from mark-vieira October 31, 2022 17:47

jakelandis marked this pull request as ready for review October 31, 2022 17:49

elasticsearchmachine added the Team:Security Meta label for security team label Oct 31, 2022

jakelandis commented Oct 31, 2022

View reviewed changes

jakelandis added 2 commits October 31, 2022 13:36

remove architecture constraint

bb55bef

fix port issues

82b2b26

jakelandis commented Oct 31, 2022

View reviewed changes

mark-vieira approved these changes Nov 2, 2022

View reviewed changes

Merge branch 'main' into arm_for_shibboleth

fb346a7

jakelandis added the auto-backport-and-merge label Nov 3, 2022

jakelandis merged commit 1f3ec5f into elastic:main Nov 3, 2022

jakelandis deleted the arm_for_shibboleth branch November 3, 2022 18:52

jakelandis mentioned this pull request Nov 3, 2022

[7.17] Build shibboleth-idp for testing purposes (#91216) #91293

Merged

jakelandis mentioned this pull request Nov 3, 2022

Replace existing identity provider test fixtures with ones that support aarch64 #71378

Closed

jakelandis mentioned this pull request Oct 4, 2023

Gradle build task failure for :x-pack:qa:saml-idp-tests:copyIdpFiles #95369

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Build shibboleth-idp for testing purposes #91216

Build shibboleth-idp for testing purposes #91216

Uh oh!

jakelandis commented Oct 31, 2022 •

edited

Loading

Uh oh!

jakelandis commented Oct 31, 2022

Uh oh!

jakelandis commented Oct 31, 2022

Uh oh!

elasticsearchmachine commented Oct 31, 2022

Uh oh!

jakelandis Oct 31, 2022

Uh oh!

jakelandis Oct 31, 2022

Uh oh!

jakelandis Oct 31, 2022

Uh oh!

jakelandis commented Oct 31, 2022

Uh oh!

jakelandis commented Oct 31, 2022

Uh oh!

jakelandis Oct 31, 2022

Uh oh!

mark-vieira Nov 2, 2022

Uh oh!

jakelandis Nov 2, 2022

Uh oh!

mark-vieira Nov 2, 2022

Uh oh!

elasticsearchmachine commented Nov 3, 2022

Uh oh!

Uh oh!

Build shibboleth-idp for testing purposes #91216

Build shibboleth-idp for testing purposes #91216

Uh oh!

Conversation

jakelandis commented Oct 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakelandis commented Oct 31, 2022

Uh oh!

jakelandis commented Oct 31, 2022

Uh oh!

elasticsearchmachine commented Oct 31, 2022

Uh oh!

jakelandis Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

jakelandis Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

jakelandis Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

jakelandis commented Oct 31, 2022

Uh oh!

jakelandis commented Oct 31, 2022

Uh oh!

jakelandis Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

mark-vieira Nov 2, 2022

Choose a reason for hiding this comment

Uh oh!

jakelandis Nov 2, 2022

Choose a reason for hiding this comment

Uh oh!

mark-vieira Nov 2, 2022

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Nov 3, 2022

💔 Backport failed

Uh oh!

Uh oh!

jakelandis commented Oct 31, 2022 •

edited

Loading