Add S3 integration tests for searchable snapshots

[tlrx]

This pull request adds REST-based integration tests that test the searchable snapshots feature on S3.

More precisely, this pull request:

• adds a new AbstractSearchableSnapshotsRestTestCase class that provides a generic test case for searchable snapshots
• adds a new FsSearchableSnapshotsIT that tests the previous test case using a filesystem repository. This test is added in the already existing qa:rest project
• adds a new S3SearchableSnapshotsIT that tests the previous test case using a S3 repository. This test is added in a new qa:s3 project that takes care of setting up an S3 fixture for the test. This test can also be run against an external service if the appropriate environment variables are provided

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[tlrx]

@mark-vieira If you have time that would be great if you could have a look at Gradle and Docker files, just in case I missed something. Thanks!

[mark-vieira]

It's not clear to me why all this is required. We seem to be effectively defining a new test fixture here. Why isn't we can't just use the existing s3-fixture like we do for other tests?

elasticsearch/plugins/repository-s3/build.gradle

Line 217 in 88ff69b

testFixtures.useFixture(':test:fixtures:s3-fixture')

[tlrx]

This QA test depends on repository-s3 project which already uses many S3 fixtures and the test fixtures plugin prevented me from reusing the existing one. Since I wanted to avoid any mention to searchables snapshots in open source code I added the current configuration. Do you think there's a better way to do that?

[mark-vieira]

I would define a separate service in the s3-fixture compose file to be used here.

[tlrx]

I would define a separate service in the s3-fixture compose file to be used here.

Sorry if I wasn't clear, I tried this solution and got a failure. Before doing what is coded in this PR I tried to define a separate service in the /test/fixtures/s3-fixture/docker-compose.yml file:

...
  s3-fixture-generic:
    build:
      context: .
      args:
        fixtureClass: fixture.s3.S3HttpFixture
        port: 80
        bucket: "bucket"
        basePath: "base_path"
        accessKey: "access_key"
      dockerfile: ./testfixtures_shared/shared/Dockerfile
    volumes:
      - ./testfixtures_shared/shared:/fixture/shared
    ports:
      - "80"

and tried to declare the usage of this fixture in the qa:s3 build file like this:

if (useFixture) {
  testFixtures.useFixture(':test:fixtures:s3-fixture', 's3-fixture-generic')
}

But it failed with the following error:

Build file '/x-pack/plugin/searchable-snapshots/qa/s3/build.gradle' line: 36

* What went wrong:
A problem occurred evaluating project ':x-pack:plugin:searchable-snapshots:qa:s3'.
> Projects :plugins:repository-s3 and :x-pack:plugin:searchable-snapshots:qa:s3 both claim all services from :test:fixtures:s3-fixture. This is not supported because it breaks running in parallel. Configure specific services in docker-compose.yml for each and add the service name to `useFixture`

In order to allow parallel test execution and to try to reuse as much as possible the existing code, I went with declaring a docker-compose.yml file in this qa:s3project that reuses the existing DockerFile and code from the test fixture.

[mark-vieira]

Understood. The problem here is that :plugins:repository-s3 declares testFixtures.useFixture(':test:fixtures:s3-fixture') which means "give me all services of the s3-fixture project. If we add a new service which that project doesn't need, we now need to be explicit say exactly which services it does need. So you'll need to modify that build script and call useFixture(':test:fixtures:s3-fixture', 's3-fixture') and so on for each service required in the compose yaml.

I apologize for the back and forth, I just want to avoid us reimplementing a bunch of logic that our test fixtures plugin does for us if possible.

[tlrx]

The problem here is that :plugins:repository-s3 declares testFixtures.useFixture(':test:fixtures:s3-fixture')

Thanks for spotting this @mark-vieira, I should have seen it by myself. I pushed 331408a to correct the fixtures.

I apologize for the back and forth, I just want to avoid us reimplementing a bunch of logic that our test fixtures plugin does for us if possible.

Nothing to apologize, I pinged you because I was suspecting something wrong and you spotted it 👍. I'm happy to not add another docker-compose.yml file to the mix :)

[mark-vieira]

Changes LGTM 👍

[mark-vieira]

Perhaps s3-fixture-searchable-snapshots?

[tlrx]

I prefer to not mention searchable snapshots in open source code, so I'll keep "other" if that's ok for you.

[tlrx]

@DaveCTurner Can you have a look at the test itself, please? Thanks

[DaveCTurner]

The tests themselves LGTM, and I've done some basic experiments to verify that they do indeed run as part of ./gradlew :x-pack:plugin:searchable-snapshots:check. Thanks @tlrx.

[tlrx]

Thanks David and Mark!