Antalya Smart tag selection for integration/runner

[strtgbb]

Changelog category (leave one):

• Build/Testing/Packaging Improvement

Allow the omission of --docker-image-version and --docker-compose-images-tags in most circumstances.

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

[ianton-ru]

Explanation required :)

[ianton-ru]

What is in this map? As I understand, it's like altinityinfra/integration-test => sha256:74ac23582ebab40378fa66806e796f2143b4b5a6956a2daf9cd98ff33d0d6117. Which digest it contains when multiple versions of image available?

[strtgbb]

This contains the digests matching the states of the docker containers in your current branch.

[ilejn]

Sorry, I don't get the idea.
@strtgbb , could you explain it more explicitly/verbosely ?
What is the sequence of actions?

[strtgbb]

When CICD runs, it calculates hashes for the dockerfiles that it builds. It then publishes the images using the hashes to form the tags.
DockerDigester.get_all_digests() calculates the tags that CICD would use. The unspecified arguments can then be populated using those tags.
It's a relatively safe assumption that CICD has already run on the base of your branch and that you have not changed the dockerfiles (and thus the calculated hashes).

[ilejn]

Makes sense.
I suggest adding the comment above to the script.

Do you know what DIND stands for?
Shouldn't we promote the change to upstream?

[Enmk]

Not only a Dockerfile, but rather a whole directory used to build the docker image, and all of it's dependent docker images.
docker/images.json describes all docker images that are used in CI/CD, excerpt:

{
    "docker/test/integration/base": {
        "only_amd64": true,
        "name": "altinityinfra/integration-test",
        "dependent": [
            "docker/test/integration/clickhouse_with_unity_catalog"
        ]
    },
...
   "docker/test/integration/clickhouse_with_unity_catalog": {
        "name": "altinityinfra/integration-test-with-unity-catalog",
        "dependent": []
    }
}

Corresponding directories:

$ ls docker/test/integration/base
Dockerfile  requirements.txt

$ ls docker/test/integration/clickhouse_with_unity_catalog
Dockerfile

So in order to get a hash for altinityinfra/integration-test docker image, DockerDigester.get_all_digests() will have to hash following files:

docker/test/integration/base/Dockerfile
docker/test/integration/base/requirements.txt
docker/test/integration/clickhouse_with_unity_catalog/Dockerfile

[strtgbb]

@ilejn
DIND stands for Docker in Docker. It means that the image will be configured for nested Docker. It's the image that you're specifying with --docker-image-version.
I don't think upstream will care for this. It's a little bit hacky, and they don't support old releases like we do.