Support building without containers (docker)

[inglor]

Is your feature request related to a problem? Please describe.
I'm a packager of OpenSearch in Arch Linux and would like to be able to build without docker since in our chroot environment we can't start docker.

Describe the solution you'd like
I'd like to have the ability to skip any docker steps which are not required for building the project even if it means skipping some tests

Describe alternatives you've considered
I tried using podman but it's not supported - I believe this is in discussion / progress. Even if podman is available as alternative the ability to build the project without containers would be beneficial and unblock me, then I can work on maybe enabling podman when it comes available.

Additional context
Add any other context or screenshots about the feature request here.

[reta]

I believe we use to skip Docker dependent steps in build but not for distribution targets:

Docker (checked [/usr/bin/docker, /usr/local/bin/docker]) is required to run the following tasks: 
:distribution:docker:buildArm64DockerImage
:distribution:docker:buildDockerImage
:distribution:docker:buildPpc64leDockerImage
:distribution:docker:buildS390xDockerImage
you can address this by attending to the reported issue, or removing the offending tasks from being executed.

Should not be too difficult to fix that. Also, for podman support, please check #6877

[minalsha]

@inglor can you try what @reta mentioned above and submit the PR for the fix ? Thanks

[inglor]

@minalsha what's the suggestion exactly? To skip the tasks. I don't think that's a real solution.

[reta]

@minalsha what's the suggestion exactly? To skip the tasks. I don't think that's a real solution.

@inglor what do you think is the solution?

[inglor]

@minalsha what's the suggestion exactly? To skip the tasks. I don't think that's a real solution.

@inglor what do you think is the solution?

The current state is that the below sub-projects require docker to build due to tests:

    repository-azure
    repository-gcs
    repository-hdfs
    repository-s3

The ideal solution would be to split tests to ones which require docker and the ones which don't, so when downstream projects building those are able to only execute tests which don't require docker with a command line flag (until podman is solved).

[dblock]

I am going to say something controversial, but I think integration tests (that require docker) should not be required to produce the distribution build. IMO that is backwards in OpenSearch, I'd want to make sure all code compiles/builds/assembles, then I want to run integration tests with the output.

Another far-fetched idea: we could go as far as removing these plugins from core repo 🤔 they bring a significant amount of annoying dependencies (e.g. AWS SDK for repository-s3).

[reta]

I am going to say something controversial, but I think integration tests (that require docker) should not be required to produce the distribution build.

Agree

Another far-fetched idea: we could go as far as removing these plugins from core repo thinking they bring a significant amount of annoying dependencies (e.g. AWS SDK for repository-s3).

We POCed that :-) #1754

[inglor]

I am going to say something controversial, but I think integration tests (that require docker) should not be required to produce the distribution build.

I agree with the above sentence 100%.

IMO that is backwards in OpenSearch, I'd want to make sure all code compiles/builds/assembles, then I want to run integration tests with the output.

Would this compile/builds/assembles includes unitTesting? As a downstream consumer we get a bit more confidence when we execute tests (even unit testing) instead of just building and packaging it.

[reta]

Would this compile/builds/assembles includes unitTesting? As a downstream consumer we get a bit more confidence when we execute tests (even unit testing) instead of just building and packaging it.

AFAIK compile/assembles - no, build - yes