Add support for reproducible builds

[inglor]

Description

As per gradle docs add support to remove timestamps and package with same order which is required from reproducible builds

Issues Resolved

As per the suggestion in the opensearch-project/anomaly-detection PR 579

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dblock]

Since we are here, is there a change we can make in https://github.com/opensearch-project/OpenSearch Gradle plugin plugin to avoid having to put this in every plugin implementation?

[inglor]

Since we are here, is there a change we can make in https://github.com/opensearch-project/OpenSearch Gradle plugin plugin to avoid having to put this in every plugin implementation?

The main project already has it in the main build.gradle and is inherited in the plugins/build.gradle. Is that what you refer to?

[dblock]

Since we are here, is there a change we can make in https://github.com/opensearch-project/OpenSearch Gradle plugin plugin to avoid having to put this in every plugin implementation?

The main project already has it in the main build.gradle and is inherited in the plugins/build.gradle. Is that what you refer to?

This PR (into a plugin) adds preserveFileTimestamps and reproducibleFileOrder into the plugin's build.gradle. I am looking for a way to enable it in the main project in a way that we don't have to make this PR and automatically apply to all plugins. The code in the plugin's build.gradle does apply plugin: 'opensearch.opensearchplugin', can it also inherit preserveFileTimestamps and reproducibleFileOrder automatically?

[inglor]

This PR (into a plugin) adds preserveFileTimestamps and reproducibleFileOrder into the plugin's build.gradle. I am looking for a way to enable it in the main project in a way that we don't have to make this PR and automatically apply to all plugins. The code in the plugin's build.gradle does apply plugin: 'opensearch.opensearchplugin', can it also inherit preserveFileTimestamps and reproducibleFileOrder automatically?

OK from what I understand there are 3 (4 with optional) places were we can the reproducible fixes

1. The main OpenSearch project, already added here
2. The plugins included with OpenSearch project, inherited already from the above as well.
3. The individual project plugins from OpenSearch like Anomaly Detection. For these I think we need to do them one by one.
4. [Optional] Add the reproducibility (made up word) to the template for new plugins which is this project here.

For number 3 I think you are saying there's another way to do it but I don't know where would that be. Please point me to the right direction and I'm happy to give it a go.

[dblock]

@inglor precisely I believe that plugins can inherit these options by including apply plugin: 'opensearch.opensearchplugin'. @prudhvigodithi do you know where to do this?

[prudhvigodithi]

Hey @inglor @dblock just acknowledging that I have seen your comment, I will provide you with some details shortly ;)
Thank you

[prudhvigodithi]

Hey @inglor so there are some common custom gradle plugins that is being used by all OpenSearch plugins, so all the common logic need not be re-written inside all OpenSearch plugins gradle code.

Example 1: opensearch.opensearchplugin
This plugin is applied to all OpenSearch plugins anomaly-detection, job-scheduler ....

Example 2: opensearch.pluginzip
anomaly-detection, job-scheduler ....

So the ask is if its possible to add the reproducible build logic into one of these custom gradle plugins, so that it need not be added manually to each and every plugin (like done for AD opensearch-project/anomaly-detection#579)

@bbarani

[prudhvigodithi]

Something like modifying AbstractArchiveTask properties PreserveFileTimestamps and ReproducibleFileOrder via those custom gradle plugins added above
Useful Ref: https://github.com/gradle/gradle/blob/master/subprojects/core/src/main/java/org/gradle/api/tasks/bundling/AbstractArchiveTask.java#L87-L88

[inglor]

@prudhvigodithi I guess something as simple as this could work:

diff --git a/buildSrc/src/main/groovy/org/opensearch/gradle/plugin/PluginBuildPlugin.groovy b/buildSrc/src/main/groovy/org/opensearch/gradle/plugin/PluginBuildPlugin.groovy
--- a/buildSrc/src/main/groovy/org/opensearch/gradle/plugin/PluginBuildPlugin.groovy	(revision 4b08d386e7e567071365029bd1d71b3bb0388cc7)
+++ b/buildSrc/src/main/groovy/org/opensearch/gradle/plugin/PluginBuildPlugin.groovy	(date 1664745672458)
@@ -29,13 +29,13 @@
 package org.opensearch.gradle.plugin
 
 import com.github.jengelman.gradle.plugins.shadow.ShadowPlugin
+import org.gradle.api.tasks.bundling.AbstractArchiveTask
 import org.opensearch.gradle.BuildPlugin
 import org.opensearch.gradle.NoticeTask
 import org.opensearch.gradle.Version
 import org.opensearch.gradle.VersionProperties
 import org.opensearch.gradle.dependencies.CompileOnlyResolvePlugin
 import org.opensearch.gradle.info.BuildParams
-import org.opensearch.gradle.plugin.PluginPropertiesExtension
 import org.opensearch.gradle.test.RestTestBasePlugin
 import org.opensearch.gradle.testclusters.RunTask
 import org.opensearch.gradle.util.Util
@@ -138,6 +138,12 @@
         project.tasks.register('run', RunTask) {
             dependsOn(project.tasks.bundlePlugin)
         }
+        project.tasks.withType(AbstractArchiveTask.class).configureEach { task ->
+            // ignore file timestamps
+            // be consistent in archive file order
+            task.preserveFileTimestamps = false
+            task.reproducibleFileOrder = true
+        }
     }
 
     private void configurePublishing(Project project, PluginPropertiesExtension extension) {

[AmiStrn]

@inglor so are we proceeding with this branch or do you want me to close it?

[inglor]

@inglor so are we proceeding with this branch or do you want me to close it?

Didn't have much time to test this properly. If you think my patch would work as is then I can raise a PR against OpenSearch project tomorrow or the day after.

Kinda busy the next days so I recon I might have more time next week for more testing.

[AmiStrn]

Didn't have much time to test this properly

No rush, test when you have time.

[inglor]

@AmiStrn opened a PR for the OpenSearch gradle plugin. Just to confirm my understanding, once that one is merged then from the next release and after any plugin is going to be using that, right? I'm just not sure when the gradle plugin is published to maven repo so it can be used from other projects.

[prudhvigodithi]

@AmiStrn opened a PR for the OpenSearch gradle plugin. Just to confirm my understanding, once that one is merged then from the next release and after any plugin is going to be using that, right? I'm just not sure when the gradle plugin is published to maven repo so it can be used from other projects.

If I may hop in :)
So all the custom gradle plugins would be part of build-tools, example job-scheduler, so once this dependency is build, which is done by the build-workflow when building OpenSearch component, this build-tools would be published to the maven repo, from then whenever a plugin is applied new changes should be picked, this setup can even be tested on local as well where the build-tools would be picked up from local maven repo. @AmiStrn @dblock Please add if i'm missing anything.
Thank you

[dblock]

@AmiStrn opened a PR for the OpenSearch gradle plugin. Just to confirm my understanding, once that one is merged then from the next release and after any plugin is going to be using that, right? I'm just not sure when the gradle plugin is published to maven repo so it can be used from other projects.

If I may hop in :) So all the custom gradle plugins would be part of build-tools, example job-scheduler, so once this dependency is build, which is done by the build-workflow when building OpenSearch component, this build-tools would be published to the maven repo, from then whenever a plugin is applied new changes should be picked, this setup can even be tested on local as well where the build-tools would be picked up from local maven repo. @AmiStrn @dblock Please add if i'm missing anything. Thank you

Yes, that's what I was after. So this PR will no longer be necessary. @inglor Let's confirm when that works and close this without merging?

[inglor]

Covered by opensearch-project/OpenSearch#4746