WIP: Make gradle the root of every dependency graph

[donoghuc]

Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate.

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

[mergify]

This pull request does not have a backport label. Could you fix it @donoghuc? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[donoghuc]

Note to self: make sure to re-evaluate https://github.com/elastic/logstash/pull/18340/files#r2488030095

[donoghuc]

run exhaustive tests

[donoghuc]

Got through PR tests, moving on the exhaustive next (Still need to do DRA and ansillary).

[donoghuc]

Seems like file descriptor inheretance is tripping up how fpm (https://github.com/jordansissel/fpm/blob/d9be49871fe040937b63d1223491fc352508ab78/lib/fpm/util.rb#L8) wants to shell out to tar. I found jruby/jruby#5249 which seems similar (but should have been resolved).

For now tasks that use fpm shell out to rake instead of using jruby via gradle.

example failure https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/2998#019b0aa3-ff39-4b6b-9f37-cfd4cf1aa73b:

2025-12-10 15:45:17 PST | [artifact:deb] building deb package for x86_64
2025-12-10 15:45:17 PST | Rake task error: Errno::EBADF: Bad file descriptor - tar
2025-12-10 15:45:27 PST | Backtrace: org/jruby/RubyProcess.java:1779:in `spawn'
2025-12-10 15:45:17 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/util.rb:145:in `execmd'
2025-12-10 15:45:17 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/util.rb:186:in `safesystem'
2025-12-10 15:44:34 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/package/deb.rb:1043:in `block in write_control_tarball'
2025-12-10 15:45:27 PST | <internal:uri:classloader:/jruby/kernel/kernel.rb>:19:in `tap'
2025-12-10 15:44:34 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/package/deb.rb:1033:in `write_control_tarball'
2025-12-10 15:45:17 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/fpm-1.17.0/lib/fpm/package/deb.rb:677:in `output'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/rakelib/artifacts.rake:853:in `package'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/rakelib/artifacts.rake:672:in `package_with_jdk'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/rakelib/artifacts.rake:313:in `block in <main>'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `block in execute'
2025-12-10 15:45:27 PST | org/jruby/RubyArray.java:2009:in `each'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `execute'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
2025-12-10 15:45:27 PST | org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
2025-12-10 15:45:27 PST | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1765410090515934184/elastic/logstash-exhaustive-tests-pipeline/vendor/bundle/jruby/3.1.0/gems/rake-13.3.1/lib/rake/task.rb:188:in `invoke'
2025-12-10 15:45:27 PST | <script>:6:in `<main>'

[donoghuc]

run exhaustive tests

[donoghuc]

Made more progress on acceptance tests... The fpm gem does not like being called from gradle/jruby 😅 #18471 (comment)

Also, there will be some follow on work with invokingh bunlder directly for acceptance tests. For now, instead of invoking rake i'm invoking bundler, but i have not decided if that is actually a win or not.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 5c08684

History

• 💚 Build #3955 succeeded 2b6945b
• 💛 Build #3952 was flaky 090182e
• 💛 Build #3949 was flaky ce8c78a
• 💚 Build #3947 succeeded 1f8697b
• 💛 Build #3943 was flaky 09be295
• 💔 Build #3942 failed 6fabba3

[donoghuc]

Today i was able to figure out the fpm issues but the problem i'm playing whack-a-mole with is the management of dowloading jdk. The exhaustive test pipeline provides an example of an issue i'm facing.

Currently as a prep step we run ./gradlew clean bootstrap artifactDeb artifactRpm to prepare artifacts for testing on linux targets which consume .rpm and .deb packages. When run in isolation (for example artifactDeb) we can prepare a complete artifact (in this case one that includes a jdk). This has a dependsOn for copyJdk and is "finalized" by a delete of that jdk. Unfortunately when run together (artifactDeb artifactRpm) the deletion does not inform gradle that copyJdk is needed again and the wrong JDK is used.

I need to figure out exactly which tasks need the copyJdk (and which explicitly need it deleted) in order to figure out how to best represent it in gradle.