Upgrade kafka_version from 3.7.1 to 4.0.0

[cwperks]

Description

Upgrade kafka_version from 3.7.1 to 4.0.0

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Maintenance

Issues Resolved

Attempt to update these dependencies. Seeing failing tests from the kafka update in separate PR: #4888

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.92%. Comparing base (d7435f0) to head (7297ff4).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5131      +/-   ##
==========================================
+ Coverage   71.90%   71.92%   +0.02%     
==========================================
  Files         378      378              
  Lines       23562    23562              
  Branches     3630     3630              
==========================================
+ Hits        16942    16948       +6     
+ Misses       4810     4803       -7     
- Partials     1810     1811       +1     

see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[nibix]

Approving this.

Yet, this is likely a change without consequences, as the Kafka audit log sink cannot be actually used (and - if I am informed correctly - never could be so far).

See:

• Issue: #4373
• The docs also do not mention a Kafka sink: https://docs.opensearch.org/docs/latest/security/audit-logs/storage-types/

However, audit logs can be configured to log to log4j which in turn can log to Kafka.

Maybe, for code hygiene and maintenance reasons, it might make sense to remove the Kafka dependency?

[cwperks]

Maybe, for code hygiene and maintenance reasons, it might make sense to remove the Kafka dependency?

I have not seen any forum posts or Github issues about kafka support so it should be ok to remove if its non-functional.

What would be required to make this function? One concern with re-using log4j as a sink is that it would get interleaved with application logs vs directly posting to kafka.