[improve] Removed unnecessary native libraries from Docker images

[merlimat]

Motivation

The RocksDB jar that we are including in our Docker image is ~55 MB in size because it includes native library of RocksDB compiled for many different platforms.

In reality, our Docker image is going to be used exactly on one single platorm, either linux/x86-64 or linux/arm64.

Therefore we can remove all the other RocksDB compiled libraries from the image. The resulting RocksDB jar is shrunk to 4.6 MB.

Similarly, we can remove Netty TC natives jar that are not relevant for the container.

Modifications

Added a script to remove the unnecessary libraries.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[lhotari]

LGTM, just had a question about musl&glibc coexistence

[eolivelli]

Great idea

[lhotari]

error message in failing containers

  2024-03-09T00:44:36,305+0000 ERROR [main] t.apachepulsar/java-test-image:latest@547 - Log output from the failed container:
  2024-03-09T00:39:40,007+0000 [main] INFO  org.apache.pulsar.PulsarStandalone - Starting BK with RocksDb metadata store
  2024-03-09T00:39:40,117+0000 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
  org.apache.pulsar.metadata.api.MetadataStoreException: Failed to load RocksDB JNI library
  	at org.apache.pulsar.metadata.impl.RocksdbMetadataStore.<init>(RocksdbMetadataStore.java:218) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.metadata.impl.RocksdbMetadataStore.get(RocksdbMetadataStore.java:113) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.metadata.impl.RocksdbMetadataStoreProvider.create(RocksdbMetadataStore.java:587) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.newInstance(MetadataStoreFactoryImpl.java:63) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.createExtended(MetadataStoreFactoryImpl.java:50) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.metadata.api.extended.MetadataStoreExtended.create(MetadataStoreExtended.java:42) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.metadata.bookkeeper.BKCluster.<init>(BKCluster.java:134) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.metadata.bookkeeper.BKCluster$BKClusterConf.build(BKCluster.java:118) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.PulsarStandalone.startBookieWithMetadataStore(PulsarStandalone.java:457) ~[org.apache.pulsar-pulsar-broker-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:297) ~[org.apache.pulsar-pulsar-broker-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:149) ~[org.apache.pulsar-pulsar-broker-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  Caused by: java.lang.ExceptionInInitializerError
  	at org.apache.pulsar.metadata.impl.RocksdbMetadataStore.<init>(RocksdbMetadataStore.java:216) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	... 10 more
  Caused by: java.lang.RuntimeException: librocksdbjni-linux64-musl.so was not found inside JAR.
  	at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:118) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?]
  	at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:102) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?]
  	at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:82) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?]
  	at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:70) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?]
  	at org.rocksdb.RocksDB.<clinit>(RocksDB.java:39) ~[org.rocksdb-rocksdbjni-7.9.2.jar:?]
  	at org.apache.pulsar.metadata.impl.RocksdbMetadataStore.<init>(RocksdbMetadataStore.java:216) ~[org.apache.pulsar-pulsar-metadata-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
  	... 10 more

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.58%. Comparing base (bbc6224) to head (a702b26).
Report is 33 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #22230      +/-   ##
============================================
+ Coverage     73.57%   73.58%   +0.01%     
+ Complexity    32624    32259     -365     
============================================
  Files          1877     1879       +2     
  Lines        139502   139792     +290     
  Branches      15299    15338      +39     
============================================
+ Hits         102638   102872     +234     
- Misses        28908    28926      +18     
- Partials       7956     7994      +38     

Flag	Coverage Δ
inttests	26.56% <ø> (+1.98%)	⬆️
systests	24.36% <ø> (+0.04%)	⬆️
unittests	72.87% <ø> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 79 files with indirect coverage changes