Fix Bouncy Castle fips incompatible issue

[jiazhai]

Descriptions of the changes in this PR:

Motivation

More details are provided in Pulsar # 10937.

In #2631, the default BouncyCastle was changed from non-fips into fips version. But the default version of BouncyCastle in Pulsar is the non-fips one(aimed to make it compatible with the old version of Pulsar).

Bouncy Castle provides both FIPS and non-FIPS versions, but in a JVM, it can not include both of the 2 versions(non-Fips and Fips), and we have to exclude the current version before including the other. This makes the backward compatible a little hard, and that's why Pulsar has to involve an individual module for Bouncy Castle.

And if we want to start BookKeeper with TLS enabled through Pulsar's binary, it will meet the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/bouncycastle/jcajce/provider/BouncyCastleFipsProvider
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at org.apache.bookkeeper.common.util.ReflectionUtils.forName(ReflectionUtils.java:49)
	at org.apache.bookkeeper.tls.SecurityProviderFactoryFactory.getSecurityProviderFactory(SecurityProviderFactoryFactory.java:39)
	at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:129)
	at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52)
	at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
	at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
	at org.apache.bookkeeper.server.Main.main(Main.java:208)
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 9 more

This fix is to use the reflection to get the loaded bc version to avoid the hard-coded bc version.

Changes

Use the reflection to get the loaded bc version to avoid the hard-coded bc version
Add backward compatible test for bc-non-fips version

[jiazhai]

@zymap @codelipenghui FYI.

[eolivelli]

@Ghatage PTAL

[eolivelli]

@jiazhai TLS tests are failing, PTAL

[sijie]

@jiazhai overall looks good to me. Can you check why TLS tests are failing?

[Ghatage]

Thanks for doing this @jiazhai, I understand this is a WIP, comments added below.

Thanks for the tag @eolivelli!

[Ghatage]

Thanks for the context @jiazhai. That helps a lot. And also thanks for doing this!.
Wouldn't an easy fix be to allow imports of BK's bouncy castle which were disallowed in @eolivelli's changes here?
I guess I may be over looking the reasons why they were made in the first place.
Your changes look good otherwise 👍
Will you be using maven or gradle for your builds?

Side note:
@pkumar-singh Could you please help us understand how build.gradle is pulling in the dependencies here?
In these lines I see bouncycastle 1.56 and bc-fips 1.0.2, both deps being pulled in. AFAIK, it's recommended to have only one or to avoid classpath conflicts and runtime errors based on which one gets loaded.
I don't see the same in the base level maven pom.xml.

[jiazhai]

Thanks for the comments @Ghatage .
From my view, @eolivelli's changes are reasonable, because the bc versions(fips/non-fips) are in conflict, we could only include one of them into the package. Usually it uses include/exclude to solve the issue.
It would help a lot if we could find a way to support both of the versions.

currently, the changes and tests are included in this PR. And by this change, we could have a choice in Pulsar.

Thanks for the context @jiazhai. That helps a lot. And also thanks for doing this!.
Wouldn't an easy fix be to allow imports of BK's bouncy castle which were disallowed in @eolivelli's changes here?
I guess I may be over looking the reasons why they were made in the first place.
Your changes look good otherwise 👍
Will you be using maven or gradle for your builds?

Side note:
@pkumar-singh Could you please help us understand how build.gradle is pulling in the dependencies here?
In these lines I see bouncycastle 1.56 and bc-fips 1.0.2, both deps being pulled in. AFAIK, it's recommended to have only one or to avoid classpath conflicts and runtime errors based on which one gets loaded.
I don't see the same in the base level maven pom.xml.

[Ghatage]

LGTM +1 @jiazhai
PTAL at the tests, other than that the changes are good and thanks for doing this!

[jiazhai]

@zymap Would you please also help take a look?

[zymap]

LGTM

[jiazhai]

@sijie to take a look

[jiazhai]

cherry-picked into branch-4.14

[eolivelli]

@jiazhai I have fixed branch-4.14 because this patch included a new pom.xml, that needed to be amended to set the version to 4.14.1-SNAPSHOT in branch-4.14.

cc @sijie @merlimat

[eolivelli]

see 69360e1

[diegosalvi]

+1 to have this released ^_^ I failed to cannot upgrade to 4.14.x due to non fips libs already in my classpath and widely used

[jiazhai]

@diegosalvi Would you please also help verify this change in your environment?

[diegosalvi]

@jiazhai I can check with this commit next week :)

[eolivelli]

@diegosalvi we are going to cut a release soon

@zymap already started a discussion on dev@bookkeeper.apache.org

[eolivelli]

Please take time to vote on the release candidate when it will be prepared