DX-88666: Add changes from branch-3.3.2-dremio to 3.3.6

This list captures the current state of non-upstream changes in our branch
that are not in the public repo.

---Changes cherry-picked to branch-3.3.6-dremio from branch-3.3.2-dremio---
The below changes were on branch-3.3.2-dremio and needed to be brought to
branch-3.3.6-dremio to prevent regressing scenarios these changes addressed.

HADOOP-18928: S3AFileSystem URL encodes twice where Path has trailing / (proposed)
DX-69726: Bumping okie from 1.6.0 to 3.4.0 (CVE-2023-3635)
DX-69726: Bumping okie from 1.6.0 to 3.4.0 (CVE-2023-3635)
DX-66470: Allow for custom shared key signer for ABFS
DX-66673: Backport HADOOP-18602. Remove netty3 dependency
DX-66673: Backport MAPREDUCE-7434. Fix ShuffleHandler tests. Contributed by Tamas Domok
DX-66673: Backport MAPREDUCE-7431. ShuffleHandler refactor and fix after Netty4 upgrade. (apache#5311)
DX-66673: Backport HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4 apache#3259. Contributed by Szilard Nemeth.
DX-66673: Backport HADOOP-17115. Replace Guava Sets usage by Hadoop's own Sets in hadoop-common and hadoop-tools (apache#2985)
HADOOP-18676. jettison dependency override in hadoop-common lib
DX-52816: Downgrade azure-data-lake-store-sdk to 2.3.3 to support dremio version.
DX-52701: Remove node based module by Naveen Kumar
DX-32012: Adding BatchList Iterator for ListFiles by “ajmeera.nagaraju”
DX-18552: Make file status check optional in S3AFileSystem create()
Add flag to skip native tests by Laurent Goujon
DX-21904: Support S3 requester-pays headers by Brandon Huang
DX-21471: Fix checking of use of OAuth credentials with AzureNativeFileSystem
DX-19314: make new kms format configurable
DX-17058 Add FileSystem to META-INF/services
DX-17317 Fix incorrect parameter passed into AzureADAuthenticator-getTokenUsingClientCreds by TiffanyLam
DX-17276 Azure AD support for StorageV1 by James Duong
DX-17276 Add Azure AD support in Dremio's hadoop-azure library for Storage V1 support
unwraps BindException in HttpServer2

---Changes picked up by moving to 3.3.6---
The below changes were changes on branch-3.3.2-dremio that did not need to
come to branch-3.3.6-dremio as the public 3.3.6 branch contained the fixes
already.

DX-67500: Backport HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files.
DX-66673: Backport HADOOP-18079. Upgrade Netty to 4.1.77. (apache#3977)
DX-66673: Backport HADOOP-11245. Update NFS gateway to use Netty4 (apache#2832) (apache#4997)
DX-64051: Bump jettison from 1.1 to 1.5.4 in hadoop/branch-3.3.2-dremio
DX-64051: Bump jettison from 1.1 to 1.5.4 in hadoop/branch-3.3.2-dremio
DX-63800 Bump commons-net from 3.6 to 3.9.0 to address CVE-2021-37533
DX-27168: removing org.codehaus.jackson

Change-Id: I6cdb968e33826105caff96e1c3d2c6313a550689