[HUDI-6825] Use UTF_8 to encode String to byte array in all places

[yihua]

Change Logs

This PR unifies the encoding of Java String to byte array in Hudi, especially for writing bytes to the storage, by using UTF_8 encoding only. There are places calling String.getBytes() which are fixed by this PR. String.getBytes() uses the platform's default charset and encoding scheme. Note that the default character encoding scheme on Windows is ANSI, while the default character encoding scheme on Linux is UTF-8. The PR has no impact on Linux systems writing and reading Hudi tables.

These are the places that used String.getBytes() before, which are fixed to use UTF_8 encoding.

MercifulJsonConverter#generateBytesTypeHandler
HoodieJsonPayload#compressData
SimpleBloomFilter#write
HFileBootstrapIndexWriter#writeNextSourceFileMapping
HoodiePartitionMetadata#writeMetafile
HoodieHFileDataBlock#serializeRecords
HoodieLogBlock#getLogMetadataBytes
HoodieDefaultTimeline#setInstants
RocksDBDAO
HoodieAvroHFileReader
HoodieAvroHFileWriter
HoodieAvroOrcWriter
HoodieTableMetadataUtil#convertMetadataToBloomFilterRecords
HoodieTableMetadataUtil#readBloomFilter

We don't intend to make these backwards compatible for Windows which can be broken because of different default charset.

Impact

Make sure the encoding of Java String and storage bytes in a Hudi table does not depend on platforms.

Risk level

low

Documentation Update

N/A

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[codope]

So this is a breaking change for Windows? Any path for users to safely migrate?

[hudi-bot]

CI report:

• 75b9207 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[yihua]

So this is a breaking change for Windows? Any path for users to safely migrate?

I've updated the PR description. We don't intend to make these backwards compatible for Windows which can be broken because of different default charset.