[ML] Fix character set finder bug with unencodable charsets #33234
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some character sets cannot be encoded and this was tripping up the binary data check in the ML log structure character set finder. The fix is to assume that if ICU4J identifies that some bytes correspond to a character set that cannot be encoded and those bytes contain zeroes then the data is binary rather than text. Fixes elastic#33227
|
Pinging @elastic/ml-core |
|
Marked as |
benwtrent
approved these changes
Aug 29, 2018
droberts195
added a commit
that referenced
this pull request
Aug 29, 2018
Some character sets cannot be encoded and this was tripping up the binary data check in the ML log structure character set finder. The fix is to assume that if ICU4J identifies that some bytes correspond to a character set that cannot be encoded and those bytes contain zeroes then the data is binary rather than text. Fixes #33227
dnhatn
added a commit
that referenced
this pull request
Aug 29, 2018
* master: Painless: Add Bindings (#33042) Update version after client credentials backport Fix forbidden apis on FIPS (#33202) Remote 6.x transport BWC Layer for `_shrink` (#33236) Test fix - Graph HLRC tests needed another field adding to randomisation exception list HLRC: Add ML Get Records API (#33085) [ML] Fix character set finder bug with unencodable charsets (#33234) TESTS: Fix overly long lines (#33240) Test fix - Graph HLRC test was missing field name to be excluded from randomisation logic Remove unsupported group_shard_failures parameter (#33208) Update BucketUtils#suggestShardSideQueueSize signature (#33210) Parse PEM Key files leniantly (#33173) INGEST: Add Pipeline Processor (#32473) Core: Add java time xcontent serializers (#33120) Consider multi release jars when running third party audit (#33206) Update MSI documentation (#31950) HLRC: create base timed request class (#33216) [DOCS] Fixes command page titles HLRC: Move ML protocol classes into client ml package (#33203) Scroll queries asking for rescore are considered invalid (#32918) Painless: Fix Semicolon Regression (#33212) ingest: minor - update test to include dissect (#33211) Switch remaining LLREST usage to new style Requests (#33171) HLREST: add reindex API (#32679)
dnhatn
added a commit
that referenced
this pull request
Aug 29, 2018
* 6.x: Fix forbidden apis on FIPS (#33202) HLRC: Add ML Get Records API (#33085) [ML] Fix character set finder bug with unencodable charsets (#33234) Tests fix - Graph HLRC client overly long line and syncing core’s copy of GraphExploreResponseTests taken from protocol. Related to #33231 Test fix - Graph HLRC test was missing field name to be excluded from randomisation logic Parse PEM Key files leniantly (#33173) Core: Add java time xcontent serializers (#33120) Consider multi release jars when running third party audit (#33206) Update MSI documentation (#31950) [DOCS] Fixes command page titles HLRC: Move ML protocol classes into client ml package (#33203) Painless: Fix Semicolon Regression (#33212)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some character sets cannot be encoded and this was tripping
up the binary data check in the ML log structure character
set finder.
The fix is to assume that if ICU4J identifies that some bytes
correspond to a character set that cannot be encoded and those
bytes contain zeroes then the data is binary rather than text.
Fixes #33227