7.7.0 meta ticket (Part 2)

[codebrain]

7.7.0 meta ticket

The following tickets tracks known (and potential) changes to the high level REST client.

Part 1: #4525
Part 2: #4533
Part 3: #4534

Elasticsearch Release Notes

The following release notes were extracted 2020 04 01.

This list needs to be reviewed for potential changes to the high level REST client. Note, it will contain references to new APIs as tracked in #4525 .

Enhancements

Aggregations

• Try to save memory on aggregations http://github.com/elastic/elasticsearch/pull/53793 [#53793]

• Speed up partial reduce of terms aggregations http://github.com/elastic/elasticsearch/pull/53216 [#53216] (issue: http://github.com/elastic/elasticsearch/issues/51857 [#51857])

• Add histogram field type support to boxplot aggs http://github.com/elastic/elasticsearch/pull/52265 [#52265] (issues: http://github.com/elastic/elasticsearch/issues/33112 [#33112], http://github.com/elastic/elasticsearch/issues/52233 [#52233])

• Implement top_metrics agg http://github.com/elastic/elasticsearch/pull/51155 [#51155] (issue: http://github.com/elastic/elasticsearch/issues/48069 [#48069])

• Bucket aggregation circuit breaker optimization. http://github.com/elastic/elasticsearch/pull/46751 [#46751]

Authentication

• Add exception metadata for disabled features http://github.com/elastic/elasticsearch/pull/52811 [#52811] (issues: http://github.com/elastic/elasticsearch/issues/47759 [#47759], http://github.com/elastic/elasticsearch/issues/52311 [#52311], http://github.com/elastic/elasticsearch/issues/55255 [#55255])

• Validate role templates before saving role mapping http://github.com/elastic/elasticsearch/pull/52636 [#52636] (issue: http://github.com/elastic/elasticsearch/issues/48773 [#48773])

• Add support for secondary authentication http://github.com/elastic/elasticsearch/pull/52093 [#52093]

• Expose API key name to the ingest pipeline http://github.com/elastic/elasticsearch/pull/51305 [#51305] (issues: http://github.com/elastic/elasticsearch/issues/46847 [#46847], http://github.com/elastic/elasticsearch/issues/49106 [#49106])

• Disallow Password Change when authenticated by Token http://github.com/elastic/elasticsearch/pull/49694 [#49694] (issue: http://github.com/elastic/elasticsearch/issues/48752 [#48752])

Authorization

• Allow kibana_system to create and invalidate API keys on behalf of other users http://github.com/elastic/elasticsearch/pull/53824 [#53824] (issue: http://github.com/elastic/elasticsearch/issues/48716 [#48716])

• Add "grant_api_key" cluster privilege http://github.com/elastic/elasticsearch/pull/53527 [#53527] (issues: http://github.com/elastic/elasticsearch/issues/48716 [#48716], http://github.com/elastic/elasticsearch/issues/52886 [#52886])

• [APM] Allow kibana to collect APM telemetry in background task http://github.com/elastic/elasticsearch/pull/52917 [#52917] (issue: http://github.com/elastic/elasticsearch/issues/50757 [#50757])

• Add the new 'maintenance' privilege containing 4 actions (#29998) http://github.com/elastic/elasticsearch/pull/50643 [#50643]

Cluster Coordination

• Describe STALE_STATE_CONFIG in ClusterFormationFH http://github.com/elastic/elasticsearch/pull/53878 [#53878] (issue: http://github.com/elastic/elasticsearch/issues/53734 [#53734])

Distributed

• Flush instead of synced-flush inactive shards http://github.com/elastic/elasticsearch/pull/49126 [#49126] (issues: http://github.com/elastic/elasticsearch/issues/31965 [#31965], http://github.com/elastic/elasticsearch/issues/48430 [#48430])

Engine

• Restore off-heap loading for term dictionary in ReadOnlyEngine http://github.com/elastic/elasticsearch/pull/53713 [#53713] (issues: http://github.com/elastic/elasticsearch/issues/43158 [#43158], http://github.com/elastic/elasticsearch/issues/51247 [#51247])

• Separate translog from index deletion conditions http://github.com/elastic/elasticsearch/pull/52556 [#52556]

• Move the terms index of _id off-heap. http://github.com/elastic/elasticsearch/pull/52405 [#52405] (issue: http://github.com/elastic/elasticsearch/issues/42838 [#42838])

• Cache completion stats between refreshes http://github.com/elastic/elasticsearch/pull/51991 [#51991] (issue: http://github.com/elastic/elasticsearch/issues/51915 [#51915])

• Use local checkpoint to calculate min translog gen for recovery http://github.com/elastic/elasticsearch/pull/51905 [#51905] (issue: http://github.com/elastic/elasticsearch/issues/49970 [#49970])

Features/CAT APIs

• /_cat/shards support path stats http://github.com/elastic/elasticsearch/pull/53461 [#53461]

• Allow _cat indices & aliases to use indices options http://github.com/elastic/elasticsearch/pull/53248 [#53248] (issue: http://github.com/elastic/elasticsearch/issues/52304 [#52304])

Features/Features

• Enable deprecation checks for removed settings http://github.com/elastic/elasticsearch/pull/53317 [#53317]

Features/ILM+SLM

• Hide ILM & SLM history aliases http://github.com/elastic/elasticsearch/pull/53564 [#53564]

• Avoid race condition in ILMHistorySotre http://github.com/elastic/elasticsearch/pull/53039 [#53039] (issues: http://github.com/elastic/elasticsearch/issues/50353 [#50353], http://github.com/elastic/elasticsearch/issues/52853 [#52853])

• Make FreezeStep retryable http://github.com/elastic/elasticsearch/pull/52540 [#52540]

• Make DeleteStep retryable http://github.com/elastic/elasticsearch/pull/52494 [#52494]

• Convert ILM and SLM histories into hidden indices http://github.com/elastic/elasticsearch/pull/51456 [#51456]

• Expose master timeout for ILM actions http://github.com/elastic/elasticsearch/pull/51130 [#51130] (issue: http://github.com/elastic/elasticsearch/issues/44136 [#44136])

• Adding best_compression http://github.com/elastic/elasticsearch/pull/49974 [#49974]

Features/Ingest

• Reduce log level for pipeline failure http://github.com/elastic/elasticsearch/pull/54097 [#54097] (issue: http://github.com/elastic/elasticsearch/issues/51459 [#51459])

• Add empty_value parameter to CSV processor http://github.com/elastic/elasticsearch/pull/51567 [#51567]

• Add pipeline name to ingest metadata http://github.com/elastic/elasticsearch/pull/50467 [#50467] (issue: http://github.com/elastic/elasticsearch/issues/42106 [#42106])

Features/Java High Level REST Client

• SourceExists HLRC uses GetSourceRequest instead of GetRequest http://github.com/elastic/elasticsearch/pull/51789 [#51789] (issue: http://github.com/elastic/elasticsearch/issues/50885 [#50885])

• Add Get Source API to the HLRC http://github.com/elastic/elasticsearch/pull/50885 [#50885] (issue: http://github.com/elastic/elasticsearch/issues/47678 [#47678])

Features/Stats

• Add analysis components and mapping types to the usage API. http://github.com/elastic/elasticsearch/pull/51031 [#51031]

Features/Watcher

• Make watch history indices hidden http://github.com/elastic/elasticsearch/pull/52962 [#52962] (issue: http://github.com/elastic/elasticsearch/issues/50251 [#50251])

• Upgrade to the latest OWASP HTML sanitizer http://github.com/elastic/elasticsearch/pull/50765 [#50765] (issue: http://github.com/elastic/elasticsearch/issues/50395 [#50395])

Infra/Core

• Allow keystore add to handle multiple settings http://github.com/elastic/elasticsearch/pull/54229 [#54229] (issue: http://github.com/elastic/elasticsearch/issues/54191 [#54191])

• Report parser name and location in XContent deprecation warnings http://github.com/elastic/elasticsearch/pull/53805 [#53805]

• Report parser name and location in XContent deprecation warnings http://github.com/elastic/elasticsearch/pull/53752 [#53752]

• Deprecate all variants of a ParseField with no replacement http://github.com/elastic/elasticsearch/pull/53722 [#53722]

• Add a helper method to parse maps of objects http://github.com/elastic/elasticsearch/pull/53096 [#53096]

• Allow specifying an exclusive set of fields on ObjectParser http://github.com/elastic/elasticsearch/pull/52893 [#52893]

• Implement hidden aliases http://github.com/elastic/elasticsearch/pull/52547 [#52547] (issue: http://github.com/elastic/elasticsearch/issues/52304 [#52304])

• Introduce system index APIs for Kibana http://github.com/elastic/elasticsearch/pull/52385 [#52385]

• Allow ObjectParsers to specify required sets of fields http://github.com/elastic/elasticsearch/pull/49661 [#49661] (issue: http://github.com/elastic/elasticsearch/issues/48824 [#48824])

Infra/Logging

• Capture stdout and stderr to log4j log http://github.com/elastic/elasticsearch/pull/50259 [#50259] (issue: http://github.com/elastic/elasticsearch/issues/50156 [#50156])

Infra/Packaging

• Introduce aarch64 Docker image http://github.com/elastic/elasticsearch/pull/53936 [#53936] (issue: http://github.com/elastic/elasticsearch/issues/53914 [#53914])

• Introduce jvm.options.d for customizing JVM options http://github.com/elastic/elasticsearch/pull/51882 [#51882] (issue: http://github.com/elastic/elasticsearch/issues/51626 [#51626])

Infra/Plugins

• Allow sha512 checksum without filename for maven plugins http://github.com/elastic/elasticsearch/pull/52668 [#52668] (issue: http://github.com/elastic/elasticsearch/issues/52413 [#52413])

Infra/Scripting

• Scripting: Context script cache unlimited compile http://github.com/elastic/elasticsearch/pull/53769 [#53769] (issue: http://github.com/elastic/elasticsearch/issues/50152 [#50152])

• Scripting: Increase ingest script cache defaults http://github.com/elastic/elasticsearch/pull/53765 [#53765] (issue: http://github.com/elastic/elasticsearch/issues/50152 [#50152])

• Scripting: Per-context script cache, default off (#52855) http://github.com/elastic/elasticsearch/pull/53756 [#53756] (issues: http://github.com/elastic/elasticsearch/issues/50152 [#50152], http://github.com/elastic/elasticsearch/issues/52855 [#52855])

• Scripting: Per-context script cache, default off http://github.com/elastic/elasticsearch/pull/52855 [#52855] (issue: http://github.com/elastic/elasticsearch/issues/50152 [#50152])

• Improve Painless compilation performance for nested conditionals http://github.com/elastic/elasticsearch/pull/52056 [#52056]

• Scripting: Add char position of script errors http://github.com/elastic/elasticsearch/pull/51069 [#51069] (issue: http://github.com/elastic/elasticsearch/issues/50993 [#50993])

Infra/Settings

• Allow keystore add-file to handle multiple settings http://github.com/elastic/elasticsearch/pull/54240 [#54240] (issue: http://github.com/elastic/elasticsearch/issues/54191 [#54191])

• Settings: AffixSettings as validator dependencies http://github.com/elastic/elasticsearch/pull/52973 [#52973] (issue: http://github.com/elastic/elasticsearch/issues/52933 [#52933])

License

• License removal leads back to a basic license http://github.com/elastic/elasticsearch/pull/52407 [#52407] (issue: http://github.com/elastic/elasticsearch/issues/45022 [#45022])

• Refactor license checking http://github.com/elastic/elasticsearch/pull/52118 [#52118] (issue: http://github.com/elastic/elasticsearch/issues/51864 [#51864])

• Add enterprise mode and refactor http://github.com/elastic/elasticsearch/pull/51864 [#51864] (issue: http://github.com/elastic/elasticsearch/issues/51081 [#51081])

MULTIPLE AREA LABELS

• Introduce formal role for remote cluster client http://github.com/elastic/elasticsearch/pull/53924 [#53924]

• [Transform][Rollup] add processing stats to record the time spent for processing results http://github.com/elastic/elasticsearch/pull/53770 [#53770]

• append index name for the source of the cluster put-mapping task http://github.com/elastic/elasticsearch/pull/52690 [#52690]

• Simplify ml license checking with XpackLicenseState internals http://github.com/elastic/elasticsearch/pull/52684 [#52684] (issue: http://github.com/elastic/elasticsearch/issues/52115 [#52115])

• Allow forcemerge in the hot phase for ILM policies http://github.com/elastic/elasticsearch/pull/52073 [#52073] (issue: http://github.com/elastic/elasticsearch/issues/43165 [#43165]) - requires update to XML docs in client

• Remove the query builder serialization from QueryShardException message http://github.com/elastic/elasticsearch/pull/51885 [#51885] (issues: http://github.com/elastic/elasticsearch/issues/48910 [#48910], http://github.com/elastic/elasticsearch/issues/51843 [#51843])

• Percentiles aggregation validation checks for range http://github.com/elastic/elasticsearch/pull/51871 [#51871] (issue: http://github.com/elastic/elasticsearch/issues/51808 [#51808])

• Shortcut query phase using the results of other shards http://github.com/elastic/elasticsearch/pull/51852 [#51852] (issues: http://github.com/elastic/elasticsearch/issues/49601 [#49601], http://github.com/elastic/elasticsearch/issues/51708 [#51708])

• Always rewrite search shard request outside of the search thread pool http://github.com/elastic/elasticsearch/pull/51708 [#51708] (issue: http://github.com/elastic/elasticsearch/issues/49601 [#49601])

• Stop policy on last PhaseCompleteStep instead of TerminalPolic… http://github.com/elastic/elasticsearch/pull/51631 [#51631] (issue: http://github.com/elastic/elasticsearch/issues/48431 [#48431])

Machine Learning

• [ML] Stratified cross validation split for classification http://github.com/elastic/elasticsearch/pull/54087 [#54087]

• [ML] Data frame analytics data counts http://github.com/elastic/elasticsearch/pull/53998 [#53998]

• Verify that the field is aggregatable before attempting cardinality aggregation http://github.com/elastic/elasticsearch/pull/53874 [#53874] (issue: http://github.com/elastic/elasticsearch/issues/53876 [#53876])

• [ML] adds multi-class feature importance support http://github.com/elastic/elasticsearch/pull/53803 [#53803]

• [ML] Data frame analytics analysis stats http://github.com/elastic/elasticsearch/pull/53788 [#53788]

• [ML] Add a model memory estimation endpoint for anomaly detection http://github.com/elastic/elasticsearch/pull/53507 [#53507] (issue: http://github.com/elastic/elasticsearch/issues/53219 [#53219])

• [ML][Inference] adds new default_field_map field to trained models http://github.com/elastic/elasticsearch/pull/53294 [#53294]

• [ML] Improve DF analytics audits and logging http://github.com/elastic/elasticsearch/pull/53179 [#53179]

• [ML] Add indices_options to datafeed config and update http://github.com/elastic/elasticsearch/pull/52793 [#52793] (issue: http://github.com/elastic/elasticsearch/issues/48056 [#48056])

• [ML] Parse and report memory usage for DF Analytics http://github.com/elastic/elasticsearch/pull/52778 [#52778]

• [ML] Adds the class_assignment_objective parameter to classification http://github.com/elastic/elasticsearch/pull/52763 [#52763] (issue: http://github.com/elastic/elasticsearch/issues/52427 [#52427])

• [ML][Inference] Add support for multi-value leaves to the tree model http://github.com/elastic/elasticsearch/pull/52531 [#52531]

• [ML] Make ml internal indices hidden http://github.com/elastic/elasticsearch/pull/52423 [#52423] (issue: http://github.com/elastic/elasticsearch/issues/52420 [#52420])

• [ML] Add _cat/ml/data_frame/analytics API http://github.com/elastic/elasticsearch/pull/52260 [#52260] (issue: http://github.com/elastic/elasticsearch/issues/51413 [#51413])

• [ML] Adds feature importance option to inference processor http://github.com/elastic/elasticsearch/pull/52218 [#52218]

• [ML] Switch poor categorization audit warning to use status field http://github.com/elastic/elasticsearch/pull/52195 [#52195] (issues: http://github.com/elastic/elasticsearch/issues/50749 [#50749], http://github.com/elastic/elasticsearch/issues/51146 [#51146], http://github.com/elastic/elasticsearch/issues/51879 [#51879])

• [ML] Retry persisting DF Analytics results http://github.com/elastic/elasticsearch/pull/52048 [#52048]

• [ML] Improve multiline_start_pattern for CSV in find_file_structure http://github.com/elastic/elasticsearch/pull/51737 [#51737]

• [ML] add _cat/ml/trained_models API http://github.com/elastic/elasticsearch/pull/51529 [#51529] (issue: http://github.com/elastic/elasticsearch/issues/51414 [#51414])

• [ML] add GET _cat/ml/datafeeds http://github.com/elastic/elasticsearch/pull/51500 [#51500] (issue: http://github.com/elastic/elasticsearch/issues/51411 [#51411])

• [ML] Use CSV ingest processor in find_file_structure ingest pipeline http://github.com/elastic/elasticsearch/pull/51492 [#51492] (issue: http://github.com/elastic/elasticsearch/issues/56038 [#56038])

• [ML] Add _cat/ml/anomaly_detectors API http://github.com/elastic/elasticsearch/pull/51364 [#51364]

• [ML][Inference] add tags url param to GET http://github.com/elastic/elasticsearch/pull/51330 [#51330]

• [ML] Make datafeeds work with nanosecond time fields http://github.com/elastic/elasticsearch/pull/51180 [#51180] (issue: http://github.com/elastic/elasticsearch/issues/49889 [#49889])

• [ML] Add audit warning for 1000 categories found early in job http://github.com/elastic/elasticsearch/pull/51146 [#51146] (issue: http://github.com/elastic/elasticsearch/issues/50749 [#50749])

• [ML] Adds support for a global calendars http://github.com/elastic/elasticsearch/pull/50372 [#50372] (issue: http://github.com/elastic/elasticsearch/issues/45013 [#45013])

Mapping

• Wildcard field - add normalizer support http://github.com/elastic/elasticsearch/pull/53851 [#53851]

• Wildcard field - added case insensitive search option http://github.com/elastic/elasticsearch/pull/53814 [#53814]

Network

• Give helpful message on remote connections disabled http://github.com/elastic/elasticsearch/pull/53690 [#53690]

• Add server name to remote info API http://github.com/elastic/elasticsearch/pull/53634 [#53634]

• Log when probe succeeds but full connection fails http://github.com/elastic/elasticsearch/pull/51304 [#51304]

• Encrypt generated key with AES http://github.com/elastic/elasticsearch/pull/51019 [#51019] (issue: http://github.com/elastic/elasticsearch/issues/50843 [#50843])

Ranking

• Adds recall@k metric to rank eval API http://github.com/elastic/elasticsearch/pull/52577 [#52577] (issue: http://github.com/elastic/elasticsearch/issues/51676 [#51676])

SQL

• SQL: jdbc debugging enhancement http://github.com/elastic/elasticsearch/pull/53880 [#53880]

• SQL: use a proper error message for queries directed at empty mapping indices http://github.com/elastic/elasticsearch/pull/52967 [#52967] (issue: http://github.com/elastic/elasticsearch/issues/52865 [#52865])

• SQL: Use calendar_interval of 1d for HISTOGRAMs with 1 DAY intervals http://github.com/elastic/elasticsearch/pull/52749 [#52749] (issue: http://github.com/elastic/elasticsearch/issues/52713 [#52713])

• SQL: use a calendar interval for histograms over 1 month intervals http://github.com/elastic/elasticsearch/pull/52586 [#52586] (issue: http://github.com/elastic/elasticsearch/issues/51538 [#51538])

• SQL: Make parsing of date more lenient http://github.com/elastic/elasticsearch/pull/52137 [#52137] (issue: http://github.com/elastic/elasticsearch/issues/49379 [#49379])

• SQL: Enhance timestamp escaped literal parsing http://github.com/elastic/elasticsearch/pull/52097 [#52097] (issue: http://github.com/elastic/elasticsearch/issues/46069 [#46069])

• SQL: Handle uberjar scenario where the ES jdbc driver file is bundled in another jar http://github.com/elastic/elasticsearch/pull/51856 [#51856] (issue: http://github.com/elastic/elasticsearch/issues/50201 [#50201])

• SQL: Verify Full-Text Search functions not allowed in SELECT http://github.com/elastic/elasticsearch/pull/51568 [#51568] (issue: http://github.com/elastic/elasticsearch/issues/47446 [#47446])

• SQL: Extend the optimisations for equalities http://github.com/elastic/elasticsearch/pull/50792 [#50792] (issue: http://github.com/elastic/elasticsearch/issues/49637 [#49637])

• SQL: add trace logging for search responses coming from server http://github.com/elastic/elasticsearch/pull/50530 [#50530]

• SQL: Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) http://github.com/elastic/elasticsearch/pull/47720 [#47720] (issue: http://github.com/elastic/elasticsearch/issues/46632 [#46632])

• SQL: Enchance output of explain for optimized plan http://github.com/elastic/elasticsearch/pull/33718 [#33718]

Search

• Add heuristics to compute pre_filter_shard_size when unspecified http://github.com/elastic/elasticsearch/pull/53873 [#53873] (issue: http://github.com/elastic/elasticsearch/issues/39835 [#39835])

• Refine SearchProgressListener internal API http://github.com/elastic/elasticsearch/pull/53373 [#53373]

• Check for query cancellation during rewrite http://github.com/elastic/elasticsearch/pull/53166 [#53166] (issue: http://github.com/elastic/elasticsearch/issues/52822 [#52822])

• Implement Cancellable DirectoryReader http://github.com/elastic/elasticsearch/pull/52822 [#52822]

• Address MinAndMax generics warnings http://github.com/elastic/elasticsearch/pull/52642 [#52642] (issue: http://github.com/elastic/elasticsearch/issues/49092 [#49092])

• Clarify when shard iterators get sorted http://github.com/elastic/elasticsearch/pull/52633 [#52633]

• Generalize how queries on _index are handled at rewrite time http://github.com/elastic/elasticsearch/pull/52486 [#52486] (issues: http://github.com/elastic/elasticsearch/issues/49254 [#49254], http://github.com/elastic/elasticsearch/issues/49713 [#49713])

• Short circuited to MatchNone for non-participating slice http://github.com/elastic/elasticsearch/pull/51207 [#51207]

• Add "did you mean" to unknown queries http://github.com/elastic/elasticsearch/pull/51177 [#51177]

• Exclude unmapped fields during max clause limit checking for querying http://github.com/elastic/elasticsearch/pull/49523 [#49523] (issue: http://github.com/elastic/elasticsearch/issues/49002 [#49002])

Snapshot/Restore

• Use Azure Bulk Deletes in Azure Repository http://github.com/elastic/elasticsearch/pull/53919 [#53919] (issue: http://github.com/elastic/elasticsearch/issues/53865 [#53865])

• Only link fd files during source-only snapshot http://github.com/elastic/elasticsearch/pull/53463 [#53463] (issue: http://github.com/elastic/elasticsearch/issues/50231 [#50231])

• Add Blob Download Retries to GCS Repository http://github.com/elastic/elasticsearch/pull/52479 [#52479] (issues: http://github.com/elastic/elasticsearch/issues/46589 [#46589], http://github.com/elastic/elasticsearch/issues/52319 [#52319])

• Better Incrementality for Snapshots of Unchanged Shards http://github.com/elastic/elasticsearch/pull/52182 [#52182]

• Add Region and Signer Algorithm Overrides to S3 Repos http://github.com/elastic/elasticsearch/pull/52112 [#52112] (issue: http://github.com/elastic/elasticsearch/issues/51861 [#51861])

• Allow Parallel Snapshot Restore And Delete http://github.com/elastic/elasticsearch/pull/51608 [#51608] (issue: http://github.com/elastic/elasticsearch/issues/41463 [#41463])

Store

• HybridDirectory should mmap postings. http://github.com/elastic/elasticsearch/pull/52641 [#52641]

[codebrain]

Merged into: #4525