[meta] 6.4.0 release

[Mpdreamz]

This is a meta issue to keep track of the work we want to do for the 6.4.0 release individual tickets with the v6.4.0 tag will be created for most of these.

Feature parity work

Deprecations

• Add deprecation warnings / changes for nGram and edgeNGram - Deprecate nGram and edgeNGram names for ngram filters elasticsearch#30209
• Investigate copy_settings default - Deprecate not copy settings and explicitly disallow elasticsearch#30404
• Default shards changing from 5 to 1 - Add deprecation warning for default shards elasticsearch#30587
• Deprecate not copy settings and explicitly disallow - Deprecate not copy settings and explicitly disallow elasticsearch#30404 (obsolete copy_settings as of 6.4 it will no longer allow false and wi… #3416)
• Deprecate accepting malformed requests in stored script API - Deprecate accepting malformed requests in stored script API elasticsearch#28939 Check, suspect we are OK
• In the field capabilities API, deprecate support for providing fields in the request body. In the field capabilities API, deprecate support for providing fields in the request body. elasticsearch#30157
• Deprecate indexing and querying a context completion field without context - Deprecates indexing and querying a context completion field without context elasticsearch#30712

New Features

• Add WeightedAvg metric aggregation - Add WeightedAvg metric aggregation elasticsearch#31037 (Add support for weighted average aggregation #3417)
• Add a MovingFunction pipeline aggregation, deprecate MovingAvg - Add a MovingFunction pipeline aggregation, deprecate MovingAvg agg elasticsearch#29594 (Add support for moving function aggregation #3419)
• Add missing_bucket option in the composite agg - Add missing_bucket option in the composite agg elasticsearch#29465
• Expose lucenes RemoveDuplicatesTokenFilter - Expose lucene's RemoveDuplicatesTokenFilter elasticsearch#31275
• Multiplexing token filter - Multiplexing token filter elasticsearch#31208
• Adds a new analysis plugin called analysis_nori that exposes the Lucene Korean analysis module - Expose the Lucene Korean analyzer module in a plugin elasticsearch#30397
• Adding a char_group tokenizer - [Feature] Adding a char_group tokenizer elasticsearch#24186
• Add Kerberos authentication support - [Kerberos] Add Kerberos authentication support elasticsearch#32263
• ML. Detectors now support custom rules that enable the user to improve machine learning results by providing some domain-specific knowledge in the form of rule. - [ML] Implement new rules design elasticsearch#31110
• ML. Reverse engineer Grok patterns from categorization results - [ML] Reverse engineer Grok patterns from categorization results elasticsearch#30125
• Add support for field aliases - Add support for field aliases. elasticsearch#32172
• Add an option to split keyword field on whitespace at query time - Add an option to split keyword field on whitespace at query time elasticsearch#30691
• The new _ignored field enables you to know which fields got ignored at index time because of the ignore_malformed option - Add a new _ignored meta field. elasticsearch#29658
• Add more contexts to painless execute api Add more contexts to painless execute api elasticsearch#30511
• Index phrases Index phrases elasticsearch#30450
• Add a format option to docvalue_fields. - Add a format option to docvalue_fields. elasticsearch#29639

Enhancements

• Add exclusion option to keep_types token filter - Add exclusion option to keep_types token filter elasticsearch#32012
• Added lenient flag for synonym token filter - Added lenient flag for synonym token filter elasticsearch#31484
• Consistent encoder names - Consistent encoder names elasticsearch#29492
• Add support for ignore_unmapped to geo sort - Add support for ignore_unmapped to geo sort elasticsearch#31153
• Add Index UUID to /_stats Response - Add Index UUID to /_stats Response elasticsearch#31871
• INGEST: Extend KV Processor - https://github.com/elastic/elasticsearch/pull/32232/files
• date_index_name processor template resolution. ingest: date_index_name processor template resolution elasticsearch#31841
• Introduction of a bytes processor - ingest: Introduction of a bytes processor elasticsearch#31733
• Extend allowed characters for grok field names - Extend allowed characters for grok field names (#21745) elasticsearch#31653
• Ingest: Add ignore_missing option to RemoveProc - Ingest: Add ignore_missing option to RemoveProc elasticsearch#31693
• Add region ISO code to GeoIP Ingest plugin - Add region ISO code to GeoIP Ingest plugin elasticsearch#31669
• Add ingest-attachment support for per document indexed_chars limit - Add ingest-attachment support for per document indexed_chars limit elasticsearch#31352
• Return statistics about forecasts as part of the job stats and usage API - [ML] Return statistics about forecasts as part of the jobsstats and usage API elasticsearch#31647
• [ML] Add description to ML filters - [ML] Add description to ML filters elasticsearch#31330
• Remove RestGetAllMappingsAction Remove RestGetAllMappingsAction elasticsearch#31129
• _cluster/state should always return cluster_uuid - _cluster/state should always return cluster_uuid elasticsearch#30143
• Rename ranking evaluation quality_level to metric_score - Rename ranking evaluation quality_level to metric_score elasticsearch#32168
• Rename ranking evaluation response unknown_docs section - Rename ranking evaluation response unknown_docs section elasticsearch#32166
• Add Expected Reciprocal Rank metric - Add Expected Reciprocal Rank metric elasticsearch#31891
• Add details section for dcg ranking metric - Add details section for dcg ranking metric elasticsearch#31177
• Add second level of field collapsing - Add second level of field collapsing elasticsearch#31808
• Added max_expansion param to span_multi - Added max_expansion param to span_multi elasticsearch#30913
• Store username on watch execution - Watcher: Store username on watch execution elasticsearch#31873

Bug fixes

• Ensure that ip_range aggregations always return bucket keys. - Ensure that ip_range aggregations always return bucket keys. elasticsearch#30701
• Add support for indexed shape routing in geo_shape query - Add support for indexed shape routing in geo_shape query elasticsearch#30760
• Add support for is_write_index in put-alias body parsing - add support for is_write_index in put-alias body parsing elasticsearch#31674
• Make sure that field collapsing supports field aliases - Make sure that field collapsing supports field aliases. elasticsearch#32648
• Get Mapping API to honour allow_no_indices and ignore_unavailable - Get Mapping API to honour allow_no_indices and ignore_unavailable elasticsearch#31507
• Fix _cluster/state to always return cluster_uuid - Fix _cluster/state to always return cluster_uuid elasticsearch#30656
• Reject forcemerge requests with a body - RestAPI: Reject forcemerge requests with a body elasticsearch#30792

New API's

There a quite a few new API's that need to be mapped:

• xpack.license.get_basic_status.json
• xpack.license.post_start_basic.json
• xpack.ml.delete_calendar.json
• xpack.ml.delete_calendar_event.json
• xpack.ml.delete_calendar_job.json
• xpack.ml.get_calendar_events.json
• xpack.ml.get_calendars.json
• xpack.ml.info.json
• xpack.ml.post_calendar_events.json
• xpack.ml.put_calendar.json
• xpack.ml.put_calendar_job.json
• xpack.ml.get_calendar_job.json
• xpack.rollup.delete_job.json
• xpack.rollup.get_jobs.json
• xpack.rollup.get_rollup_caps.json ([Rollup] Add new capabilities endpoint based on concrete rollup indices elasticsearch#30401)
• xpack.rollup.put_job.json
• xpack.rollup.rollup_search.json
• xpack.rollup.start_job.json
• xpack.rollup.stop_job.json
• xpack.sql.clear_cursor.json (Add support for the SQL API's #3414)
• xpack.sql.query.json (Add support for the SQL API's #3414)
• xpack.sql.translate.json (Add support for the SQL API's #3414)
• xpack.ssl.certificates.json
• scripts_painless_execute.json
• xpack.ml.update_filter.json
• xpack.rollup.get_rollup_index_caps.json
• xpack.security.delete_privileges.json
• xpack.security.get_privileges.json
• xpack.security.has_privileges.json
• xpack.security.put_privilege.json
• xpack.security.put_privileges.json

Tests.Throughput

An unscientific throughput application to measure our different IConnection implementations on how many messages we can pump. For now we only plan to measure _bulk and _search and we'll use the machine learning data set MachineLearningCluster already pulls down.

Will warrant its own [meta] ticket which will be linked here later.

Netcoreapp2.1

or 2.2 whichever comes first 😸.

• Explicitly ship this TFM and get some numbers how the managed handler performs on linux/windows through Tests.Throughput vs the already existing implementations/
• Move our dotnet core IHttpConnection over to HttpClientFactory. We currently do much of the same but if we don't have to maintain our own pooling of HttpClient instances that would eleviate some complexity in our codebase.

Elasticsearch.Net.Abstractions

• Use differ from this repository see: Port Differ to console application elasticsearch-net-abstractions#13
• Move ApiGeneration over.
• Move DocGeneration over.

[Mpdreamz]

Add deprecation warnings / changes for nGram and edgeNGram - elastic/elasticsearch#30209

No action needed we are using the non deprecated formats 😸

[Mpdreamz]

Default shards changing from 5 to 1 - elastic/elasticsearch#30587
No action needed on our end, integration tests still run successfully where we do not explicitly state the number of shards.

[Mpdreamz]

Deprecate accepting malformed requests in stored script API - elastic/elasticsearch#28939

No action need on our end.

[Mpdreamz]

In the field capabilities API, deprecate support for providing fields in the request body. - elastic/elasticsearch#30157

Our mapping of this API always uses GET so no action needed

[Mpdreamz]

Deprecate indexing and querying a context completion field without context - elastic/elasticsearch#30712

We expose a handy CompletionField that folks can use on their POJO's but the validation for this should live on the server.

[Mpdreamz]

Consistent encoder names - elastic/elasticsearch#29492

Ascii docs change only.

[Mpdreamz]

The new _ignored field enables you to know which fields got ignored at index time because of the ignore_malformed option - elastic/elasticsearch#29658

This new field gets automatically stored. Doesn't like like an action is needed on our end.

[Mpdreamz]

date_index_name processor template resolution. - elastic/elasticsearch#31841

Doesn't look like this requires an action on our end (no additional properties).

[Mpdreamz]

Extend allowed characters for grok field names - elastic/elasticsearch#31653

No action needed on our end.

[Mpdreamz]

Add region ISO code to GeoIP Ingest plugin - elastic/elasticsearch#31669

No action needed unless we want to include another default implementation for the GeoIP target field class.

[Mpdreamz]

Remove RestGetAllMappingsAction - elastic/elasticsearch#31129

This has no consequences on our end afaict.

[Mpdreamz]

Added max_expansion param to span_multi - elastic/elasticsearch#30913
No action needed on our part, PR ended up reusing and adhering to using bool max clauses rather than introducing a new option.

[Mpdreamz]

Add support for indexed shape routing in geo_shape query- elastic/elasticsearch#30760

Our IFieldLookup already exposes routing 😸

[Mpdreamz]

Make sure that field collapsing supports field aliases - elastic/elasticsearch#32648

Requires no changes on our end.

[Mpdreamz]

Fix _cluster/state to always return cluster_uuid - elastic/elasticsearch#30656

No action needed, fixes a behaviour from a previous PR which we covered earlier as well.