[meta] 6.5.0 Release

[Mpdreamz]

This is a meta issue to keep track of the work we want to do for the 6.4.0 release individual tickets with the v6.4.0 tag will be created for most of these.

Feature parity work

Deprecations

New Features

6.4.0 leftover work

• Add Kerberos authentication support - [Kerberos] Add Kerberos authentication support elasticsearch#32263
• ML. Detectors now support custom rules that enable the user to improve machine learning results by providing some domain-specific knowledge in the form of rule. - [ML] Implement new rules design elasticsearch#31110
• ML. Reverse engineer Grok patterns from categorization results - [ML] Reverse engineer Grok patterns from categorization results elasticsearch#30125
• Add a format option to docvalue_fields. - Add a format option to docvalue_fields. elasticsearch#29639

Enhancements

6.4.0 leftover work

• Return statistics about forecasts as part of the job stats and usage API - [ML] Return statistics about forecasts as part of the jobsstats and usage API elasticsearch#31647
• [ML] Add description to ML filters - [ML] Add description to ML filters elasticsearch#31330
• Rename ranking evaluation quality_level to metric_score - Rename ranking evaluation quality_level to metric_score elasticsearch#32168
• Rename ranking evaluation response unknown_docs section - Rename ranking evaluation response unknown_docs section elasticsearch#32166
• Add Expected Reciprocal Rank metric - Add Expected Reciprocal Rank metric elasticsearch#31891
• Add details section for dcg ranking metric - Add details section for dcg ranking metric elasticsearch#31177

Bug fixes

New API's

6.4.0 leftover work

• xpack.license.get_basic_status.json
• xpack.license.post_start_basic.json
• xpack.ml.delete_calendar.json
• xpack.ml.delete_calendar_event.json
• xpack.ml.delete_calendar_job.json
• xpack.ml.get_calendar_events.json
• xpack.ml.get_calendars.json
• xpack.ml.info.json
• xpack.ml.post_calendar_events.json
• xpack.ml.put_calendar.json
• xpack.ml.put_calendar_job.json
• xpack.ml.get_calendar_job.json
• xpack.ssl.certificates.json
• xpack.ml.update_filter.json
• xpack.rollup.get_rollup_index_caps.json
• xpack.security.delete_privileges.json
• xpack.security.get_privileges.json
• xpack.security.has_privileges.json
• xpack.security.put_privilege.json
• xpack.security.put_privileges.json
• nodes.reload_secure_settings.json

6.5.0 new API

• delete_by_query_rethrottle.json
• update_by_query_rethrottle.json
• xpack.ml.delete_forecast.json
• xpack.ml.find.file_structure.json
• xpack.security.get_user_privileges.json
• ccr.delete_auto_follow_pattern.json
• ccr.follow.json
• ccr.follow_stats.json
• ccr.get_auto_follow_pattern.json
• ccr.pause_follow.json
• ccr.put_auto_follow_pattern.json
• ccr.resume_follow.json
• ccr.stats.json
• ccr.unfollow.json

From the release notes:

• Handles exists query in composite aggs Handles exists query in composite aggs elasticsearch#35758
• [Scripting] Use Number as a return value for BucketAggregationScript [Scripting] Use Number as a return value for BucketAggregationScript elasticsearch#35653
• Require combine and reduce scripts in scripted metrics aggregation Require combine and reduce scripts in scripted metrics aggregation elasticsearch#33452
• Disallow "enabled" attribute change for types in mapping update (The enabled setting is not updatable (and probably shouldn't) elasticsearch#33566) Disallow "enabled" attribute change for types in mapping update (#33566) elasticsearch#33933
• Introduce index store plugins Introduce index store plugins elasticsearch#32375
• Deprecates the standard filter Deprecates the standard filter elasticsearch#33468
• Deprecate negative query boost (Disallow negative query boost elasticsearch#34486) Deprecate negative query boost (#34486) elasticsearch#34512
• Deprecate negative weight in Function Score Query Deprecate negative weight in Function Score Query elasticsearch#33624
• Switch WritePipelineResponse to AcknowledgedResponse Switch WritePipelineResponse to AcknowledgedResponse elasticsearch#32722
• Adds a new auto-interval date histogram Adds a new auto-interval date histogram elasticsearch#28993
• Add script_filter tokenfilter Add script_filter tokenfilter elasticsearch#33431
• Add conditional token filter to elasticsearch Add conditional token filter to elasticsearch elasticsearch#31958
• Add get-user-privileges API Add get-user-privileges API elasticsearch#33928
• Add support for "authorization_realms" Add support for "authorization_realms" elasticsearch#33262
• [CCR] Added auto follow patterns feature [CCR] Added auto follow patterns feature elasticsearch#33118
• Rollup adding support for date field metrics (Add date value type support for certain metrics in Rollup elasticsearch#34185) Rollup adding support for date field metrics (#34185) elasticsearch#34200
• Add early termination support for min/max aggregations Add early termination support for min/max aggregations elasticsearch#33375
• Add early termination support to BucketCollector Add early termination support to BucketCollector elasticsearch#33279
• Add interval response parameter to AutoDateInterval histogram Add interval response parameter to AutoDateInterval histogram elasticsearch#33254
• Token API supports the client_credentials grant Token API supports the client_credentials grant elasticsearch#33106
• Improved IndexNotFoundException's default error message Improved IndexNotFoundException's default error message elasticsearch#34649
• Introduce index settings version Introduce index settings version elasticsearch#34429
• Introduce mapping version to index metadata Introduce mapping version to index metadata elasticsearch#33147
• ingest: processor stats ingest: processor stats  elasticsearch#34724
• ingest: better support for conditionals with simulate?verbose ingest: better support for conditionals with simulate?verbose elasticsearch#34155
• ingest: correctly measure chained pipeline stats ingest: correctly measure chained pipeline stats elasticsearch#33912
• ingest: support simulate with verbose for pipeline processor ingest: support simulate with verbose for pipeline processor elasticsearch#33839
• ingest: Introduce the dissect processor ingest: Introduce the dissect processor elasticsearch#32884
• [ingest] geo-ip performance improvements [ingest] geo-ip performance improvements elasticsearch#33029
• INGEST: Add Pipeline Processor INGEST: Add Pipeline Processor elasticsearch#32473
• Ingest: Add conditional per processor Ingest: Add conditional per processor elasticsearch#32398
• Introduce the dissect library Introduce the dissect library elasticsearch#32297
• INGEST: Enable default pipelines INGEST: Enable default pipelines elasticsearch#32286
• INGEST: Implement Drop Processor INGEST: Implement Drop Processor elasticsearch#32278
• Ingest: Support integer and long hex values in convert Ingest: Support integer and long hex values in convert elasticsearch#32213
• Add region ISO code to GeoIP Ingest plugin Add region ISO code to GeoIP Ingest plugin elasticsearch#31669
• [ML] Add an ingest pipeline definition to structure finder [ML] Add an ingest pipeline definition to structure finder elasticsearch#34350
• [ML] Add a timeout option to file structure finder [ML] Add a timeout option to file structure finder elasticsearch#34117
• [ML] Allow asynchronous job deletion [ML] Allow asynchronous job deletion elasticsearch#34058
• Make certain ML node settings dynamic ([ML] Review and upgrade static node settings and into dynamic cluster settings elasticsearch#33565) Make certain ML node settings dynamic (#33565) elasticsearch#33961
• Adding node_count to ML Usage ([ML] Add the number of ML nodes to the ML feature set usage info elasticsearch#33850) Adding node_count to ML Usage (#33850) elasticsearch#33863
• [ML] Minor improvements to categorization Grok pattern creation [ML] Minor improvements to categorization Grok pattern creation elasticsearch#33353
• Preserve the order of nested documents in the Lucene index Preserve the order of nested documents in the Lucene index elasticsearch#34225
• [ML] Delete forecast API ([ML] Delete forecast API elasticsearch#31134) [ML] Delete forecast API (#31134) elasticsearch#33218
• Use the global doc id to generate random scores Use the global doc id to generate random scores elasticsearch#33599
• Add minimal sanity checks to custom/scripted similarities. (backport) Add minimal sanity checks to custom/scripted similarities. (backport) elasticsearch#33893
• Propagate auto_id_timestamp in primary-replica resync Propagate auto_id_timestamp in primary-replica resync elasticsearch#33964
• [Rollup] Add support for date histo format when searching [Rollup] Add support for date histo format when searching elasticsearch#34537
• [Rollup] Only allow aggregating on multiples of configured interval [Rollup] Only allow aggregating on multiples of configured interval elasticsearch#32052
• Introduce a search_throttled threadpool Introduce a search_throttled threadpool elasticsearch#33732
• Expose max_concurrent_shard_requests in _msearch Expose max_concurrent_shard_requests in _msearch elasticsearch#33016
• Add a limit for graph phrase query expansion Add a limit for graph phrase query expansion elasticsearch#34031
• Introduce private settings Introduce private settings elasticsearch#33327
• Add user-defined cluster metadata Add user-defined cluster metadata elasticsearch#33325
• Add _source-only snapshot repository Add _source-only snapshot repository elasticsearch#32844
• Make XContentBuilder in AliasActions build is_write_index field Make XContentBuilder in AliasActions build is_write_index field elasticsearch#35071

Netcoreapp2.1

or 2.2 whichever comes first 😸.

• Explicitly ship this TFM and get some numbers how the managed handler performs on linux/windows through Tests.Throughput vs the already existing implementations/
• Move our dotnet core IHttpConnection over to HttpClientFactory. We currently do much of the same but if we don't have to maintain our own pooling of HttpClient instances that would eleviate some complexity in our codebase.
• Span based JSON serializer implementation?

Elasticsearch.Net.Abstractions

• Use differ from this repository see: Port Differ to console application elasticsearch-net-abstractions#13
• Move ApiGeneration over.
• Move DocGeneration over.

[Mpdreamz]

Differ is now using the differ we release from: https://github.com/nullean/AssemblyDiffer

https://github.com/elastic/elasticsearch-net/blob/master/build/scripts/Targets.fsx#L68-L73

[Mpdreamz]

Document kerberos integration: #3519

[Mpdreamz]

Format for doc_values fields already went out the door with #3491

[Mpdreamz]

#3520 => Deprecates the standard filter elastic/elasticsearch#33468

[Mpdreamz]

Diff.zip

Attached is the output for AssemblyDiffer for NEST and Elasticsearch.net against 6.4.2 as released on Nuget.

LGTM could use extra pair of eyes to validate @russcam @codebrain

[Mpdreamz]

Moved open bullet points over to #3552

[codebrain]

@Mpdreamz - checked both XMLs, they look good.