7.4 meta ticket

[codebrain]

Ticket to track progress on 7.4 client release.

Breaking changes

Infra/REST API

• Update the schema for the REST API specification #42346 (issue: #35262)

Search

• Use float instead of double for query vectors. #46004

Snapshot/Restore

• Provide an Option to Use Path-Style-Access with S3 Repo #41966 (issue: #41816)

Transforms

• Improve response format of transform stats endpoint #44350 (issue: #43767)

Breaking Java changes

Geo

• Geo: Change order of parameter in Geometries to lon, lat #45332 (issue: #45048)

Deprecations

Infra/Core

• Add node.processors setting in favor of processors #45855

Machine Learning

• Deprecate the ability to update datafeed’s job_id. #44691 (issue: #44615)

New features

Aggregations

• Support Range Fields in Histogram and Date Histogram #45395
• Add Cumulative Cardinality agg (and Data Science plugin) #43661 (issue: #43550)

Analysis

• Add support for inlined user dictionary in the Kuromoji plugin #45489 (issue: #25343)

Authorization

• Add granular privileges for API keys #42020 no changes needed as PR was closed

Features/ILM

• Add Snapshot Lifecycle Management #43934 (issue: #38461)

Features/Watcher

• Add max_iterations configuration to watcher action with foreach execution #45715 (issues: #41997, #45169)

Geo

• New shape field type for indexing Cartesian Geometries #43644 - via [SPATIAL] Backport new ShapeFieldMapper and ShapeQueryBuilder to 7x elasticsearch#45363 - PR: Implement shape property type mapping. #4154
• [SPATIAL] New ShapeQueryBuilder for querying indexed cartesian geometry #45108 (issue: #44980) - PR: Implement shape queries. #4155
• [GEO] New ShapeFieldMapper for indexing cartesian geometries #44980 no client changes needed, accounted for in Implement shape queries. #4155 and Implement shape property type mapping. #4154
• Add Circle Processor #43851 (issue: #43554) - PR: Add Circle processor #4161

Enhancements

Aggregations

• Add more flexibility to MovingFunction window alignment #44360 (issue: #42181)
• Allow pipeline aggs to select specific buckets from multi-bucket aggs #44179

Analysis

• Allow all token/char filters in normalizers #43803 (issue: #43758)

CCR

• Include leases in error message when operations no longer available #45681

CRUD

• Return seq_no and primary_term for noop update #44603 (issue: #42497)

Features/ILM

• Add node setting for disabling SLM #46794 (issue: #38461)
• Include in-progress snapshot for a policy with get SLM policy API #45245
• Add option to filter ILM explain response #44777 (issue: #44189)
• Expose index age in ILM explain output #44457 (issue: #38988)

Features/Indices APIs

• Add Clone Index API #44267 (issue: #44128)
• Add description to force-merge tasks #41365 (issue: #15975)

Features/Watcher

• Watcher reporting: add email warning if CSV attachment contains values that may be interperted as formulas #44460

Machine Learning

• Introduce formal node ML role #45174 (issues: #29943, #43175)
• Improve CSV header row detection in find_file_structure #45099 (issue: #45047) N/A as client doesn't implement find_file_structure
• Outlier detection should only fetch docs that have the analyzed fields #44944 N/A to client
• Persist DatafeedTimingStats with RefreshPolicy.NONE by default #44940 (issue: #44792) N/A to client
• Add result_type field to TimingStats and DatafeedTimingStats documents #44812 - PR: Add results_type field to TimingStats object. #4164
• Implement exponential average search time per hour statistics. #44683 (issue: #29857)
• Add r_squared eval metric to regression #44248 N/A: relates to dataframes APIs
• Adds support for regression.mean_squared_error to eval API #44140 No client changes needed, dataframes are not implemented.
• Add DatafeedTimingStats.average_search_time_per_bucket_ms and TimingStats.total_bucket_processing_time_ms stats #44125 (issue: #29857) @russcam
• Add DatafeedTimingStats to datafeed GetDatafeedStatsAction.Response #43045 (issue: #29857) Not implemented in client

Ranking

• Search enhancement: pinned queries #44345 (issue: #44074)
• Add l1norm and l2norm distances for vectors #44116 (issue: #37947)

Search

• Adds usage stats for vectors: #44512 - PR: Implement vector usage #4151
• Associate sub-requests to their parent task in multi search API #44492 no client changes needed
• Cancel search task on connection close #43332

Snapshot/Restore

• add disable_chunked_encoding configuration #44052 - PR: Implement disable_chucked_encoding s3 repository setting. #4146
• Repository Cleanup Endpoint #43900 - - PR: Implement snapshot repository cleanup #4145

Bug fixes

Analysis

• Fix AnalyzeAction response serialization #44284 (issue: #44078) Doesn't look like this change affects the client.

CRUD

• Do not allow version in Rest Update API #43516 (issue: #42497) no client changes needed

Features/ILM

• Updated slm API spec parameters and URL #44797

Infra/REST API

• Add is_write_index column to cat.aliases #44772 no client changes needed, code change was implemented in 6.4.

Machine Learning

• Rename outlier detection method values knn and tnn to distance_kth_nn and distance_knn respectively to match the API. #598 no changes needed, contained in Dataframes and this is not implemented in the client.

Mapping

• Add include_type_name in indices.exitst REST API spec #43910 (issue: #43905) no changes needed, PR was closed and spec change not implemented in 7.4.

Rollup

• Fix Rollup job creation to work with templates #43943 no changes needed.

Snapshot/Restore

• Make Timestamps Returned by Snapshot APIs Consistent #43148 (issue: #43074) no changes needed.

Transforms

• Respond with 409 status code when failing _stop #44231 (issue: #44103) no changes needed.
• Audit message missing for autostop #43984 (issue: #43977) no changes needed.

[Mpdreamz]

Checked:

• Improve response format of transform stats endpoint #44350 (issue: #43767)

Since we do not support data frame just yet

[Mpdreamz]

• Geo: Change order of parameter in Geometries to lon, lat #45332 (issue: #45048)

Does not affect us.

[Mpdreamz]

• Add node.processors setting in favor of processors #45855

Node settings will be passed to on DynamicDictionary

[Mpdreamz]

• Allow pipeline aggs to select specific buckets from multi-bucket aggs #44179

No impact on us

[Mpdreamz]

• Allow all token/char filters in normalizers #43803 (issue: #43758)

We already pass everything through for the server to validate

[Mpdreamz]

• Include leases in error message when operations no longer available #45681

No impact on us

[Mpdreamz]

• Return seq_no and primary_term for noop update #44603 (issue: #42497)

This actually fixes a bug for us, we assume these were always returned and are non nullable in our mapping

[Mpdreamz]

• Add node setting for disabling SLM #46794 (issue: #38461)

No impact on us.

[Mpdreamz]

TODO elastic/elasticsearch#45245 also mentions last_success which we currently do not map for GetSnapshotLifecycle

[Mpdreamz]

• Add description to force-merge tasks #41365 (issue: #15975)

description already comes through on get task responses

[Mpdreamz]

• Watcher reporting: add email warning if CSV attachment contains values that may be interperted as formulas #44460

Does not affect us afaict.

[russcam]

Support WKT point conversion to geo_point type #44107 (issue: #41821)

For this, I was thinking we can extend GeoLocation to support deserializing from this, similar to what we have done for GeoShapes i.e. deserialize from WKT (and serialize to WKT when has been deserialized from WKT).

[Mpdreamz]

• Cancel search task on connection close #43332

Aborting or closing a request in the client will take care of this.

An abort in .NET results in a tcp reset which does not put the connection in TIME_WAIT a dispose will result in a FIN which does result in a TIME_WAIT.

Either way the connection will be closed and the search task will be aborted on the server

Validated on with WireShark on Linux netcoreapp2.2 which uses the ManagedSocketsHandler.

[Mpdreamz]

• Add l1norm and l2norm distances for vectors #44116 (issue: #37947)

Painless changes only