Java high-level REST client completeness

[javanna]

This is a meta issue to track completeness of the Java REST high-level Client in terms of supported API. The following list includes all the REST API that Elasticsearch exposes to date, and that are also exposed by the Transport Client. The ones marked as done are already supported by the high-level REST client, while the others need to be added. Every group is sorted based on an estimation around how important the API is, from more important to less important. Each API is also assigned a rank (easy, medium, hard) that expresses how difficult adding support for it is expected to be.

The API listed as "Not Required" won't need to be supported before the transport client is removed from the master branch (next major version). Such API are mainly administrative API that are not likely to be used from a Java application. They generally return heavy responses and make it hard to reuse response objects from the transport client as they expose internal objects that in some cases cannot even be parsed back entirely based on the information returned at REST. We considered returning those as maps of maps but that’s also easy to achieve using the low-level REST client hence we decided to not implement them for the time being.

Top-level APIs

• ping (easy)
• info (easy)
• index (medium)
• update (medium)
• delete (medium)
• bulk (hard)
• get (medium)
• exists (easy)
• multi get (medium) Add multi get api to the high level rest client #27337
• search (very hard)
• search scroll (easy)
• clear scroll (easy)
• multi search (hard) Add msearch api to high level client #27274
• update by query (medium) @sohaibiftikhar
• delete by query (medium) @sohaibiftikhar
• reindex (medium) @sohaibiftikhar
• [] reindex with wait_for_completion=false creates task @pgomulka
• rethrottle (reindex, update by query, delete by query) HLRC: Add throttling for update & delete-by-query #33951
• search template (medium) Add support for search templates to the high-level REST client. #30473
• render search template (easy) (included in search template API) Add support for search templates to the high-level REST client. #30473
• multi search templates (medium) Add MultiSearchTemplate support to High Level Rest client #30836
• term vectors (hard) HLRC API for _termvectors #33447
• multi term vectors (hard) @mayya-sharipova HLRC for _mtermvectors #35266
• explain (medium) Add rest highlevel explain API #31387
• field caps (easy) Add support for field capabilities to the high-level REST client. #29664
• put stored script (easy) Add put stored script support to high-level rest client #31323
• delete stored script (easy) Add get stored script and delete stored script to high level REST API #31355
• get stored script (medium) Add get stored script and delete stored script to high level REST API #31355

Indices API

• create index (easy)
• delete index (easy)
• indices exist (easy) Add support for indices exists to REST high level client #27384
• update alias (medium) Add Indices Aliases API to the high level REST client #27876
• exists alias (easy) REST high-level client: add support for exists alias #28332
• get alias (medium) Add Get Aliases API to the high-level REST client #28799
• [ ] types exist (easy)
• put mapping (easy) Added Put Mapping API to high-level Rest client (#27205) #27869
• open index (easy)
• close index (easy)
• refresh (easy) Add Refresh API for RestHighLevelClient #27799
• flush (easy) REST high-level client: add flush API #28852
• update index settings (easy) REST high-level client: add support for Indices Update Settings API #28892
• get index settings (easy) Add Get Settings API support to java high-level rest client #29229
• clear cache (easy) REST high-level client: add clear cache API #28866
• force merge (easy) REST high-level client: add force merge API #28896
• shrink (easy) REST high-level client: add support for split and shrink index API #28425
• split (easy) REST high-level client: add support for split and shrink index API #28425
• rollover (easy) REST high-level client: add support for Rollover Index API #28698
• synced flush (medium) (exposes ShardRouting, hard to reconstruct the whole response from info returned via REST) REST high-level client: add synced flush API (2) #30650
• get index (medium) REST high-level client: add get index API #31703
• get mappings (easy) Add get mappings support to high-level rest client #30889
• get field mappings (medium) Add get field mappings to High Level REST API Client #31423
• put index template (medium) Add put index template api to high level rest client #30400
• delete index template (easy) HLRC: Add delete template API #36320
• get index templates (medium) HLRest: Add get index templates API #31161
• validate query (medium) REST high-level client: add validate query API #31077
• analyze (hard) Add analyze API to high-level rest client #31577

Not required

• shard stores (medium)
• upgrade (easy) (to be removed?)
• upgrade status (easy) (to be removed?)
• segments (hard) (exposes ShardRouting)
• recoveries (hard) (exposes ShardRouting, DiscoveryNode)
• indices stats (hard) (exposes ShardRouting and a lot of other objects)

Snapshot API

• get repositories Add GET Repository High Level REST API #30362
• create repository Add PUT Repository High Level REST API #30501
• verify repository Add Verify Repository High Level REST API #30934
• delete repository Add Delete Repository High Level REST API #30666
• create snapshot (medium) Add Create Snapshot to High-Level Rest Client #31215
• snapshots status (medium) Add Snapshots Status API to High Level Rest Client #31515
• get snapshots (medium) Add Get Snapshots High Level REST API #31537
• delete snapshot (easy) Add Delete Snapshot High Level REST API #31393
• restore snapshot (medium) Add Restore Snapshot High Level REST API #32155

Ingest API

• put ingest pipeline (easy) REST high-level client: add put ingest pipeline API #30793
• delete ingest pipeline (easy) REST high-level client: add delete ingest pipeline API #30865
• get ingest pipeline (easy) REST high-level client: add get ingest pipeline API #30847
• simulate ingest pipeline (medium) REST high-level client: add simulate pipeline API #31158

Tasks API

• list tasks (medium) Rest High Level client: Add List Tasks #29546
• get task (medium) HLRC support for getTask #35166
• cancel task (easy) high level REST api: cancel task #30745

Cluster API

• cluster health (medium) REST high-level client: add Cluster Health API #29331
• update cluster settings (easy) Add Cluster Put Settings API to the high level REST client #28633
• get cluster settings (medium) REST high-level client: add cluster get settings API #31706 (doesn't have its own Response object, exposed at REST only)

Not required

• search shards (medium) (exposes ShardRouting, DiscoveryNode and requires parsing back QueryBuilder)
• pending cluster tasks (easy)
• allocation explain (hard) (exposes ShardRouting)
• cluster state (hard) (exposes ClusterState)
• reroute (easy if done after cluster state API, returns the entire cluster state)
• nodes info (hard) (exposes DiscoveryNode and a lot of other objects)
• nodes stats (hard) (exposes ShardRouting and a lot of other objects)
• cluster stats (hard) (exposes DiscoveryNode, requires nodes info and nodes stats)
• hot threads (easy) (exposes DiscoveryNode)
• nodes usage (medium) (exposes DiscoveryNode)

REST only API

There are a number of API that are exposed via REST but not via the Transport Client. They don't necessarily have to be implemented if the goal is feature parity with the Transport Client, yet we should probably have a look at why they were not added to the Transport Client and whether it makes sense to add their support to the high-level REST Client or not. I don't think it makes sense to add support for cat API and ingest processor grok, hence I took them out already.

• cluster remote info
• count Add document _count API support to Rest High Level Client.  #31868
• get source
• source exists HLRC - add support for source exists API #34519
• delete alias
• indices template exist @andyb-elastic ([hlrc] add index templates exist API #36132)
• get upgrade
• ingest processor grok
• cat API: aliases, allocation, count, fielddata, health, help, indices, master, nodeattrs, nodes, pending tasks, plugins, recovery, repositories, segments, shards, snapshots, tasks, templates, threadpool

How to add support for a new API

Look at some of the already supported API and existing PRs that have been merged:

• Add Index API to High Level Rest Client (Add Index API to High Level Rest Client #23040)
• Add BulkRequest support to High Level Rest client (Add BulkRequest support to High Level Rest client #23312)
• Add delete API to the High Level Rest Client (Add delete API to the High Level Rest Client #23187)
• Add UpdateRequest support to High Level Rest client (Add UpdateRequest support to High Level Rest client #23266)
• Added Delete Index support to high-level REST client (Added Delete Index support to high-level REST client #27019)

The common tasks in each of the above PRs are:

• add fromXContent method to existing response class currently used by transport client and corresponding unit tests that make use of fields shuffling as well as random fields insertion (in order to test forward compatibility). That usually means adding a test for the response object that extends AbstractXContentTestCase where supportsUnknownFields() returns true as well as assertToXContentEquivalence. There are cases where we can't insert random fields everywhere, which then require to also override the getRandomFieldsExcludeFilter() method which returns path that should be excluded when injecting random fields. Given the randomizations applied, it makes sense to run this type of test locally with -Dtests.iters=50 argument just to make sure that it is consistently green.
• add new method to Request class which translates the input request into the internal REST request representation that holds method, url, endpoint, params etc. and add corresponding tests to RequestTests
• add new method to RestHighLevelClient, possibly also its async variant when it makes sense, we may not want to add async variants to every single method, so we decide case by case. The name of the new method must match what is defined in our REST spec including the namespace.
• add integration test that extends ESRestHighLevelClientTestCase that tests the new method end-to-end by sending REST requests to an external cluster.
• add docs page. To check how docs are rendered and whether the links between docs pages and docs snippets work ok, run the following command from the root of your local checkout of the Elasticsearch repository: /path/to/elastic/docs/build_docs.pl --doc docs/java-rest/index.asciidoc --chunk 1 --out ~/temp/asciidoc --open . This requires also a local checkout of the docs repository, where the perl script is located.

Relates #29827

[javanna]

I updated the description of the issue by assigning each API a rank from 1 to 3 based on how difficult it should be to add support for it to the high-level REST client. Criterias were mainly how big the request is to serialize and how big the response is to parse back.

[clintongormley]

thanks @javanna - I've separated the APIs into "important" and "optional" lists, where optional APIs are ones that will seldom be used from applications other than monitoring applications or tests. If anybody disagrees with my selection, feel free to mention which APIs should be marked as important.

[slovdahl]

Not using the high-level REST client yet, but I would really have expected that multi-get was supported.

[hariso]

@javanna This might be a bloody stupid question, but: In which way does someone pick up an API and starts working on it? Without risking that someone did the same.: )

[nik9000]

@javanna This might be a bloody stupid question, but: In which way does someone pick up an API and starts working on it? Without risking that someone did the same.: )

You add a comment here saying you are working on it.

[hariso]

Thanks @nik9000 !