-
Notifications
You must be signed in to change notification settings - Fork 729
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Added client helpers * Updated test * The search helper should return only the documents * Added code comments * Fixed bug * Updated test * Removed bulkSize and added flushBytes * Updated test * Added concurrency * Updated test * Added support for 429 handling in the scroll search helper * Updated test * Updated stats count * Updated test * Fix test * Use client maxRetries as default * Updated type definitions * Refactored bulk helper to be more consistent with the client api * Updated test * Improved error handling, added refreshOnCompletion option and forward additinal options to the bulk api * Updated type definitions * Updated test * Fixed test on Node v8 * Updated test * Added TODO * Updated docs * Added Node v8 note * Updated scripts * Removed useless files * Added helpers to integration test * Fix cli argument position * Moar fixes * Test run elasticsearch in github actions * Use master action version * Add vm.max_map_count step * Test new action setup * Added Configure sysctl limits step * Updated action to latest version * Don't run helpers integration test in jenkins * Run helpers integratino test also with Node v10 * Updated docs * Updated docs * Updated helpers type definitions * Added test for helpers type definitions * Added license header
- Loading branch information
Showing
24 changed files
with
7,654 additions
and
11 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,272 @@ | ||
| [[client-helpers]] | ||
| == Client Helpers | ||
|
|
||
| The client comes with an handy collection of helpers to give you a more comfortable experience with some APIs. | ||
|
|
||
| CAUTION: The client helpers are experimental, and the API may change in the next minor releases. | ||
| If you are using the client with Node.js v8 you should run your code with the `--harmony-async-iteration` argument. + | ||
| eg: `node --harmony-async-iteration index.js` | ||
|
|
||
| === Bulk Helper | ||
| Running Bulk requests can be complex due to the shape of the API, this helper aims to provide a nicer developer experience around the Bulk API. | ||
|
|
||
| ==== Usage | ||
| [source,js] | ||
| ---- | ||
| const { createReadStream } = require('fs') | ||
| const split = require('split2') | ||
| const { Client } = require('@elastic/elasticsearch') | ||
| const client = new Client({ node: 'http://localhost:9200' }) | ||
| const result = await client.helpers.bulk({ | ||
| datasource: createReadStream('./dataset.ndjson').pipe(split()), | ||
| onDocument (doc) { | ||
| return { | ||
| index: { _index: 'my-index' } | ||
| } | ||
| } | ||
| }) | ||
| console.log(result) | ||
| // { | ||
| // total: number, | ||
| // failed: number, | ||
| // retry: number, | ||
| // successful: number, | ||
| // time: number, | ||
| // bytes: number, | ||
| // aborted: boolean | ||
| // } | ||
| ---- | ||
|
|
||
| To create a new instance of the Bulk helper, you should access it as shown in the example above, the configuration options are: | ||
| [cols=2*] | ||
| |=== | ||
| |`datasource` | ||
| a|An array or a readable stream with the data you need to index/create/update/delete. | ||
| It can be an array of strings or objects, but also a stream of json strings or JavaScript objects. + | ||
| If it is a stream, we recommend to use the https://www.npmjs.com/package/split2[`split2`] package, that will split the stream on new lines delimiters. + | ||
| This parameter is mandatory. | ||
| [source,js] | ||
| ---- | ||
| const { createReadStream } = require('fs') | ||
| const split = require('split2') | ||
| const b = client.helpers.bulk({ | ||
| // if you just use split(), the data will be used as array of strings | ||
| datasource: createReadStream('./dataset.ndjson').pipe(split()) | ||
| // if you need to manipulate the data, you can pass JSON.parse to split | ||
| datasource: createReadStream('./dataset.ndjson').pipe(split(JSON.parse)) | ||
| }) | ||
| ---- | ||
|
|
||
| |`onDocument` | ||
| a|A function that will be called for each document of the datasource. Inside this function you can manipulate the document and you must return the operation you want to execute with the document. Look at the link:{ref}/docs-bulk.html[Bulk API documentation] to see the supported operations. + | ||
| This parameter is mandatory. | ||
| [source,js] | ||
| ---- | ||
| const b = client.helpers.bulk({ | ||
| onDocument (doc) { | ||
| return { | ||
| index: { _index: 'my-index' } | ||
| } | ||
| } | ||
| }) | ||
| ---- | ||
|
|
||
| |`onDrop` | ||
| a|A function that will be called for everytime a document can't be indexed and it has reached the maximum amount of retries. | ||
| [source,js] | ||
| ---- | ||
| const b = client.helpers.bulk({ | ||
| onDrop (doc) { | ||
| console.log(doc) | ||
| } | ||
| }) | ||
| ---- | ||
|
|
||
| |`flushBytes` | ||
| a|The size of the bulk body in bytes to reach before to send it. Default of 5MB. + | ||
| _Default:_ `5000000` | ||
| [source,js] | ||
| ---- | ||
| const b = client.helpers.bulk({ | ||
| flushBytes: 1000000 | ||
| }) | ||
| ---- | ||
|
|
||
| |`concurrency` | ||
| a|How many request will be executed at the same time. + | ||
| _Default:_ `5` | ||
| [source,js] | ||
| ---- | ||
| const b = client.helpers.bulk({ | ||
| concurrency: 10 | ||
| }) | ||
| ---- | ||
|
|
||
| |`retries` | ||
| a|How many times a document will be retried before to call the `onDrop` callback. + | ||
| _Default:_ Client max retries. | ||
| [source,js] | ||
| ---- | ||
| const b = client.helpers.bulk({ | ||
| retries: 3 | ||
| }) | ||
| ---- | ||
|
|
||
| |`wait` | ||
| a|How much time to wait before retries in milliseconds.+ | ||
| _Default:_ 5000. | ||
| [source,js] | ||
| ---- | ||
| const b = client.helpers.bulk({ | ||
| wait: 3000 | ||
| }) | ||
| ---- | ||
|
|
||
| |`refreshOnCompletion` | ||
| a|If `true`, at the end of the bulk operation it will run a refresh on all indices or on the specified indices. + | ||
| _Default:_ false. | ||
| [source,js] | ||
| ---- | ||
| const b = client.helpers.bulk({ | ||
| refreshOnCompletion: true | ||
| // or | ||
| refreshOnCompletion: 'index-name' | ||
| }) | ||
| ---- | ||
|
|
||
| |=== | ||
|
|
||
| ==== Abort a bulk operation | ||
| If needed, you can abort a bulk operation at any time. The bulk helper returns a https://promisesaplus.com/[thenable], which has an `abort` method. | ||
|
|
||
| NOTE: The abort method will stop the execution of the bulk operation, but if you are using a concurrency higher than one, the operations that are already running will not be stopped. | ||
|
|
||
| [source,js] | ||
| ---- | ||
| const { createReadStream } = require('fs') | ||
| const split = require('split2') | ||
| const { Client } = require('@elastic/elasticsearch') | ||
| const client = new Client({ node: 'http://localhost:9200' }) | ||
| const b = client.helpers.bulk({ | ||
| datasource: createReadStream('./dataset.ndjson').pipe(split()), | ||
| onDocument (doc) { | ||
| return { | ||
| index: { _index: 'my-index' } | ||
| } | ||
| }, | ||
| onDrop (doc) { | ||
| b.abort() | ||
| } | ||
| }) | ||
| console.log(await b) | ||
| ---- | ||
|
|
||
| ==== Passing custom options to the Bulk API | ||
| You can pass any option supported by the link:{ref}/docs-bulk.html#docs-bulk-api-query-params[Bulk API] to the helper, and the helper will use those options in conjuction with the Bulk | ||
| API call. | ||
|
|
||
| [source,js] | ||
| ---- | ||
| const result = await client.helpers.bulk({ | ||
| datasource: [...] | ||
| onDocument (doc) { | ||
| return { | ||
| index: { _index: 'my-index' } | ||
| } | ||
| }, | ||
| pipeline: 'my-pipeline' | ||
| }) | ||
| ---- | ||
|
|
||
| === Search Helper | ||
| A simple wrapper around the search API. Instead of returning the entire `result` object it will return only the search documents result. | ||
|
|
||
| [source,js] | ||
| ---- | ||
| const documents = await client.helpers.search({ | ||
| index: 'stackoverflow', | ||
| body: { | ||
| query: { | ||
| match: { | ||
| title: 'javascript' | ||
| } | ||
| } | ||
| } | ||
| }) | ||
| for (const doc of documents) { | ||
| console.log(doc) | ||
| } | ||
| ---- | ||
|
|
||
| === Scroll Search Helper | ||
| This helpers offers a simple and intuitive way to use the scroll search API. Once called, it returns an https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of[async iterator] which can be used in conjuction with a for-await...of. + | ||
| It handles automatically the `429` error and uses the client's `maxRetries` option. | ||
|
|
||
| [source,js] | ||
| ---- | ||
| const scrollSearch = await client.helpers.scrollSearch({ | ||
| index: 'stackoverflow', | ||
| body: { | ||
| query: { | ||
| match: { | ||
| title: 'javascript' | ||
| } | ||
| } | ||
| } | ||
| }) | ||
| for await (const result of scrollSearch) { | ||
| console.log(result) | ||
| } | ||
| ---- | ||
|
|
||
| ==== Clear a scroll search | ||
|
|
||
| If needed, you can clear a scroll search by calling `result.clear()`: | ||
|
|
||
| [source,js] | ||
| ---- | ||
| for await (const result of scrollSearch) { | ||
| if (condition) { | ||
| await result.clear() | ||
| } | ||
| } | ||
| ---- | ||
|
|
||
| ==== Quickly getting the documents | ||
|
|
||
| If you only need the documents from the result of a scroll search, you can access them via `result.documents`: | ||
|
|
||
| [source,js] | ||
| ---- | ||
| for await (const result of scrollSearch) { | ||
| console.log(result.documents) | ||
| } | ||
| ---- | ||
|
|
||
| === Scroll Documents Helper | ||
|
|
||
| It works in the same way as the scroll search helper, but it returns only the documents instead. Note, every loop cycle will return you a single document, and you can't use the `clear` method. | ||
|
|
||
| [source,js] | ||
| ---- | ||
| const scrollSearch = await client.helpers.scrollDocuments({ | ||
| index: 'stackoverflow', | ||
| body: { | ||
| query: { | ||
| match: { | ||
| title: 'javascript' | ||
| } | ||
| } | ||
| } | ||
| }) | ||
| for await (const doc of scrollSearch) { | ||
| console.log(doc) | ||
| } | ||
| ---- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.