Skip to content
Merged

v1.0 #2098

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
8bc871e
v1.0
maryamsulemani97 Jan 10, 2023
324f006
update docker
maryamsulemani97 Jan 10, 2023
9916eb5
`too_many_open_files` error
maryamsulemani97 Jan 10, 2023
f51c093
remove two-step dump import
maryamsulemani97 Jan 11, 2023
ad3e065
v1.0: Reject master keys less than 16 bytes
maryamsulemani97 Jan 11, 2023
2b7ef45
v1.0: primary key inference
maryamsulemani97 Jan 11, 2023
c3e0b14
v1.0: Multi-word synonyms are treated as phrases
maryamsulemani97 Jan 11, 2023
d7eb6e5
v1.0: add new error tye `system`
maryamsulemani97 Jan 11, 2023
295c9e8
update postman collection file
guimachiavelli Jan 12, 2023
5cab29b
search: clarify query term normalization
guimachiavelli Jan 12, 2023
ffca789
update `updating.md`
maryamsulemani97 Jan 16, 2023
c3c56b8
remove mentions of `--disable-auto-batching`
guimachiavelli Jan 16, 2023
284c8fe
update `system` error
maryamsulemani97 Jan 16, 2023
2a4d7ff
v1.0: update to add * for date filters in task API
maryamsulemani97 Jan 16, 2023
723da74
fix typo+spacing
maryamsulemani97 Jan 16, 2023
24671ee
v1.0: Accepted OS versions for the binary
maryamsulemani97 Jan 16, 2023
275af5a
dumps: rename CLI/env var option name
guimachiavelli Jan 16, 2023
83b3615
remove database size options and update disk space usage documentation
guimachiavelli Jan 17, 2023
5f9a87a
update based on review
maryamsulemani97 Jan 17, 2023
2ee4e7f
Apply suggestions from code review
maryamsulemani97 Jan 17, 2023
9ba542b
snapshots: remove `snapshot-interval-sec`, update `schedule-snapshot`
guimachiavelli Jan 17, 2023
7ffe4da
Merge branch 'main' into v1.0
guimachiavelli Jan 17, 2023
696173b
update config file link
guimachiavelli Jan 17, 2023
d587495
v1.0: update `error_codes.md`
maryamsulemani97 Jan 17, 2023
a111eb1
telemetry: remove database size datapoints
guimachiavelli Jan 17, 2023
84d64f1
telemetry: update `--schedule-snapshot` datapoint
guimachiavelli Jan 17, 2023
55d07ea
update master key instance option+add autogenerated master key
maryamsulemani97 Jan 18, 2023
f7a43a7
remove extra space
maryamsulemani97 Jan 18, 2023
c720619
update `supported_os.md`
maryamsulemani97 Jan 18, 2023
1bebc67
improve `--schedule-snapshot` wording
guimachiavelli Jan 18, 2023
0528d70
update error codes
maryamsulemani97 Jan 18, 2023
a186da8
replace export with --master-key
maryamsulemani97 Jan 19, 2023
30c62e2
Apply suggestions from code review
maryamsulemani97 Jan 19, 2023
10c2bbe
update based on review
maryamsulemani97 Jan 19, 2023
62770a7
Apply suggestions from code review
maryamsulemani97 Jan 19, 2023
23b528c
update based on review
maryamsulemani97 Jan 19, 2023
784e884
database size limits: address reviewer feedback
guimachiavelli Jan 19, 2023
26ead3e
quick start: improve auto-batching wording
guimachiavelli Jan 19, 2023
9314d95
update based on review
maryamsulemani97 Jan 23, 2023
a3606bc
Update learn/configuration/synonyms.md
maryamsulemani97 Jan 23, 2023
91f101d
update based on review
maryamsulemani97 Jan 23, 2023
df95292
Update learn/configuration/synonyms.md
maryamsulemani97 Jan 23, 2023
fb32b78
Update learn/advanced/known_limitations.md
guimachiavelli Jan 23, 2023
31155ef
update based on review
maryamsulemani97 Jan 24, 2023
384e8d5
Apply suggestions from code review
maryamsulemani97 Jan 24, 2023
9c1be36
list errors in alphabetical order
maryamsulemani97 Jan 24, 2023
248b26a
Update reference/errors/error_codes.md
maryamsulemani97 Jan 24, 2023
7d307f0
update order
maryamsulemani97 Jan 24, 2023
a317607
update based on review
maryamsulemani97 Jan 24, 2023
9654a8c
Apply suggestions from code review
maryamsulemani97 Jan 24, 2023
09f8cd2
update based on review
maryamsulemani97 Jan 24, 2023
6a94a54
quick start: auto-batching: address reviewer feedback
guimachiavelli Jan 24, 2023
58f51a7
`--schedule-snapshot`: address reviewer feedback
guimachiavelli Jan 24, 2023
ca9e547
update remaining tables
maryamsulemani97 Jan 25, 2023
67f590a
Apply suggestions from code review
guimachiavelli Jan 25, 2023
f4da775
Update learn/what_is_meilisearch/telemetry.md
guimachiavelli Jan 25, 2023
e574971
instance options: add note container
guimachiavelli Jan 25, 2023
ca68da9
update based on review
maryamsulemani97 Jan 25, 2023
b5d439f
remove `index_not_accessible`
maryamsulemani97 Jan 30, 2023
4d01424
Merge #2123
bors[bot] Feb 1, 2023
0436bd3
Merge #2104
bors[bot] Feb 1, 2023
a46d081
update Features page
guimachiavelli Feb 1, 2023
d9abc91
fix broken link
guimachiavelli Feb 1, 2023
b1a1fb2
Remove "Features" intro paragraph
guimachiavelli Feb 1, 2023
625d927
faq: remove mentions of things we'll change when v1 arrives
guimachiavelli Feb 1, 2023
54188c9
faq: improve wording
guimachiavelli Feb 1, 2023
d79ba16
Improve Features page wording
guimachiavelli Feb 1, 2023
2f9f3df
Merge #2100
bors[bot] Feb 2, 2023
efcb55b
Update reference/errors/error_codes.md
maryamsulemani97 Feb 2, 2023
cf98e6f
Update resources/faq.md
guimachiavelli Feb 2, 2023
2d9efa1
address reviewer feedback
guimachiavelli Feb 2, 2023
12e429d
Update learn/what_is_meilisearch/features.md
guimachiavelli Feb 2, 2023
4986cf6
address reviewer feedback
guimachiavelli Feb 2, 2023
af242a9
Apply suggestions from code review
guimachiavelli Feb 2, 2023
1ced95e
Merge #2099 #2107
bors[bot] Feb 2, 2023
5ce66fe
Merge #2102
bors[bot] Feb 2, 2023
4707e94
Merge #2103
bors[bot] Feb 2, 2023
1cb282b
Merge #2106 #2108
bors[bot] Feb 2, 2023
d612dcf
Merge branch 'v1.0' into v1.0--Multi-word-synonyms-are-treated-as-phr…
guimachiavelli Feb 2, 2023
0d9048e
Merge #2105
bors[bot] Feb 2, 2023
157a893
Merge #2110 #2111
bors[bot] Feb 2, 2023
da30c24
Merge branch 'v1.0' into v1.0--Accepted-OS-versions-to-run-the-Meilis…
guimachiavelli Feb 2, 2023
5d9cdc1
Merge #2112
bors[bot] Feb 2, 2023
7c77780
Merge branch 'v1.0' into v1-index-size-limits
guimachiavelli Feb 2, 2023
6d4a8fe
Merge #2114
bors[bot] Feb 2, 2023
c5dc37c
Merge #2118
bors[bot] Feb 2, 2023
079ae18
Merge #2120 #2121
bors[bot] Feb 2, 2023
8db86a0
Merge branch 'main' into v1.0
guimachiavelli Feb 2, 2023
1f26b7d
address reviewer feedback
guimachiavelli Feb 6, 2023
cc17a82
Merge #2142
bors[bot] Feb 6, 2023
0bb031d
Merge #2141
bors[bot] Feb 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .vale/styles/Vocab/word_list/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,5 @@ AMIs
[Cc]ancelation
Brotli
zlib
glibc
autogenerate
5 changes: 3 additions & 2 deletions .vuepress/config.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
const ogprefix = 'og: http://ogp.me/ns#'
module.exports = {
title: 'Meilisearch Documentation v0.30',
title: 'Meilisearch Documentation v1.0',
description: 'Open source Instant Search Engine',
theme: 'default-prefers-color-scheme',
themeConfig: {
Expand Down Expand Up @@ -105,6 +105,7 @@ module.exports = {
path: '/learn/getting_started/quick_start.html',
collapsable: false,
children: [
'/learn/getting_started/supported_os',
'/learn/getting_started/quick_start',
{
title: 'Meilisearch 101',
Expand Down Expand Up @@ -341,7 +342,7 @@ module.exports = {
pattern: '**/*.md',
strictExt: true,
ignoreFilePattern: ['document_structure'],
ignoreHashPattern: ['actions-2', 'primary-key-2', 'query-parameters-2', 'query-parameters-3'],
ignoreHashPattern: ['actions-2', 'primary-key-2', 'query-parameters-2', 'query-parameters-3', 'body-2', 'body-3', 'phrase-search-2'],
exitLevel: 'warn',
},
],
Expand Down
9 changes: 5 additions & 4 deletions .vuepress/public/postman/meilisearch-collection.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"info": {
"_postman_id": "a4f9078b-c91f-4399-9a98-5e4adc5099a2",
"name": "Meilisearch v0.30",
"_postman_id": "bcc2fb5b-d130-4a7b-9acf-c481248b2b9d",
"name": "Meilisearch v1.0",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
Expand Down Expand Up @@ -481,7 +481,8 @@
"{{url}}"
],
"path": [
"tasks"
"tasks",
"cancel"
],
"query": [
{
Expand Down Expand Up @@ -2035,7 +2036,7 @@
"bearer": [
{
"key": "token",
"value": "masterKey",
"value": "aSampleMasterKey",
"type": "string"
}
]
Expand Down
8 changes: 2 additions & 6 deletions learn/advanced/dumps.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Creating a dump is also referred to as exporting it, whereas launching Meilisear

## Creating a dump

To create a dump of your dataset, to use the [create a dump endpoint](/reference/api/dump.md#create-a-dump):
To create a dump of your dataset, use the [create a dump endpoint](/reference/api/dump.md#create-a-dump):

<CodeSamples id="post_dump_1" />

Expand Down Expand Up @@ -50,7 +50,7 @@ This command should return an object with detailed information about the dump op

The dump creation process is an asynchronous task that takes time proportional to the size of your dataset. All indexes of the current instance are exported along with their documents and settings and saved as a single `.dump` file.

After dump creation is finished—when `status` is `succeeded`—the dump file is added to the dump directory. By default, this folder is named `dumps` and can be found in the same directory as your Meilisearch binary. You can customize [this using the `--dumps-dir` configuration option](/learn/configuration/instance_options.md#dumps-directory). **If the dump directory does not already exist when the dump creation process is called, Meilisearch will create it.**
After dump creation is finished—when `status` is `succeeded`—the dump file is added to the dump directory. By default, this folder is named `dumps` and can be found in the same directory as your Meilisearch binary. You can customize [this using the `--dump-dir` configuration option](/learn/configuration/instance_options.md#dump-directory). **If the dump directory does not already exist when the dump creation process is called, Meilisearch will create it.**

If a dump file is visible in the file system, the dump process was successfully completed. **Meilisearch will never create a partial dump file**, even if you interrupt an instance while it is generating a dump.

Expand Down Expand Up @@ -80,10 +80,6 @@ As the data contained in the dump needs to be indexed, the process will take som
./meilisearch --import-dump /dumps/20200813-042312213.dump
```

### Importing a dump for v0.20 or below

If you are using Meilisearch v0.20 or below, migration should be done in two steps. First, import your v0.20 dump into an instance running any version of Meilisearch between v0.21 and v0.25. Second, export another dump from this instance and import it to a final instance running your targeted version.

## Use cases

Dumps are used to restore your database after [updating Meilisearch](/learn/advanced/updating.md) or to copy your database to other Meilisearch instances without having to worry about their respective versions. For more on this subject, see a [comparison of snapshots and dumps](/learn/advanced/snapshots_vs_dumps.md).
10 changes: 5 additions & 5 deletions learn/advanced/geosearch.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Our restaurant dataset looks like this once we add geopositioning data:
```

::: warning
Trying to index a dataset with one or more documents containing badly formatted `_geo` values will cause Meilisearch to throw an [`invalid_geo_field`](/reference/errors/error_codes.md#invalid-geo-field) error. In this case, the update will fail and no documents will be added or modified.
Trying to index a dataset with one or more documents containing badly formatted `_geo` values will cause Meilisearch to throw an [`invalid_document_geo_field`](/reference/errors/error_codes.md#invalid-document-geo-field) error. In this case, the update will fail and no documents will be added or modified.
:::

### Using `_geo` with CSV
Expand Down Expand Up @@ -133,12 +133,12 @@ First, ensure your documents contain valid geolocation data and that you have ad
_geoRadius(lat, lng, distance_in_meters)
```

`lat` and `lng` must be floating point numbers indicating a geographic position. `distance_in_meters` must be an integer indicating the radius covered by the `_geoRadius` filter. If any of these three parameters are invalid or missing, Meilisearch will return an [`invalid_filter`](/reference/errors/error_codes.md#invalid-filter) error.
`lat` and `lng` must be floating point numbers indicating a geographic position. `distance_in_meters` must be an integer indicating the radius covered by the `_geoRadius` filter. If any of these three parameters are invalid or missing, Meilisearch will return an [`invalid_search_filter`](/reference/errors/error_codes.md#invalid-search-filter) error.

[You can read more about using `filter` in our dedicated guide.](/learn/advanced/filtering_and_faceted_search.md#using-filters)

::: warning
`_geo`, `_geoDistance`, and `_geoPoint` are not valid filter rules. Trying to use any of them with the `filter` search parameter will result in an [`invalid_filter`](/reference/errors/error_codes.md#invalid-filter) error.
`_geo`, `_geoDistance`, and `_geoPoint` are not valid filter rules. Trying to use any of them with the `filter` search parameter will result in an [`invalid_search_filter`](/reference/errors/error_codes.md#invalid-search-filter) error.
:::

### Examples
Expand Down Expand Up @@ -222,12 +222,12 @@ _geoPoint(0.0, 0.0):asc

`_geoPoint` requires two floating point numbers indicating a location's latitude and longitude. You must also specify whether the sort should be ascending (`asc`) or descending (`desc`). Ascending sort will prioritize results closer to the specified location, while descending sort will put the most distant results first.

If either `lat` or `lng` is invalid or missing, Meilisearch will return an [`invalid_sort`](/reference/errors/error_codes.md#invalid-sort) error. An error will also be thrown if you fail to indicate a sorting order.
If either `lat` or `lng` is invalid or missing, Meilisearch will return an [`invalid_search_sort`](/reference/errors/error_codes.md#invalid-search-sort) error. An error will also be thrown if you fail to indicate a sorting order.

[You can read more about sorting in our dedicated guide.](/learn/advanced/sorting.md#sorting-results-at-search-time)

::: warning
`_geo`, `_geoDistance`, and `_geoRadius` are not valid `sort` values. Trying to use any of them with the `sort` search parameter will result in an [`invalid_sort`](/reference/errors/error_codes.md#invalid-sort) error.
`_geo`, `_geoDistance`, and `_geoRadius` are not valid `sort` values. Trying to use any of them with the `sort` search parameter will result in an [`invalid_search_sort`](/reference/errors/error_codes.md#invalid-search-sort) error.
:::

### Examples
Expand Down
14 changes: 13 additions & 1 deletion learn/advanced/known_limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Currently, Meilisearch has a number of known limitations. Some of these limitations are the result of intentional design trade-offs, while others can be attributed to [LMDB](/learn/advanced/storage.md), the key-value store that Meilisearch uses under the hood.

This guide covers hard limits that cannot be altered. Meilisearch also has some default limits that _can_ be changed, such as a [default payload limit of 100MB](/learn/configuration/instance_options.md#payload-limit-size), a [default database size limit of 100GiB](/learn/configuration/instance_options.md#max-index-size), and a [default search limit of 20 hits](/reference/api/search.md#limit).
This guide covers hard limits that cannot be altered. Meilisearch also has some default limits that _can_ be changed, such as a [default payload limit of 100MB](/learn/configuration/instance_options.md#payload-limit-size) and a [default search limit of 20 hits](/reference/api/search.md#limit).

## Maximum number of query words

Expand Down Expand Up @@ -112,3 +112,15 @@ user = 1 OR user = 2 […] OR user = 1500 OR user = 1501 […] OR user = 2000 OR
**Limitation:** Meilisearch might throw an internal error when indexing large batches of documents.

**Explanation:** Indexing a large batch of documents, such as a JSON file over 3.5GB in size, can result in Meilisearch opening too many file descriptors. Depending on your machine, this might reach your system's default resource usage limits and trigger an internal error. Use [`ulimit`](https://www.ibm.com/docs/en/aix/7.1?topic=u-ulimit-command) or a similar tool to increase resource consumption limits before running Meilisearch. For example, call `ulimit -Sn 3000` in a UNIX environment to raise the number of allowed open file descriptors to 3000.

## Maximum database size

**Limitation:** The maximum size of an index is 500GiB, and the maximum size of the task database is 10GiB.

**Explanation:** Meilisearch allocates all the virtual memory it requires upfront. The maximum database size ensures instances can contain several large indexes without reaching operating system limits on the amount of virtual memory available to a single process.

## Maximum number of indexes in an instance

**Limitation:** A single Meilisearch instance can safely contain around 180 indexes in Linux and macOS environments, and around 15 indexes in Windows environments.

**Explanation:** Operating systems restrict the amount of virtual memory available to a single process. Since Meilisearch allocates the maximum index size for each index upfront, this effectively creates a limit on how many indexes an instance can contain. The values provided are indicative: because of the particularities of each specific setup, the actual maximum number of indexes per instance will vary from machine to machine. A Linux install might contain 200 indexes without issue, or return allocation failures at 181 indexes depending on the runtime environment.
6 changes: 3 additions & 3 deletions learn/advanced/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,17 +24,17 @@ Now snapshots are created in `mySnapShots/` directory.

The first snapshot is created on launching Meilisearch. After that, snapshots are created routinely on a set interval until you deactivate snapshots by ending the Meilisearch instance. By default, one snapshot is taken every 24 hours.

The amount of time between each new snapshot can be modified with [`--snapshot-interval-sec`](/learn/configuration/instance_options.md#snapshot-interval):
The interval between each new snapshot can be modified by providing an integer to [`--schedule-snapshot`](/learn/configuration/instance_options.md#schedule-snapshot-creation):

```bash
meilisearch --schedule-snapshot --snapshot-interval-sec 3600
meilisearch --schedule-snapshot=3600
```

After running the above code, a snapshot is created every hour (3600 seconds).

During snapshot creation, old snapshots are **automatically overwritten**. This means only the most recent snapshot should be present in the folder at any given time.

[[More about snapshots flags and environment variables]](/learn/configuration/instance_options.md#schedule-snapshot-creation)
[More about snapshots flags and environment variables](/learn/configuration/instance_options.md#schedule-snapshot-creation)

## Starting from a snapshot

Expand Down
2 changes: 1 addition & 1 deletion learn/advanced/sorting.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ By default, Meilisearch focuses on ordering results according to their relevancy
This can be useful in many situations, such as when a user wants to see the cheapest products available in a webshop.

::: tip
Sorting at search time can be particularly effective when combined with [placeholder searches](/learn/what_is_meilisearch/features.md#placeholder-search).
Sorting at search time can be particularly effective when combined with [placeholder searches](/reference/api/search.md#placeholder-search).
:::

## Configuring Meilisearch for sorting at search time
Expand Down
4 changes: 1 addition & 3 deletions learn/advanced/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ The choice of LMDB comes with certain pros and cons, especially regarding databa

#### Database size

When deleting documents from a Meilisearch index, you may notice disk space usage remains the same. This happens because LMDB internally marks that space as free, but does not make it available for the operating system at large. This design choice leads to better performance, as there is no need for periodic compaction operations. As a result, disk space occupied by LMDB (and thus by Meilisearch) tends to increase over time.

It is not currently possible to calculate the precise maximum amount of space a Meilisearch instance can occupy. However, disk space usage is dictated in part by index and task database size. This means you can have a general idea of the upper size boundaries by taking into account the number of indexes in your instance together with the values configured for `--max-index-size` and `--max-task-db`.
When deleting documents from a Meilisearch index, you may notice disk space usage remains the same. This happens because LMDB internally marks that space as free, but does not make it available for the operating system at large. This design choice leads to better performance, as there is no need for periodic compaction operations. As a result, disk space occupied by LMDB (and thus by Meilisearch) tends to increase over time. It is not possible to calculate the precise maximum amount of space a Meilisearch instance can occupy.

#### Memory usage

Expand Down
10 changes: 3 additions & 7 deletions learn/advanced/updating.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ This guide only works for versions v0.15 and above. If you are using an older ve

This section contains instructions for upgrading from specific versions. Most versions don't require version-specific steps and you should be able to upgrade directly. If the version you are upgrading from isn't listed here, no additional steps are required.

- If you are updating **from v0.20 or below**, please ensure all updates finish processing before creating the dump. `enqueued` updates will not be exported and may result in data loss.

- If you are using **v0.24 or below**, use the `X-Meili-API-Key: API_KEY` authorization header:

<CodeSamples id="updating_guide_check_version_old_authorization_header" />
Expand Down Expand Up @@ -79,7 +81,7 @@ Once the status is `processed`, you're good to go. Repeat this process for all i

### Create the dump

Before creating your dump, make sure that your [dump directory](/learn/configuration/instance_options.md#dumps-directory) is somewhere accessible. By default, dumps are created in a folder called `dumps` at the root of your Meilisearch directory.
Before creating your dump, make sure that your [dump directory](/learn/configuration/instance_options.md#dump-directory) is somewhere accessible. By default, dumps are created in a folder called `dumps` at the root of your Meilisearch directory.

**Cloud platforms** like DigitalOcean, AWS, and GCP are configured to store dumps in the `/var/opt/meilisearch/dumps` directory.

Expand Down Expand Up @@ -258,12 +260,6 @@ For **cloud platforms**, move the new Meilisearch binary to the `/usr/bin` direc
mv meilisearch /usr/bin/meilisearch
```

::: warning
If you are using Meilisearch v0.20 or below, migration should be done in two steps. First, import your dump into an instance running any version of Meilisearch from v0.21 to v0.24, inclusive. Second, export another dump from this instance and import it to a final instance running your targeted version.

Once Meilisearch v1 is released, this two-step process won't be necessary as v1 will be compatible with dumps from all previous versions.
:::

## Step 3: Import data

### Launch Meilisearch and import the dump
Expand Down
Loading