This repository has been archived by the owner on Mar 21, 2024. It is now read-only.
Indexing CSVs #28
Merged
Indexing CSVs #28
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gmourier
added
the
Draft
gmourier
added
In Progress
|
Should this be a joint spec with json line? Support already exist, and just needs to be exposed |
|
I personally advocate for a separation of two distinct specs as we can use the merged pull requests to reflect what the engine is capable of as a single source of truth (like a spec dictionary?). Nonetheless, we need to take a decision about this statement. Do we use it as a task tracker or do we use it as a support for mapping the whole Meilisearch capabilities at a given point in time? I've added a commit on the last specification guide proposal to elaborate a specification lifecycle (github automation can't do all this steps, so it will need a lot of manual work I think). What do you mean by saying "expose"? Thinking about this one, I'm wondering if we need to expose it as an SDK's method to pass CSV file through http. It would be nice to have for sure. Especially to propose an interface for the future dashboard or DX but maybe we can just use it as a discovery tool to measure usage and payload size modifications on the CLI level. It still permits for MeiliSearch to iterate in the future. |
|
What I mean by expose is that this feature is already implemented and supported by the engine, and is disabled for use by the user. I get your point about isolation of specs, I'm just wondering if that since this is already implemented, and is basically the same thing, we should maybe ship the two at once, and avoid unecessary overhead. As for the sdk, what they decide to implement usually follows what MeiliSearch exposes in it's api. |
MarinPostma
reviewed
Apr 11, 2021
gmourier
added
Ready For Review
|
Just to follow up with you @curquiza, I will update the tech part as soon as I've updated the PR related to the change in spec definition. |
|
As mentioned on meilisearch/transplant#142, tech part needs to be updated to mention that. |
gmourier
added
In Progress
gmourier
added
Ready For Review
|
As mentioned in meilisearch/transplant#142 it will be a candidate for v0.22 |
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
bors bot
added a commit
to meilisearch/milli
that referenced
this pull request
May 31, 2021
184: Transfer numbers and strings facets into the appropriate facet databases r=Kerollmops a=Kerollmops This pull request is related to #152 and changes the layout of the facets values, numbers and strings are now in dedicated databases and the user no more needs to define the type of the fields. No more conversion between the two types is done, numbers (floats and integers converted to f64) go to the facet float database and strings go to the strings facet database. There is one related issue that I found regarding CSVs, the values in a CSV are always considered to be strings, [meilisearch/specifications#28](https://github.com/meilisearch/specifications/blob/d916b57d748628c3b45500e9dfa46b50dfa2ad3f/text/0028-indexing-csv.md) fixes this issue by allowing the user to define the fields types using `:` in the "CSV Formatting Rules" section. All previous tests on facets have been modified to pass again and I have also done hand-driven tests with the 115m songs dataset. Everything seems to be good! Fixes #192. Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Kerollmops <clement@meilisearch.com>
MarinPostma
reviewed
Jun 2, 2021
Co-authored-by: Clément Renault <clement@meilisearch.com>
Kerollmops
approved these changes
Jun 16, 2021
gmourier
added
Ready To Be Implemented
gmourier
added a commit
that referenced
this pull request
Aug 17, 2021
* Initiate csv indexation support specification * update spec file name to match pull request id * update csv indexation spec * fix code examples and typos * fix typos * update spec name * Update header part to match MeiliSearch Tracking-Issues * update spec from the equivalent ndjson spec reviews * update --data sample examples * add information about giving application/json content-type or not for a json payload * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> * Change curl --data param to --binary-data in examples Co-authored-by: Clément Renault <clement@meilisearch.com> * updates error codes * moved behavior about missing content-type in explanation part * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Clément Renault <clement@meilisearch.com>
gmourier
added a commit
that referenced
this pull request
Aug 17, 2021
* Initiate csv indexation support specification * update spec file name to match pull request id * update csv indexation spec * fix code examples and typos * fix typos * update spec name * Update header part to match MeiliSearch Tracking-Issues * update spec from the equivalent ndjson spec reviews * update --data sample examples * add information about giving application/json content-type or not for a json payload * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> * Change curl --data param to --binary-data in examples Co-authored-by: Clément Renault <clement@meilisearch.com> * updates error codes * moved behavior about missing content-type in explanation part * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Clément Renault <clement@meilisearch.com>
gmourier
added a commit
that referenced
this pull request
Sep 13, 2021
* Initiate csv indexation support specification * update spec file name to match pull request id * update csv indexation spec * fix code examples and typos * fix typos * update spec name * Update header part to match MeiliSearch Tracking-Issues * update spec from the equivalent ndjson spec reviews * update --data sample examples * add information about giving application/json content-type or not for a json payload * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> * Change curl --data param to --binary-data in examples Co-authored-by: Clément Renault <clement@meilisearch.com> * updates error codes * moved behavior about missing content-type in explanation part * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Clément Renault <clement@meilisearch.com>
gmourier
added
v0.23
Ready To Be Implemented
gmourier
added a commit
that referenced
this pull request
Oct 11, 2021
* Initiate csv indexation support specification * update spec file name to match pull request id * update csv indexation spec * fix code examples and typos * fix typos * update spec name * Update header part to match MeiliSearch Tracking-Issues * update spec from the equivalent ndjson spec reviews * update --data sample examples * add information about giving application/json content-type or not for a json payload * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> * Change curl --data param to --binary-data in examples Co-authored-by: Clément Renault <clement@meilisearch.com> * updates error codes * moved behavior about missing content-type in explanation part * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Clément Renault <clement@meilisearch.com>
gmourier
added a commit
that referenced
this pull request
Oct 12, 2021
* Indexing NDJSONs (#29) * initialize a draft for json lines indexation support specification * update filename number to match related pull-request * update specs * update link to CSV spec * update spec name * Apply typos correction from code review Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com> * fix typo * update impact on documentation part * replace file by data * add information about giving application/json content-type or not for a json payload * updates error codes, curl instructions * moved behavior about missing content-type in explanation part Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com> * Indexing CSVs (#28) * Initiate csv indexation support specification * update spec file name to match pull request id * update csv indexation spec * fix code examples and typos * fix typos * update spec name * Update header part to match MeiliSearch Tracking-Issues * update spec from the equivalent ndjson spec reviews * update --data sample examples * add information about giving application/json content-type or not for a json payload * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> * Change curl --data param to --binary-data in examples Co-authored-by: Clément Renault <clement@meilisearch.com> * updates error codes * moved behavior about missing content-type in explanation part * Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Clément Renault <clement@meilisearch.com> * Geosearch (#59) * Initialize draft specification for geo-search feature * add future possibilities * Update specification * mention errors and aspects about filterableAttributes and sortableAttributes * Add measure and finalized key changes * Add description in OpenApi * remove old falsy sentence * Add definition and explanation for error * fix rebase on develop * Specify missing edge cases (#63) * Initialize draft specification for geo-search feature * add future possibilities * Update specification * mention errors and aspects about filterableAttributes and sortableAttributes * Add measure and finalized key changes * Add description in OpenApi * remove old falsy sentence * Add definition and explanation for error * fix rebase on develop * update open-api.yml with description on _geoPoint built-in sort rule and _geo field * Apply suggestions from code review Co-authored-by: gui machiavelli <gui@meilisearch.com> * remove - char in geo-search * update invalid_geo_field error definition Co-authored-by: gui machiavelli <gui@meilisearch.com> * Patch GeoSearch specification to mention technical limit on `desc` ordering around a _geoPoint (#66) * mention decision and expected behavior for a desc ordering around a geoPoint * add desc ordering around a geoPoint as a future possibility * Patch error codes for csv and ndjson formats specs (#64) * specify error codes dedicated to payload format for post/put documents endpoints * Udpdate error codes naming * Add errors definition * update errors and cURL examples * Add alternative message for reserved keyword and update invalid_criterion error definition (#67) * add alternative message for reserved keyword and update invalid_criterion error * update error name in link field for invalid_ranking_rule error * update invalid_geo_field error message * fix typo * Add future possibilities from irevoire to geosearch spec file (#74) * add descending order capability for _geoPoint built-in sort (#77) * Add variant message to ensure _geoPoint and _geoRadius expressions are not used as a ranking rule to help the user (#78) * Mention the supported separator character (#81) * Add content-type header requirements * update bump.yml configuration Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com> Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: gui machiavelli <gui@meilisearch.com>
gmourier
added
Implemented
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of this PR is to specify CSV file indexing feature.