Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Searchable Snapshot] Implement the IndexInput classes for block-based storage #3961

Closed
Tracked by #2919
kotwanikunal opened this issue Jul 20, 2022 · 2 comments
Closed
Tracked by #2919
Assignees
Labels
enhancement Enhancement or improvement to existing feature or request Indexing & Search v2.4.0 'Issues and PRs related to version v2.4.0'

Comments

@kotwanikunal
Copy link
Member

Is your feature request related to a problem? Please describe.

  • The snapshot mechanisms in OpenSearch restore the entire index including the segment files for restoring an index
  • Searchable snapshot adds support for searching an index without downloading the entire files to disk
  • The above described protocol needs a new mechanism which will download parts of the file as required by the query

Describe the solution you'd like

  • The IndexInput interface can be utilized to create a virtual file mechanism which will download blocks of data as needed by the query
  • The underlying blocks will be cached onto the disk as files which can utilize the seek mechanism to load the appropriate block in memory using the IndexInput interfaces

Describe alternatives you've considered
N/A

Additional context
#3869

@kotwanikunal
Copy link
Member Author

POC feature commit: andrross#101

@kotwanikunal kotwanikunal removed their assignment Aug 19, 2022
@aabukhalil
Copy link
Contributor

I'm working on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request Indexing & Search v2.4.0 'Issues and PRs related to version v2.4.0'
Projects
Status: Done
Development

No branches or pull requests

4 participants