Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Codesearch: use less memory when indexing #7508

Merged
merged 3 commits into from
Sep 19, 2024
Merged

Codesearch: use less memory when indexing #7508

merged 3 commits into from
Sep 19, 2024

Conversation

tylerwilliams
Copy link
Member

@tylerwilliams tylerwilliams commented Sep 19, 2024 •
edited
Loading

  • Don't allocate giant sparse sets in the tokenizer
  • Don't copy between []uint64 and posting lists
  • Use less memory by encoding directly into the pebble batch without additional allocs

@tylerwilliams tylerwilliams changed the title use less memory when indexing Codesearch: use less memory when indexing Sep 19, 2024
@tylerwilliams tylerwilliams merged commit 96c8c94 into master Sep 19, 2024
15 checks passed
@tylerwilliams tylerwilliams deleted the use_less_memory_when_indexing branch September 19, 2024 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jdhollen jdhollen jdhollen approved these changes

@siggisim siggisim Awaiting requested review from siggisim

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tylerwilliams @jdhollen