Use full-text index for semantic document filtering. Improve Doc Queries for Perf#253
Merged
Merged
Conversation
Replace ILIKE-based library search text filtering in semantic retrieval with a search_vector full-text scope so PostgreSQL can use the existing GIN index, and remove a redundant deleted filter from document listing. Co-authored-by: Cursor <cursoragent@cursor.com>
Add concurrent partial indexes on library plus created_at and updated_at for non-deleted documents to speed high-volume library-scoped list and cleanup queries. Co-authored-by: Cursor <cursoragent@cursor.com>
- Change schema version to 2026_07_03_183252. - Replace existing partial indexes on documents with new indexes for active documents based on created_at and updated_at, improving query performance for non-deleted documents. - Reintroduce the `doc_lookback_days` column in the questions table to enhance question-answer retrieval capabilities. Co-authored-by: Cursor <cursoragent@cursor.com>
- Changed the method for selecting document columns from `select` to `reselect` in the BaseDocumentsController to enhance query performance. - Updated the embedding status display in the document view to provide clearer information on availability, replacing the raw embedding output with a user-friendly message. - Added a new script for loading test documents, allowing for bulk creation of documents with customizable parameters for testing purposes.
Drop the ad hoc document seeding script so this PR only contains the intended application and index changes. Co-authored-by: Cursor <cursoragent@cursor.com>
Switch document listing back to select to avoid changing relation select semantics while we continue performance validation. Co-authored-by: Cursor <cursoragent@cursor.com>
vswamidass-sfdc
changed the title
chester-vanhuang
approved these changes
Jul 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
document ILIKEtosearch_vector @@ plainto_tsqueryvia a newDocument.full_text_matchscope.not_deletedfrom default documents index scope to avoid duplicated soft-delete predicatesTest plan
library_search_textpresent and confirm results still returnindex_documents_on_search_vectorMade with Cursor