Skip to content

feat: implement incremental tpuf reindexing #5712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
sahil485 wants to merge 39 commits into
base: app
Choose a base branch
from
Open

feat: implement incremental tpuf reindexing #5712

sahil485 wants to merge 39 commits into from
+714 −33

Conversation

@sahil485
Copy link
Contributor

@sahil485 sahil485 commented Dec 6, 2025 •
edited
Loading

Fixes FER-

Short description of the changes made

  • implement incremental tpuf reindexing for _fern_docs index and _query index (through FAI sync)
  • implement pagination on batching endpoints to prevent unnecessarily large reads/writes
  • upgrade turbopuffer dep to ^1.10.0
  • add posthog feature flag for to control which domains go through incremental reindexing
  • add robust chunk-level analytics to check how many pages + chunks have been added, updated, and deleted per reindex requests (sent to posthog)

What was the motivation & context behind this PR?

  • unnecessary to delete + reindex the entire tpuf namespace every time reindex is called since most things wouldn't have changed

@vercel
Copy link
Contributor

vercel bot commented Dec 6, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Updated (UTC)
dev.ferndocs.com Ready Ready Preview Dec 11, 2025 8:14am
fern-dashboard Ready Ready Preview Dec 11, 2025 8:14am
fern-dashboard-dev Ready Ready Preview Dec 11, 2025 8:14am
prod-assets.ferndocs.com Ready Ready Preview Dec 11, 2025 8:14am
prod.ferndocs.com Ready Ready Preview Dec 11, 2025 8:14am
1 Skipped Deployment
Project Deployment Preview Updated (UTC)
fern-platform Ignored Ignored Dec 11, 2025 8:14am

@vercel vercel bot temporarily deployed to Preview – prod.ferndocs.com December 6, 2025 17:25 Inactive
@vercel vercel bot temporarily deployed to Preview – prod-assets.ferndocs.com December 6, 2025 17:25 Inactive
@vercel vercel bot temporarily deployed to Preview – prod-assets.ferndocs.com December 10, 2025 16:27 Inactive
@vercel vercel bot temporarily deployed to Preview – prod.ferndocs.com December 10, 2025 16:27 Inactive
@vercel vercel bot temporarily deployed to Preview – prod-assets.ferndocs.com December 11, 2025 07:48 Inactive
@vercel vercel bot temporarily deployed to Preview – prod.ferndocs.com December 11, 2025 07:48 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sahil485