feat: implement streaming export for large databases#99
Open
769066112-ops wants to merge 1 commit intoouterbase:mainfrom
Open
feat: implement streaming export for large databases#99769066112-ops wants to merge 1 commit intoouterbase:mainfrom
769066112-ops wants to merge 1 commit intoouterbase:mainfrom
Conversation
Replace in-memory data loading with ReadableStream-based streaming for all export endpoints (SQL dump, CSV, JSON). Changes: - Add forEachPage() helper for paginated LIMIT/OFFSET queries (1000 rows/page) - Add tableExists() and createStreamingExportResponse() utilities - Rewrite dump.ts to stream SQL dump page-by-page per table - Rewrite csv.ts to stream CSV rows with proper escaping - Rewrite json.ts to stream JSON array incrementally - Preserve backward-compatible getTableData() and createExportResponse() This prevents memory overflow and request timeouts (30s limit) when exporting large databases (up to 10GB on Durable Objects). Fixes outerbase#59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #59 — Database dumps do not work on large databases.
Problem
All export endpoints (
/export/dump,/export/csv/:table,/export/json/:table) load the entire dataset into memory before creating the response. For large databases (up to 10GB on Durable Objects), this causes:Solution
Replace in-memory buffering with
ReadableStream-based streaming combined with paginated database queries (LIMIT/OFFSET, 1000 rows per page).Changes
src/export/index.ts— New streaming utilities:forEachPage()— Paginated query helper that fetches rows in batches viaLIMIT/OFFSETand invokes a callback per pagetableExists()— Lightweight table existence checkcreateStreamingExportResponse()— Creates a chunked-transferResponsefrom aReadableStreamgetTableData()andcreateExportResponse()preserved for backward compatibilitysrc/export/dump.ts— Streaming SQL dump:src/export/csv.ts— Streaming CSV export:escapeCsvValue()helper for cleaner escaping logicsrc/export/json.ts— Streaming JSON export:[, row,,\n, row, ...])Design Decisions
handler.tsTesting
The streaming approach can be verified by:
/export/dump,/export/csv/:table,/export/json/:tableBounty: $250 · Issue #59