Skip to content

Buffered writes#122

Open
tharropoulos wants to merge 31 commits into
typesense:masterfrom
tharropoulos:buffered-writes
Open

Buffered writes#122
tharropoulos wants to merge 31 commits into
typesense:masterfrom
tharropoulos:buffered-writes

Conversation

@tharropoulos

@tharropoulos tharropoulos commented Apr 9, 2025

Copy link
Copy Markdown
Collaborator

TLDR

Add buffer-based operations for Typesense to improve reliability and performance.

Change Summary

  • This change adds a buffering layer to improve reliability of Typesense operations
  • Buffer stores operations in Firestore before processing them in batches
  • Provides retry capabilities for failed requests and better error handling
  • Configurable via extension parameters for flexibility

Added Features:

  1. Buffer functionality for Typesense operations:

    • Added option to use buffering for Typesense operations with a new config parameter: TYPESENSE_USE_BUFFER
    • Buffer stores operations in Firestore before processing them in batches
    • Implemented automatic retry mechanism for failed operations

  2. New Configuration Parameters in extension.yaml:

    • TYPESENSE_USE_BUFFER: Toggle buffering on/off
    • TYPESENSE_BUFFER_COLLECTION_IN_FIRESTORE: Collection name for buffer storage
    • TYPESENSE_BUFFER_BATCH_SIZE: Number of documents to process in a batch
    • TYPESENSE_BUFFER_MAX_RETRIES: Maximum retry attempts for failed operations
    • TYPESENSE_BUFFER_FLUSH_INTERVAL: Cron-style interval for buffer processing

  3. New files:

    • processBuffer.js: Scheduled function to process buffered operations
    • Added test files: indexOnWriteWithBuffer.spec.js and indexOnWriteSubcollectionWithBuffer.spec.js

Code Changes:

  1. In indexOnWrite.js:

    • Modified to check for buffer setting and route operations accordingly
    • Split functionality into realTimeWrites and bufferedWrites
    • Added handling for path parameters in buffered operations

  2. In processBuffer.js:

    • Implemented scheduled function to process buffer in batches
    • Added retry logic for failed operations
    • Added status tracking (pending → processing → completed/retrying → failed)

  3. In config.js:

    • Added new configuration parameters with defaults

  4. Test Environment Updates:

    • Added support for custom Typesense fields in test environment
    • Created comprehensive test suites for buffered operations
    • Added test environment configurations for buffer testing

  5. Fix: Double URL Encoding:

    • Removed redundant encodeURIComponent calls when accessing Typesense collections after updating typesense-js
    • Updated all test files to match this change

Dependencies:

  1. Updated Typesense client:
    • Bumped from v1.8.2 to v2.1.0-3 for improved functionality

PR Checklist

@tharropoulos
feat(config): add buffer configuration parameters
56be3b5 
- Add `typesenseBufferCollectionInFirestore` to specify Firestore collection for buffering
- Add `typesenseBufferBatchSize` to control batch size for buffer processing
- Add `typesenseBufferFlushInterval` with default of 3 minutes for buffer flushing
@tharropoulos
feat: implement buffer processing scheduler
194d6fc 
- add scheduled function to process pending operations in buffer
- handle batch upserts and deletes to typesense collection
- implement retry mechanism for failed operations
- track operation status with firestore updates
@tharropoulos
chore: add max retries for buffer
093d064
@tharropoulos
refactor: move cron functionality to dedicated function for testing
312162a 
- move the function logic into a separate function for invoking on test
  files
@tharropoulos
fix: check for documents where the retry is equal to the maximum
843c7d4
@tharropoulos
refactor: change failedIds from set into a map
9118dbd
@tharropoulos
fix: log generic on import fails if not of type of ImportError
531c6b1
@tharropoulos
refactor: use last error for import fails
92d73aa
@tharropoulos
refactor: change the logic order of max retries on imports
66a7160
@tharropoulos
fix: use the whole delete batch if num_deleted = 0
50edb65 
- Typesense will not return the ids if none are deleted
@tharropoulos
fix: fix doc ref access on deletion fails
957a92b
@tharropoulos
fix: use iterator for failed deletions instead of array.forEach
af9c88b
@tharropoulos
refactor: change failed deletions retry logic
47a67f9
@tharropoulos
fix: init app on processBuffer
87a2986
@tharropoulos
feat(test): add support for custom typesense fields in TestEnvironment
a54bfe6 
- Add `typesenseFields` parameter to `TestEnvironment` constructor
- Store custom fields in class instance variable
- Use custom fields when creating typesense collection
- Default to wildcard field config when no custom fields provided
@tharropoulos
test: add tests for buffer processing operations
cdb0408 
- Create test suite to verify typesense buffer processing functionality
- Test successful upsert operations from buffer to typesense
- Test successful delete operations from buffer to typesense
- Implement retry logic tests for failed operations
- Verify proper status transitions (pending → completed/retrying → failed)
@tharropoulos
refactor: improve buffer processing code organization
52f9600 
- split monolithic function into smaller, focused helper functions
- rename main function from `based` to descriptive `processTypesenseBuffer`
- add jsdoc comments to document function purposes and parameters
- extract common patterns into reusable functions like `markDocumentsAsCompleted`
- improve error handling with consistent status update pattern
- rename filter function from `filterByDelete` to `createDeleteFilter`
@tharropoulos
chore: bump typesense dep
595e463
@tharropoulos
fix(test): remove double encoding on test environment
9c8e4c1 
- after upgrading to 2.1.0, encoding is happening on typesense's side
@tharropoulos
chore: include typesense-data in gitignore
33343b2
@tharropoulos
chore: bump typesense
7862a33
@tharropoulos
fix: remove double encoding of collection name
a49c08c
@tharropoulos
build: read from environment vars for buffer operations
09d7c7b
@tharropoulos
chore: add envs for buffers
25d3c50
@tharropoulos
fix(buffer): use the path snapshot params of the document when importing
373a481
@tharropoulos
feat(buffering): add buffering option to indexOnWrite
92ab894
@tharropoulos
test(buffer): test buffering on indexOnWrite
aeabaee
@tharropoulos
fix: change default buffer interval syntax
e7b523c
@tharropoulos
build: add new env params to yaml file
c1e1d07
@tharropoulos

tharropoulos commented Apr 9, 2025

Copy link
Copy Markdown
Collaborator Author

The tests are going to be failing because the buffered deletion feature relies on this PR. After a new tag is released on Dockerhub, I'll update the Typesense version running on CI.

@tharropoulos tharropoulos marked this pull request as draft April 9, 2025 11:34
@tharropoulos tharropoulos requested a review from jasonbosco April 9, 2025 11:35
@tharropoulos tharropoulos marked this pull request as ready for review April 14, 2025 11:39
@tharropoulos

tharropoulos commented Apr 14, 2025

Copy link
Copy Markdown
Collaborator Author

@jasonbosco Ready for review! 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jasonbosco jasonbosco Awaiting requested review from jasonbosco

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tharropoulos