merge lantern_extras and lantern repos #338
Draft
+50,316
−127
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Added lantern-cli binary and cli option for embedding generation * Update CI/CD to build CLI package * Fix CI env var name * Improve error handling and logging, update README * Make image downloading parallel, update README * Add data using clone * Add more status logs * Fix error messages for image downloads * Update README * Fix output for bge models * Get CLS embeddings from bert models * Refactor and cleanup build/package script * Add schema support, make pk field generic * Update README, bump version * Get approximate count of rows * Add README.md and LICENSE into release package * Update README * Change lantern-cli name to contain architecture and platform * Add schema in table size estimation, make input column value optional
* Add lantern-cli daemon mode * Add memory optimizations and comment code * Add Dockerfile for cli * Add logger, add task id when logging * Fix db exporter * Fix logger.warn * Update README add logger in create index * Add dockerfile for CUDA * Add print_raw method for logger * Update onnxruntime version in README * Update GPU dockerfile * Update README for docker usage * Add update listener, continous embedding generation, startup checks * Fix startup query * Optimize tokenizer * Implement stop handler, optimize code, make fault tolerant * Remove unnecessary clones on references * Set data path based on OS * Add restarts on connection loss * Implement row locking and batching for client table inserts * Rename client to client_jobs * Check write permission on target table, add streaming option in embeddings * Send pending jobs only to update channel * Fix model column name typo * Fix startup job init * Fix init_finished_at update on startup * Add comment * Change gh runner ubuntu version * Update versions and README * Fix updates for rows with non-numeric ids * Update BERT large model to v1.5 * Add more BERT models * Make tokenizer and inputs for bert model dynamic
* Add docker publish action for CLI * Optimize dockerfiles
* Better handle sql quotting, update README for docker run * Bump cli version
* Skip column creation if specified, check permissions from grants, improve error logging for client task * Add schema name to table permission check
* Fix handling names for client identifier * Update cli version
* - Create lock table in lantern schema - Hash client trigger function and trigger names so they won't exceed the character limit * Bump cli version
* Make row locking non-blocking * Remove unnecessary deref
* Add command to measure model generation speed * Fix GPU memory leak, merge process_text and process_image into one function * Add min/max/avg speeds * Update min time for test * Update version
* Fix data_producer deadloc, remove unwraps * Bump versions
* Index autotune WIP * Add import functionality for lantern index * add index autotune to cli * Update README
* Fix find_bes_variant, add comments, add existing result reuse functionality * Refactor daemon, add autotune to daemon * Make float to 2 decimal points, change internal schema name * Bump version
… selects optimizations based on CPU in runtime, print hardware acceleration being used, update compiler versions in dockerfiles to clang-18
…ich makes postmaster error if loaded in shared_preload libs
…deamon bgworker jobs
…h the new project structure
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
modify CI scripts to work with the new project structure