Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: separate celery workers #2948

Merged
merged 87 commits into from
Aug 6, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
6f205a4
add: auth dropbox
chloedia Jul 15, 2024
483db0a
fix: dropbox sync creds
chloedia Jul 15, 2024
c587456
add: list dropbox items
chloedia Jul 15, 2024
1d9070f
add: weblink
chloedia Jul 16, 2024
d55f411
feat: poetry lock
StanGirard Jul 16, 2024
9d98792
feat: Add Dropbox sync functionality
StanGirard Jul 16, 2024
78d161f
fix: DropBox integration link & front dropbox
chloedia Jul 16, 2024
ac0d211
add: notification to DropBox
chloedia Jul 16, 2024
36a4ecd
fix: delete Session middleware
chloedia Jul 17, 2024
98c216a
fix: refacto syncs
chloedia Jul 17, 2024
6631975
fix: remove useless debug & fix dropbox refresh
chloedia Jul 17, 2024
3b00e66
fix: merge problems
chloedia Jul 17, 2024
7625182
feat: Update sync datetime formats for Google Drive and Azure Drive &…
chloedia Jul 18, 2024
806af03
add: retry fetch_files Azure
chloedia Jul 18, 2024
f002a09
add : notion auth
chloedia Jul 18, 2024
e1c2742
feat: Add Notion sync functionality and update dependencies
chloedia Jul 18, 2024
1a7a961
add: download file for notion
chloedia Jul 19, 2024
e02d273
add: notion front
chloedia Jul 19, 2024
c90f08b
fix: add notion icon url
chloedia Jul 19, 2024
6e76108
add: is_subfolder, fix: notif failures,
chloedia Jul 22, 2024
36ae406
add: icon
chloedia Jul 22, 2024
8f07c79
add: front show emoji
chloedia Jul 22, 2024
8d18ba9
add: add async to notion integration
chloedia Jul 23, 2024
df73259
feat: Add Notion sync functionality and update dependencies
chloedia Jul 23, 2024
86a4721
add: redis
chloedia Jul 23, 2024
693d9c3
add: store in db + get_files from db
chloedia Jul 26, 2024
4e8b8f0
add: migration notion sync
chloedia Jul 30, 2024
4fa4f6e
tests sync notion
AmineDiro Jul 30, 2024
3443586
fix test notion session sync
AmineDiro Jul 30, 2024
9971d2b
fix test sync
AmineDiro Jul 30, 2024
b6bb55d
feat(frontend): handle no brain selection (#2932)
Zewed Jul 31, 2024
24672c0
moved parsed to worker and update packages
AmineDiro Jul 31, 2024
8819379
add: get_file, upload, add notion page to db
chloedia Jul 31, 2024
8ca9e53
add: get_file, upload, add notion page to db
chloedia Jul 31, 2024
5b50f80
add: get_file, upload, add notion page to db
chloedia Jul 31, 2024
794e3ad
fix: merge with main
chloedia Jul 31, 2024
95bf927
fix: merge with main
chloedia Jul 31, 2024
c0a7fec
fix: merge with main
chloedia Jul 31, 2024
03fa348
feat: add sync logic
chloedia Jul 31, 2024
2d64962
fix: processor quivr version (#2934)
AmineDiro Jul 31, 2024
13709b6
init quivr_worker
AmineDiro Aug 1, 2024
0d3b698
Merge remote-tracking branch 'origin/main' into feat/separate-celery-…
AmineDiro Aug 1, 2024
d9c1f3a
fix: quivr core fix tests (#2935)
AmineDiro Aug 1, 2024
837f33d
moved parsers
AmineDiro Aug 1, 2024
bd8763f
merged main
AmineDiro Aug 1, 2024
a4150f1
refacto file
AmineDiro Aug 1, 2024
de8ae64
add: sync in sync cron job
chloedia Aug 1, 2024
0584a79
moved pdf parsing to megaparse
Aug 2, 2024
5a9d27a
moved md5 in core to sha1
Aug 2, 2024
cd01abc
moved check_premium + changed qfile to sha1
Aug 2, 2024
b3debee
chore(main): release core 0.0.13 (#2930)
StanGirard Aug 2, 2024
e775644
finish: regular notion sync
chloedia Aug 2, 2024
9a93dbb
finish: regular notion sync
chloedia Aug 2, 2024
1397d49
quivr_core qfile changes
AmineDiro Aug 2, 2024
67bf3a0
refacto processing + tasks
AmineDiro Aug 2, 2024
b1d5f45
parse simple tests
Aug 2, 2024
207e4f5
added llama-parse dep
Aug 2, 2024
071670b
added processing audio to parsers + tests
Aug 3, 2024
e8b7423
crawler celery workers
Aug 3, 2024
5a38a98
dockerfile.dev workers
Aug 3, 2024
7a463f5
dockerfile.dev workers
Aug 4, 2024
e918c73
Merge branch 'main' into feat/notion-oauth-sync
chloedia Aug 4, 2024
3c2cae2
fix: double mime type in file name
chloedia Aug 4, 2024
94f7948
working docker-compose
AmineDiro Aug 4, 2024
dbd2191
patch json dump tests
AmineDiro Aug 4, 2024
99491e7
logger + placeholder test
AmineDiro Aug 4, 2024
499b907
Delete backend/supabase/migrations/20240107152745_ollama.sql
AmineDiro Aug 4, 2024
6c29231
add: normalize async for BaseSync & SyncUtils
chloedia Aug 4, 2024
1a27e23
remove prints
chloedia Aug 4, 2024
e8ac203
changed dockerfile
AmineDiro Aug 5, 2024
64207c4
default venv
AmineDiro Aug 5, 2024
52b5037
fix original file_name
AmineDiro Aug 5, 2024
5144275
skip all known processors checks
AmineDiro Aug 5, 2024
12c2d92
rolled back crawler
AmineDiro Aug 5, 2024
61d9b47
production docker images
AmineDiro Aug 5, 2024
c129cd0
ignore tox env
AmineDiro Aug 5, 2024
9934a7a
feat: Add GitHub sync functionality to sync router (#2871)
chloedia Aug 5, 2024
1c63608
refactor: Remove syncGitHub function from useSync.ts (#2942)
StanGirard Aug 5, 2024
71733e4
rolled back crawler
AmineDiro Aug 5, 2024
803f425
fix: sync update only modified files
chloedia Aug 5, 2024
704b47d
merge with main
chloedia Aug 5, 2024
3f4f323
sync utils rewrite
AmineDiro Aug 6, 2024
0463c8a
push new lock
chloedia Aug 6, 2024
4016da9
push new lock
chloedia Aug 6, 2024
adc71b7
push new lock
chloedia Aug 6, 2024
46df016
Merged feat/notion-oauth
AmineDiro Aug 6, 2024
dafda78
lock files ok
AmineDiro Aug 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor: Remove syncGitHub function from useSync.ts (#2942)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
  • Loading branch information
StanGirard authored Aug 5, 2024
commit 1c63608b4ca414698c9e1bb98a36acbde84c53ec
2 changes: 0 additions & 2 deletions frontend/lib/api/sync/useSync.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ import {
getUserSyncs,
syncDropbox,
syncFiles,
syncGitHub,
syncGoogleDrive,
syncSharepoint,
updateActiveSync
Expand Down Expand Up @@ -54,7 +53,6 @@ export const useSync = () => {
syncGoogleDrive(name, axiosInstance),
syncSharepoint: async (name: string) => syncSharepoint(name, axiosInstance),
syncDropbox: async (name: string) => syncDropbox(name, axiosInstance),
syncGitHub: async (name: string) => syncGitHub(name, axiosInstance),
getUserSyncs: async () => getUserSyncs(axiosInstance),
getSyncFiles: async (userSyncId: number, folderId?: string) =>
getSyncFiles(axiosInstance, userSyncId, folderId),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ interface ConnectionCardsProps {
export const ConnectionCards = ({
fromAddKnowledge,
}: ConnectionCardsProps): JSX.Element => {
const { syncGoogleDrive, syncSharepoint, syncDropbox, syncGitHub } = useSync();
const { syncGoogleDrive, syncSharepoint, syncDropbox } = useSync();

return (
<div
Expand Down
Loading