Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release: kms 0.1 #3259

Draft
wants to merge 82 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
eaca4a0
remove sync/sync active simplify
AmineDiro Sep 17, 2024
3daef91
sync repository
AmineDiro Sep 17, 2024
5803fb9
sync service
AmineDiro Sep 17, 2024
13b288f
oauth2 state
AmineDiro Sep 18, 2024
ae70171
oauth2 refacto
AmineDiro Sep 18, 2024
c5d372b
ruff formatter vscode
AmineDiro Sep 18, 2024
b0e76e3
syncs + cleanupdb
AmineDiro Sep 18, 2024
339c6b6
sync list return knowledge info
AmineDiro Sep 19, 2024
625dfa2
brain dto join
AmineDiro Sep 20, 2024
08d7b33
sync knowledge
AmineDiro Sep 20, 2024
9bf4519
knowledge link to brain DONE
AmineDiro Sep 20, 2024
6d1dd8d
fixed sync children none
AmineDiro Sep 20, 2024
6e91583
link knowledge on creation
AmineDiro Sep 23, 2024
50a0800
merged n+1 query optimization
AmineDiro Sep 23, 2024
211fcaa
processor class worker
AmineDiro Sep 23, 2024
51421c0
working process knowledge
AmineDiro Sep 24, 2024
5b39026
all tests working
AmineDiro Sep 25, 2024
e99beab
merged main
AmineDiro Sep 25, 2024
0a8faaf
fixed tests
AmineDiro Sep 25, 2024
0654860
link brain appends unique brains
AmineDiro Sep 25, 2024
8fc8219
added sync status
AmineDiro Sep 25, 2024
c9d8a50
process file sync
AmineDiro Sep 25, 2024
ae3a5ef
worker tests working
AmineDiro Sep 25, 2024
da160ae
small refacto
AmineDiro Sep 25, 2024
e318ad6
rollback on error working
AmineDiro Sep 25, 2024
6c2ba27
rollback on error working
AmineDiro Sep 25, 2024
5a88bf0
knowledge sync
AmineDiro Sep 25, 2024
bb18d13
Merge pull request #3226 from QuivrHQ/feature/ent-67-backend-remove-s…
AmineDiro Sep 25, 2024
1d5761d
added test sync file in different brain
AmineDiro Sep 26, 2024
1b3915c
unlink from brain
AmineDiro Sep 26, 2024
74a8989
profiler for tests
AmineDiro Sep 26, 2024
d17d796
Merge pull request #3268 from QuivrHQ/feat/remove-km-kms
AmineDiro Sep 26, 2024
868058f
notifier update
AmineDiro Sep 26, 2024
970bbac
readme update
AmineDiro Sep 26, 2024
84b1901
updated readme
AmineDiro Sep 26, 2024
42a2dbb
readme
AmineDiro Sep 26, 2024
e9e0dbd
readme
AmineDiro Sep 26, 2024
5139294
Update README.md
AmineDiro Sep 26, 2024
4f36519
root list join on children only
AmineDiro Sep 27, 2024
c7491b8
serialization benchmark
AmineDiro Sep 27, 2024
f08f653
url tests
AmineDiro Sep 27, 2024
23a3c6d
Merge pull request #3269 from QuivrHQ/feat/notifier-kms
AmineDiro Sep 27, 2024
055a033
load tested kms
AmineDiro Sep 27, 2024
fa46f30
fix awaitable attr knowledge
AmineDiro Sep 27, 2024
8d0ca96
pool size as param + locust on_start/on_stop setup
AmineDiro Sep 28, 2024
edda350
Merge pull request #3272 from QuivrHQ/release/load-test-kms
AmineDiro Sep 30, 2024
ae5f6a8
sync init
AmineDiro Sep 30, 2024
95df919
sync status added
AmineDiro Sep 30, 2024
6cc61bc
Merge pull request #3281 from QuivrHQ/fix/sync-status-added
AmineDiro Sep 30, 2024
62ff887
Merge remote-tracking branch 'origin/fix/sync-status-added' into feat…
AmineDiro Sep 30, 2024
8d729de
get files last synced at
AmineDiro Sep 30, 2024
7fbfab8
sync knowledge service
AmineDiro Sep 30, 2024
f8db112
sync utils
AmineDiro Sep 30, 2024
0b7eecd
working update + tests
AmineDiro Oct 1, 2024
3ed0de5
test sync update single file
AmineDiro Oct 1, 2024
b99db0f
celery worker update syncs
AmineDiro Oct 1, 2024
4180026
celery worker time limits
AmineDiro Oct 1, 2024
53ba2bb
modify Readme.ms
AmineDiro Oct 1, 2024
22bd331
Merge pull request #3303 from QuivrHQ/feat/sync-knowledges
AmineDiro Oct 2, 2024
e8e5de5
merged main
AmineDiro Oct 4, 2024
713fa86
added field brains
AmineDiro Oct 4, 2024
fdf51c6
Merge pull request #3323 from QuivrHQ/feature/core-216-fix-add-all-br…
AmineDiro Oct 7, 2024
e7cdf6a
file size limit
Oct 7, 2024
1f826c9
Merge pull request #3332 from QuivrHQ/feat/kms-file-size-limit
AmineDiro Oct 7, 2024
233b496
fix update knowledge (#3334)
AmineDiro Oct 7, 2024
24f672f
fix arg length
AmineDiro Oct 7, 2024
b7f49d5
Merge pull request #3337 from QuivrHQ/fix/logger-args-len-sync
AmineDiro Oct 7, 2024
6075b83
update km date
AmineDiro Oct 7, 2024
1bb25eb
Merge pull request #3338 from QuivrHQ/fix/km-update-dat
AmineDiro Oct 7, 2024
aae36f4
sort knowledge, remove bulk_id
AmineDiro Oct 8, 2024
4c4cd6d
modified test
AmineDiro Oct 8, 2024
a81f553
updated benchmark with routes
AmineDiro Oct 8, 2024
9076729
Merge pull request #3348 from QuivrHQ/fix/sort-knowledge-update-brain
AmineDiro Oct 9, 2024
87a7116
Merge pull request #3350 from QuivrHQ/feat/locust-kms
AmineDiro Oct 9, 2024
fcf2b34
merged release/kms
AmineDiro Oct 14, 2024
2fe24d2
Merge pull request #3271 from QuivrHQ/feat/url-add-knowledge
AmineDiro Oct 14, 2024
4e88b7e
sort brain name and knowledge list local
AmineDiro Oct 16, 2024
b241103
Merge pull request #3379 from QuivrHQ/fix/sort-brain-list-km-only
AmineDiro Oct 16, 2024
505a765
fix: sync provider naming (#3380)
AmineDiro Oct 16, 2024
939d28d
fixed wrong parsing file extension
AmineDiro Oct 16, 2024
f899cd1
Merge pull request #3386 from QuivrHQ/fix/knowledge-parsing
AmineDiro Oct 17, 2024
fe16805
feat(frontend): display all knowledge in a kms (#3411)
Zewed Oct 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: sync provider naming (#3380)
# Description
  • Loading branch information
AmineDiro authored Oct 16, 2024
commit 505a76548e6e329dc595c0cfbaa50e7946d6f8ad
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from quivr_api.middlewares.auth import AuthBearer, get_current_user
from quivr_api.modules.dependencies import get_service
from quivr_api.modules.sync.dto.inputs import SyncStatus, SyncUpdateInput
from quivr_api.modules.sync.dto.outputs import SyncProvider
from quivr_api.modules.sync.service.sync_service import SyncsService
from quivr_api.modules.sync.utils.oauth2 import parse_oauth2_state
from quivr_api.modules.user.entity.user_identity import UserIdentity
Expand Down Expand Up @@ -63,7 +64,7 @@ async def authorize_azure(
)
logger.debug(f"Authorizing Azure sync for user: {current_user.id}")
state = await syncs_service.create_oauth2_state(
provider="Azure", name=name, user_id=current_user.id
provider=SyncProvider.AZURE, name=name, user_id=current_user.id
)
flow = client.initiate_auth_code_flow(
scopes=SCOPE,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from quivr_api.middlewares.auth import AuthBearer, get_current_user
from quivr_api.modules.dependencies import get_service
from quivr_api.modules.sync.dto.inputs import SyncStatus, SyncUpdateInput
from quivr_api.modules.sync.dto.outputs import SyncProvider
from quivr_api.modules.sync.service.sync_service import SyncsService
from quivr_api.modules.sync.utils.oauth2 import parse_oauth2_state
from quivr_api.modules.user.entity.user_identity import UserIdentity
Expand Down Expand Up @@ -64,7 +65,7 @@ async def authorize_dropbox(
scope=SCOPE,
)
state = await syncs_service.create_oauth2_state(
provider="DropBox", name=name, user_id=current_user.id
provider=SyncProvider.DROPBOX, name=name, user_id=current_user.id
)
authorize_url = auth_flow.start(state.model_dump_json())
logger.info(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from quivr_api.middlewares.auth import AuthBearer, get_current_user
from quivr_api.modules.dependencies import get_service
from quivr_api.modules.sync.dto.inputs import SyncStatus, SyncUpdateInput
from quivr_api.modules.sync.dto.outputs import SyncProvider
from quivr_api.modules.sync.service.sync_service import SyncsService
from quivr_api.modules.sync.utils.oauth2 import parse_oauth2_state
from quivr_api.modules.user.entity.user_identity import UserIdentity
Expand Down Expand Up @@ -54,7 +55,7 @@ async def authorize_github(
"""
logger.debug(f"Authorizing GitHub sync for user: {current_user.id}")
state = await syncs_service.create_oauth2_state(
provider="Github", name=name, user_id=current_user.id
provider=SyncProvider.GITHUB, name=name, user_id=current_user.id
)
authorization_url = (
f"https://github.com/login/oauth/authorize?client_id={CLIENT_ID}"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from quivr_api.middlewares.auth import AuthBearer, get_current_user
from quivr_api.modules.dependencies import get_service
from quivr_api.modules.sync.dto.inputs import SyncStatus, SyncUpdateInput
from quivr_api.modules.sync.dto.outputs import SyncProvider
from quivr_api.modules.sync.service.sync_service import SyncsService
from quivr_api.modules.sync.utils.oauth2 import parse_oauth2_state
from quivr_api.modules.user.entity.user_identity import UserIdentity
Expand Down Expand Up @@ -89,7 +90,7 @@ async def authorize_google(
)

state = await syncs_service.create_oauth2_state(
provider="Google", name=name, user_id=current_user.id
provider=SyncProvider.GOOGLE, name=name, user_id=current_user.id
)
authorization_url, state = flow.authorization_url(
access_type="offline",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
from quivr_api.middlewares.auth import AuthBearer, get_current_user
from quivr_api.modules.dependencies import get_service
from quivr_api.modules.sync.dto.inputs import SyncStatus, SyncUpdateInput
from quivr_api.modules.sync.dto.outputs import SyncProvider
from quivr_api.modules.sync.service.sync_service import SyncsService
from quivr_api.modules.sync.utils.oauth2 import parse_oauth2_state
from quivr_api.modules.user.entity.user_identity import UserIdentity
Expand Down Expand Up @@ -57,7 +58,7 @@ async def authorize_notion(
"""
logger.debug(f"Authorizing Notion sync for user: {current_user.id}, name : {name}")
state = await syncs_service.create_oauth2_state(
provider="Notion", name=name, user_id=current_user.id
provider=SyncProvider.NOTION, name=name, user_id=current_user.id
)
# Finalize the state
authorize_url = str(NOTION_AUTH_URL) + f"&state={state.model_dump_json()}"
Expand Down
12 changes: 6 additions & 6 deletions backend/api/quivr_api/modules/sync/controller/sync_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from quivr_api.modules.sync.controller.google_sync_routes import google_sync_router
from quivr_api.modules.sync.controller.notion_sync_routes import notion_sync_router
from quivr_api.modules.sync.dto import SyncsDescription
from quivr_api.modules.sync.dto.outputs import AuthMethodEnum
from quivr_api.modules.sync.dto.outputs import AuthMethodEnum, SyncProvider
from quivr_api.modules.sync.entity.sync_models import SyncFile
from quivr_api.modules.sync.service.sync_service import SyncsService
from quivr_api.modules.user.entity.user_identity import UserIdentity
Expand Down Expand Up @@ -51,31 +51,31 @@

# Google sync description
google_sync = SyncsDescription(
name="Google",
name=SyncProvider.GOOGLE,
description="Sync your Google Drive with Quivr",
auth_method=AuthMethodEnum.URI_WITH_CALLBACK,
)

azure_sync = SyncsDescription(
name="Azure",
name=SyncProvider.AZURE,
description="Sync your Azure Drive with Quivr",
auth_method=AuthMethodEnum.URI_WITH_CALLBACK,
)

dropbox_sync = SyncsDescription(
name="DropBox",
name=SyncProvider.DROPBOX,
description="Sync your DropBox Drive with Quivr",
auth_method=AuthMethodEnum.URI_WITH_CALLBACK,
)

notion_sync = SyncsDescription(
name="Notion",
name=SyncProvider.NOTION,
description="Sync your Notion with Quivr",
auth_method=AuthMethodEnum.URI_WITH_CALLBACK,
)

github_sync = SyncsDescription(
name="GitHub",
name=SyncProvider.GITHUB,
description="Sync your GitHub Drive with Quivr",
auth_method=AuthMethodEnum.URI_WITH_CALLBACK,
)
Expand Down
4 changes: 2 additions & 2 deletions backend/api/quivr_api/modules/sync/service/sync_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
SyncStatus,
SyncUpdateInput,
)
from quivr_api.modules.sync.dto.outputs import SyncsOutput
from quivr_api.modules.sync.dto.outputs import SyncProvider, SyncsOutput
from quivr_api.modules.sync.repository.sync_repository import SyncsRepository
from quivr_api.modules.sync.utils.oauth2 import Oauth2BaseState, Oauth2State

Expand Down Expand Up @@ -54,7 +54,7 @@ async def get_from_oauth2_state(self, state: Oauth2State) -> SyncsOutput:

async def create_oauth2_state(
self,
provider: str,
provider: SyncProvider,
name: str,
user_id: UUID,
additional_data: dict[str, Any] = {},
Expand Down
Loading