Skip to content

Conversation

@mawandm
Copy link
Contributor

@mawandm mawandm commented Apr 16, 2024

Description
This PR introduces the concept of tasks within the API module. A use case for task is running an ingestion pipeline from a datasource.

The result of this PR is that we will not need to current uniform scheduling mechanism. Rather, users are able to pick when to schedule the ingestion of a specific datasource.

Testing this PR
To test this PR,

  1. Run the RAG Microservice (Set the env variable NESIS_RAG_EMBEDDING_DIMENSIONS = 384 in order to use HuggingFace's embeddings models)
  2. Run the API Microservice
  3. Create a session with
curl -X POST -H "Content-Type: application/json"  -k http://localhost:6000/v1/sessions -d '{"email":"test.user@domain.com","password":"password"}'
  1. Create a datasource with (Use token from step 3. above
curl -d '{"type": "minio", "name": "cloud-documents", "connection": {"user": "your_username","password": "your_password","endpoint": "http://localhost:59000","dataobjects": "boc"}}' -H "Content-Type: application/json"  -H "Authorization: Bearer <token>" -X POST http://127.0.0.1:6000/v1/datasources
  1. Create a task to run immediately with replace parent_id and datasource.id with the datasource id from step 4.
curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer <token>" -k http://localhost:6000/v1/tasks -d '{"type":"ingest_datasource","parent_id":"<datasource-id>",  "definition":{"datasource":{"id":"<datasource-id>"}}}'

Ingestion should commence shortly

Closes #8

@mawandm mawandm marked this pull request as draft April 16, 2024 18:28
@mawandm mawandm requested review from akizito and removed request for akizito April 18, 2024 01:00
@mawandm mawandm marked this pull request as ready for review April 18, 2024 09:39
@mawandm mawandm requested a review from akizito April 18, 2024 09:39
@mawandm mawandm changed the title feat(api): add task scheduling feature feat(api)(frontend): add task scheduling feature Apr 19, 2024
@mawandm mawandm changed the title feat(api)(frontend): add task scheduling feature feat(api,frontend): add task scheduling feature Apr 19, 2024
@mawandm mawandm changed the title feat(api,frontend): add task scheduling feature feat(api)(frontend): add task scheduling feature Apr 19, 2024
@mawandm mawandm self-assigned this Apr 19, 2024
@mawandm mawandm merged commit 5099ffb into main Apr 19, 2024
@mawandm mawandm deleted the 8-introduce-task-scheduling branch April 19, 2024 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add ability to schedule a datasource ingestion job

3 participants