Renku as a Service: Datasets

[jsam]

Description

The renku service should offer functionality to allow the following operations on data sets:

• create datasets
• add files to datasets
• remove files from datasets
• delete datasets

Implementation

To provide this functionality, the renku service needs to do several additional things:

• clone the project repo
• push the project repo back to server
• receive uploaded files
• receive uploaded directories

Design

To develop a PoC quickly, some design choices have been made:

• The service will respond to requests synchronously

Original

With #690 being merged we have enabled the ability to implement all dataset management endpoints within a renku-python as a service.

Work which has been done in https://github.com/SwissDataScienceCenter/renku-core/issues should be ported into renku-python and implemented as a service.

related: #731

[ciyer]

From today's discussion

File/folder hierarchies

If the user just uploads an individual file, then current handling is fine. But, users may want to upload folder/file hierarchies. The easiest solution is to let the user structure things the way they want on their file system and then select the root of what they want to upload. The UI will create an archive and upload that and include a flag as part of the request stating that the contents need to be unarchived.

[ciyer]

From today's discussion

Endpoints

datasets

• /datasets/list
• /datasets/create
• /datasets/files-add
• /datasets/files-list

projects/files

• /cache/projects-list
• /cache/project-clone
• /cache/files-list
• /cache/files-upload

[ciyer]

From today's discussion

/cache/files-list

The return should be an object with at least

{
  "id": [service-file-id],
  "path": [path of the file],
  "size": [size of the file]
}

/dataset/files

The return should be an object with at least

{
  "path": [path of the file],
  "size": [size of the file]
}

[rokroskar]

As discussed yesterday, we should immediately have a plan for enabling the service to handle requests asynchronously. To me it looks like Celery would be a good option because it would still allow us to use Flask as the web framework and not have to resort to too many exotic python libraries/dependencies/restrictions. As also discussed with @ciyer, using coroutines would be of limited value in cases where the long-running tasks were executing within python (e.g. git tasks in renku-python) anyway.

[rokroskar]

This epic has been completed - new, finer-grained epics should be opened as needed. 🚀