Skip to content

Implement Task Manager - #84

Merged
JJFinnerty merged 24 commits into
developfrom
feature/53-dispatch-tasks
May 17, 2022
Merged

Implement Task Manager#84
JJFinnerty merged 24 commits into
developfrom
feature/53-dispatch-tasks

Conversation

@mocsharp

@mocsharp mocsharp commented Apr 8, 2022 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Fixes #53.

Description

#53 Implement Task Manager with a stubbed input message queue until Workflow Manager is ready.

  • Provides ITaskRunner interface and RunnerBase base class with ExecuteTask(...) and GetStatus(...) methods for executing a task plug-in and getting the status of a task.

#54 Implements Argo plug-in that requires 2 templates, the main workflow template (kind: WorkflowTemplate), and an exit hook template (kind: WorkflowTemplate). The Argo plug-in creates a kind: Workflow with references to the 2 provided templates. It also creates any Kubernetes credentials required to connect the Argo workflow to the MinIO buckets and deletes them on completion. Due to this Argo bug, this implementation creates an Argo Workflow (kind: Workflow) based on the provided Argo workflow template (kind: WorkflowTemplate).

The Argo plug-in copies the entrypoint template and any child templates (through dags or steps) into a new Workflow and injects s3 information provided by the Workflow Manager & Task Manager. See #54 for details.

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • All tests passed locally.
  • Documentation comments included/updated.
  • User guide updated.
  • I have updated the changelog

@mocsharp mocsharp added the enhancement New feature or request label Apr 8, 2022
@mocsharp mocsharp added this to the POC milestone Apr 8, 2022
@mocsharp mocsharp self-assigned this Apr 8, 2022
@codecov

codecov Bot commented Apr 8, 2022 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #84 (0bc3926) into develop (748a5a1) will not change coverage.
The diff coverage is n/a.

❗ Current head 0bc3926 differs from pull request most recent head bd96ffd. Consider uploading reports for the commit bd96ffd to get more accurate results

Impacted file tree graph

@@      Coverage Diff      @@
##   develop   #84   +/-   ##
=============================
=============================

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 748a5a1...bd96ffd. Read the comment docs.

@mocsharp
mocsharp force-pushed the feature/53-dispatch-tasks branch 4 times, most recently from e4ef86a to aa33394 Compare April 14, 2022 21:27
@mocsharp
mocsharp marked this pull request as ready for review April 18, 2022 19:19
@mocsharp
mocsharp force-pushed the feature/53-dispatch-tasks branch from f1b0b2f to ad1f429 Compare April 18, 2022 20:46
@mocsharp
mocsharp marked this pull request as draft April 23, 2022 02:08
@mocsharp
mocsharp force-pushed the feature/53-dispatch-tasks branch 2 times, most recently from e00066d to 68c442c Compare April 27, 2022 00:18
@mocsharp
mocsharp marked this pull request as ready for review April 27, 2022 00:21
@mocsharp mocsharp mentioned this pull request May 3, 2022
Closed
mocsharp added a commit that referenced this pull request May 3, 2022
@mocsharp
gh-84 Use admin credentials for artifact mapping
7a15920 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
mocsharp added a commit that referenced this pull request May 5, 2022
@mocsharp
gh-84 Update messaging lib version
9377e60 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
mocsharp added a commit that referenced this pull request May 5, 2022
@mocsharp
gh-84 Fix unit test
17a15e8 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
mocsharp force-pushed the feature/53-dispatch-tasks branch from 17a15e8 to a8fcee8 Compare May 6, 2022 17:20
mocsharp added a commit that referenced this pull request May 6, 2022
@mocsharp
gh-84 Use admin credentials for artifact mapping
26614a4 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
mocsharp added a commit that referenced this pull request May 6, 2022
@mocsharp
gh-84 Update messaging lib version
419b268 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
mocsharp added a commit that referenced this pull request May 6, 2022
@mocsharp
gh-84 Fix unit test
ac3f1ba 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
mocsharp added a commit that referenced this pull request May 6, 2022
@mocsharp
gh-84 Rebase from develop
a8fcee8 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
mocsharp requested review from RemakingEden, jackschofield23 and joshliberty and removed request for jackschofield23 May 6, 2022 17:20
joshliberty
joshliberty reviewed May 9, 2022
View reviewed changes
Comment thread src/TaskManager/Runner/Program.cs
joshliberty
joshliberty reviewed May 9, 2022
View reviewed changes
Comment thread src/TaskManager/Runner/Program.cs
mocsharp added 19 commits May 16, 2022 11:45
@mocsharp
gh-54 Define TaskRunerAPI
d372357 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
Update Monai.Deploy.Messaging version
ab1235a 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-54 Implement Argo and Kubernetes providers
8323458 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-54 Add TaskManager runner
81bd1ee 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-53 gh-54 Add unit test for Task Manager & Argo plug-in
1fd7d3d 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-53 gh-54 fix typos
87b3d25 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-53 Subscribe to md.tasks.dispatch directly using Messaging lib
1a7afce 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-54 Create Workflow from provided WorkflowTemplate
9637bb2 
- include end hook template
- handle TaskCallbackEvent

Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-54 Fix unit test
1c7f153 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-54 Rebase with develop
71e63a0 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-47 gh-53 gh-54 Clean ups to match issue requirements
19dc027 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-54 Wait for Argo to be in the final state
168602c 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
Update dependency_decisions.yml
4000245 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-54 Warn missing storage mapping instead of throw
800be96 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-84 Use admin credentials for artifact mapping
6901747 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-84 Update messaging lib version
b5194f1 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-84 Fix unit test
db202e3 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-84 Update messaging lib
0a97ba0 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-84 Support dag/step with intermediate output
98972b0 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
mocsharp force-pushed the feature/53-dispatch-tasks branch from 5fa1767 to 4b94978 Compare May 16, 2022 19:42
mocsharp added a commit that referenced this pull request May 16, 2022
@mocsharp
gh-84 Populate credentials for intermediate storage
4b94978 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
gh-84 Populate credentials for intermediate storage
761dffd 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
mocsharp force-pushed the feature/53-dispatch-tasks branch from 4b94978 to 761dffd Compare May 16, 2022 20:14
mocsharp added 2 commits May 16, 2022 13:43
@mocsharp
approve liceses
ecf1022 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@mocsharp
Move test projects
bd96ffd 
Signed-off-by: Victor Chang <vicchang@nvidia.com>
@sonarqubecloud

sonarqubecloud Bot commented May 16, 2022 •
edited
Loading

Copy link
Copy Markdown

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 11 Code Smells

77.8% 77.8% Coverage
0.1% 0.1% Duplication

@JJFinnerty
JJFinnerty self-requested a review May 17, 2022 09:33
@JJFinnerty
JJFinnerty merged commit 3088b16 into develop May 17, 2022
@mocsharp
mocsharp deleted the feature/53-dispatch-tasks branch May 17, 2022 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joshliberty joshliberty joshliberty left review comments
@jackschofield23 jackschofield23 jackschofield23 approved these changes
@JJFinnerty JJFinnerty JJFinnerty approved these changes
@RemakingEden RemakingEden Awaiting requested review from RemakingEden

Assignees

@mocsharp mocsharp

Labels

enhancement New feature or request

Projects

Clinical Data Pipelines
Status: Done

Milestone

POC

Development

Successfully merging this pull request may close these issues.

Dispatch tasks

4 participants

@mocsharp @jackschofield23 @joshliberty @JJFinnerty