Skip to content

Commit 58c5adb

Browse files
committed
Add how to run batch job with steps
1 parent 10fd58f commit 58c5adb

File tree

1 file changed

+9
-0
lines changed

1 file changed

+9
-0
lines changed

blog/build-machine-learning-workflow.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,9 @@ vbatch -f -P cpu-8 python my_long_running_script.py
4747

4848
Let's build a simple ML workflow with three stages: data processing, model training, and evaluation. We can chain these tasks together using the `--after-success` flag, which ensures a task starts only after its dependencies have successfully completed. Assuming you already have scripts for each step, creating the pipeline in Velda is as simple as running these commands:
4949

50+
`training_pipeline.sh`:
5051
```
52+
#!/bin/bash
5153
# Process the data, e.g. data cleaning
5254
vbatch -P cpu-16 --name process python process_data.py
5355
@@ -59,6 +61,13 @@ vbatch -P cpu-8 --name eval --after-success train python evaluate_model.py
5961
```
6062

6163
This creates a linear pipeline where each step is executed in sequence.
64+
To execute the pipeline, run:
65+
```
66+
vbatch ./training_pipeline.sh
67+
```
68+
69+
All tasks created within a vbatch script will be automatically grouped as sub-tasks, and the parent task
70+
is only marked as complete when all children tasks complete.
6271

6372
<img src="https://cdn-images-1.medium.com/max/1600/1*UFUi0CM4fZaN1ey4MnZfmQ.png" alt="Task list view" />
6473
<center><small>Web page for task list view</small></center>

0 commit comments

Comments
 (0)