In this module we will create a Cloud Function that executes a Vertex AI pipeline on-demand based off of a pipeline JSON in GCS. This module takes 15 minutes to review, and almost an hour to run.
- Successful testing of pipeline template JSON
- Customized Vertex AI Spark ML model training template JSON in GCS
We completed #1 in the prior module. #2 is already available for you in GCS.
Read the documentation for scheduling Vertex AI pipelines ahead of working on the next step to better understand on-demand execution through a simpler example than the one in the lab.
https://cloud.google.com/vertex-ai/docs/pipelines/schedule-cloud-scheduler
The Cloud Function is already deployed in your environment. The folowing is the author's deployment from the Terraform script. Yours should be identical.
The latest requirements.txt is avialable here-
https://github.com/anagha-google/s8s-spark-mlops-lab/blob/main/02-scripts/cloud-functions/requirements.txt
The latest source code is avialable here-
https://github.com/anagha-google/s8s-spark-mlops-lab/blob/main/02-scripts/cloud-functions/main.py
The Cloud Function is generation 2 and does not have a "click to test" button feature yet. We need to grab the command line execution from the UI and run it in Cloud Shell.
Follow at least a couple steps through completion.
This concludes the module. In the next module, we will create a Cloud Scheduler job for time based execution of the model training pipeline.