Merge pull request #35 from avast/mlops-123

mlops-123 auto-generated docs for models + cmek fix for jobs

docs/tutorial/docker.md

@@ -11,8 +11,10 @@ date: 2022-04-06
 Multiple resources created by WANNA rely on Docker containers. We make it easy for you to
 build your images either locally or using GCP Cloud Build.
 
-
 ### Types of docker images
+
+::: wanna.core.models.docker.DockerImageModel
+
 We currently support three types of docker images:
 
 - `provided_image` - you supply a link to the docker image in the registry. We don't build anything,
@@ -55,20 +57,13 @@ and there is no need to push the images over the network. That makes it suitable
 However, building images in the cloud is not allowed for production.
 
 ### Build configuration
+
+::: wanna.core.models.docker.DockerBuildConfigModel
+
 When building locally, we offer you a way to set additional build parameters. These parameters
 must be specified in a separate yaml file in path `WANNA_DOCKER_BUILD_CONFIG`. If this is not set,
 it defaults to the `dockerbuild.yaml` in the working directory.
 
-You can set:
-
-- `build_args: Dict[str, str]` 
-- `labels: Dict[str, str]`
-- `network: Optional[str]`
-- `platforms: Optional[List[str]]`
-- `secrets: Union[str, List[str]]`
-- `ssh: Optional[str]`
-- `target: Optional[str]`
-
 These parameters refer to [standard docker build parameters](https://github.com/docker/buildx#buildx-bake-options-target).
 
 One example use case can be when you want to git clone your internal repository during
@@ -89,6 +84,9 @@ RUN --mount=type=ssh,id=github git clone git@git.your.company.com:your_profile/y
 ```
 
 ### Parameters for docker section
+
+::: wanna.core.models.docker.DockerModel
+
 docker section takes following parameters:
 - `images` - list of docker images, see below
 - `repository`- GCP Artifact Registry repository for pushing images

docs/tutorial/job.md

@@ -9,9 +9,14 @@ date: 2022-04-06
 
 # WANNA Job
 
+::: wanna.core.models.training_custom_job.BaseCustomJobModel
+    options:
+      members: false
+
+## Hyper-parameter tuning
 
+::: wanna.core.models.training_custom_job.HyperparameterTuning
 
-## Hyper-parameter tuning
 A custom job can be simply converted to a hyper-parameter tuning job just by adding 
 one extra parameter called `hp_tuning`. This will start a series of jobs (instead of just one job) 
 and try to find the best combination of hyper-parameters in regard to a target variable that you specify.

docs/tutorial/managed-notebook.md

@@ -12,6 +12,8 @@ date: 2022-06-03
 It offers a simple way of deploying Jupyter Notebooks on GCP, with minimum environment
 set up, automatic mount of all GCS buckets in the project, connection do Dataproc clusters and more.
 
+::: wanna.core.models.notebook.ManagedNotebookModel
+
 ### Obligatory fields
 - `name`- Custom name for this instance
 - `owner` - Currently only the owner will be able to access the notebook

docs/tutorial/notebook.md

@@ -11,6 +11,8 @@ date: 2022-04-06
 We offer a simple way of managing Jupyter Notebooks on GCP, with multiple ways
 to set your environment, mount a GCS bucket, and more.
 
+::: wanna.core.models.notebook.NotebookModel
+
 ### Notebook Environments
 There are two distinct possibilities for your environment.
 

docs/tutorial/pipeline.md

@@ -11,7 +11,6 @@ date: 2022-04-06
 
 WANNA ML Pipelines aim at reducing the friction on development cycle to release whilst providing project organization for independent and testable components. It wrapps Kubeflow V2 pipelines with a build and deployment tools for managed GCP VertexAI ML Pipelines service. It has several utils and conventions to reduce boilerplate and speed up development.
 
-
 ## Tutorial Get started
 
 In this tutorial we will go over several steps that will bootstrap and run a Vertex AI Pipeline(Kubeflow V2) through wanna cli.
@@ -193,6 +192,8 @@ You can now see that you have a self contained component and testable, ready to
 
 ## Pipeline
 
+::: wanna.core.models.pipeline.PipelineModel
+
 WANNA template creates two files that allow to put together the Kubeflow V2 pipeline that will then be deployed to GCP Vertex AI Pipelines.
 
 `pipeline/config.py` captures wanna compile time exposed environment variables and provides this as configuration to `pipeline/pipeline.py`.

docs/tutorial/profile.md

@@ -29,6 +29,11 @@ either `WANNA_GCP_PROFILE_NAME=my-profile-name` or `--profile=my-profile-name`.
 When the selected WANNA Profile is not found, we throw an error.
 
 ### WANNA Profile parameters
+
+::: wanna.core.models.gcp_profile.GCPProfileModel
+    options:
+      members: false
+
 `wanna_profile` section of the yaml config consists of the following inputs:
 
 - `profile_name` - name of the WANNA GCP Profile, `default` will be used if not specified otherwise.

docs/tutorial/project.md

@@ -10,6 +10,8 @@ date: 2022-04-06
 # WANNA project
 WANNA project settings set some basic values about your project. 
 
+::: wanna.core.models.wanna_project.WannaProjectModel
+
 `wanna_project` section of the yaml config consists of the following inputs:
 
 - `name` - the name of the wanna project should be unique, this name will be used in the docker service 

docs/tutorial/tensorboard.md

@@ -15,6 +15,8 @@ Tensorboards can either be used as a separate resource with `wanna tensorboard c
 or you can use them similarly to docker images as a dependency of other resources with `tensorboard_ref` key. In the second case,
 Tensorboards are automatically created when needed and used when already existing.
 
+::: wanna.core.models.tensorboard.TensorboardModel
+
 ### Tensorboard parameters
 Tensorboards take only two parameters:
 

mkdocs.yaml

@@ -67,6 +67,7 @@ nav:
 
 plugins:
 - search
+- mkdocstrings
 
 markdown_extensions:
     - mkdocs-typer

pyproject.toml

@@ -70,6 +70,8 @@ typer = "0.6.1"
 waiting = "^1.4.1"
 rich = "^12.5.1"
 types-setuptools = "^63.2.3"
+mkdocstrings = "^0.19.0"
+mkdocstrings-python = "^0.7.1"
 
 [tool.poetry.dev-dependencies]
 black = "^22.3.0"

src/wanna/core/deployment/models.py

@@ -92,7 +92,7 @@ class JobResource(GCPResource, Generic[JOB]):
     tensorboard: Optional[str]
     network: Optional[str]
     job_config: JOB
-    encryption_spec_key_name: Optional[str]
+    encryption_spec: Optional[str]
 
 
 class PushArtifact(BaseModel):

src/wanna/core/deployment/vertex_jobs.py

@@ -68,7 +68,7 @@ def run_custom_job(self, manifest: JobResource[CustomJobModel], sync: bool) -> N
                 max_trial_count=manifest.job_config.hp_tuning.max_trial_count,
                 parallel_trial_count=manifest.job_config.hp_tuning.parallel_trial_count,
                 search_algorithm=manifest.job_config.hp_tuning.search_algorithm,
-                encryption_spec_key_name=manifest.encryption_spec_key_name,
+                encryption_spec_key_name=manifest.encryption_spec,
             )
         else:
             runable = custom_job  # type: ignore
@@ -112,12 +112,14 @@ def run_training_job(
 
         if manifest.job_config.worker and manifest.job_config.worker.container:
             training_job = CustomContainerTrainingJob(
-                model_encryption_spec_key_name=manifest.encryption_spec_key_name,
+                training_encryption_spec_key_name=manifest.encryption_spec,
+                model_encryption_spec_key_name=manifest.encryption_spec,
                 **manifest.job_payload,
             )
         elif manifest.job_config.worker and manifest.job_config.worker.python_package:
             training_job = CustomPythonPackageTrainingJob(
-                model_encryption_spec_key_name=manifest.encryption_spec_key_name,
+                training_encryption_spec_key_name=manifest.encryption_spec,
+                model_encryption_spec_key_name=manifest.encryption_spec,
                 **manifest.job_payload,
             )  # type: ignore
         else: