Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDK - Compiler - Make it possible to create more portable pipelines #2271

Commits on Sep 30, 2019

  1. SDK - Compiler - Allow creating portable pipelines

    This change allows directly passing the PipelineConf instance to compiler or launcher which makes it easier to create portable pipelines by allowing the environment-specific configuration to be directly passed to the environment-specific launcher.
    
    Background:
    PipelineConf holds all pipeline-level configuration including `op_transformers`, `image_pull_secrets` etc. Some of these are specific to particular execution environment (e.g. GCP secret or Argo artifact location or Kubernetes-specific options).
    Previously, the only way to modify `PipelineConf` was to do it inside the piepline function. That tied the pipeline function to specific execution environment (e.g. GCP, Argo or Kubernetes)
    
    Solution: This change allows directly passing the PipelineConf instance to compiler or launcher. This allows writing portable enlauncher and environment agnostic pipeline functions. All environment-specific configurations can be moved to launching stage.
    
    Before:
    ```python
    # Defining pipeline
    def my_pipeline():
        # portable pipeline code
    
        dsl.get_pipeline_conf().add_op_transformer(gcp.use_gcp_secret('user-gcp-sa'))
    
    # Launching pipeline
    kfp.Clinet().create_run_from_pipeline_func(my_pipeline, arguments={})
    ```
    
    After:
    ```python
    # Defining pipeline
    def my_pipeline():
        # portable pipeline code
    
    # Launching pipeline
    pipeline_conf = dsl.PipelineConf()
    pipeline_conf.add_op_transformer(gcp.use_gcp_secret('user-gcp-sa'))
    kfp.Clinet().create_run_from_pipeline_func(my_pipeline, arguments={}, pipeline_conf=pipeline_conf)
    ```
    
    After 2 *(launching same portable pipeline using different launchers):
    ```python
    # Loading portable pipeline
    from portable_pipeline import my_pipeline
    
    # Launching pipeline on Kubeflow
    pipeline_conf = dsl.PipelineConf()
    pipeline_conf.add_op_transformer(gcp.use_gcp_secret('user-gcp-sa'))
    kfp.Clinet().create_run_from_pipeline_func(my_pipeline, arguments={}, pipeline_conf=pipeline_conf)
    
    # Launching pipeline on locally (not implemented yet)
    kfp.run_pipeline_func_locally(my_pipeline, arguments={})
    ```
    Ark-kun committed Sep 30, 2019
    Configuration menu
    Copy the full SHA
    d99525d View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2019

  1. Added parameter docstring

    Ark-kun committed Oct 3, 2019
    Configuration menu
    Copy the full SHA
    d6a64a2 View commit details
    Browse the repository at this point in the history