Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose an API for appending params/names/descriptions in a programmable way. #2082

Merged

Conversation

numerology
Copy link

@numerology numerology commented Sep 10, 2019

An effort towards #1942

Add an API kfp.compiler.Compiler.create_workflow_from_func(func, name, description, params_list)
Usage:
Create workflow spec from pipeline function and specified pipeline params/metadata.

Now we can do something like the following:

ppl_param = dsl.PipelineParam(name='arg_specified_at_runtime')

def some_op():
  '''Example containerOp'''
  return dsl.ContainerOp(
      name='some op',
      image='busybox',
      command=['sleep 1'],
      arguments=[ppl_param]
  )

def pipeline_func():
  task = some_op()

compiler.Compiler().create_workflow(
    func=pipeline_func,
    name='Example Pipeline',
    description='Example description',
    params_list=[ppl_param])

It returns the workflow spec of the pipeline, then it can be dumped to yaml spec.


This change is Reviewable

@numerology
Copy link
Author

/retest

@numerology
Copy link
Author

/test kubeflow-pipeline-sample-test

sdk/python/kfp/compiler/compiler.py Outdated Show resolved Hide resolved
sdk/python/kfp/compiler/compiler.py Outdated Show resolved Hide resolved
sdk/python/kfp/compiler/compiler.py Show resolved Hide resolved
@numerology
Copy link
Author

/retest

@numerology
Copy link
Author

/test kubeflow-pipeline-e2e-test

@neuromage
Copy link
Contributor

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neuromage

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@numerology
Copy link
Author

/test kubeflow-pipeline-e2e-test

@k8s-ci-robot k8s-ci-robot merged commit 497d016 into kubeflow:master Sep 11, 2019
@numerology numerology deleted the expose-an-api-for-params-appending branch September 11, 2019 03:07
magdalenakuhn17 pushed a commit to magdalenakuhn17/pipelines that referenced this pull request Oct 22, 2023
* Added target and metric to components
Added scaleTarget and scaleMetric to ComponentSpec
Made changes to ksvc_reconciler to add annotations to component as required

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Added validation

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Fixed validation bugs

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Updated comments

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Updated python sdk and openapi

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Added test for autoscaling

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Added tests for autoscaling

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Added e2e for autoscaling changes
Removed validation for metric concurrency

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Updated CRD

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Fixed liniting issues

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Made ScaleMetric as enum
Added a test for raw deployment

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Fixed issues with type after changes in ScaleMetric type

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Updated crd

Updated crd

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Updated test case.
Modified to check deployment mode before checking hpa or kpa.

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Added debug logs to debug e2e failure in workflow alone

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Disabling all tests temporarily to debug failing e2e

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Reverting debugging changes

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Temporary changes to verify if e2e is passing after Dan's port forwarding

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Removed e2e changes.
Removed comments.
Updated python sdk version to debug test failure.

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Updated test variable

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Fixed linting error
Added test logs

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Adding logs to print spec

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Added more debugging logs

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Reverting python definition log in run e2e script

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Updated python sdk and docs
Fixed liniting error

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Fix for failing protobuf issue

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Added debug logs for kserve controller

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Added logs to check controller changes show up

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Added knative hpa

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Updated resource requirements

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Moved knative hpa installation to setup-deps

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

Removed skip for tests

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>
Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* Mutate and add deployment mode annotation if RawDeployment or ModelMesh

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

Co-authored-by: Dan Sun <dsun20@bloomberg.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants