Allow Python functions defined in a pipeline to be run as entrypoint for operations.

[FaustineLi]

It would be great if lightweight components could be created in a pipeline .py file from functions defined inside that pipeline file. I am envisioning something like this:

def sum_num(num):
      sum = 0
      for i in range(num):
            sum += i
      return sum

op1 = dsl.ContainerOp(
    name='sum',
    image='<some_base_image>'
    function_entrypoint=sum_num,
    arguments=[{'num': 5}]
  )

For many tasks it would reduce the number of containers to build and maintain while speeding up the iteration loop. Currently there's support for a dockerless workflow in notebooks. Since our organization is trying to move code out of notebooks for production work, I would love to see it supported in other ways. I don't see this replacing large components, but the ability to author a component that does a short, simple thing seems really powerful.

[Ark-kun]

JFYI, Python supports putting everything in a function.
You can put functions, classes, import statements etc inside one main function:

def my_component_func(num1: int, num2: int) -> int:
  import numpy as mp
  def sum_num(num):
      sum = 0
      for i in range(num):
            sum += i
      return sum
  return sum_num(num1) + sum_num(num2)

Does that solve your problem?

BTW, you can probably put the my_component_func in a separate .py file and then just import it.

from my_module import my_component_func
my_op = func_to_container_op(my_component_func)

JFYI, you can use my_op = func_to_container_op(my_component_func, output_component_file='component.yaml') to write the component to a file on disk that can be shared between different pipelines or people.

[FaustineLi]

Awesome, I had no idea that func_to_container_op was a thing!

[Ark-kun]

It would be great if lightweight components could be created in a pipeline .py file from functions defined inside that pipeline file.
Awesome, I had no idea that func_to_container_op was a thing!

Interesting. Did you get the term "lightweight components" from the documentation or did you just call the feature like that by yourself?

The Lightweight python components are shown in this sample notebook which is referenced in the (scarce) documentation.

[gaoning777]

Note that we also have a component decorator(

pipelines/sdk/python/kfp/dsl/_component.py

Line 56 in d2f5cc9

def component(func):

). Maybe use that instead of the func_to_container_op because we currently use the func_to_container_op internally.

[vicaire]

Looks like this is resolved. Please reopen if this is not the case.