[feature] [SDK] Extract Executor to its own package

[leseb]

Feature Area

/area sdk

What feature would you like to see?

To improve modularity and reduce dependency overhead in Kubeflow Pipelines, I propose extracting the executor component into a standalone package. Currently, the executor is tightly coupled with other SDK components, resulting in unnecessary dependencies during execution. By isolating the executor into its own package, we can minimize the dependency footprint and enhance flexibility for users seeking a more lightweight solution.

Something like kfp.executor package would be nice. Thanks for considering this proposal.

Love this idea? Give it a 👍.

[leseb]

@HumairAK FYI

[HumairAK]

past attempt on this: #9710

[chensun]

Something like kfp.executor package would be nice. Thanks for considering this proposal.

Second part of this effort was #9738, which was later reverted by #9874 due to some technical challenges IIRC. @connor-mccarthy do you recall the exact issue?

[HumairAK]

In today's community call it was decided that this approach makes sense and got a +1 to move forward.

[chensun]

And what we ended up as a shortcut was to install kfp without any dependencies:

https://github.com/connor-mccarthy/pipelines/blob/fcdff294a6323f6cb1c0e574fc7aa5ccc25e420b/sdk/python/kfp/dsl/component_factory.py#L146-L149

which is arguably the same level of dependency as installing a subpackage like kfp.runtime.

[leseb]

Could we get a list of the technical challenges that caused the initial revert of the previous attempt? This will help us avoid repeating the same issues. Thanks!