[SDK-compiler] Refactor Compiler to expose an API to write out yaml spec of pipeline. #2146
Conversation
sdk/python/kfp/compiler/compiler.py
Outdated
| def compile(self, pipeline_func, package_path, type_check=True): | ||
| """Compile the given pipeline function into workflow yaml. | ||
| @staticmethod | ||
| def write_workflow(workflow: Dict[Text, Any], package_path: Text = None): |
There was a problem hiding this comment.
I'm not sure this method should be public.
There was a problem hiding this comment.
I think here we have two options:
- We can expose this API so that TFX side can use it to dump workflow spec to arbitrary format, or;
- we can duplicate this part of logic to TFX's code base, which should not be a very serious problem because this is not very KFP-specific function. It's just a yaml dump utility.
WDYT?
There was a problem hiding this comment.
I think that the Compiler user should not see or handle the compiled pipeline code dictionary. If this is the case, then the user never has anything they can pass to the write_workflow method. Thus, I think, it should be private.
What are the cases where TFX code might want to run write_workflow, but not compile? Can they call private function/method?
There was a problem hiding this comment.
I think that the
Compileruser should not see or handle the compiled pipeline code dictionary. If this is the case, then the user never has anything they can pass to thewrite_workflowmethod. Thus, I think, it should be private.
What are the cases where TFX code might want to runwrite_workflow, but notcompile? Can they call private function/method?
See here. Currently TFX is calling create_workflow in order to plug in PipelineParam dynamically. I think they can call private function if bypass the lint check.
There was a problem hiding this comment.
Currently TFX is calling create_workflow
Maybe create_workflow should write the pipeline package the same way the compile does?
There was a problem hiding this comment.
That's a good suggestion. Will let you know once I'm done. Thanks!
There was a problem hiding this comment.
BTW, I'm in favor of extracting the _write_workflow function. I just think that it should be private.
|
Some thoughts about possible solution to the
|
…xtract-api-to-writeout-workflow-spec � Conflicts: � sdk/python/kfp/compiler/compiler.py
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Ark-kun The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Signed-off-by: Suresh Nakkeran <suresh.n@ideas2it.com>
This change will unblock customize file extension issue in tfx.
This change is