Fix PipelineParam pattern bug

[elikatsis]

If a PipelineParam was found twice in an Op, e.g. in arguments and in volumes, having different patterns (None vs anything) messed the inputs attribute (which uses set() on the extracted PipelineParams).
The different patterns may occur since arguments (as well as command) gets the str() representation of the PipelineParam before it gets sanitized.

I chose to generate a pattern in the constructor, if one is not provided.
Otherwise, we could use the pattern attribute in __hash__ method, but it messes tests because it generally leads to duplicate entries.

---

This change is 

[k8s-ci-robot]

Hi @elikatsis. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

Hi @elikatsis. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Ark-kun]

PipelineParam.pattern is news to me. It should not exist and it definitely should not be public. We should remove it.

But for this PR
/lgtm
/approve

[Ark-kun]

/approve cancel

[Ark-kun]

Sorry, can you please add a test demonstrating the issue that you've fixed? It might be an _op_to_template test...

[elikatsis]

Sorry, can you please add a test demonstrating the issue that you've fixed? It might be an _op_to_template test...

Oh yes, good idea!
It had to be a whole pipeline test because the problem occurs when sanitization happens (loop through all Ops in _compile() Compiler's method).

[Ark-kun]

For future reference: Here is the error:

~/_projects/pipelines/sdk/python$ 
PYTHONPATH=. python3 -c '

import kfp.dsl as dsl


@dsl.pipeline(
    name="Param Substitutions",
    description="Test the same PipelineParam getting substituted in multiple "
                "places"
)
def param_substitutions():
    vop = dsl.VolumeOp(
        name="create_volume",
        resource_name="data",
        size="1Gi"
    )

    op = dsl.ContainerOp(
        name="cop",
        image="image",
        arguments=["--param", vop.output],
        pvolumes={"/mnt": vop.volume}
    )


if __name__ == "__main__":
    import kfp.compiler as compiler
    compiler.Compiler().compile(param_substitutions, "bla.pipeline.yaml")

'
Traceback (most recent call last):
  File "<string>", line 28, in <module>
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/compiler.py", line 665, in compile
    workflow = self._compile(pipeline_func)
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/compiler.py", line 650, in _compile
    workflow = self._create_pipeline_workflow(args_list_with_defaults, p)
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/compiler.py", line 514, in _create_pipeline_workflow
    templates = self._create_templates(pipeline)
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/compiler.py", line 484, in _create_templates
    templates.append(_op_to_template(op))
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/_op_to_template.py", line 183, in _op_to_template
    processed_op = _process_base_ops(op)
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/_op_to_template.py", line 106, in _process_base_ops
    setattr(op, key, _process_obj(getattr(op, key), map_to_tmpl_var))
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/_op_to_template.py", line 73, in _process_obj
    setattr(obj, key, _process_obj(getattr(obj, key), map_to_tmpl_var))
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/_op_to_template.py", line 50, in _process_obj
    return [_process_obj(item, map_to_tmpl_var) for item in obj]
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/_op_to_template.py", line 50, in <listcomp>
    return [_process_obj(item, map_to_tmpl_var) for item in obj]
  File "/usr/local/google/home/avolkov/_projects/pipelines/sdk/python/kfp/compiler/_op_to_template.py", line 46, in _process_obj
    obj = re.sub(param_tuple.pattern, map_to_tmpl_var[param_tuple.pattern], obj)
KeyError: '{{pipelineparam:op=create_volume;name=name;value=;type=;}}'

[Ark-kun]

/lgtm
Waiting for the tests to complete

Thanks again for the catch!

[Ark-kun]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• sdk/OWNERS [Ark-kun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• sdk/OWNERS [Ark-kun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment