Running into TypeError exception with batch_predict component

[sahilkhanna129]

What steps did you take:

I am using batch_predict component to schedule prediction job on AI platform. I was referring to steps here. https://github.com/kubeflow/pipelines/tree/master/components/gcp/ml_engine/batch_predict

• I downloaded the component from https://raw.githubusercontent.com/kubeflow/pipelines/1.0.0/components/gcp/ml_engine/batch_predict/component.yaml.
• loaded the component with ml_op = load_component_from_file(<local-path>)
• created pipeline component with ml_op(<attributes-mentioned-below>)
• scheduled the pipeline on kubeflow

Job Parameters:

prediction_input: {'runtimeVersion': '2.1'}
project_id: <gcs-project>
region: <region-name>
model_path: <gcs-path>
input_paths: [<list-of-input-paths>]
input_data_format: TF_RECORD_GZIP
output_path: <gcs-path>
output_data_format: JSON

What happened:

Running into type error while generating the job_id.

Issue:

It is raising an exception while generating id to schedule the job. This line is failing https://github.com/kubeflow/pipelines/blob/1.0.0/components/gcp/container/component_sdk/python/kfp_component/google/common/_utils.py#L44 with this error TypeError: expected string or buffer. I am not passing any job_id and job_id_prefix. It is using the default values for generating the name.

Error trace:

INFO:root:Start KFP context with ID: 858b3ff01cdeed5c0b0b7fd9d2655641.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/ml/kfp_component/launcher/__main__.py", line 45, in <module>
    main()
  File "/ml/kfp_component/launcher/__main__.py", line 42, in main
    launch(args.file_or_module, args.args)
  File "kfp_component/launcher/launcher.py", line 45, in launch
    return fire.Fire(module, command=args, name=module.__name__)
  File "/usr/local/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
    component_trace = _Fire(component, args, context, name)
  File "/usr/local/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
    component, remaining_args)
  File "/usr/local/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
    result = fn(*varargs, **kwargs)
  File "kfp_component/google/ml_engine/_batch_predict.py", line 70, in batch_predict
    create_job(project_id, job, job_id_prefix, wait_interval)
  File "kfp_component/google/ml_engine/_create_job.py", line 48, in create_job
    return CreateJobOp(project_id, job, job_id_prefix, job_id, wait_interval
  File "kfp_component/google/ml_engine/_create_job.py", line 63, in execute_and_wait
    self._set_job_id(ctx.context_id())
  File "kfp_component/google/ml_engine/_create_job.py", line 74, in _set_job_id
    job_id = gcp_common.normalize_name(job_id)
  File "kfp_component/google/common/_utils.py", line 45, in normalize_name
    invalid_char_placeholder, name)
  File "/usr/local/lib/python2.7/re.py", line 155, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or buffer

What did you expect to happen:

I was expecting the component to schedule the job with auto generated job id.

Environment:

How did you deploy Kubeflow Pipelines (KFP)?

• I used Kustomize to deploy standalone kubeflow pipeline manifests on GKE.

KFP version:
1.0.0

KFP SDK version:
kfp==0.5.1

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug

[Ark-kun]

Thank you fr your bug report.
I've found the issue:
It was introduced in #3850 . That PR has added a new parameter in the middle of the the create_job function signature which can cause breaking changes as the parameter ordering changes.

[sahilkhanna129]

Hi @Ark-kun ,

Thanks for the quick response. Are you planning to release a new version of the component after merging the linked PR?

I am also wondering if there are any plans to migrate these components to python3?

Thanks

[Ark-kun]

Are you planning to release a new version of the component after merging the linked PR?

Yes.

I am also wondering if there are any plans to migrate these components to python3?

There are some plans, but they're not prioritized. I think these components already support python 3. Is these any issue with them using python 2 right now?

[sahilkhanna129]

For ML Engine Components, I am not running into any issue but Dataflow component is running the application code with Python2. My applications are written is python3, I am not sure if there are any workarounds for it. I am currently thinking of writing a custom component for Dataflow jobs to run application in python3.

[sahilkhanna129]

Hi @Ark-kun ,

I ran into similar issue with deploy component as well.

  File "kfp_component/google/ml_engine/_deploy.py", line 68, in deploy
    wait_interval)
  File "kfp_component/google/ml_engine/_create_version.py", line 64, in create_version
    replace_existing, wait_interval).execute_and_wait()
  File "kfp_component/google/ml_engine/_create_version.py", line 86, in execute_and_wait
    self._set_version_name(ctx.context_id())
  File "kfp_component/google/ml_engine/_create_version.py", line 114, in _set_version_name
    name = gcp_common.normalize_name(name)
  File "kfp_component/google/common/_utils.py", line 45, in normalize_name
    invalid_char_placeholder, name)
  File "/usr/local/lib/python2.7/re.py", line 155, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or buffer

Could you please change the function signature for _deploy.py in your PR as well?

Thanks