Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDK - Compiler - Fixed ParallelFor argument resolving #3029

Conversation

Ark-kun
Copy link
Contributor

@Ark-kun Ark-kun commented Feb 10, 2020

This PR rewrites the ParallelFor loop item argument resolving to solve many issues discovered by our users.

The ParallelFor argument reference resolving was really broken.
The old code implemented the following faulty logic: If the name of the referenced output contained the name of the loop collection source output, then it was considered to be the reference to the loop item.
This broke lots of scenarios especially in cases where there were multiple components with same output name (e.g. the default "Output" output name). The logic also did not distinguish between references to the loop collection item vs. references to the loop collection source itself.

I've rewritten the argument resolving logic, to fix the issues.

I need to make couple of checks to verify that everything works as expected.

Fixes #2943
Fixes #2936
Fixes #2569
Fixes #2829


This change is Reviewable

The ParallelFor argument reference resolving was really broken.
The logic "worked" like this - of the name of the referenced output
contained the name of the loop collection source output, then it was
considered to be the reference to the loop item.
This broke lots of scenarios especially in cases where there were
multiple components with same output name (e.g. the default "Output"
output name). The logic also did not distinguish between references to
the loop collection item vs. references to the loop collection source
itself.

I've rewritten the argument resolving logic, to fix the issues.
@Ark-kun Ark-kun force-pushed the SDK---Compiler---Fixed-ParallelFor-name-clashes branch from 2992fe9 to 8bd6620 Compare February 10, 2020 20:22
@Ark-kun Ark-kun changed the title [WIP]SDK - Compiler - Fixed ParallelFor name clashes SDK - Compiler - Fixed ParallelFor name clashes Feb 10, 2020
@Ark-kun Ark-kun changed the title SDK - Compiler - Fixed ParallelFor name clashes SDK - Compiler - Fixed ParallelFor argument resolving Feb 10, 2020
@Ark-kun Ark-kun force-pushed the SDK---Compiler---Fixed-ParallelFor-name-clashes branch 2 times, most recently from 183c9e4 to 16493ab Compare February 10, 2020 23:14
@Ark-kun Ark-kun force-pushed the SDK---Compiler---Fixed-ParallelFor-name-clashes branch from 16493ab to 5f626c1 Compare February 11, 2020 19:32
@Ark-kun
Copy link
Contributor Author

Ark-kun commented Feb 11, 2020

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@numerology
Copy link

/lgtm

@k8s-ci-robot k8s-ci-robot merged commit 4a1b282 into kubeflow:master Feb 11, 2020
Jeffwan pushed a commit to Jeffwan/pipelines that referenced this pull request Dec 9, 2020
* SDK - Compiler - Fixed ParallelFor name clashes

The ParallelFor argument reference resolving was really broken.
The logic "worked" like this - of the name of the referenced output
contained the name of the loop collection source output, then it was
considered to be the reference to the loop item.
This broke lots of scenarios especially in cases where there were
multiple components with same output name (e.g. the default "Output"
output name). The logic also did not distinguish between references to
the loop collection item vs. references to the loop collection source
itself.

I've rewritten the argument resolving logic, to fix the issues.

* Argo cannot use {{item}} when withParams items are dicts

* Stabilize the loop template names

* Renamed the test case
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants