Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TypeTransformer] Support non-Any Python types as Any input in workflows #2432

Closed
wants to merge 18 commits into from

Conversation

Future-Outlier
Copy link
Member

@Future-Outlier Future-Outlier commented May 20, 2024

Tracking issue

flyteorg/flyte#5366

Why are the changes needed?

We want to allow the Any type to accept all kinds of inputs and outputs, not just the types we currently can't handle.

For example, workflow like this.

@task
def foo(a: Any) -> int:
    if type(a) == int:
        return a + 1
    return 0

@workflow
def wf(a: int) -> int:
    return foo(a=a)

Note: This PR only supports Any in python, if we want to support java, we will need to add more code.

What changes were proposed in this pull request?

Check if the literal's scalar has a primitive type.
If it does, convert it to a Python value using the corresponding type transformer.

How was this patch tested?

unit test and remote cluster

from flytekit import task, workflow
from typing import Any

@task
def foo(a: Any) -> int:
    if type(a) == int:
        return a + 1
    return 0

@workflow
def wf(a: int) -> int:
    return foo(a=a)
FROM python:3.9-slim-buster
USER root
WORKDIR /root
ENV PYTHONPATH /root
RUN apt-get update && apt-get install build-essential -y
RUN apt-get install git -y

RUN pip install -U git+https://github.com/flyteorg/flytekit.git@3804a5155a523e2028ece8e7e581f794ebabd788

Setup process

pyflyte run --remote --image localhost:30000/any:0522 any_task.py wf --a 1

Screenshots

image image

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

flyteorg/flyte#5408

@Future-Outlier Future-Outlier changed the title [TypeTransformer] support input Non-Any Any Type [WIP] [TypeTransformer] support input Non-Any Any Type May 20, 2024
@Future-Outlier Future-Outlier marked this pull request as draft May 20, 2024 07:19
@Future-Outlier Future-Outlier changed the title [WIP] [TypeTransformer] support input Non-Any Any Type [TypeTransformer] support input Non-Any Any Type May 22, 2024
@Future-Outlier Future-Outlier self-assigned this May 22, 2024
@Future-Outlier Future-Outlier marked this pull request as ready for review May 22, 2024 08:49
@Future-Outlier Future-Outlier changed the title [TypeTransformer] support input Non-Any Any Type [TypeTransformer] support Non-Any to Any Conversion May 22, 2024
@Future-Outlier Future-Outlier changed the title [TypeTransformer] support Non-Any to Any Conversion [TypeTransformer] Support Python non-any type as any input and output in workflow May 22, 2024
@Future-Outlier Future-Outlier changed the title [TypeTransformer] Support Python non-any type as any input and output in workflow [TypeTransformer] Support non-Any Python types as Any input and output in workflows May 22, 2024
@Future-Outlier
Copy link
Member Author

Update: I'm not sure if it's ok to support primitive types only.

@Future-Outlier Future-Outlier changed the title [TypeTransformer] Support non-Any Python types as Any input and output in workflows [TypeTransformer] Support non-Any Python types as Any output in workflows May 24, 2024
@Future-Outlier Future-Outlier changed the title [TypeTransformer] Support non-Any Python types as Any output in workflows [TypeTransformer] Support non-Any Python types as Any input in workflows May 24, 2024
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
@Future-Outlier
Copy link
Member Author

Future-Outlier commented May 28, 2024

It can only run in local execution and pyflyte run --remote, since I haven't discovered a way to create metadata at flyteconsole.

image
@dataclass
class Datum:
    x: int
    y: str
    z: dict[int, str]
    w: list[int]
    
  @task
def foo_dataclass(a: Datum) -> int:
    if type(a) == Datum:
        return a.x
    return 0

@workflow
def foo_wf(a: Datum) -> int:
    return foo_dataclass(a=a)
pyflyte run --remote --image localhost:30000/any:0528-1445 any_task.py foo_wf --a '{ "x": 1, "y": "2", "z": { "1": "one", "2": "two" }, "w": [1, 2, 3] }'

Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
@Future-Outlier
Copy link
Member Author

Note:
python dotted path can solve type for Dataclass, while eval can solve most of python type,
we should use both of these ways

Signed-off-by: Future-Outlier <eric901201@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant