-
Notifications
You must be signed in to change notification settings - Fork 626
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Core feature] Flytekit should support using output with Non-Any
type as the input with Any
type.
#5366
Comments
Thank you for opening your first issue here! 🛠 |
To support using output with
Here's an example implementation: from flytekit import task, workflow, TypeEngine, context_manager
from flytekit.types.structured import StructuredDataset
from flytekit.models import types as _type_models
from flytekit.models.literals import Literal, Scalar, Blob, BlobMetadata
class AnyTypeTransformer(TypeEngine.TypeTransformer):
def __init__(self):
super().__init__(name="AnyType", t=object)
def get_literal_type(self, t):
return _type_models.LiteralType(simple=_type_models.SimpleType.STRUCT)
def to_literal(self, ctx, python_val, python_type, expected):
local_file = ctx.file_access.get_random_local_path() + ".json"
with open(local_file, "w") as f:
json.dump(python_val, f)
remote_file = ctx.file_access.get_random_remote_path()
ctx.file_access.upload(local_file, remote_file)
return Literal(scalar=Scalar(blob=Blob(uri=remote_file, metadata=BlobMetadata(type=self.get_literal_type(python_type)))))
def to_python_value(self, ctx, lv, expected_python_type):
local_file = ctx.file_access.get_random_local_path() + ".json"
ctx.file_access.download(lv.scalar.blob.uri, local_file)
with open(local_file, "r") as f:
return json.load(f)
TypeEngine.register(AnyTypeTransformer())
@task
def foo(a: Any) -> int:
if isinstance(a, int):
return a + 1
return 0
@workflow
def wf(a: int) -> int:
return foo(a=a)
@workflow
def wf1(a: Any) -> int:
a1 = foo(a=a)
return foo(a=a1) This code defines a custom type transformer for Would you like more details on any specific part? Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not. Mention @RunLLM to ask follow-up questions. See runllm.com for more! |
Related to #5319. |
This is important to my LangChain Agent PR, so I will take over it. |
NP |
Motivation: Why do you think this is important?
Intuitively,
Any
type should take every possible type as the input. Flytekit today supports using all Python types as the input, but it doesn't supporttask
.workflow
.Goal: What should the final outcome look like, ideally?
The following scenarios should succeed:
Scenario 1:
Scenario 2:
Describe alternatives you've considered
Adding some tags as type hints for the transformer?
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: