Input through file and pipe

[mao3267]

Tracking issue

Related to flyteorg/flyte#5365

Why are the changes needed?

1. We want to input multiple parameters through a single JSON/YAML file. eg. pyflyte run wf.py --inputs-file input.json
2. We want to support input from pipe as well. eg. cat input.json | pyflyte run wf.py wf

What changes were proposed in this pull request?

1. Add a new option --inputs-file to specify the input file.
2. Read from file or pipe and transform into click options
3. Add unit test (support all types, replace the input from file/pipe if there's an option specify the same parameter)

How was this patch tested?

Currently, two unit tests are performed on this feature. test_all_types_with_file_input and test_replace_file_inputs are added in tests/flytekit/unit/cli/pyflyte/test_run.py

Missing value in custom class

wf.py

from flytekit import task, workflow
from dataclasses import dataclass
import typing
from dataclasses_json import DataClassJsonMixin


@dataclass
class myclass(DataClassJsonMixin):
    i: int
    j: typing.List[int]


@task
def t1(a: myclass) -> myclass:
    return a


@workflow
def wf(a: myclass) -> myclass:
    return t1(a=a)

input.json

{
    "a": {
        "j": [1, 2, 3]
    }
}

Output:

Missing input parameter

wf.py

from flytekit import task, workflow

@task
def t1(a: int, b: int) -> int:
    return a + b

@workflow
def wf(a: int, b: int) -> int:
    return t1(a=a, b=b)

input.json

{
    "a": 1,
}

Output:

Help message

More tests can be added if needed.

Setup process

git clone https://github.com/mao3267/flytekit.git
git checkout input-through-file-and-pipe
make setup && pip install -e .

Screenshots

None

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed. (test_run.py)
• All commits are signed-off.

Related PRs

#2583

Docs link

ToDo

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[kumare3]

@mao3267 can you please DCO sign-off, This is awesome would love to get it merged

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.76%. Comparing base (69445ff) to head (ff92807).
Report is 11 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2552      +/-   ##
==========================================
+ Coverage   79.24%   83.76%   +4.52%     
==========================================
  Files         196        3     -193     
  Lines       19785      154   -19631     
  Branches     4008        0    -4008     
==========================================
- Hits        15678      129   -15549     
+ Misses       3407       25    -3382     
+ Partials      700        0     -700     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[welcome]

Congrats on merging your first pull request! 🎉

[mao3267]

Thanks for the help from @pingsutw, @kumare3 and other reviewers! Glad to see it merged.

[kumare3]

Can we add docs for this - this is a very good and important feature for folks experimenting continuously

[pingsutw]

@mao3267 yeah, thanks for the contribution. Could you help update this doc too?