Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Prevent local files from being moved when using FlyteFile on local executions #2476

Merged
merged 7 commits into from
Jun 27, 2024

Conversation

ggydush
Copy link
Collaborator

@ggydush ggydush commented Jun 12, 2024

Tracking issue

N/A

Why are the changes needed?

When running local executions, local files that exist should not need to be copied. When running large files, this can become a bottleneck.

What changes were proposed in this pull request?

Check if running a local task execution when converting FlyteFile to literal.

How was this patch tested?

from flytekit import task
from flytekit.types.file import FlyteFile


@task
def example(ff: FlyteFile) -> str:
    downloaded_path = ff.download()
    print(downloaded_path)
    return str(ff.path)


example(ff=__file__)

Before the fix, this printed:

/var/folders/d1/n1tj86t11pj_drgfg7_br5n00000gn/T/flyte-qd4bk3t0/raw/0fee47924014b3cb5f12295a48330c0f/tmp.py

After the fix, this printed:

/Users/ggydush/dev/delve/flytekit/flytekit/types/file/tmp.py

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Signed-off-by: ggydush <greggydush@gmail.com>
Signed-off-by: ggydush <greggydush@gmail.com>
Signed-off-by: ggydush <greggydush@gmail.com>
@ggydush ggydush changed the title Ggydush/fix flytefile local files fix: Prevent local files from being moved when using FlyteFile on local executions Jun 12, 2024
Signed-off-by: ggydush <greggydush@gmail.com>
Copy link
Collaborator

@eapolinario eapolinario left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, @ggydush . This is a great idea IMO. Can you take a look at the test failures?

Signed-off-by: ggydush <greggydush@gmail.com>
@ggydush ggydush requested a review from eapolinario June 12, 2024 18:15
@ggydush
Copy link
Collaborator Author

ggydush commented Jun 12, 2024

Thanks for the PR, @ggydush . This is a great idea IMO. Can you take a look at the test failures?

Yep fixed! Looks like there's still issue with lint/mac builds but doesn't seem related.

Copy link

codecov bot commented Jun 12, 2024

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 50.68%. Comparing base (647b071) to head (9e2510c).
Report is 1 commits behind head on master.

Current head 9e2510c differs from pull request most recent head 078a47d

Please upload reports for the commit 078a47d to get more accurate results.

Files Patch % Lines
flytekit/types/file/file.py 0.00% 4 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (647b071) and HEAD (9e2510c). Click for more details.

HEAD has 14 uploads less than BASE | Flag | BASE (647b071) | HEAD (9e2510c) | |------|------|------| ||15|1|
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #2476       +/-   ##
===========================================
- Coverage   91.30%   50.68%   -40.63%     
===========================================
  Files          78      182      +104     
  Lines        3968    18508    +14540     
  Branches        0     3642     +3642     
===========================================
+ Hits         3623     9380     +5757     
- Misses        345     8650     +8305     
- Partials        0      478      +478     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ggydush and others added 2 commits June 13, 2024 15:55
Signed-off-by: ggydush <greggydush@gmail.com>
…l-files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
@eapolinario eapolinario merged commit bd4c708 into master Jun 27, 2024
45 of 46 checks passed
bgedik pushed a commit to bgedik/flytekit that referenced this pull request Jul 3, 2024
…al executions (flyteorg#2476)

* fix: Do not copy local files when using FlyteFile

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Prevent copying of local files when running local execution

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Revert

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Fix another location of should upload

Signed-off-by: ggydush <greggydush@gmail.com>

* test: Fix failing test cases

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Fix to still handle uploads

Signed-off-by: ggydush <greggydush@gmail.com>

---------

Signed-off-by: ggydush <greggydush@gmail.com>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: bugra.gedik <bugra.gedik@predera.ai>
fiedlerNr9 pushed a commit that referenced this pull request Jul 25, 2024
…al executions (#2476)

* fix: Do not copy local files when using FlyteFile

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Prevent copying of local files when running local execution

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Revert

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Fix another location of should upload

Signed-off-by: ggydush <greggydush@gmail.com>

* test: Fix failing test cases

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Fix to still handle uploads

Signed-off-by: ggydush <greggydush@gmail.com>

---------

Signed-off-by: ggydush <greggydush@gmail.com>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: Jan Fiedler <jan@union.ai>
mao3267 pushed a commit to mao3267/flytekit that referenced this pull request Jul 29, 2024
…al executions (flyteorg#2476)

* fix: Do not copy local files when using FlyteFile

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Prevent copying of local files when running local execution

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Revert

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Fix another location of should upload

Signed-off-by: ggydush <greggydush@gmail.com>

* test: Fix failing test cases

Signed-off-by: ggydush <greggydush@gmail.com>

* fix: Fix to still handle uploads

Signed-off-by: ggydush <greggydush@gmail.com>

---------

Signed-off-by: ggydush <greggydush@gmail.com>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: mao3267 <chenvincent610@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants