-
Notifications
You must be signed in to change notification settings - Fork 297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Prevent local files from being moved when using FlyteFile on local executions #2476
Conversation
Signed-off-by: ggydush <greggydush@gmail.com>
Signed-off-by: ggydush <greggydush@gmail.com>
Signed-off-by: ggydush <greggydush@gmail.com>
Signed-off-by: ggydush <greggydush@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR, @ggydush . This is a great idea IMO. Can you take a look at the test failures?
Signed-off-by: ggydush <greggydush@gmail.com>
Yep fixed! Looks like there's still issue with lint/mac builds but doesn't seem related. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2476 +/- ##
===========================================
- Coverage 91.30% 50.68% -40.63%
===========================================
Files 78 182 +104
Lines 3968 18508 +14540
Branches 0 3642 +3642
===========================================
+ Hits 3623 9380 +5757
- Misses 345 8650 +8305
- Partials 0 478 +478 ☔ View full report in Codecov by Sentry. |
Signed-off-by: ggydush <greggydush@gmail.com>
…l-files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
…al executions (flyteorg#2476) * fix: Do not copy local files when using FlyteFile Signed-off-by: ggydush <greggydush@gmail.com> * fix: Prevent copying of local files when running local execution Signed-off-by: ggydush <greggydush@gmail.com> * fix: Revert Signed-off-by: ggydush <greggydush@gmail.com> * fix: Fix another location of should upload Signed-off-by: ggydush <greggydush@gmail.com> * test: Fix failing test cases Signed-off-by: ggydush <greggydush@gmail.com> * fix: Fix to still handle uploads Signed-off-by: ggydush <greggydush@gmail.com> --------- Signed-off-by: ggydush <greggydush@gmail.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: bugra.gedik <bugra.gedik@predera.ai>
…al executions (#2476) * fix: Do not copy local files when using FlyteFile Signed-off-by: ggydush <greggydush@gmail.com> * fix: Prevent copying of local files when running local execution Signed-off-by: ggydush <greggydush@gmail.com> * fix: Revert Signed-off-by: ggydush <greggydush@gmail.com> * fix: Fix another location of should upload Signed-off-by: ggydush <greggydush@gmail.com> * test: Fix failing test cases Signed-off-by: ggydush <greggydush@gmail.com> * fix: Fix to still handle uploads Signed-off-by: ggydush <greggydush@gmail.com> --------- Signed-off-by: ggydush <greggydush@gmail.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Jan Fiedler <jan@union.ai>
…al executions (flyteorg#2476) * fix: Do not copy local files when using FlyteFile Signed-off-by: ggydush <greggydush@gmail.com> * fix: Prevent copying of local files when running local execution Signed-off-by: ggydush <greggydush@gmail.com> * fix: Revert Signed-off-by: ggydush <greggydush@gmail.com> * fix: Fix another location of should upload Signed-off-by: ggydush <greggydush@gmail.com> * test: Fix failing test cases Signed-off-by: ggydush <greggydush@gmail.com> * fix: Fix to still handle uploads Signed-off-by: ggydush <greggydush@gmail.com> --------- Signed-off-by: ggydush <greggydush@gmail.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: mao3267 <chenvincent610@gmail.com>
Tracking issue
N/A
Why are the changes needed?
When running local executions, local files that exist should not need to be copied. When running large files, this can become a bottleneck.
What changes were proposed in this pull request?
Check if running a local task execution when converting FlyteFile to literal.
How was this patch tested?
Before the fix, this printed:
/var/folders/d1/n1tj86t11pj_drgfg7_br5n00000gn/T/flyte-qd4bk3t0/raw/0fee47924014b3cb5f12295a48330c0f/tmp.py
After the fix, this printed:
/Users/ggydush/dev/delve/flytekit/flytekit/types/file/tmp.py
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link