Add an error for file size exceeded to prevent system retries

[wild-endeavor]

Tracking issue

See the original slack thread.

Why are the changes needed?

Even when upload of a file fails because the file is too large, we retry many times. This is not needed.

What changes were proposed in this pull request?

• Add a new error type and test for it, returning a not IsRecoverable error.

How was this patch tested?

Tested locally with the following workflow

from flytekit import task, workflow


@task
def print_arrays(arr1: str) -> None:
    print(f"Array 1: {arr1}")


@task  # (container_image=normal_image)
def increase_size_of_of_arrays(n: int) -> str:
    arr1 = "a" * n * 1024
    return arr1


@workflow
def simple_pipeline(n: int) -> int:
    arr1 = increase_size_of_of_arrays(n=n)
    print_arrays(arr1)
    return 2

Setup process

Screenshots

Before

After

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Tangentially related to this PR: flyteorg/flytekit#2700, which solves a similar issue but from within the container.

Docs link

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 50.77%. Comparing base (a058fd1) to head (4bbae21).
Report is 43 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #5725       +/-   ##
===========================================
+ Coverage   36.21%   50.77%   +14.56%     
===========================================
  Files        1303     1158      -145     
  Lines      109560    73757    -35803     
===========================================
- Hits        39675    37451     -2224     
+ Misses      65765    32320    -33445     
+ Partials     4120     3986      -134     

Flag	Coverage Δ
unittests-datacatalog	51.37% <ø> (ø)
unittests-flyteadmin	55.57% <ø> (-0.07%)	⬇️
unittests-flytecopilot	12.17% <ø> (ø)
unittests-flytectl	62.21% <ø> (+0.04%)	⬆️
unittests-flyteidl	?
unittests-flyteplugins	53.45% <100.00%> (+0.10%)	⬆️
unittests-flytepropeller	42.04% <100.00%> (+0.28%)	⬆️
unittests-flytestdlib	55.37% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.