Imgspec/copy auto

[wild-endeavor]

Why are the changes needed?

We introduced a new switch, --copy, to the pyflyte command recently to help standardize control of the fast register experience. This switch also changed the behavior slightly for discovering and copying the files into the fast register tar file. This PR changes the ImageSpec build process to also rely on that new file discovery and copying behavior, and updates the default behavior to copy only loaded Python modules rather than all files. This make all file finding/copying the same across flytekit.

What changes were proposed in this pull request?

• Added a new option called copy to the ImageSpec object to mirror the option on the switches. This is the first time ImageSpec has a non-primitive option but I think this is okay.
• Moved the enum that back this option to its own package - because it's used by by both the tools and core side it was causing too many circular imports.
• Behavior of ImageSpec when copying is needed is now to copy 'auto' (only loaded Python modules). Before it was copying all files.
• Move shared logic in containertask/pythonautocontainertask into a helper function.

Couple notes:

• The tag and id properties of ImageSpec rely on the fields source_root and now copy to compute the tag and id correctly. This is fine but because these values are mutated by the update_image_spec_copy_handling (this was happening before this pr also), this means that when a user calls these properties will affect their return values. We can make update_image_spec_copy_handling not mutate, but that doesn't work since tag/id need the correct values for source_root and copy in order to correctly compute their values. Basically the question is how to make sure serialization settings has been taken into account before these two properties are called.
• Noticed in multiple places that deref_symlinks and any ignore settings are not piped all the way through. We should clean this up as a follow-up.

How was this patch tested?

Unit tests, tested locally with sandbox, confirmed behavior before and after. Tested both default and envd builders. ls files inside image, ran a test workflow, etc.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

Attention: Patch coverage is 33.96226% with 35 lines in your changes missing coverage. Please review.

Project coverage is 49.75%. Comparing base (a9c9c46) to head (3dd1b32).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
flytekit/core/python_auto_container.py	14.28%	11 Missing and 1 partial ⚠️
flytekit/image_spec/default_builder.py	16.66%	10 Missing ⚠️
flytekit/image_spec/image_spec.py	27.27%	8 Missing ⚠️
flytekit/tools/script_mode.py	25.00%	3 Missing ⚠️
flytekit/core/container_task.py	33.33%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (a9c9c46) and HEAD (3dd1b32). Click for more details.

HEAD has 7 uploads less than BASE

Flag	BASE (a9c9c46)	HEAD (3dd1b32)
	9	2

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2731       +/-   ##
===========================================
- Coverage   78.57%   49.75%   -28.83%     
===========================================
  Files         193      194        +1     
  Lines       19672    19718       +46     
  Branches     4100     4109        +9     
===========================================
- Hits        15458     9811     -5647     
- Misses       3492     9379     +5887     
+ Partials      722      528      -194     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pingsutw]

a.txt:auto. is there a plan to support this?
it looks nicer honestly

[wild-endeavor]

separate pr?