refactor: Added a generic FileStream (still in active development!)

[edgarrmondragon]

References:

• https://filesystem-spec.readthedocs.io/en/latest/api.html#base-classes

---

📚 Documentation preview 📚: https://meltano-sdk--2654.org.readthedocs.build/en/2654/

[codspeed-hq]

CodSpeed Performance Report

Merging #2654 will not alter performance

_{Comparing 2648-feat-add-a-generic-filestream-interface (4628468) with main (e997deb)}

Summary

✅ 6 untouched benchmarks

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (main@f1e8114). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2654   +/-   ##
=======================================
  Coverage        ?   90.46%           
=======================================
  Files           ?       62           
  Lines           ?     4992           
  Branches        ?      974           
=======================================
  Hits            ?     4516           
  Misses          ?      330           
  Partials        ?      146           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[edgarrmondragon]

This is ready for review. Everything here is subject to change: naming conventions, implementation, abstractions so feel free to comment on those.

[edgarrmondragon]

Suggested change

	# https://github.com/boto/boto3/issues/3889
	# "ignore:datetime\\.datetime\\.utcnow\\(\\) is deprecated:DeprecationWarning:botocore",

[edgarrmondragon]

I suggest reviewing this module in split view. All that's left after the refactor are the get_schema and read_file implementations.

[edgarrmondragon]

I suggest reviewing this module in split view.

CSV-specific settings were added and custom discovery was removed in favor of the default implementation.

[edgarrmondragon]

Using partitions allows us to track state for each individual file in merge mode.

[edgarrmondragon]

These are working now

FTP

{
    "filesystem": "ftp",
    "path": "fixtures/csv",
    "read_mode": "one_stream_per_file",
    "delimiter": "\t",
    "ftp": {
        "host": "127.0.0.1",
        "port": 21,
        "username": "my_ftp_user",
        "password": "my_ftp_password"
    }
}

SFTP

{
    "filesystem": "sftp",
    "path": "fixtures/csv",
    "read_mode": "one_stream_per_file",
    "delimiter": "\t",
    "sftp": {
        "host": "127.0.0.1",
        "port": 2022,
        "username": "my_ftp_user",
        "password": "my_ftp_password"
    }
}