connector(filesystem): Refactor and bundle readers and implementations

[amotl]

About

Group all ingredients related to filesystem format readers and filesystem implementations. Continue to discriminate between "local" and "remote" filesystems.

References

• Discuss and resolve ambiguities with file and table formats #106 (comment)
• connector(filesystem): Refactor and bundle blobstorage, sftp, and utils #156

[amotl]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[codecov]

Codecov Report

❌ Patch coverage is 91.17647% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.93%. Comparing base (d5b181d) to head (438ffa9).

Files with missing lines	Patch %	Lines
omniload/source/filesystem/impl/core.py	68.75%	5 Missing ⚠️
omniload/source/filesystem/impl/remote.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #160      +/-   ##
==========================================
+ Coverage   51.89%   51.93%   +0.03%     
==========================================
  Files         193      197       +4     
  Lines        9139     9145       +6     
==========================================
+ Hits         4743     4749       +6     
  Misses       4396     4396              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderabbitai]

Walkthrough

Filesystem source code is reorganized: reader-resolution logic and path utilities move from adapter.py, model.py, and router.py into new format/registry.py, format/settings.py, impl/core.py, and impl/util.py modules. Import paths across api.py, local.py, remote.py, and tests are updated accordingly.

Changes

Filesystem module restructuring

Layer / File(s)	Summary
Format registry and settings extraction omniload/source/filesystem/format/registry.py, omniload/source/filesystem/format/settings.py, omniload/source/filesystem/format/helpers.py, omniload/source/filesystem/format/readers.py, omniload/source/filesystem/router.py, omniload/source/filesystem/model.py	Adds FORMAT_TO_READER, reader_for_format, supported_file_format_message and DEFAULT_CHUNK_SIZE in dedicated modules; removes duplicated logic from router.py and model.py; updates helpers.py/readers.py imports accordingly.
Path normalization utilities omniload/source/filesystem/impl/util.py	Adds Windows drive detection, file:// URL path conversion, absolute-path detection, and glob-aware directory/glob splitting helpers.
resource_for_reader relocation omniload/source/filesystem/impl/core.py, omniload/source/filesystem/adapter.py	Adds resource_for_reader (with read_csv_headless special-casing) in impl/core.py; removes the old transformer aliases and helper from adapter.py.
Wiring into local/remote sources omniload/source/filesystem/api.py, omniload/source/filesystem/impl/local.py, omniload/source/filesystem/impl/remote.py	Updates imports to absolute impl.* paths and switches resource_for_reader/path-splitting calls to the new modules.
Test updates tests/main/filesystem/test_local.py, tests/main/filesystem/test_remote.py	Updates imports and mock patch targets to reference the relocated modules.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LocalFilesystemSource
  participant GCSSource
  participant core as impl.core.resource_for_reader
  participant readers as filesystem.readers

  LocalFilesystemSource->>core: resource_for_reader(bucket_url, credentials, file_glob, reader_name, column_types)
  GCSSource->>core: resource_for_reader(bucket_url, credentials, file_glob, reader_name, column_types)
  alt reader_name == read_csv_headless
    core->>core: build read_csv_headless_with_cols transformer
  else other reader_name
    core->>readers: readers(bucket_url, credentials, file_glob).with_resources(reader_name)
  end
  core-->>LocalFilesystemSource: configured resource

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately summarizes the main refactor: bundling filesystem readers and implementation code.
Description check	✅ Passed	The description matches the PR’s filesystem refactor and the local/remote split.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch refactor-filesystem-readers

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}