Pre commit file fetcher#13982
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a concrete FileFetcher implementation for the pre_commit ecosystem so Dependabot can discover and download pre-commit configuration files, gated behind the beta-ecosystem experiment flag.
Changes:
- Implement
Dependabot::PreCommit::FileFetcherwith a regex-based config filename matcher, beta-ecosystem gating, andexclude_paths-aware filtering. - Add an RSpec suite for the pre-commit file fetcher covering required file detection, error messaging, fetch behaviour (including missing files and beta disabled), and ecosystem versions.
- Add GitHub API and pre-commit config fixtures to support the new tests.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
pre_commit/lib/dependabot/pre_commit/file_fetcher.rb |
Implements the pre-commit FileFetcher, including config filename pattern, beta gating, and final exclude_paths filtering. |
pre_commit/spec/dependabot/pre_commit/file_fetcher_spec.rb |
Adds tests for required files detection, error messaging, fetch behaviour under various conditions, exclude_paths, and ecosystem_versions. |
pre_commit/spec/fixtures/pre_commit_configs/basic.yaml |
Provides a canonical pre-commit configuration fixture used to validate decoded file content. |
pre_commit/spec/fixtures/github/repo_contents.json |
Mocks GitHub repository contents API response including .pre-commit-config.yaml and a README for list-directory behaviour tests. |
pre_commit/spec/fixtures/github/pre_commit_config.json |
Mocks GitHub file contents API response for .pre-commit-config.yaml, including base64-encoded YAML content. |
| describe ".required_files_in?" do | ||
| subject { described_class.required_files_in?(filenames) } | ||
|
|
||
| context "with a .pre-commit-config.yaml file" do | ||
| let(:filenames) { %w(.pre-commit-config.yaml README.md) } | ||
|
|
||
| it { is_expected.to be(true) } | ||
| end | ||
|
|
||
| context "with a .pre-commit-config.yml file" do | ||
| let(:filenames) { %w(.pre-commit-config.yml README.md) } | ||
|
|
||
| it { is_expected.to be(true) } | ||
| end | ||
|
|
||
| context "with a .pre-commit.yaml file" do | ||
| let(:filenames) { %w(.pre-commit.yaml README.md) } | ||
|
|
||
| it { is_expected.to be(true) } | ||
| end | ||
|
|
||
| context "with a .pre-commit.yml file" do | ||
| let(:filenames) { %w(.pre-commit.yml README.md) } | ||
|
|
||
| it { is_expected.to be(true) } | ||
| end | ||
|
|
||
| context "with uppercase .Pre-Commit-Config.YAML file" do | ||
| let(:filenames) { %w(.Pre-Commit-Config.YAML README.md) } | ||
|
|
||
| it { is_expected.to be(true) } | ||
| end | ||
|
|
||
| context "without a pre-commit config file" do | ||
| let(:filenames) { %w(README.md setup.py) } | ||
|
|
||
| it { is_expected.to be(false) } | ||
| end | ||
| end |
There was a problem hiding this comment.
We exercise all supported pre-commit config filename variations in .required_files_in?, but #fetch_files is only tested with .pre-commit-config.yaml. This means that regressions in config_file_name (e.g., failing to locate .pre-commit.yml or .pre-commit.yaml in repo_contents) would not be caught; consider adding at least one integration-style #fetch_files example that stubs repo_contents with a non--config filename and verifies that the correct file is fetched.
| it "excludes the matching file" do | ||
| expect(file_fetcher_instance.files).to be_empty |
There was a problem hiding this comment.
The expectation in this context appears inconsistent with Dependabot::FileFetchers::Base#files: when fetch_files returns an empty array (as it will when the only manifest is excluded by FileFiltering.should_exclude_path?), #files raises Dependabot::DependencyFileNotFound with "No files found in ..." rather than returning an empty array. To test the exclude_paths behaviour without conflicting with the base class contract, this example should either assert that file_fetcher_instance.files raises DependencyFileNotFound, or call fetch_files directly and assert that it returns an empty array.
| it "excludes the matching file" do | |
| expect(file_fetcher_instance.files).to be_empty | |
| it "raises a DependencyFileNotFound error when all files are excluded" do | |
| expect { file_fetcher_instance.files } | |
| .to raise_error(Dependabot::DependencyFileNotFound) do |error| | |
| expect(error.message).to include("No files found in") | |
| end |
AbhishekBhaskar
left a comment
There was a problem hiding this comment.
LGTM, but copilot's suggestion to add another integration style spec seemed relevant to me.
…cosystems and enhance file fetching tests for various pre-commit config formats
|
Apologies for butting in on an already merged pull request, but I believe pre-commit does not actually support |
|
@MajorTanya No need to apologise, open source is a collaborative effort and input is always welcome! We're supporting both |
* Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats
…14135) * pre-commit scaffold * removing bazel example * Update file_parser_spec to use PreCommit::FileParser * Pre commit file fetcher (#13982) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser (#13991) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser * Add PreCommit ecosystem support with package manager and requirement classes * Set typed: true for spec files to enable type checking * add pre-commit to docker-dev-shell * Change type annotation from strict to strong in multiple files * lint * Add pre commit fixtures to yaml linting ignore list * Fix newline at end of file in .yamllint.yaml for pre_commit_configs ignore rule * Refactor PackageManager and FileParser initialization for improved clarity * lint * Add pre commit update checker functionality (#14019) * add pre commit update checker functionality * fix lint errors * add more specs * fix lint errors and failing specs * add test for package details fetcher and latest version finder * fix lint errors * fix lint errors * Add pre commit file updater functionality (#14020) * add pre commit file updater * fix same version in different repo issue * fix lint errors * update specs * remove unecessary comments in code * Fix no files changed error (#14053) * fix no files changed error * add spec for no files changed error * add additional deps architecture * add additional deps architecture * add additional deps architecture * add additional support for python * Removing additional depenency parsering, and using the python ecosystem's parsers * Implement base architecture for additional dependencies support for ecosystems (#14115) * add additional deps architecture * add additional deps architecture * add additional deps architecture * Add additional dependencies support for python (#14117) * add additional deps architecture * add additional deps architecture * add additional deps architecture * add additional support for python * fix sorbet and lint erros * restore updater gemfile and lockfile * call python requirements updater for update process * call python requirements updater for update process * Update pre_commit/lib/dependabot/pre_commit/additional_dependency_checkers/python.rb Co-authored-by: Rob Aiken <6567647+robaiken@users.noreply.github.com> * fix merge conflicts --------- Co-authored-by: Rob Aiken <6567647+robaiken@users.noreply.github.com>
* pre-commit scaffold * removing bazel example * Update file_parser_spec to use PreCommit::FileParser * Pre commit file fetcher (#13982) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser (#13991) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser * Add PreCommit ecosystem support with package manager and requirement classes * Set typed: true for spec files to enable type checking * add pre-commit to docker-dev-shell * Change type annotation from strict to strong in multiple files * lint * Add pre commit fixtures to yaml linting ignore list * Fix newline at end of file in .yamllint.yaml for pre_commit_configs ignore rule * Refactor PackageManager and FileParser initialization for improved clarity * lint * Add pre commit update checker functionality (#14019) * add pre commit update checker functionality * fix lint errors * add more specs * fix lint errors and failing specs * add test for package details fetcher and latest version finder * fix lint errors * fix lint errors * Add pre commit file updater functionality (#14020) * add pre commit file updater * fix same version in different repo issue * fix lint errors * update specs * remove unecessary comments in code * Fix no files changed error (#14053) * fix no files changed error * add spec for no files changed error * add additional dependencies support for go modules * remove comments and fix lint erros * fix sorbet errors * remove comments * fix merge conflicts --------- Co-authored-by: Rob Aiken <6567647+robaiken@users.noreply.github.com>
* pre-commit scaffold * removing bazel example * Update file_parser_spec to use PreCommit::FileParser * Pre commit file fetcher (#13982) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser (#13991) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser * Add PreCommit ecosystem support with package manager and requirement classes * Set typed: true for spec files to enable type checking * add pre-commit to docker-dev-shell * Change type annotation from strict to strong in multiple files * lint * Add pre commit fixtures to yaml linting ignore list * Fix newline at end of file in .yamllint.yaml for pre_commit_configs ignore rule * Refactor PackageManager and FileParser initialization for improved clarity * lint * Add pre commit update checker functionality (#14019) * add pre commit update checker functionality * fix lint errors * add more specs * fix lint errors and failing specs * add test for package details fetcher and latest version finder * fix lint errors * fix lint errors * Add pre commit file updater functionality (#14020) * add pre commit file updater * fix same version in different repo issue * fix lint errors * update specs * remove unecessary comments in code * Fix no files changed error (#14053) * fix no files changed error * add spec for no files changed error * add additional deps support for ruby * fix merge conflicts * revert removed comment --------- Co-authored-by: Rob Aiken <6567647+robaiken@users.noreply.github.com>
* Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats
* pre-commit scaffold * removing bazel example * Update file_parser_spec to use PreCommit::FileParser * Pre commit file fetcher (#13982) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser (#13991) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser * Add PreCommit ecosystem support with package manager and requirement classes * Set typed: true for spec files to enable type checking * add pre-commit to docker-dev-shell * Change type annotation from strict to strong in multiple files * lint * Add pre commit fixtures to yaml linting ignore list * Fix newline at end of file in .yamllint.yaml for pre_commit_configs ignore rule * Refactor PackageManager and FileParser initialization for improved clarity * lint * Add pre commit update checker functionality (#14019) * add pre commit update checker functionality * fix lint errors * add more specs * fix lint errors and failing specs * add test for package details fetcher and latest version finder * fix lint errors * fix lint errors * Add pre commit file updater functionality (#14020) * add pre commit file updater * fix same version in different repo issue * fix lint errors * update specs * remove unecessary comments in code * Fix no files changed error (#14053) * fix no files changed error * add spec for no files changed error * add additional deps support for ruby * fix merge conflicts * revert removed comment --------- Co-authored-by: Rob Aiken <6567647+robaiken@users.noreply.github.com>
* pre-commit scaffold * removing bazel example * Update file_parser_spec to use PreCommit::FileParser * Pre commit file fetcher (#13982) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser (#13991) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser * Add PreCommit ecosystem support with package manager and requirement classes * Set typed: true for spec files to enable type checking * add pre-commit to docker-dev-shell * Change type annotation from strict to strong in multiple files * lint * Add pre commit fixtures to yaml linting ignore list * Fix newline at end of file in .yamllint.yaml for pre_commit_configs ignore rule * Refactor PackageManager and FileParser initialization for improved clarity * lint * Add pre commit update checker functionality (#14019) * add pre commit update checker functionality * fix lint errors * add more specs * fix lint errors and failing specs * add test for package details fetcher and latest version finder * fix lint errors * fix lint errors * Add pre commit file updater functionality (#14020) * add pre commit file updater * fix same version in different repo issue * fix lint errors * update specs * remove unecessary comments in code * Fix no files changed error (#14053) * fix no files changed error * add spec for no files changed error * pre-commit scaffold * removing bazel example * Update file_parser_spec to use PreCommit::FileParser * Pre commit file fetcher (#13982) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser (#13991) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser * Add PreCommit ecosystem support with package manager and requirement classes * Set typed: true for spec files to enable type checking * add pre-commit to docker-dev-shell * Change type annotation from strict to strong in multiple files * lint * Add pre commit fixtures to yaml linting ignore list * Fix newline at end of file in .yamllint.yaml for pre_commit_configs ignore rule * Refactor PackageManager and FileParser initialization for improved clarity * lint * Add pre commit update checker functionality (#14019) * add pre commit update checker functionality * fix lint errors * add more specs * fix lint errors and failing specs * add test for package details fetcher and latest version finder * fix lint errors * fix lint errors * Add pre commit file updater functionality (#14020) * add pre commit file updater * fix same version in different repo issue * fix lint errors * update specs * remove unecessary comments in code * Fix no files changed error (#14053) * fix no files changed error * add spec for no files changed error * Implement base architecture for additional dependencies support for ecosystems (#14115) * add additional deps architecture * add additional deps architecture * add additional deps architecture * Add additional dependencies support for python (#14117) * add additional deps architecture * add additional deps architecture * add additional deps architecture * add additional support for python * fix sorbet and lint erros * Refactor Ecosystem Parsing in Pre-Commit (#14123) * Removing additional depenency parsering, and using the python ecosystem's parsers * reverting gem * Refactor lower bound operator handling in version extraction * Fix edge case failures in pre-commit additional dependencies support (#14135) * pre-commit scaffold * removing bazel example * Update file_parser_spec to use PreCommit::FileParser * Pre commit file fetcher (#13982) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser (#13991) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser * Add PreCommit ecosystem support with package manager and requirement classes * Set typed: true for spec files to enable type checking * add pre-commit to docker-dev-shell * Change type annotation from strict to strong in multiple files * lint * Add pre commit fixtures to yaml linting ignore list * Fix newline at end of file in .yamllint.yaml for pre_commit_configs ignore rule * Refactor PackageManager and FileParser initialization for improved clarity * lint * Add pre commit update checker functionality (#14019) * add pre commit update checker functionality * fix lint errors * add more specs * fix lint errors and failing specs * add test for package details fetcher and latest version finder * fix lint errors * fix lint errors * Add pre commit file updater functionality (#14020) * add pre commit file updater * fix same version in different repo issue * fix lint errors * update specs * remove unecessary comments in code * Fix no files changed error (#14053) * fix no files changed error * add spec for no files changed error * add additional deps architecture * add additional deps architecture * add additional deps architecture * add additional support for python * Removing additional depenency parsering, and using the python ecosystem's parsers * Implement base architecture for additional dependencies support for ecosystems (#14115) * add additional deps architecture * add additional deps architecture * add additional deps architecture * Add additional dependencies support for python (#14117) * add additional deps architecture * add additional deps architecture * add additional deps architecture * add additional support for python * fix sorbet and lint erros * restore updater gemfile and lockfile * call python requirements updater for update process * call python requirements updater for update process * Update pre_commit/lib/dependabot/pre_commit/additional_dependency_checkers/python.rb Co-authored-by: Rob Aiken <6567647+robaiken@users.noreply.github.com> * fix merge conflicts --------- Co-authored-by: Rob Aiken <6567647+robaiken@users.noreply.github.com> * Add support for Node.js additional dependencies (#14138) * feat: add support for Node.js additional dependencies in pre-commit hooks - Implemented `parse_dep_string` method in `Dependabot::NpmAndYarn::Requirement` to parse package names and versions. - Created `Node` class in `Dependabot::PreCommit::AdditionalDependencyCheckers` to handle Node.js additional dependencies. - Updated `FileParser` to recognize Node.js additional dependencies and parse them correctly. - Added integration tests for Node.js additional dependencies in pre-commit hooks. - Updated Dockerfile and gemspec to include necessary dependencies for Node.js support. - Created fixtures for testing Node.js additional dependencies in pre-commit configurations. * replace StanderdError with specific errors * Add support for Rust additional dependencies (#14147) * Add Rust support for additional dependencies in pre-commit hooks * Add Rust support for additional dependencies in pre-commit hooks * adding back comments * Fixing specs * Add additional dependencies support for go modules (#14144) * pre-commit scaffold * removing bazel example * Update file_parser_spec to use PreCommit::FileParser * Pre commit file fetcher (#13982) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser (#13991) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser * Add PreCommit ecosystem support with package manager and requirement classes * Set typed: true for spec files to enable type checking * add pre-commit to docker-dev-shell * Change type annotation from strict to strong in multiple files * lint * Add pre commit fixtures to yaml linting ignore list * Fix newline at end of file in .yamllint.yaml for pre_commit_configs ignore rule * Refactor PackageManager and FileParser initialization for improved clarity * lint * Add pre commit update checker functionality (#14019) * add pre commit update checker functionality * fix lint errors * add more specs * fix lint errors and failing specs * add test for package details fetcher and latest version finder * fix lint errors * fix lint errors * Add pre commit file updater functionality (#14020) * add pre commit file updater * fix same version in different repo issue * fix lint errors * update specs * remove unecessary comments in code * Fix no files changed error (#14053) * fix no files changed error * add spec for no files changed error * add additional dependencies support for go modules * remove comments and fix lint erros * fix sorbet errors * remove comments * fix merge conflicts --------- Co-authored-by: Rob Aiken <6567647+robaiken@users.noreply.github.com> * Add additional dependencies support for Ruby (#14160) * pre-commit scaffold * removing bazel example * Update file_parser_spec to use PreCommit::FileParser * Pre commit file fetcher (#13982) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser (#13991) * Implement pre-commit file fetching * Add exclude path * Refactor file fetcher specs to use experiment registration for beta ecosystems and enhance file fetching tests for various pre-commit config formats * Implement PreCommit file parser * Add PreCommit ecosystem support with package manager and requirement classes * Set typed: true for spec files to enable type checking * add pre-commit to docker-dev-shell * Change type annotation from strict to strong in multiple files * lint * Add pre commit fixtures to yaml linting ignore list * Fix newline at end of file in .yamllint.yaml for pre_commit_configs ignore rule * Refactor PackageManager and FileParser initialization for improved clarity * lint * Add pre commit update checker functionality (#14019) * add pre commit update checker functionality * fix lint errors * add more specs * fix lint errors and failing specs * add test for package details fetcher and latest version finder * fix lint errors * fix lint errors * Add pre commit file updater functionality (#14020) * add pre commit file updater * fix same version in different repo issue * fix lint errors * update specs * remove unecessary comments in code * Fix no files changed error (#14053) * fix no files changed error * add spec for no files changed error * add additional deps support for ruby * fix merge conflicts * revert removed comment --------- Co-authored-by: Rob Aiken <6567647+robaiken@users.noreply.github.com> * Adding pre_commit gem in gem * Bump dependabot-pre_commit and related dependencies to version 0.361.1 * Add dependabot-pre_commit gem and update Gemfile.lock * Add pre_commit ecosystem to Dockerfile for source management * Add pre_commit volume mappings to devcontainer configuration * revert incorrect gem lock file changes * add dependabot-bundler in gemspec * add dependabot-bundler in gemspec * add comment in dockerfile updater core to invalidate cache from ghcr registry * fix lint error * add arg in dockerfile updater core to force cache invalidation * add pre_commit dependency to setup * fix order of COPY commands in Dockerfile for consistency * remove pre_commit volume mapping from dependabot script * revert updater gemfile and lockfile changes b1546fe * revert changes in updater setup file --------- Co-authored-by: Abhishek <abhishekbhaskar@github.com>
What are you trying to accomplish?
Add the
FileFetcherclass for the pre-commit ecosystem, enabling Dependabot to fetch pre-commit configuration files from repositories.Anything you want to highlight for special attention from reviewers?
CONFIG_FILE_PATTERNregex (/\.pre-commit(-config)?\.ya?ml$/i) matches all valid pre-commit config file variationsALLOW_BETA_ECOSYSTEMS=trueto be enabledHow will you know you've accomplished your goal?
We can fetch pre-commit files
Checklist