Skip to content

Report all validation errors instead of stopping at first failure#107

Open
eshaffer321 wants to merge 2 commits intoRoadieHQ:mainfrom
eshaffer321:upstream-pr
Open

Report all validation errors instead of stopping at first failure#107
eshaffer321 wants to merge 2 commits intoRoadieHQ:mainfrom
eshaffer321:upstream-pr

Conversation

@eshaffer321
Copy link
Copy Markdown

@eshaffer321 eshaffer321 commented Oct 22, 2025

what's this about

Right now if you validate a bunch of files and one fails, you have to fix it and run again to find the next error. Rinse and repeat until all issues are found. Kind of annoying in CI.

This PR makes the validator check all files before exiting, so you see every problem in one go.

Before:

$ validate-entity file1.yaml file2.yaml file3.yaml
Failed to validate file1.yaml: Invalid schema
# stops here, never checks the other files

After:

$ validate-entity file1.yaml file2.yaml file3.yaml

============================================================
Validation Summary:
  Total files: 3
  ✓ Passed: 1
  ✗ Failed: 2
============================================================

Failed files:
  ✗ file1.yaml
    Invalid schema
  ✗ file3.yaml
    Missing required field

what changed

  • validate() now collects errors and keeps going instead of bailing early
  • added a summary at the end (total/passed/failed)
  • verbose mode shows the full error list
  • quiet mode still shows inline errors plus the summary
  • exit codes stay the same (0 = all good, 1 = something failed)
  • github actions mode calls setFailed once with all the errors

testing

added tests for multi-file validation, summary output, and verbose/quiet modes. all existing tests still pass.

backwards compat

exit codes unchanged. output is additive - you just get more info now.

eshaffer321 pushed a commit to eshaffer321/backstage-entity-validator that referenced this pull request Oct 22, 2025
Changes validator behavior from fail-fast to comprehensive error reporting.

Previous behavior:
- Stopped validation at first error
- Only showed one error at a time
- Required multiple runs to see all issues

New behavior:
- Validates ALL files before exiting
- Collects and reports ALL errors
- Shows summary statistics (Total/Passed/Failed)
- Exit code 1 if any failures, 0 if all pass
- Verbose mode: detailed error list for each failure
- Quiet mode: errors as they occur + summary

Benefits:
- Improves CI/CD efficiency (see all errors in one run)
- Better developer experience (fix all issues at once)
- Consistent with modern validators (ESLint, yamllint, etc.)

Testing:
- Added comprehensive test suite using TDD
- 5 tests covering all scenarios
- Verified with real sample files
- Backward compatible (exit codes unchanged)

Implementation:
- src/index.js: Error collection + summary output
- src/index.test.js: Complete test coverage
- dist/: Rebuilt with ncc
@punkle
Copy link
Copy Markdown
Contributor

punkle commented Jan 15, 2026

@eshaffer321 this is a great idea. It looks like my recent updates/changes have caused some conflicts unfortunately.

Merged latest upstream changes with the validate-all-files feature.
Updated tests to use upstream's improved mocking structure while
preserving the new behavior that validates all files instead of
stopping at first failure.
@eshaffer321
Copy link
Copy Markdown
Author

@punkle fixed the merge conflicts

@punkle
Copy link
Copy Markdown
Contributor

punkle commented Jan 22, 2026

@eshaffer321 I believe we need to increment the version in the package.json in order for it to be properly released when it is merged. Could you do that. I think we can call this minor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants