Repo upgrade#112
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR modernizes the repository’s testing, documentation, and compliance tooling alignment by migrating the test suite to xUnit v3 and introducing a new verification-documentation set that maps requirements to test evidence, while also restructuring CI document outputs into generated/ and a consolidated docs/generated/ artifact folder.
Changes:
- Migrated unit/integration tests from MSTest to xUnit v3 (attributes + assertion updates, test initialization updates).
- Added a new
docs/verification/document collection (definition, title, and per-system/subsystem/unit/OTS verification pages). - Updated CI/document tooling to write build outputs into
generated/folders and publish final PDFs intodocs/generated/, plus updated standards/config to ignore generated outputs.
Reviewed changes
Copilot reviewed 90 out of 94 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/DemaConsulting.SarifMark.Tests/Utilities/UtilitiesTests.cs | Migrates Utilities subsystem tests from MSTest to xUnit. |
| test/DemaConsulting.SarifMark.Tests/Utilities/PathHelpersTests.cs | Migrates PathHelpers unit tests from MSTest to xUnit. |
| test/DemaConsulting.SarifMark.Tests/SelfTest/ValidationTests.cs | Migrates Validation unit tests from MSTest to xUnit and adjusts assertions. |
| test/DemaConsulting.SarifMark.Tests/SelfTest/SelfTestTests.cs | Migrates SelfTest subsystem tests from MSTest to xUnit. |
| test/DemaConsulting.SarifMark.Tests/Sarif/SarifTests.cs | Migrates SARIF subsystem tests to xUnit and renames scenarios. |
| test/DemaConsulting.SarifMark.Tests/Sarif/SarifRunTests.cs | Migrates SarifRun unit tests from MSTest to xUnit. |
| test/DemaConsulting.SarifMark.Tests/Sarif/SarifFindingTests.cs | Migrates SarifFinding unit tests from MSTest to xUnit. |
| test/DemaConsulting.SarifMark.Tests/ProgramTests.cs | Migrates Program unit tests to xUnit and adds coverage for --silent. |
| test/DemaConsulting.SarifMark.Tests/IntegrationTests.cs | Migrates end-to-end integration tests to xUnit and updates initialization pattern. |
| test/DemaConsulting.SarifMark.Tests/DemaConsulting.SarifMark.Tests.csproj | Switches test framework packages to xUnit v3 and updates implicit usings. |
| test/DemaConsulting.SarifMark.Tests/Cli/CliTests.cs | Migrates CLI subsystem tests to xUnit and adds depth/log behavior coverage. |
| test/DemaConsulting.SarifMark.Tests/AssemblyInfo.cs | Updates assembly-level parallelization settings for xUnit. |
| src/DemaConsulting.SarifMark/Program.cs | Adds documentation/remarks and tightens intended internal API surface. |
| src/DemaConsulting.SarifMark/Cli/Context.cs | Improves XML doc exception/remarks detail for parsing/log/error behavior. |
| requirements.yaml | Updates ReqStream include paths and switches OTS test framework requirement to xUnit v3. |
| docs/verification/title.txt | Adds verification document metadata front-matter for Pandoc. |
| docs/verification/sarifmark/utilities/path-helpers.md | Adds unit-level verification evidence narrative for PathHelpers. |
| docs/verification/sarifmark/utilities.md | Adds subsystem-level verification narrative for Utilities. |
| docs/verification/sarifmark/self-test/validation.md | Adds unit-level verification narrative for Validation. |
| docs/verification/sarifmark/self-test.md | Adds subsystem-level verification narrative for SelfTest. |
| docs/verification/sarifmark/sarif/sarif-run.md | Adds unit-level verification narrative for SarifRun. |
| docs/verification/sarifmark/sarif/sarif-results.md | Adds unit-level verification narrative for SarifResults. |
| docs/verification/sarifmark/sarif/sarif-finding.md | Adds unit-level verification narrative for SarifFinding. |
| docs/verification/sarifmark/sarif.md | Adds subsystem-level verification narrative for Sarif. |
| docs/verification/sarifmark/program.md | Adds unit-level verification narrative for Program. |
| docs/verification/sarifmark/cli/context.md | Adds unit-level verification narrative for Context. |
| docs/verification/sarifmark/cli.md | Adds subsystem-level verification narrative for CLI. |
| docs/verification/sarifmark.md | Adds system-level verification narrative for SarifMark. |
| docs/verification/ots/xunit-v3.md | Adds OTS verification narrative for xUnit v3. |
| docs/verification/ots/weasyprint.md | Updates/introduces OTS verification narrative for WeasyPrint. |
| docs/verification/ots/versionmark.md | Updates/introduces OTS verification narrative for VersionMark. |
| docs/verification/ots/sonarmark.md | Updates/introduces OTS verification narrative for SonarMark. |
| docs/verification/ots/reviewmark.md | Updates/introduces OTS verification narrative for ReviewMark. |
| docs/verification/ots/reqstream.md | Updates/introduces OTS verification narrative for ReqStream. |
| docs/verification/ots/pandoc.md | Updates/introduces OTS verification narrative for Pandoc. |
| docs/verification/ots/fileassert.md | Updates/introduces OTS verification narrative for FileAssert. |
| docs/verification/ots/buildmark.md | Updates/introduces OTS verification narrative for BuildMark. |
| docs/verification/ots.md | Adds top-level OTS verification section wrapper. |
| docs/verification/introduction.md | Adds verification collection introduction and navigation guidance. |
| docs/verification/definition.yaml | Adds Pandoc build definition for the verification document. |
| docs/requirements_report/definition.yaml | Moves trace matrix input to generated/ output path. |
| docs/requirements_doc/definition.yaml | Moves requirements doc input to generated/ output path. |
| docs/reqstream/sarifmark/utilities.yaml | Adds Utilities subsystem requirements file in new ReqStream structure. |
| docs/reqstream/sarifmark/self-test.yaml | Adds SelfTest subsystem requirements file in new ReqStream structure. |
| docs/reqstream/sarifmark/sarif.yaml | Adds SARIF subsystem requirements file in new ReqStream structure. |
| docs/reqstream/sarifmark/program.yaml | Updates Program requirement wording for validate behavior. |
| docs/reqstream/sarifmark/cli/context.yaml | Adjusts requirement wording and ensures WriteError has explicit tests mapping. |
| docs/reqstream/sarifmark/cli.yaml | Adds CLI subsystem requirements file in new ReqStream structure. |
| docs/reqstream/sarifmark.yaml | Updates system requirements with children + adds new system requirements. |
| docs/reqstream/ots/xunit-v3.yaml | Adds xUnit v3 OTS requirements (replacing MSTest OTS). |
| docs/reqstream/ots/mstest.yaml | Removes MSTest OTS requirements (replaced by xUnit v3). |
| docs/design/sarifmark/utilities/path-helpers.md | Updates design headings/structure to comply with heading depth rules. |
| docs/design/sarifmark/utilities.md | Updates design headings/structure to comply with heading depth rules. |
| docs/design/sarifmark/self-test/validation.md | Updates design headings/structure to comply with heading depth rules. |
| docs/design/sarifmark/self-test.md | Updates design headings/structure and cross-reference style. |
| docs/design/sarifmark/sarif/sarif-run.md | Updates design headings/structure to comply with heading depth rules. |
| docs/design/sarifmark/sarif/sarif-results.md | Updates design headings/structure to comply with heading depth rules. |
| docs/design/sarifmark/sarif/sarif-finding.md | Updates design headings/structure to comply with heading depth rules. |
| docs/design/sarifmark/sarif.md | Updates design headings/structure to comply with heading depth rules. |
| docs/design/sarifmark/program.md | Updates design headings/structure and clarifies requirement satisfaction mapping. |
| docs/design/sarifmark/cli/context.md | Updates design headings/structure to comply with heading depth rules. |
| docs/design/sarifmark/cli.md | Updates design headings/structure to comply with heading depth rules. |
| docs/design/sarifmark.md | Updates system design doc references/structure and adds external dependencies section. |
| docs/design/introduction.md | Updates introduction content and adds companion artifact structure section. |
| docs/design/definition.yaml | Updates Pandoc input list to new design doc paths. |
| docs/code_review_report/definition.yaml | Moves ReviewMark report input to generated/. |
| docs/code_review_plan/definition.yaml | Moves ReviewMark plan input to generated/. |
| docs/code_quality/definition.yaml | Moves code quality report inputs to generated/. |
| docs/build_notes/definition.yaml | Moves build notes inputs to generated/. |
| CONTRIBUTING.md | Updates documented test framework and assertion guidance to xUnit v3. |
| AGENTS.md | Updates structure matrix, adds verification docs, and enforces “no generated file access”. |
| .yamllint.yaml | Updates ignore patterns and adds generated/ exclusion. |
| .reviewmark.yaml | Updates review sets to include verification docs and new requirements/design paths. |
| .markdownlint-cli2.yaml | Ignores generated/ folders. |
| .github/workflows/build.yaml | Writes all document outputs into generated/ and publishes PDFs/outputs into docs/generated/. |
| .github/standards/verification-documentation.md | Adds a new standard defining required structure for verification docs. |
| .github/standards/technical-documentation.md | Updates Pandoc layout rules, heading depth rule, and generated-folder handling. |
| .github/standards/software-items.md | Adds verification design as a first-class artifact type and expands OTS guidance. |
| .github/standards/reviewmark-usage.md | Updates ReviewMark guidance to the new generated paths and includes verification sets. |
| .github/standards/requirements-principles.md | Clarifies what vs how guidance for unit naming. |
| .github/standards/reqstream-usage.md | Updates required folder layout and generated output paths. |
| .github/standards/design-documentation.md | Updates required design doc structure and heading depth rule alignment. |
| .github/standards/csharp-testing.md | Updates C# testing standard from MSTest guidance to xUnit v3 guidance. |
| .github/standards/csharp-language.md | Updates documentation example and emphasizes XmlDoc + literate coding. |
| .github/standards/coding-principles.md | Expands documentation expectations and contract guidance for all symbols. |
| .github/pull_request_template.md | Updates recommended build/lint commands and documentation checklist wording. |
| .github/agents/repo-consistency.agent.md | Removes the repo-consistency agent definition. |
| .github/agents/lint-fix.agent.md | Updates markdownlint guidance for wide tables. |
| .github/agents/formal-review.agent.md | Updates formal-review agent to include verification-documentation standard. |
| .github/agents/developer.agent.md | Minor wording/format updates to developer agent workflow. |
| .fileassert.yaml | Updates FileAssert document checks for new generated/ output locations and adds verification doc checks. |
| .cspell.yaml | Updates spelling allowlist and ignores generated/. |
Comments suppressed due to low confidence (3)
docs/reqstream/sarifmark.yaml:132
SarifMark-System-ReportHeadingis currently traced toSarifMark_HelpFlag_OutputsUsageInformation, which only verifies help text includes flags and does not verify that a custom heading is actually accepted/applied in generated markdown reports. This leaves the requirement effectively untested/mis-traced; add or reference a test that runs report generation with--headingand asserts the output uses the custom heading, then update thetests:mapping accordingly.
docs/reqstream/sarifmark.yaml:142SarifMark-System-ValidateResultsis traced toSarifMark_ValidateFlag_RunsSelfValidation, but that integration test does not supply--resultsand therefore cannot verify that a results file is written. Add/point to a test that invokes--validate --results <path>(and asserts the file exists/contains expected markers), then update thetests:list so ReqStream traceability reflects actual coverage.
docs/design/sarifmark/self-test.md:28- In the subsystem interface table,
Validation.Runis marked asOutbound, but it is an entrypoint invoked by other parts of the tool (e.g.,Programvia the--validatepath), which makes it an inbound interface to the SelfTest subsystem. Update the Direction column to reflect the actual call direction so the design documentation stays consistent.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Member
Author
|
@copilot apply changes based on the comments in this thread Also the build failed with: Please check the pandoc title.txt metadata against the .fileassert.yaml checks. |
Agent-Logs-Url: https://github.com/demaconsulting/SarifMark/sessions/f649a3b0-798f-4331-b37b-b76fdad256e4 Co-authored-by: Malcolmnixon <1863707+Malcolmnixon@users.noreply.github.com>
Contributor
Addressed in commit I updated |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 90 out of 94 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (2)
docs/reqstream/sarifmark.yaml:132
SarifMark-System-ReportHeadingis linked toSarifMark_HelpFlag_OutputsUsageInformation, but that integration test does not assert--headingis present (or that the heading affects generated report content). To keep system-level traceability accurate, either (1) add an integration test that runs the tool with--headingand validates the generated report, or (2) link this requirement to existing tests that actually validate the heading behavior (CLI parsing + report generation).
docs/reqstream/sarifmark.yaml:142SarifMark-System-ValidateResultsdescribes--resultsbehavior, but it is currently linked only toSarifMark_ValidateFlag_RunsSelfValidation, which runs--validatewithout--resultsand does not assert any results file is written. Consider adding a system-level integration test that passes--validate --results <path>and asserts the file exists/content, or update thetests:mapping to point at the existingSelfTest_ResultsFile_*scenarios that actually validate results file output.
Member
Author
|
@copilot apply changes based on the comments in this thread |
Agent-Logs-Url: https://github.com/demaconsulting/SarifMark/sessions/0e126d25-ab62-4c27-a41d-823ff72d6c99 Co-authored-by: Malcolmnixon <1863707+Malcolmnixon@users.noreply.github.com>
Contributor
Applied in commit I corrected the two verification mappings in No UI changes (documentation-only), so there is no UI screenshot. |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 90 out of 94 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
docs/reqstream/sarifmark.yaml:142
- These system requirements are mapped to tests that do not validate the stated behavior:
SarifMark-System-ReportHeadingis mapped to a help test, andSarifMark-System-ValidateResultsis mapped to a validate-mode test without asserting a results file is written. Update thetests:mapping to scenarios that actually exercise--headingaffecting report output and--resultscreating TRX/JUnit files (or add such tests) so the traceability matrix reflects real verification evidence.
Member
Author
|
@copilot the Build failed with: Error: Only 113 of 146 requirements are satisfied with tests.
|
Agent-Logs-Url: https://github.com/demaconsulting/SarifMark/sessions/f2514cb6-4d4f-4327-bdbd-d777cddab7ed Co-authored-by: Malcolmnixon <1863707+Malcolmnixon@users.noreply.github.com>
Contributor
Fixed in commit I updated the ReqStream requirement mappings in:
All affected entries now reference the current xUnit v3 test method names, which restores traceability for the previously unsatisfied CLI/SARIF/report requirements. No UI changes were made, so no screenshot applies. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
Agent updates, verification document, xUnit, generated document folder.
Type of Change
Related Issues
Closes #
Pre-Submission Checklist
Before submitting this pull request, ensure you have completed the following:
Build and Test
dotnet build --configuration Releasedotnet test --configuration Releasedotnet run --project src/DemaConsulting.SarifMark --configuration Release --framework net10.0--no-build -- --validateCode Quality
dotnet format --verify-no-changesQuality Checks
Please run the following checks before submitting:
./lint.sh(Unix/macOS) orcmd /c lint.bat/./lint.bat(Windows)Testing
Documentation
Additional Notes