feat: Repository abstraction layer + Saga pattern with compensating transactions by Copilot · Pull Request #79 · DeeDee1103/StorageLens

Copilot · 2026-04-03T19:30:27Z

Services were directly coupled to EF Core DbContext, and multi-step workflow failures left orphaned data with no recovery path. This PR introduces a repository abstraction across all data-bearing services and adds saga-style compensating transactions to the Orchestrator workflow.

Repository Abstraction Layer

Extracted IXxxRepository + XxxRepository for all 5 services — all relational/in-memory conditional logic (IsRelational()) now lives in the repository, not the service:

Service	Interface	Location
Locations	`ILocationRepository`	`Repositories/`
ScanJobs	`IScanJobRepository`	`Repositories/`
FileInventory	`IFileInventoryRepository`	`Repositories/`
Duplicates	`IDuplicateRepository`	`Repositories/`
Orchestrator	`IWorkflowRepository`	`Repositories/`

Services now take their repository interface via constructor injection. All repositories registered as Scoped in DI.

Saga Pattern — Compensating Transactions

IWorkflowCompensator / WorkflowCompensator (Sagas/): invoked automatically by ProcessWorkflowAsync on failure. Issues best-effort cleanup per completed step:
- Scanning completed → DELETE api/files/by-scan-job/{id} (FileInventory)
- DetectingDuplicates completed → DELETE api/duplicates/by-scan-job/{id} (Duplicates)
Compensation is fire-and-log per step — one step's failure doesn't block others.

Checkpoint / Resume

ProcessWorkflowAsync now builds completedStepNames from persisted WorkflowStepExecution records before running. Already-completed steps are skipped on retry — Scanner won't re-scan a scan job that already finished, preventing duplicate file ingestion.

// On retry, steps with a prior Completed execution are skipped
var completedStepNames = workflow.Steps
    .Where(s => s.Status == WorkflowStepStatus.Completed.ToString())
    .Select(s => s.StepName)
    .ToHashSet(StringComparer.OrdinalIgnoreCase);

if (!completedStepNames.Contains("Scanning"))
    await RunStepAsync(workflow, "Scanning", ...);

New Endpoints

Method	Route	Purpose
`DELETE`	`api/files/by-scan-job/{scanJobId}`	FileInventory compensation
`DELETE`	`api/duplicates/by-scan-job/{scanJobId}`	Duplicates compensation
`POST`	`api/orchestrator/workflows/{id}/compensate`	Operator-triggered cleanup

Bug Fix

SetWorkflowStatus(Completed) was called before the scan-job update HTTP calls in the success path. If those calls threw, the catch block couldn't transition Completed → Failed. Reordered so the workflow status is set last, after all downstream updates succeed.

Tests

Added WorkflowSagaTests.cs (4 tests): step-resume skipping, Scanning compensation call, DetectingDuplicates compensation call, and compensation triggered from ProcessWorkflowAsync on step failure.

…ting transactions - Repository pattern: ILocationRepository, IScanJobRepository, IFileInventoryRepository, IDuplicateRepository, IWorkflowRepository with EF Core implementations. All 5 data- bearing services now take their repository interface instead of DbContext directly. Program.cs files register repos as Scoped. Tests updated to construct via repository. - Saga pattern: IWorkflowCompensator/WorkflowCompensator issues best-effort DELETE calls to FileInventory and Duplicates services when a workflow fails, undoing side-effects of completed steps. Added checkpoint/resume: already-completed steps are skipped on retry. - New endpoints: DELETE api/files/by-scan-job/{scanJobId}, DELETE api/duplicates/by-scan-job/{scanJobId}, POST api/orchestrator/workflows/{id}/compensate. - Added FileInventory upstream to OrchestratorUpstreamOptions. - Fixed SetWorkflowStatus ordering so downstream update failures don't leave the workflow in Completed before transitioning to Failed. - Added WorkflowSagaTests.cs with 4 new tests. Fixed pre-existing StorageLocationDto constructor calls in WorkflowOrchestratorServiceTests. - Updated docs/CHANGELOG.md. Agent-Logs-Url: https://github.com/DeeDee1103/StorageLens/sessions/d461ea0f-fcf6-4f0e-95b2-d97a1be74510 Co-authored-by: DeeDee1103 <61173060+DeeDee1103@users.noreply.github.com>

Agent-Logs-Url: https://github.com/DeeDee1103/StorageLens/sessions/d461ea0f-fcf6-4f0e-95b2-d97a1be74510 Co-authored-by: DeeDee1103 <61173060+DeeDee1103@users.noreply.github.com>

Copilot

Pull request overview

This PR decouples several services from direct EF Core DbContext usage by introducing repository interfaces/implementations, and extends the Orchestrator workflow with saga-style compensation plus checkpoint/resume semantics to improve recovery from partial failures.

Changes:

Added repository abstraction layers (I*Repository / *Repository) and updated services + DI wiring to depend on repositories rather than DbContext.
Implemented Orchestrator saga compensation (IWorkflowCompensator) and an operator-triggered compensation endpoint; added downstream DELETE endpoints to support cleanup.
Added/updated unit tests to reflect new constructors and to validate saga resume/compensation behavior.

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/StorageLens.Services.ScanJobs.Tests/ScanJobsServiceTests.cs	Updates tests to construct `ScanJobsService` via `ScanJobRepository`.
tests/StorageLens.Services.Orchestrator.Tests/WorkflowSagaTests.cs	Adds new saga-focused tests (resume skipping + compensation behaviors).
tests/StorageLens.Services.Orchestrator.Tests/WorkflowOrchestratorServiceTests.cs	Updates service construction for repository/compensator; adjusts test HttpClient factory behavior.
tests/StorageLens.Services.Orchestrator.Tests/OrchestratorControllerTests.cs	Updates controller construction to include compensator dependency.
tests/StorageLens.Services.Locations.Tests/LocationServiceTests.cs	Updates tests to construct `LocationService` via `LocationRepository`.
tests/StorageLens.Services.FileInventory.Tests/FileInventoryServiceTests.cs	Updates tests to construct `FileInventoryService` via `FileInventoryRepository`.
tests/StorageLens.Services.Duplicates.Tests/DuplicatesServiceTests.cs	Updates tests to construct `DuplicatesService` via `DuplicateRepository`.
src/StorageLens.Services.ScanJobs/Services/ScanJobsService.cs	Refactors service to depend on `IScanJobRepository` and map entities to DTOs.
src/StorageLens.Services.ScanJobs/Repositories/ScanJobRepository.cs	Introduces EF-backed repository (including relational fast paths).
src/StorageLens.Services.ScanJobs/Repositories/IScanJobRepository.cs	Defines ScanJobs repository contract.
src/StorageLens.Services.ScanJobs/Program.cs	Registers `IScanJobRepository` in DI.
src/StorageLens.Services.Orchestrator/Services/WorkflowOrchestratorService.cs	Switches to `IWorkflowRepository`, adds checkpoint/resume skipping and auto-compensation on failure.
src/StorageLens.Services.Orchestrator/Sagas/WorkflowCompensator.cs	Implements best-effort compensation HTTP deletes to FileInventory/Duplicates.
src/StorageLens.Services.Orchestrator/Sagas/IWorkflowCompensator.cs	Defines saga compensation contract.
src/StorageLens.Services.Orchestrator/Repositories/WorkflowRepository.cs	Introduces EF-backed workflow repository (workflow load/save + step/error adds).
src/StorageLens.Services.Orchestrator/Repositories/IWorkflowRepository.cs	Defines workflow repository contract.
src/StorageLens.Services.Orchestrator/Program.cs	Adds `fileinventory` upstream client + registers repository and compensator in DI and health checks.
src/StorageLens.Services.Orchestrator/Controllers/OrchestratorController.cs	Adds operator-triggered `POST .../compensate` endpoint and compensator dependency.
src/StorageLens.Services.Orchestrator/Clients/OrchestratorUpstreamOptions.cs	Adds `FileInventory` upstream URL option.
src/StorageLens.Services.Locations/Services/LocationService.cs	Refactors service to depend on `ILocationRepository` and map entities to DTOs.
src/StorageLens.Services.Locations/Repositories/LocationRepository.cs	Introduces EF-backed Locations repository (including relational fast paths).
src/StorageLens.Services.Locations/Repositories/ILocationRepository.cs	Defines Locations repository contract.
src/StorageLens.Services.Locations/Program.cs	Registers `ILocationRepository` in DI.
src/StorageLens.Services.FileInventory/Services/FileInventoryService.cs	Refactors service to depend on `IFileInventoryRepository`; exposes delete-by-scan-job for compensation.
src/StorageLens.Services.FileInventory/Repositories/IFileInventoryRepository.cs	Defines FileInventory repository contract.
src/StorageLens.Services.FileInventory/Repositories/FileInventoryRepository.cs	Introduces EF-backed FileInventory repository including delete-by-scan-job.
src/StorageLens.Services.FileInventory/Program.cs	Registers `IFileInventoryRepository` in DI.
src/StorageLens.Services.FileInventory/Interfaces/IFileInventoryService.cs	Adds `DeleteByScanJobAsync` to service interface for compensation support.
src/StorageLens.Services.FileInventory/Controllers/FilesController.cs	Adds `DELETE api/files/by-scan-job/{scanJobId}` endpoint for compensation cleanup.
src/StorageLens.Services.Duplicates/Services/DuplicatesService.cs	Refactors service to depend on `IDuplicateRepository`; adds delete-by-scan-job.
src/StorageLens.Services.Duplicates/Repositories/IDuplicateRepository.cs	Defines Duplicates repository contract + aggregate record.
src/StorageLens.Services.Duplicates/Repositories/DuplicateRepository.cs	Introduces EF-backed Duplicates repository including delete/replace.
src/StorageLens.Services.Duplicates/Program.cs	Registers `IDuplicateRepository` in DI.
src/StorageLens.Services.Duplicates/Interfaces/IDuplicatesService.cs	Adds `DeleteByScanJobAsync` to service interface for compensation support.
src/StorageLens.Services.Duplicates/Controllers/DuplicatesController.cs	Adds `DELETE api/duplicates/by-scan-job/{scanJobId}` endpoint for compensation cleanup.
docs/CHANGELOG.md	Documents repository abstraction and saga/compensation additions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-03T19:35:18Z

+        var completedStepNames = workflow.Steps
+            .Where(s => s.Status == WorkflowStepStatus.Completed.ToString())


Checkpoint/resume skipping is based on existing Completed step executions, but the failure-path auto-compensation deletes FileInventory records when Scanning is marked Completed. On a retry, Scanning will be skipped while its side-effects were already undone, so later steps may run with no files to hash/dedupe and the workflow can incorrectly complete with missing data. Consider making compensation and resume semantics consistent (e.g., don’t auto-compensate Scanning when you intend to resume from later steps, or record a separate 'Compensated'/'Invalidated' state and exclude those steps from completedStepNames so they rerun).

Suggested change

var completedStepNames = workflow.Steps

.Where(s => s.Status == WorkflowStepStatus.Completed.ToString())

//

// Scanning is intentionally excluded from resume-skipping. Its side effects

// (FileInventory records) may be removed by failure-path compensation, so a

// previously Completed Scanning step is not a reliable checkpoint for retry.

// Rerunning Scanning on retry keeps resume semantics consistent with

// compensation and prevents downstream steps from running against missing

// inventory.

var completedStepNames = workflow.Steps

.Where(s =>

s.Status == WorkflowStepStatus.Completed.ToString() &&

!string.Equals(s.StepName, "Scanning", StringComparison.OrdinalIgnoreCase))

Copilot · 2026-04-03T19:35:18Z

+            await workflowRepository.SaveChangesAsync(cancellationToken);
+
+            // Trigger best-effort compensating transactions to undo completed steps.
+            await compensator.CompensateAsync(workflow, cancellationToken);


The compensator invocation is not guarded. If an injected IWorkflowCompensator implementation throws, it will escape ProcessWorkflowAsync after the workflow is already marked Failed, potentially crashing the background worker and skipping subsequent queued workflows. Wrap CompensateAsync in its own try/catch and only log on failure to keep compensation strictly best-effort.

Suggested change

await compensator.CompensateAsync(workflow, cancellationToken);

try

{

await compensator.CompensateAsync(workflow, cancellationToken);

}

catch (Exception compensationEx)

{

logger.LogError(

compensationEx,

"Compensation failed for workflow {WorkflowId} after workflow execution failure",

workflowId);

}

Copilot · 2026-04-03T19:35:18Z

+    [HttpPost("workflows/{id:guid}/compensate")]
+    public async Task<IActionResult> Compensate(Guid id, CancellationToken cancellationToken)
+    {
+        var workflow = await workflowService.GetWorkflowAsync(id, cancellationToken);
+        if (workflow is null)


The operator-triggered compensation endpoint will run regardless of workflow state. If called for a Running/Pending workflow it can delete FileInventory/Duplicates data mid-flight and leave the system inconsistent. Consider restricting compensation to terminal workflows (e.g., Failed/Cancelled) or returning 409/400 when the workflow is runnable/in progress.

Copilot · 2026-04-03T19:35:19Z

    private sealed class StubHttpClientFactory(Dictionary<string, HttpClient> clients) : IHttpClientFactory
    {
        public HttpClient CreateClient(string name)
        {
-            if (!clients.TryGetValue(name, out var client))
+            if (clients.TryGetValue(name, out var client))
            {
-                throw new InvalidOperationException($"No client configured for '{name}'.");
+                return client;
            }

-            return client;
+            // Return a no-op client for optional clients (e.g. fileinventory used by compensator)
+            return new HttpClient(new StubHttpMessageHandler((_, _) =>
+                Task.FromResult(new HttpResponseMessage(System.Net.HttpStatusCode.OK))))
+            { BaseAddress = new Uri("http://localhost") };
        }


StubHttpClientFactory now returns a no-op HttpClient for any unconfigured name. This can hide regressions where WorkflowOrchestratorService starts depending on a new upstream client unexpectedly, because tests will still pass with an OK response. Prefer keeping strict behavior and only allowing a fallback for explicitly optional clients (e.g., only "fileinventory"), or have the fallback throw unless the missing name is in an allow-list.

Copilot AI and others added 2 commits April 3, 2026 12:51

fix: Correct typo in WorkflowSagaTests method name (Trigges -> Triggers)

30f3ff1

Agent-Logs-Url: https://github.com/DeeDee1103/StorageLens/sessions/d461ea0f-fcf6-4f0e-95b2-d97a1be74510 Co-authored-by: DeeDee1103 <61173060+DeeDee1103@users.noreply.github.com>

Copilot AI assigned Copilot and DeeDee1103 Apr 3, 2026

Copilot created this pull request from a session on behalf of DeeDee1103 April 3, 2026 19:30 View session

DeeDee1103 marked this pull request as ready for review April 3, 2026 19:30

Copilot AI review requested due to automatic review settings April 3, 2026 19:30

DeeDee1103 merged commit b4d7741 into develop Apr 3, 2026
1 check passed

Copilot started reviewing on behalf of DeeDee1103 April 3, 2026 19:31 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Repository abstraction layer + Saga pattern with compensating transactions#79

feat: Repository abstraction layer + Saga pattern with compensating transactions#79
DeeDee1103 merged 2 commits intodevelopfrom
copilot/code-evaluation-1-to-10

Copilot AI commented Apr 3, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		var completedStepNames = workflow.Steps
		.Where(s => s.Status == WorkflowStepStatus.Completed.ToString())

-        var completedStepNames = workflow.Steps
-            .Where(s => s.Status == WorkflowStepStatus.Completed.ToString())
+        //
+        // Scanning is intentionally excluded from resume-skipping. Its side effects
+        // (FileInventory records) may be removed by failure-path compensation, so a
+        // previously Completed Scanning step is not a reliable checkpoint for retry.
+        // Rerunning Scanning on retry keeps resume semantics consistent with
+        // compensation and prevents downstream steps from running against missing
+        // inventory.
+        var completedStepNames = workflow.Steps
+            .Where(s =>
+                s.Status == WorkflowStepStatus.Completed.ToString() &&
+                !string.Equals(s.StepName, "Scanning", StringComparison.OrdinalIgnoreCase))

-            await compensator.CompensateAsync(workflow, cancellationToken);
+            try
+            {
+                await compensator.CompensateAsync(workflow, cancellationToken);
+            }
+            catch (Exception compensationEx)
+            {
+                logger.LogError(
+                    compensationEx,
+                    "Compensation failed for workflow {WorkflowId} after workflow execution failure",
+                    workflowId);
+            }

Conversation

Copilot AI commented Apr 3, 2026

Repository Abstraction Layer

Saga Pattern — Compensating Transactions

Checkpoint / Resume

New Endpoints

Bug Fix

Tests

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants