Merged
Conversation
Merge pull request #56 from DeeDee1103/develop
…lth probing - Created `capture-dashboard.js` to automate screenshot capture and logging of dashboard details. - Added `compose-dev.ps1` for managing Docker containers with options for clean restarts and full recovery. - Introduced `probe-fileinventory-health.ps1` to check the health of the file inventory API and page, logging results to files.
- Implement FlowManager.cshtml.cs to handle dashboard data retrieval and service availability logging. - Create GlobalView.cshtml and GlobalView.cshtml.cs for displaying global storage metrics and visualizations. - Integrate Leaflet for map visualization and Chart.js for data representation in Global View. - Ensure proper handling of service unavailability in both pages.
- Created a Data Dictionary to describe the persisted data model, including schemas, tables, and important fields. - Added a Glossary of Terms for business users to clarify common terminology in plain language. - Introduced a Technical Glossary to define technical terms used across StorageLens services and operations. - Established a Terminology Consistency Matrix to ensure uniformity in term usage across documentation. - Developed the Insights page to provide operational insights for troubleshooting. - Implemented the Logging page to review structured logs, including filters for severity and service. - Added backend logic for logging insights with sample log entries for demonstration.
- Added auto-refresh interval for Scan Jobs based on AppearanceOptions. - Implemented OnGetSnapshotAsync method to retrieve scan job statistics. - Added methods to cancel and resume scan jobs. - Updated layout with mobile web app capabilities and theme color. - Redirected after starting a scan job to the Scan Jobs page. - Improved CSS for mobile touch interactions and scan jobs UI elements. - Added new styles for scan job statuses and actions. - Introduced new JSON files for runtime job data.
…lience strategies - Updated Docker Compose configuration to replace `service_started` conditions with `service_healthy` for better service readiness checks. - Added health checks for multiple services including analytics, orchestrator, and SQL server to ensure they are ready before dependent services start. - Enhanced `AnalyticsService` to use a `SafeFetchAsync` method for upstream calls, providing fallback data in case of failures. - Improved error handling in `OrchestratorController` to return a bad request for empty location IDs. - Added resilience tests for `AnalyticsService` to verify fallback behavior when upstream services are unavailable. - Introduced a PowerShell script to automate scan flow checks, capturing job statuses and results. - Created runtime check files to log scan flow outcomes for integration testing. - Updated integration tests to use a consistent in-memory database name for isolation.
- Created a new script `probe-all-service-health.ps1` to probe the health of multiple services and log their status. - Added a new script `probe-web-health-routes.ps1` to check specific web health routes and log the results. - Introduced a new file `runtime-web-health-routes.txt` to store the health check results for web routes.
…ions - Introduced a new section for quick action buttons in the top bar, allowing users to access key features quickly. - Implemented JavaScript functionality to render quick action buttons based on user preferences stored in local storage. - Updated CSS styles for the quick action buttons and other UI components to improve visual consistency and responsiveness. - Removed deprecated section collapse functionality to streamline the user experience.
- Added a function to apply hover text to buttons and interactive elements based on data attributes and existing text content. - Implemented Bootstrap tooltip initialization for elements with title or data-bs-title attributes. - Introduced a MutationObserver to dynamically apply hover text and tooltips to newly added elements in the DOM. - Ensured compatibility with elements that have specific attributes to prevent tooltip initialization.
- Implemented `capture-all-pages.js` to automate screenshot capture of various application routes, generating a manifest of results. - Created `smoke-scan-isolation.ps1` to initiate scheduled and immediate scan jobs, capturing their statuses and workflow details for analysis.
- Created new runtime status files to capture compose status after redeploy and final status in both UTF-8 and plain text formats. - Added a JSON file summarizing duplicate files and groups after a fix. - Generated smoke test results for various application pages. - Implemented health check routes and their statuses in both UTF-8 and plain text formats, including an error status for the Index route. - Updated runtime status files to reflect the health of services after an open fix. - Added migration to the Locations service to include new metadata fields: SiteLocation, Latitude, and Longitude in the StorageLocations table.
There was a problem hiding this comment.
Pull request overview
Improves StorageLens’ local development/deployment reliability and documentation by adding Docker Compose health checks + dependency gating, and expanding docs around new location storage stats synchronization and terminology.
Changes:
- Added healthchecks and
depends_on: condition: service_healthyorchestration across Docker Compose services, plus updated mount guidance for cross-platform scanning. - Expanded documentation for Global View/live location totals synchronization (new endpoints, data flow, operational checks) and introduced new glossary/reference documents.
- Updated correlation header casing across strategy docs and refreshed changelogs/screenshot documentation.
Reviewed changes
Copilot reviewed 32 out of 591 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/user-guide/user-guide.md | Updates user journey/UI areas and links to new glossary references. |
| docs/user-guide/duplicate-analysis-guide.md | Documents new default “Interactive” duplicates experience. |
| docs/user-guide/dashboard-guide.md | Documents Global View access and new global distribution behavior. |
| docs/technical/terminology-consistency-matrix.md | Adds canonical terminology matrix for consistent naming across docs/code/headers. |
| docs/technical/technical-specification.md | Updates workflow steps to include per-location totals synchronization. |
| docs/technical/service-contracts.md | Updates contract locations and lists new DTOs/requests. |
| docs/technical/database-schema.md | Notes synchronized totals and reinforces schema ownership and API-only cross-service sync. |
| docs/technical/configuration-guide.md | Adds routing/API-key considerations for global distribution synchronization. |
| docs/technical/api-specification.md | Documents auth headers and adds new per-location stats endpoints and data path notes. |
| docs/strategies/LoggingStrategy.md | Updates correlation header casing to X-Correlation-ID in example code. |
| docs/strategies/ImplementationRoadmap.md | Updates contract file-path guidance to directory-based structure. |
| docs/strategies/ErrorHandlingStrategy.md | Updates correlation header casing to X-Correlation-ID in example code. |
| docs/screenshots/capture-manifest.json | Adds a generated screenshot manifest with metadata and file sizes. |
| docs/screenshots/README.md | Updates screenshot descriptions to match new UX (home/global view, interactive duplicates, CSV exports, pricing). |
| docs/operations/monitoring-logging.md | Adds audit logging + new sync/drift metrics and alert suggestions. |
| docs/operations/docker-deployment.md | Adds troubleshooting steps for stale global location totals. |
| docs/operations/deployment-guide.md | Adds deployment validation checks for new stats endpoints and drift/0B issues. |
| docs/glossary-terms-technical.md | Adds technical glossary for architecture/service/data-flow terminology. |
| docs/glossary-terms-business-users.md | Adds business glossary for non-technical readers. |
| docs/developer/developer-guide.md | Adds pointers to Data Dictionary and terminology references. |
| docs/developer/contributing.md | Adds requirement to update data dictionary when schemas/entities change. |
| docs/data-dictionary.md | Adds source-of-truth persisted data model reference across services/schemas. |
| docs/architecture/system-overview.md | Documents live global distribution and location totals synchronization. |
| docs/architecture/security-architecture.md | Documents API key auth, correlation IDs, rate limiting, audit logging; tweaks recommendations. |
| docs/architecture/microservices-architecture.md | Clarifies service count, analytics statelessness, and API-driven sync behavior. |
| docs/architecture/data-architecture.md | Documents authoritative per-location totals endpoint + sync back to Locations. |
| docs/README.md | Updates “Recent Updates” and adds links to new glossary/data dictionary references. |
| docs/CHANGELOG.md | Adds unreleased entries and corrects references to correlation header casing and service count phrasing. |
| docker-compose.yml | Adds healthchecks, health-based dependency gating, and updated Windows/Linux mount instructions for scanner/hashing. |
| ERROR | Adds a captured PowerShell parser error output (likely accidental). |
| CHANGELOG.md | Updates root changelog with new endpoints and behavior notes. |
| .claude/settings.local.json | Adds helper Docker commands for starting/inspecting SQL container health. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+74
to
+79
| healthcheck: | ||
| test: ["CMD", "dotnet", "--info"] | ||
| interval: 10s | ||
| timeout: 4s | ||
| retries: 6 | ||
| start_period: 15s |
There was a problem hiding this comment.
dotnet --info does not validate that the service process is accepting traffic (and it can remain healthy even if the app fails to bind, crashes after startup, or returns non-200 from /health). To make depends_on: condition: service_healthy meaningful, switch these app service healthchecks to hit the container’s actual health/readiness endpoint (e.g., an HTTP GET to /health on localhost) or an internal port check that reflects real readiness.
Comment on lines
+11
to
+16
| healthcheck: | ||
| test: ["CMD-SHELL", "exit 0"] | ||
| interval: 10s | ||
| timeout: 3s | ||
| retries: 5 | ||
| start_period: 10s |
There was a problem hiding this comment.
This Jaeger healthcheck always succeeds (exit 0), which defeats the purpose of having a healthcheck and can mask failures. Either remove the healthcheck entirely (and rely on condition: service_started), or replace it with a real readiness probe (e.g., checking the Jaeger HTTP endpoint) so container health reflects actual availability.
Suggested change
| healthcheck: | |
| test: ["CMD-SHELL", "exit 0"] | |
| interval: 10s | |
| timeout: 3s | |
| retries: 5 | |
| start_period: 10s |
Comment on lines
+230
to
+238
| # Windows host path access for scanner auto-mapping: | ||
| # C:\... -> /host/drives/c/... and \\server\share -> /host/unc/server/share/... | ||
| # On Windows Docker Desktop (WSL2 or Hyper-V): each drive must be mounted explicitly. | ||
| # Docker Desktop file sharing must be enabled for these drives in Settings > Resources > File Sharing. | ||
| - C:/:/host/drives/c | ||
| # Add more drives as needed, e.g.: - D:/:/host/drives/d | ||
| # On a Linux Docker host: replace the lines above with: - /host/drives:/host/drives | ||
| # UNC network shares must be mounted explicitly: | ||
| - //DouglasCludEx2/Development:/host/unc/DouglasCludEx2/Development |
There was a problem hiding this comment.
Mounting the host root drive (C:/) and a specific UNC share by default is a high-risk default (exposes broad host filesystem contents to the container) and also makes the compose file machine-specific (//DouglasCludEx2/Development). These mounts should be opt-in and configurable (e.g., documented as commented examples, driven by env vars/profiles, or moved to an override compose file that developers create locally).
Suggested change
| # Windows host path access for scanner auto-mapping: | |
| # C:\... -> /host/drives/c/... and \\server\share -> /host/unc/server/share/... | |
| # On Windows Docker Desktop (WSL2 or Hyper-V): each drive must be mounted explicitly. | |
| # Docker Desktop file sharing must be enabled for these drives in Settings > Resources > File Sharing. | |
| - C:/:/host/drives/c | |
| # Add more drives as needed, e.g.: - D:/:/host/drives/d | |
| # On a Linux Docker host: replace the lines above with: - /host/drives:/host/drives | |
| # UNC network shares must be mounted explicitly: | |
| - //DouglasCludEx2/Development:/host/unc/DouglasCludEx2/Development | |
| # Windows host path access for scanner auto-mapping (optional, not enabled by default): | |
| # C:\... -> /host/drives/c/... | |
| # \\server\share -> /host/unc/server/share/... | |
| # | |
| # WARNING: Mounting entire host drives or UNC shares is high risk and machine-specific. | |
| # To use these mappings, copy them into a local docker-compose.override.yml | |
| # (which is not committed) or uncomment them only on your own machine as needed. | |
| # | |
| # Example Windows drive mapping: | |
| # - C:/:/host/drives/c | |
| # Add more drives as needed, e.g.: - D:/:/host/drives/d | |
| # | |
| # On a Linux Docker host, consider using a narrower, explicit directory mapping instead, | |
| # e.g.: - /path/to/data:/host/drives/data | |
| # | |
| # Example UNC network share mapping: | |
| # - //DouglasCludEx2/Development:/host/unc/DouglasCludEx2/Development |
Comment on lines
1
to
16
| At line:1 char:639 | ||
| + ... agelens-dev-key' }; $result += $u = $($_.Exception.Message) } } }; $r ... | ||
| + ~ | ||
| + ... agelens-dev-key' }; $result += $u = | ||
| $($_.Exception.Message) } } }; $r ... | ||
| + | ||
| ~ | ||
| The Try statement is missing its Catch or Finally block. | ||
| At line:1 char:642 | ||
| + ... ens-dev-key' }; $result += $u = $($_.Exception.Message) } } }; $resul ... | ||
| + ~ | ||
| + ... ens-dev-key' }; $result += $u = | ||
| $($_.Exception.Message) } } }; $resul ... | ||
| + | ||
| ~ | ||
| Unexpected token '}' in expression or statement. | ||
| + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException | ||
| + CategoryInfo : ParserError: (:) [], Parent | ||
| ContainsErrorRecordException | ||
| + FullyQualifiedErrorId : MissingCatchOrFinally | ||
|
|
There was a problem hiding this comment.
This looks like an accidental commit of a local PowerShell parser error log rather than a tracked project artifact. Recommend removing this file from the PR/repo (and adding it to .gitignore if it’s generated by a common workflow) to avoid noise and confusion.
Comment on lines
+15
to
+17
| | Canonical Term | Business Form | Technical Form | Code/Data Form | Header/Param Form | | ||
| | --- | --- | --- | --- | --- | | ||
| | Scan Job | Scan Job | Scan Job | ScanJob | scanJobId | |
There was a problem hiding this comment.
The tables use || at the start of each row, which renders an extra empty first column in standard Markdown table syntax. Replace the leading || with a single | throughout the tables so they render as intended.
Comment on lines
+30
to
+31
| - `GET /api/files/stats/by-location` (FileInventory) | ||
| - `PUT /api/locations/{id}/stats` (Locations) |
There was a problem hiding this comment.
Lines 30–31 use tab indentation (\t), which can render inconsistently in Markdown and can fail formatting checks if the repo enforces spaces. Use spaces for nested bullets to keep Markdown rendering consistent across editors.
Suggested change
| - `GET /api/files/stats/by-location` (FileInventory) | |
| - `PUT /api/locations/{id}/stats` (Locations) | |
| - `GET /api/files/stats/by-location` (FileInventory) | |
| - `PUT /api/locations/{id}/stats` (Locations) |
Comment on lines
+1
to
+5
| { | ||
| "generatedAtUtc": "2026-03-26T01:26:28.7928893Z", | ||
| "totalScreenshots": 26, | ||
| "screenshots": [ | ||
| { |
There was a problem hiding this comment.
This JSON appears to include a UTF-8 BOM (line 1 shows an invisible leading character) and a run-specific timestamp (generatedAtUtc). Both can create unnecessary churn/diffs and BOMs can break some parsers/tools. Consider removing the BOM and either omitting generatedAtUtc (or moving it to build artifacts) so the file is stable if it’s intended to be committed.
Suggested change
| { | |
| "generatedAtUtc": "2026-03-26T01:26:28.7928893Z", | |
| "totalScreenshots": 26, | |
| "screenshots": [ | |
| { | |
| { | |
| "totalScreenshots": 26, | |
| "screenshots": [ | |
| { | |
| { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant improvements to the development and deployment experience for StorageLens, primarily by adding robust health checks to all Docker Compose services and enhancing service dependency management. It also introduces new API endpoints and synchronization logic for location storage statistics, updates the changelog with recent features and fixes, and clarifies documentation and mounting instructions for cross-platform compatibility.
Docker Compose health checks and dependency management:
healthcheckconfigurations to all major services indocker-compose.yml, including custom commands and intervals for SQL Server, Redis, RabbitMQ, .NET services, and Jaeger, improving reliability and startup sequencing. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]depends_onfor all services to usecondition: service_healthywhere applicable, ensuring services only start when their dependencies are ready, which prevents race conditions during startup. [1] [2] [3] [4] [5] [6] [7] [8] [9]Location storage stats API and synchronization:
LocationStorageStatsDto,UpdateStorageLocationStatsRequest,GET /api/files/stats/by-location, andPUT /api/locations/{id}/stats. [1] [2]AnalyticsService.GetDashboardAsyncto enrich location totals from grouped FileInventory stats and synchronize these back to Locations metadata. [1] [2]0 Bin Global Storage Distribution despite having inventory data. [1] [2]Global View and map plotting improvements:
Documentation and changelog updates:
CHANGELOG.mdanddocs/CHANGELOG.mdto reflect new features (marketing home lead capture, CSV exports, new APIs), behavioral changes (routing, pricing, map plotting), and recent fixes. Also clarified mounting instructions for Windows and Linux hosts in Docker Compose. [1] [2] [3] [4] [5] [6]Development tooling:
.claude/settings.local.jsonwith Docker commands to start and inspect the SQL container, aiding local development and troubleshooting.