Repository navigation
feat(application): add transport-independent snapshot-normalizer-core crate to component 507 - #831
Merged
Merged
Conversation
… crate to component 507 - build canonical image_snapshot v1 envelope from raw JPEG bytes - publish image-snapshot-v1.schema.json as the tolerant-reader contract - add bounded dedup, FNV-1a payload digest, fixed-cardinality counters - register the crate in both rust-tests.yml matrices 📸 - Generated by Copilot
📚 Documentation Health ReportGenerated on: 2026-10-01 12:34:01 UTC 📈 Documentation Statistics
🏗️ Three-Tree Architecture Status
🔍 Quality Metrics
This report is automatically generated by the Documentation Automation workflow. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #831 +/- ##
==========================================
+ Coverage 31.79% 38.70% +6.91%
==========================================
Files 40 41 +1
Lines 6017 6696 +679
==========================================
+ Hits 1913 2592 +679
Misses 4104 4104
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
🔒 - Generated by Copilot
- add MQTT snapshot normalizer container and Helm release - add portable local ONNX inference chart - add neutral Media connector blueprint profile ✨ - Generated by Copilot
🔧 - Generated by Copilot
📚 Documentation Health ReportGenerated on: 2026-10-01 12:59:35 UTC 📈 Documentation Statistics
🏗️ Three-Tree Architecture Status
🔍 Quality Metrics
This report is automatically generated by the Documentation Automation workflow. |
📚 Documentation Health ReportGenerated on: 2026-10-01 13:02:23 UTC 📈 Documentation Statistics
🏗️ Three-Tree Architecture Status
🔍 Quality Metrics
This report is automatically generated by the Documentation Automation workflow. |
Contributor
Author
|
Thanks for reviewing this migration slice. CI triage found that the required validation gate is currently blocked by failures outside this PR's changed paths:
The snapshot-normalizer Rust test, coverage, vulnerability-scan, CodeQL, documentation, and registration jobs pass. The branch is also one commit behind |
📚 Documentation Health ReportGenerated on: 2026-10-05 13:22:26 UTC 📈 Documentation Statistics
🏗️ Three-Tree Architecture Status
🔍 Quality Metrics
This report is automatically generated by the Documentation Automation workflow. |
9 of 29 tasks
Contributor
Author
|
Status update on the required Rust Clippy — resolved. The earlier failure was Security Scan — fix in #839. Grype fails on
#839 overrides Next steps: once #839 merges, I'll update this branch from |
Marcel Bindseil (bindsi)
requested changes
Oct 5, 2026
Marcel Bindseil (bindsi)
left a comment
Marcel Bindseil (bindsi)
left a comment
Member
There was a problem hiding this comment.
Changes are requested for the reviewed runtime and deployment scope: content-only dedup can suppress continuing fresh captures, the inference chart does not resolve its mounted default model, and the health Service exposes unauthenticated inference handlers. Also align metrics exposure with an actual listener and make MQTT client identities unique across overlapping pods. CI and readiness were intentionally excluded. Helm lint/render passed; local Rust tests were blocked by Linux-only MQTT SDK symbols on macOS. These AI-assisted findings require human validation.
…er-core - revert MQTT adapter, Dockerfile, Helm charts, and vision blueprint example - revert chart YAML lint follow-up and unrelated .gitleaks.toml exception - keep the approved library, schema, README, and CI registration unchanged 🔒 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Scope update: I trimmed this PR back to the library-only change its description covers (f6c3195). Reverted, with history preserved:
The PR is now 7 files: the MQTT transport, packaging, and connector integration will come in a separate follow-up once the topic contract (#835) and image ownership are settled. |
📚 Documentation Health ReportGenerated on: 2026-10-05 14:36:33 UTC 📈 Documentation Statistics
🏗️ Three-Tree Architecture Status
🔍 Quality Metrics
This report is automatically generated by the Documentation Automation workflow. |
📐 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
📚 Documentation Health ReportGenerated on: 2026-10-05 15:00:03 UTC 📈 Documentation Statistics
🏗️ Three-Tree Architecture Status
🔍 Quality Metrics
This report is automatically generated by the Documentation Automation workflow. |
Alain Uyidi (auyidi1)
added a commit
that referenced
this pull request
Oct 5, 2026
# Pull Request ## Description `main` currently fails the Grype step of the **Security Scan** job, which fails the required `PR Validation Gate` on unrelated PRs (for example #831, #834, #825, #824, #797, and #786). Running Grype 0.109.1 with `.grype.yaml` against `main` (`128cd7ae`) reports three findings at or above the `high` threshold: | Package | Version | Advisory | Fix | Location | |---------|---------|----------|-----|----------| | `basic-ftp` | 5.3.1 | GHSA-c475-qrg2-pj4r (published 2026-10-01) | 6.2.1 | `.github/scripts/markdown-link-check/package-lock.json` | | `braces` | 3.0.3 | GHSA-vfj7-8cjw-p6xm (published 2026-09-18) | None | `docs/docusaurus/package-lock.json` | | `http-cache-semantics` | 4.2.0 | GHSA-ch52-4w7c-c8xp (published 2026-09-18) | None | `docs/docusaurus/package-lock.json` | This PR remediates the fixable finding and documents the two unfixable ones. ## Related Issue Relates to #831 ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Blueprint modification or addition - [ ] Component modification or addition - [ ] Documentation update - [x] CI/CD pipeline change - [ ] Other (please describe): ## Implementation Details ### `basic-ftp` override * Latest `get-uri` (8.0.1) still requires `basic-ftp ^5.3.1`, so there is no upstream upgrade path. * Added an npm `overrides` entry for `basic-ftp: ^6.2.2` in `.github/scripts/markdown-link-check/package.json` and updated the existing root `overrides` and `resolutions` from `^5.3.0` to `^6.2.2`. 6.2.2 also fixes GHSA-5rfr-xx34-2xxv. * `get-uri` only calls `access`, `lastMod`, `list`, `downloadTo`, and `close`. The only 6.0.0 breaking change disables separate transfer hosts by default to prevent FTP bounce attacks; none of these call sites depend on it. * Lockfile changes are limited to the `basic-ftp` entry in each lockfile. ### Scoped Grype ignores Both advisories have no patched release. The ignores follow the existing `image-size` precedent and are scoped by package type and name: * `braces` (GHSA-vfj7-8cjw-p6xm): reached through Docusaurus build-time file watching and glob matching (`chokidar`, `micromatch`) with repository-controlled patterns. * `http-cache-semantics` (GHSA-ch52-4w7c-c8xp): reached only through Docusaurus's update notifier (`update-notifier` → `latest-version` → `package-json` → `got` → `cacheable-request`), a single-user CLI version check. The repository does not run a shared HTTP cache. Both ignores should be removed when patched releases are available. ## Testing Performed - [ ] Terraform plan/apply - [ ] Blueprint deployment test - [ ] Unit tests - [ ] Integration tests - [ ] Bug fix includes regression test (see [Test Policy](docs/contributing/testing-validation.md)) - [x] Manual validation - [x] Other: Grype 0.109.1 with `.grype.yaml` (same version and config as CI) ## Validation Steps 1. `grype dir:. --config .grype.yaml` on `main`: exit 2 with the three high findings above. 2. Same command on this branch: exit 0 with no high or critical findings. 3. `yamllint -c .yamllint.yml .grype.yaml`: pass. 4. `npm ci --prefix .github/scripts/markdown-link-check` installs `basic-ftp` 6.2.2, and `markdown-link-check` runs and checks links normally. ## Checklist - [x] I have updated the documentation accordingly - [ ] I have added tests to cover my changes - [ ] All new and existing tests passed - [ ] I have run `terraform fmt` on all Terraform code - [ ] I have run `terraform validate` on all Terraform code - [ ] I have run `az bicep format` on all Bicep code - [ ] I have run `az bicep build` to validate all Bicep code - [x] I have checked for any sensitive data/tokens that should not be committed - [x] Lint checks pass (run applicable linters for changed file types) ## Security Review - [x] No credentials, secrets, or tokens are hardcoded or logged - [ ] RBAC and identity changes follow least-privilege principles - [ ] No new network exposure or public endpoints introduced without justification - [x] Dependency additions or updates have been reviewed for known vulnerabilities - [ ] Container image changes use pinned digests or SHA references ## Additional Notes No Terraform, Bicep, or application code changes. The two Grype ignores are risk acceptances and need maintainer review. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
📚 Documentation Health ReportGenerated on: 2026-10-05 15:34:02 UTC 📈 Documentation Statistics
🏗️ Three-Tree Architecture Status
🔍 Quality Metrics
This report is automatically generated by the Documentation Automation workflow. |
- document that identical fresh captures hash equally and stay suppressed - advise keying payload_hash scope on a producer capture identifier 📝 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Marcel Bindseil (bindsi)
approved these changes
Oct 5, 2026
Marcel Bindseil (bindsi)
left a comment
Marcel Bindseil (bindsi)
left a comment
Member
There was a problem hiding this comment.
Re-reviewed commit 982bbf2. All five earlier runtime/deployment findings are now removed/not applicable because the PR reverted to a pure library; this is not an upstream fix to the existing inference service. One Medium issue remains in the new schema: its Base64 pattern accepts malformed padding that consumers reject. The 70 unit tests, formatting check and strict Clippy passed. CI/readiness remained excluded, and standards skill-trace coverage is limited. The diff-scoped assessment is approve with comments. Approving the narrowed library-only scope with the remaining schema finding noted as non-blocking.
📚 Documentation Health ReportGenerated on: 2026-10-05 16:43:35 UTC 📈 Documentation Statistics
🏗️ Three-Tree Architecture Status
🔍 Quality Metrics
This report is automatically generated by the Documentation Automation workflow. |
…chema - require complete quartets, a valid final padded group, and canonical pad bits - match what the component 507 consumer's standard Base64 decoder accepts 🐛 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pshot-normalizer-core
📚 Documentation Health ReportGenerated on: 2026-10-05 17:47:28 UTC 📈 Documentation Statistics
🏗️ Three-Tree Architecture Status
🔍 Quality Metrics
This report is automatically generated by the Documentation Automation workflow. |
7 of 29 tasks
Alain Uyidi (auyidi1)
added a commit
that referenced
this pull request
Oct 8, 2026
…er (#852) # Pull Request ## Description Adds a **Custom Workload Messaging Contracts** section to `docs/solution-technology-paper-library/aio-messaging-design.md`. The section gives product-neutral guidance for custom workloads that publish to or consume from the AIO MQTT broker: * **Versioned topics:** a major version segment, opaque non-identifying segments, and concrete publish topics. It explains how the grammar relates to ADR asset and UNS topics. * **Publish loops:** recommends that workloads never publish to a topic matching any of their own subscription filters. This mirrors the AIO 2609 MQTT connector rule that rejects identical source and destination topics, and notes that the check doesn't cover wildcard overlap. * **Message envelopes:** carried as CloudEvents attributes through the MQTT binding (binary mode, MQTTv5 user properties), consistent with the paper's existing [Data Source](https://github.com/microsoft/edge-ai/blob/main/docs/solution-technology-paper-library/aio-messaging-design.md#data-source-acquisition) convention. A table maps each field to its attribute and defines the version split between the topic and `dataschema`. Existing schemas such as the 507 `image_snapshot` v1 schema from #831 are referenced, not redefined. * **Workload identity and least-privilege authorization:** * TLS-only listeners and default-deny only when authorization is bound through `authorizationRef`. * Projected service account tokens or X.509, with client IDs that are unique per replica and stable for persistent sessions. * `Connect` rules scoped to each principal's client IDs, with principal token substitution for per-workload topic scoping. * AIO 2609 as the minimum for shared subscriptions, and TLS plus authentication for external brokers. * **Payload-safe diagnostics:** randomly generated IDs, keyed digests only, and language-neutral guidance on string and debug representations. * **Synthetic contract validation:** broker-independent unit cases, a separate authorization integration check, and deduplication scope, window, and replica-sharing rules. * **Contract ownership:** producer, consumer, deployment-owner, and security-review responsibilities. The guidance is documentation only. It adds no broker configuration, runtime code, or environment-specific values. The PR also updates the paper's existing schema-generation tool link (the old URL returns 404), its frontmatter description, keywords, and reading time, and adds `SUBACK` to the IoT Operations spelling dictionary. ## Related Issue Refs #835. Once the owner approvals below are recorded on this PR, #835 can be closed. ### Owner Decisions From #835 | # | Decision | Status | |---|---|---| | D1 | Destination | ✅ @bindsi: extending `aio-messaging-design.md` is fine | | D2 | Security owner | @bindsi will review. His three required changes are applied: authorization bound through `authorizationRef`, `Connect` rules scoped to each principal's client IDs, and TLS for listeners and external brokers. **Awaiting sign-off.** | | D3 | Envelope alignment with CloudEvents | Applied: the envelope is carried as CloudEvents attributes through the MQTT binding, with a field mapping. **Awaiting confirmation.** | | D4 | Privacy review | @bindsi: not needed if identifier rules are tightened. Applied: `{producer}` and `{resource-id}` are opaque, logged topics exclude identifiers, IDs are generated randomly, and digests are keyed (HMAC). **Awaiting confirmation.** | | D5 | Principal substitution in topic rules | Included: `{principal.attributes.<name>}` on the `{producer}` segment, and `{principal.clientId}` only with client-ID-scoped `Connect` rules. **Awaiting confirmation.** | | D6 | Scope | Product-neutral content, synthetic examples, public Microsoft Learn and release-note sources | | D7 | Component 507 non-conflict | ✅ Resolved 2026-10-07 by @auyidi1. The section references the `image_snapshot` v1 schema and doesn't replace any component-owned schema. | ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Blueprint modification or addition - [ ] Component modification or addition - [x] Documentation update - [ ] CI/CD pipeline change - [ ] Other (please describe): ## Implementation Details Sources: * [Configure MQTT broker authentication](https://learn.microsoft.com/azure/iot-operations/manage-mqtt-broker/howto-configure-authentication): SAT audience, token refresh, X.509 * [Configure MQTT broker authorization](https://learn.microsoft.com/azure/iot-operations/manage-mqtt-broker/howto-configure-authorization): * allow-only rules; * `authorizationRef` binding, which also requires a linked BrokerAuthentication; * disabling authorization by omitting `authorizationRef`; * token substitution; * client-ID-scoped `Connect` rules * [Shared subscriptions](https://learn.microsoft.com/azure/iot-operations/develop-edge-apps/overview-edge-apps#shared-subscriptions) * [AIO 2609 (1.4.112) release notes](https://github.com/Azure/azure-iot-operations/releases/tag/v1.4.112): publish-loop prevention, rejection of anonymous auth to external brokers, and the shared-subscription authorization fix * [MQTT QoS ADR](https://github.com/microsoft/edge-ai/blob/main/docs/solution-adr-library/mqtt-qos.md): QoS 2 limitation * [CloudEvents MQTT protocol binding](https://github.com/cloudevents/spec/blob/main/cloudevents/bindings/mqtt-protocol-binding.md) ## Testing Performed - [ ] Terraform plan/apply - [ ] Blueprint deployment test - [ ] Unit tests - [ ] Integration tests - [ ] Bug fix includes regression test (see [Test Policy](docs/contributing/testing-validation.md)) - [x] Manual validation - [x] Other: documentation linting ## Validation Steps 1. markdownlint, markdown-table-formatter, and cspell pass for the changed file. 2. markdown-link-check passes for the changed file, including the updated tool link, the release-notes and Learn links, and the relative link to `image-snapshot-v1.schema.json`. 3. The frontmatter validator passes. 4. Gitleaks finds no leaks. 5. Manual validation compared each broker-behavior claim with the linked Learn pages and the 2609 release notes. This covered the default-deny binding, token substitution, `Connect` client-ID scoping, SAT audience and token refresh, and the three 2609 changes. Each envelope mapping was checked against the CloudEvents attributes in the paper's Data Source section. ## Checklist - [x] I have updated the documentation accordingly - [ ] I have added tests to cover my changes - [ ] All new and existing tests passed - [ ] I have run `terraform fmt` on all Terraform code - [ ] I have run `terraform validate` on all Terraform code - [ ] I have run `az bicep format` on all Bicep code - [ ] I have run `az bicep build` to validate all Bicep code - [x] I have checked for any sensitive data/tokens that should not be committed - [x] Lint checks pass (run applicable linters for changed file types) ## Security Review - [x] No credentials, secrets, or tokens are hardcoded or logged - [ ] RBAC and identity changes follow least-privilege principles: N/A, no RBAC change. The authorization guidance itself awaits security-owner sign-off (D2). - [ ] No new network exposure or public endpoints introduced without justification: N/A, no network change - [ ] Dependency additions or updates have been reviewed for known vulnerabilities - [ ] Container image changes use pinned digests or SHA references ## Additional Notes * The frontmatter validator warns about `ms.topic: solution-article` on this file. The warning predates this change, and I left the value alone so the paper's documentation-site placement doesn't change. * The new section keeps contractions, following the repository writing-style guidance. Happy to expand them to match the rest of the paper if you prefer. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Oct 8, 2026
Alain Uyidi (auyidi1)
added a commit
that referenced
this pull request
Oct 9, 2026
…867) # Pull Request ## Description Hardens the component 507 inference service and its Kubernetes manifests. It applies the packaging requirements raised during review of #831 (FR-02 to FR-04) and fixes a subscription gap so 507 receives output from the snapshot normalizer in #866. * **Model paths resolve inside the mounted claim (FR-02).** * `InferenceConfig::validate()` checked default models at `models_directory.join(path)`, but `InferenceEngine::initialize()` loaded the bare relative path from the process working directory. Validation and loading looked in different places, and a failed load only logged a warning. The engine now joins with `models_directory`, the same as validation. * `DEFAULT_MODELS=tiny-yolov2` mapped to `default.onnx`, which nothing writes. It now maps to `tiny-yolov2/tinyyolov2-8.onnx`, the file `charts/model-downloader-job.yaml` downloads; `yolov4` maps to `yolov4/yolov4.onnx`. Any other value is a relative model file path instead of silently becoming `default.onnx`. * **Startup and readiness require a loaded model (FR-02).** `/readyz` and `/startup` return 200 only once the backend is initialized and at least one model has loaded, so a pod started before the model download finishes fails startup after the 5-minute window and the kubelet restarts it. The base Deployment's startup, readiness, and liveness probes were commented out and are now enabled. The README documents applying the model downloader Job before the Deployment. * **Probe-only listener by default (FR-03).** `/test/inference` (unauthenticated inference on bodies up to 10 MB) and `/process-files` respond 404 unless the new `ENABLE_TEST_ENDPOINTS` is `true`, and the service logs a warning when it is. The base Deployment sets it to `false` explicitly. * **No phantom metrics port (FR-04).** `main.rs` never compiles `metrics.rs`, so nothing listens on `METRICS_PORT`. Removed the `metrics` container and Service port, `METRICS_PORT` from the manifests and both Dockerfiles, the unused `metrics_port` config field, and the metrics port-forward hint in `deploy.sh`. * **Subscribes to every configured input filter.** The service subscribed only to the first `MQTT_INPUT_TOPICS` entry. It now subscribes to each entry, waits for the SUBACK, and retries any filter the broker rejects. Topic matching uses the SDK's `TopicName::matches_topic_filter`. * The binary default, manifests, and docker-compose set `edge-ai/+/+/camera/snapshots,edge-ai/v1/+/camera/+/snapshots`: existing 5-level publishers keep working, and the pinned v1 filter follows the #852 `{domain}/{version}/{producer}/{resource-kind}/{resource-id}/{message-kind}` grammar, which #866's `edge-ai/v1/snapshot-normalizer/camera/{camera-id}/snapshots` topic uses. * **Payload-safe intake.** Received payloads are no longer sliced or logged; logs carry the topic and payload length only. Payloads that aren't valid UTF-8, aren't JSON, or carry a `schema_version` whose major isn't `1` are dropped with a bounded reason. This removes a panic (and, with `panic = "abort"`, a process abort) when a multibyte character straddled byte 100. * **Model names and paths.** Unknown `DEFAULT_MODELS` entries are named by file stem instead of all being `default`, duplicate names fail startup, and absolute or `..` model paths are rejected in both the service and the crate. ## Related Issue Follows up #831 review requirements FR-02, FR-03, and FR-04. Companion to #866, which covers FR-01 and FR-05. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Blueprint modification or addition - [x] Component modification or addition - [x] Documentation update - [ ] CI/CD pipeline change - [ ] Other (please describe): ## Implementation Details * `health_simple.rs`: `is_ready` is a pure function over `BackendStatus` used by both `/startup` and `/readyz`. Test routes sit behind a `warp` guard filter that rejects with 404 when disabled, so the route set and reply types are unchanged. * `mqtt.rs`: `input_patterns()` trims and drops empty entries and falls back to the two default filters. One `process_payload()` path validates, dispatches, and counts every received message. Each filter keeps its own subscribe-retry task, which checks the SUBACK reason codes. * `ai-edge-inference-crate`: new `resolve_model_path()` is shared by `validate()`, engine initialization, and the model registry. * Docs: the 507 README environment table now lists the binary defaults, a new *Health, Readiness, and Test Endpoints* section lists every unauthenticated endpoint, and *Topic Structure* shows the real input filters and the `{TOPIC_PREFIX}/inference/{model_type}/{model_name}[/{priority}]` output topic. The service README replaces stale metrics and port examples and fixes two dead relative links: the `CONTRIBUTING.md` fix is identical to #866's, so the two branches merge cleanly. * Behavior change for operators who call `/test/inference` or `/process-files`: set `ENABLE_TEST_ENDPOINTS=true`. ## Testing Performed - [ ] Terraform plan/apply - [ ] Blueprint deployment test - [x] Unit tests - [x] Integration tests - [x] Bug fix includes regression test (see [Test Policy](docs/contributing/testing-validation.md)) - [x] Manual validation - [ ] Other: * `ai-edge-inference`: `cargo clippy --all-targets -- -D warnings` is clean and `cargo test` passes all 21 tests: model path mapping, stem naming, duplicate and traversal rejection, readiness and startup gates, input-filter selection, SDK topic matching, SUBACK reason handling, a non-UTF-8 payload whose replacement character straddles byte 100, and `schema_version` major checks. * `ai-edge-inference-crate`: clippy is clean and the 26 lib tests pass. The new `startup_path_resolves_relative_default_models_in_models_directory` test loads `identity.onnx` given only a relative path; before the fix the load resolved against the working directory. New `resolve_model_path` tests reject absolute and `..` paths. * `cargo fmt --check` already fails on `main` for both crates; the changed hunks are rustfmt-clean. * **Runtime, against Mosquitto with the built binary:** | Scenario | `/startup` | `/readyz` | `/healthz` | `/test/inference` | `/process-files` | |---|---|---|---|---|---| | Valid model | 200 | 200 | 200 | 404 | 404 | | Invalid model file | 200 | **503** | 200 | 404 | 404 | | `ENABLE_TEST_ENDPOINTS=true` | 200 | 200 | 200 | 400 (bad image) | 200 | * With both filters configured, `image_snapshot` messages on `edge-ai/site/gateway/camera/snapshots` and on a versioned normalizer topic were received and processed. A 7-level topic was not delivered. * This runtime pass predates the review follow-up commit, which changes the invalid-model `/startup` result to 503 and pins the v1 filter; both are covered by unit tests. * `kubectl kustomize charts` renders 4 resources, all valid under `kubeconform -strict`. * markdownlint, markdown-table-formatter, cspell, and markdown-link-check pass on the changed Markdown. gitleaks finds no leaks. ## Validation Steps 1. `cd src/500-application/507-ai-inference/services/ai-edge-inference && cargo test` 2. `cd ../ai-edge-inference-crate && cargo test --lib engine` 3. `kubectl kustomize src/500-application/507-ai-inference/charts` and confirm one `health` port, three probes, `ENABLE_TEST_ENDPOINTS: "false"`, and the `edge-ai/+/+/camera/snapshots,edge-ai/v1/+/camera/+/snapshots` input filters. ## Checklist - [x] I have updated the documentation accordingly - [x] I have added tests to cover my changes - [x] All new and existing tests passed - [ ] I have run `terraform fmt` on all Terraform code - [ ] I have run `terraform validate` on all Terraform code - [ ] I have run `az bicep format` on all Bicep code - [ ] I have run `az bicep build` to validate all Bicep code - [x] I have checked for any sensitive data/tokens that should not be committed - [x] Lint checks pass (run applicable linters for changed file types) ## Security Review - [x] No credentials, secrets, or tokens are hardcoded or logged - [x] RBAC and identity changes follow least-privilege principles - [x] No new network exposure or public endpoints introduced without justification - [x] Dependency additions or updates have been reviewed for known vulnerabilities - [x] Container image changes use pinned digests or SHA references ## Additional Notes Merge before #866, whose normalizer publishes to the pinned v1 topic this PR subscribes to. Out of scope and unchanged: * The Dockerfile `HEALTHCHECK` runs `ai-edge-mqtt-publisher --health-check`, but `main.rs` doesn't parse arguments, so the check starts a second service instance. Kubernetes ignores Docker `HEALTHCHECK`, so the probes above are what matter in the cluster. * The unused `metrics.rs` module is left in place for a future metrics listener. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Bill Berry <WilliamBerryiii@users.noreply.github.com>
Alain Uyidi (auyidi1)
added a commit
that referenced
this pull request
Oct 9, 2026
…#866) # Pull Request ## Description > **Depends on #867.** Merge #867 first: it makes the 507 inference service subscribe to the pinned `edge-ai/v1/+/camera/+/snapshots` filter this adapter publishes to. Adds the component 507 **snapshot normalizer**: an MQTT adapter that subscribes to a binary JPEG snapshot topic, such as a media connector `snapshot-to-mqtt` stream, and republishes each accepted snapshot as an `image_snapshot` v1 request for the 507 inference service. It's built on the transport-free `snapshot-normalizer-core` library from #831 and follows the custom workload messaging contract from #852. * **New crate `services/snapshot-normalizer`.** It's a binary crate that depends on the core library by path, so the core stays transport-free as its README promises. * **Versioned output topic.** The default is `edge-ai/v1/snapshot-normalizer/camera/{camera-id}/snapshots`, following the #852 `{domain}/{version}/{producer}/{resource-kind}/{resource-id}/{message-kind}` grammar pinned to `v1`. It matches 507's pinned `edge-ai/v1/+/camera/+/snapshots` filter, which #867 adds to the binary default and the shipped manifests alongside the legacy 5-level filter; #867 also rejects envelopes whose `schema_version` major isn't `1`. The adapter refuses to start when its output topic matches its own input filter, including through wildcards. * **CloudEvents attributes as MQTTv5 user properties** come from the SDK's validated `CloudEventBuilder`: `specversion`, `type`, `source`, `id`, `time`, `subject`, `datacontenttype` (also the MQTTv5 Content Type), an optional `dataschema`, and a propagated W3C `traceparent`. * **Producer-keyed deduplication (FR-01).** The key is the input's CloudEvents `id`, scoped to the source topic and `source`, and bounded by both entry count and time window. Content is never a key. A key is recorded only after the PUBACK. Output `id` and `correlation_id` are random UUIDv4 values, per #852, never derived from producer identifiers. * **Delivery (at least once).** Inputs are acknowledged manually after handling. Publishes get up to `PUBLISH_ATTEMPTS` (1-10) attempts with exponential backoff capped at 30 s; a broker failure reason is not retried. Failed publishes are counted and dropped rather than left unacknowledged, because the SDK delivers acks in receive order. Only an input left unacknowledged by a restart is redelivered, and it can then be published twice. * **Session.** The adapter fails at startup when the SUBACK carries a failure reason, so the pod crash-loops visibly. It advertises a receive maximum of 8 and a maximum packet size of `MAX_JPEG_BYTES` plus 64 KiB, and subscribes with retain handling "do not send" so retained snapshots aren't replayed. * **Payload-safe logging.** Logs contain counters, reasons, and lengths only; no payload, Base64 data, or producer identifiers. * **Helm chart `charts/snapshot-normalizer` (FR-05):** * A single-replica StatefulSet, so `AIO_MQTT_CLIENT_ID` (prefix plus `metadata.name` through the Downward API) is unique per release and stable at `<fullname>-0` across restarts, and the persistent session resumes. * `AIO_MQTT_SESSION_EXPIRY` set explicitly from `mqtt.sessionExpirySeconds` (3600 by default). * A dedicated service account without an automounted API token, a projected `aio-internal` SAT over TLS, and the `aio-broker-auth/workload` attribute annotation for least-privilege rules. * No ports or probes; a read-only root filesystem with dropped capabilities and `RuntimeDefault` seccomp. * A placeholder image registry. * **507 consumer fix.** `IncomingMessage::ImageSnapshot.metadata` now defaults when absent. The `image_snapshot` v1 schema makes `metadata` optional and the core omits it when empty, but the consumer rejected such messages with `missing field 'metadata'`. A regression test deserializes the schema examples and a normalizer-shaped envelope without `metadata`. * **Local compose.** The `snapshot-normalizer` service loads `services/snapshot-normalizer/.env.template`, then an optional git-ignored `.env`, instead of hard-coded values. The inference chart hardening items (FR-02 model paths and readiness, FR-03 probe-only listener, FR-04 metrics port) are left to a separate follow-up PR. ## Related Issue Follows up #831 (PCM-03 packaging) and applies the custom workload messaging contract from #852 (#835). ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Blueprint modification or addition - [x] Component modification or addition - [x] Documentation update - [x] CI/CD pipeline change - [ ] Other (please describe): ## Implementation Details * Uses `azure_iot_operations_mqtt` 1.2.3: `recv_manual_ack`, `TopicName::matches_topic_filter`, and `aio::cloud_event`. Other 507 and 50x crates are unchanged. * `config.rs` reads configuration through an injected lookup. It validates the camera ID as a lowercase URL-safe topic segment, the topics, the CloudEvents `source` and `dataschema`, and positive numeric bounds. * `pipeline.rs` is pure: `prepare()` returns `Rejected`, `Duplicate`, or `Publish` without I/O. `main.rs` only owns the session, acks, retries, counters, and SIGTERM handling. * The chart renders numeric values through `int64`, so large byte limits aren't emitted in float notation (for example `4.194304e+06`). * Registered the crate in both `rust-tests.yml` matrices. `codecov.yml` already covers `507-ai-inference/**`. * Made the existing root-absolute `CONTRIBUTING.md` link in the 507 README relative so the link checker resolves it. ## Testing Performed - [ ] Terraform plan/apply - [ ] Blueprint deployment test - [x] Unit tests - [x] Integration tests - [x] Bug fix includes regression test (see [Test Policy](docs/contributing/testing-validation.md)) - [x] Manual validation - [x] Other: container image build and smoke test * `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, and `cargo test` pass: 33 tests for the new crate, including SUBACK rejection, retain handling, capped backoff, the `PUBLISH_ATTEMPTS` bound, random v4 IDs, and the v1 topic and loop guard. The `ai-edge-inference` crate passes `cargo clippy --all-targets -- -D warnings` and its 8 tests with the `metadata` fix. * A local merge of this branch onto #867 is conflict-free and passes the combined `ai-edge-inference` tests, clippy, `docker compose config`, and `kubectl kustomize`. * **End-to-end against an MQTTv5 broker (Mosquitto):** * 5 inputs produced 3 published requests, 1 duplicate (same `id`), 1 `unkeyed`, and 1 `not_jpeg` rejection. * Output arrived on the inference service's versioned input filter with the expected CloudEvents properties. This end-to-end pass predates the review follow-up commit that moved the topic to `edge-ai/v1/snapshot-normalizer/camera/{camera-id}/snapshots`. * Producer `time` and `traceparent` were propagated. * A standalone serde probe of the 507 `IncomingMessage` confirmed that an envelope without `metadata` failed before the fix. * The release image built from `services/` runs as uid 1000 on a read-only root filesystem, published to the broker, and logged final counters on SIGTERM. * `helm lint --strict` and `helm template` with default and override values pass and render a ServiceAccount and a StatefulSet. * `Validate-RustCrateRegistration.ps1` passes (18 crates). Grype finds no vulnerabilities in the new `Cargo.lock`. gitleaks finds no leaks. * markdownlint, markdown-table-formatter, cspell, and markdown-link-check pass on the changed Markdown. ## Validation Steps 1. `cd src/500-application/507-ai-inference/services/snapshot-normalizer && cargo test && cargo clippy --all-targets -- -D warnings` 2. `helm template t src/500-application/507-ai-inference/charts/snapshot-normalizer` and confirm a `StatefulSet` with `replicas: 1`, `AIO_MQTT_CLIENT_ID` using `$(POD_NAME)`, and `AIO_MQTT_SESSION_EXPIRY: "3600"`. 3. Optionally run a local MQTTv5 broker and publish a JPEG with an `id` user property to `INPUT_TOPIC`, as described in the crate README. ## Checklist - [x] I have updated the documentation accordingly - [x] I have added tests to cover my changes - [x] All new and existing tests passed - [ ] I have run `terraform fmt` on all Terraform code - [ ] I have run `terraform validate` on all Terraform code - [ ] I have run `az bicep format` on all Bicep code - [ ] I have run `az bicep build` to validate all Bicep code - [x] I have checked for any sensitive data/tokens that should not be committed - [x] Lint checks pass (run applicable linters for changed file types) ## Security Review - [x] No credentials, secrets, or tokens are hardcoded or logged - [x] RBAC and identity changes follow least-privilege principles - [x] No new network exposure or public endpoints introduced without justification - [x] Dependency additions or updates have been reviewed for known vulnerabilities - [x] Container image changes use pinned digests or SHA references ## Additional Notes * No image is published. The chart uses `your-registry.azurecr.io/snapshot-normalizer` as a placeholder. * Microsoft Learn doesn't document whether the media connector sets a CloudEvents `id` on `snapshot-to-mqtt` messages. Without one, messages pass through un-deduplicated and are counted as `unkeyed`. * Deduplication state is in memory, which is why the chart runs a single-replica StatefulSet. Scaling to shared subscriptions would need shared dedup state, per the #852 guidance. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Bill Berry <WilliamBerryiii@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
snapshot-normalizer-core, a pure, transport-independent Rust library that builds the canonicalimage_snapshotv1 request value from raw JPEG bytes, and publishes the matching JSON Schema as the contract for that envelope.The crate is a library only. It is not a service, not a container, and not a deployable unit. Envelope construction is pure: no clock, no randomness, no I/O, no network, and no transport dependency. The same input always yields the same envelope.
MQTT, topic-derived identity, Helm packaging, and connector integration are explicitly out of scope for this change.
Related Issue
No tracking issue; opening as a standalone contribution. Happy to file one if the team prefers.
Type of Change
Implementation Details
Public surface:
is_jpeg,JPEG_SOIRejectReasonSizeLimitsencode_jpegbuild_envelope,SnapshotEnvelope,serialize_envelopeBoundedDeduppayload_hashCounters,CountersSnapshotDesign notes worth reviewer attention:
additionalProperties: trueand the type does not usedeny_unknown_fields, so adding a field later stays non-breaking. Strict validation lives in tests rather than the runtime parser. This matches the existingIncomingMessage::ImageSnapshotconsumer inai-edge-inference, which already accepts unknown fields.timestampis an integer epoch (i64), not RFC 3339, matching the existing consumer.locationis contract-only. It is accepted by consumers and documented in the schema as optional, but this v1 producer never emits it.SizeLimits, keeping the library deployment-neutral. The schema documents 4 MiB raw / 8 MiB envelope as recommended starting values, not enforced limits.Debugis hand-written on both payload-bearing types so it reports lengths rather than rendering raw or Base64 payload content.Dependencies are
base64,serde, andserde_jsononly - all crates.io, nounsafe, no feature flags.Testing Performed
70 unit tests, all synthetic hand-built byte arrays and neutral identifiers. No test requires a broker, cluster, GPU, camera, model, network, or secret.
Schema conformance was verified by generating real
serialize_envelopeoutput and validating it against the published schema under a draft 2020-12 validator, including negative controls: a missing required field, an RFC 3339 timestamp, a three-elementlocation, and a wrongmessage_typeare each correctly rejected, while a document carrying an unknown member is accepted.Validation Steps
Checklist
No Terraform or Bicep is touched by this change, so those checklist items do not apply.
Security Review
This change introduces no network surface, no credentials, and no container images.
cargo auditreports no advisories across the 13-package dependency closure. Payload bytes are never retained, logged, or rendered.This PR does not touch
SECURITY.md,src/000-cloud/010-security-identity/, ordeploy/, so thesecurity-reviewedlabel requirement should not apply.Additional Notes
The crate registration adds one entry to each of the two matrices in
.github/workflows/rust-tests.yml, keeping them 1:1 bynamesince the coverage job uploads an SBOM artifact that the vulnerability-scan job downloads by the same interpolation.codecov.ymlneeds no change:flags.rust.pathsalready coverssrc/500-application/507-ai-inference/**.Two known follow-ups, deliberately left out to keep this change focused:
serde_jsontypes appear in the public API without apub use serde_json;re-export, so consumers must take a semver-compatible direct dependency.RejectReason::Emptyis currently only reachable by a caller that checks emptiness itself.Happy to fold either into this PR if reviewers would prefer them here rather than as a follow-up.