Skip to content

Bug: Eager in-memory repository startup validates trusted embedded seed twice #1576

Closed

Description

@JoshuaRowePhantom

Summary

Creating a MainWindowViewModel currently starts repository creation immediately. For an in-memory repository, bootstrap loads the trusted embedded corpus through a stack that performs schema validation twice, so constructor-heavy, single-threaded tests repeatedly pay about 481 ms before their own work. The fix is not a process-wide snapshot: make broker creation lazy/single-flight, seed in-memory storage through a narrowly scoped trusted-embedded mode that retains merge and referential processing while bypassing only schema composition/evaluation, then discard every seed-time wrapper/accessor and expose a fresh normal validating pipeline.

Measured Cost Breakdown

Measured path / phase Median Current entities Interpretation
SchemaPopulator directly to InMemoryDataAccessLayer ~12 ms 156 Fast but incorrect; skips generated folders and other pipeline behavior.
SchemaAccessor registry build after raw seed ~13 ms 156 Registry construction alone is not the dominant cost.
Population through one validating layer ~247 ms 156 Schema composition/meta-schema evaluation dominates one validation pass.
Merge + referential processing with schema evaluation bypassed ~56 ms 176 Chosen trusted-seed path; retains the 20 generated folders.
Current production core pipeline ~481 ms 176 Two validation passes plus merge/referential work.
First validating write through a fresh post-seed wrapper ~11 ms — One-time lazy registry/accessor work remains acceptable after seed.
Subsequent cached validating write ~0.27 ms — Normal runtime writes benefit from existing cache behavior.

Within one validating layer, instrumentation attributed approximately 71% to schema-entity composition/meta-schema evaluation, 18.5% to ordinary entity evaluation, and 10.5% to other work.

Methodology and limitations

These are medians from an instrumented in-memory reproduction of the current code paths, not a machine-independent CI budget. Absolute timings will vary by runtime and machine; the relative attribution and entity-count difference are the actionable findings. The rows isolate different paths and therefore should not be treated as strictly additive. Entity counts are current stored entities after the measured population path, not update-operation counts. The measurements do not establish an infinite JSON-schema cycle and do not include all UI initialization or a complete test-suite run.

Root Cause

Eager broker creation

Phantom.Workspaces/ViewModels/MainWindowViewModel.cs, MainWindowViewModel.MainWindowViewModel (lines 111-128), starts work during construction:

this.entityBrokerTask = EntityBroker.CreateInitializedAsync(
    repositorySource,
    userComputerProfileOverride: configuration?.UserComputerProfileOverride);

InitializeAsync (lines 637-639) only awaits the already-running task. Constructor-only tests therefore create and seed a repository even when they never initialize the view model.

Phantom.Workspaces/EntityBroker.cs, CreateInitializedAsync (lines 90-101), immediately calls EntityRepository.CreateAsync, initializes the broker, and builds its interest catalog.

The local repository stack validates every population change twice

Phantom.Workspaces/EntityRepository.cs, CreateAsync (lines 43-76), currently composes this non-web stack (lines 56-60):

new MergeProcessingDataAccessLayer(
    new ReferentialIntegrityDataAccessLayer(
        new SchemaValidatingDataAccessLayer(underlyingDataAccessLayer, schemaAccessor),
        schemaAccessor));

ReferentialIntegrityDataAccessLayer inherits SchemaValidatingDataAccessLayer (Phantom.Workspaces.Data.Core/ReferentialIntegrityDataAccessLayer.cs, lines 11-29). Its UpdateAsync rewrites the request and then calls base.UpdateAsync (lines 32-53), which performs one schema-validation pass. That base call forwards to the nested SchemaValidatingDataAccessLayer, which performs the same validation path again.

SchemaValidatingDataAccessLayer.UpdateAsync (Phantom.Workspaces.Data.Core/SchemaValidatingDataAccessLayer.cs, lines 38-101) builds/uses a registry, checks entity-type discriminators, and invokes ValidateChangeAsync for every change. ValidateChangeAsync (lines 135-225) resolves applicable schemas, composes a schema, and evaluates the entity. This is correct for runtime writes but unnecessarily repeated for trusted embedded bootstrap data.

SchemaAccessor.BuildSchemaRegistryAsync (Phantom.Workspaces.Data.Core/SchemaAccessor.cs, lines 56-82) caches its registry; InvalidateOnSchemaUpdate (lines 86-100) clears all caches after a schema write. During bootstrap, however, the accessor is created before the lower store contains the embedded schemas and is reused after population. A runtime accessor must instead be constructed only after seed processing completes.

Population is one large pipeline update

SchemaPopulator.Populate (Phantom.Workspaces.Data.Core/SchemaPopulator.cs, lines 20-45) loads the embedded JsonEntities, applies current concurrency tags, and sends one UpdateRequest. LoadEntityChanges (lines 85-139) materializes the embedded corpus. EntityRepository.EnsureSeedDataIfNeededAsync (Phantom.Workspaces/EntityRepository.cs, lines 204-215) sends that request through the stack above on every fresh in-memory repository.

Why Direct Raw Population Is Incorrect

Sending SchemaPopulator directly to the raw in-memory DAL produces 156 entities, while merge/referential processing produces 176. The missing 20 are generated folder entities. In ReferentialIntegrityDataAccessLayer.BuildRewrittenChangesAsync (lines 55-147), the layer manages reference relationships, applies folder-prefix changes via ApplyFolderPrefixEntityChangesAsync (lines 428-553), coalesces relationships, and validates references (beginning at line 555). Bypassing that layer changes repository contents and omits correctness checks.

Accordingly, fast seed mode must still execute:

  • merge processing;
  • referential-integrity rewriting and reference validation;
  • discriminator checks;
  • generated folder creation; and
  • the normal raw in-memory write.

Only expensive schema composition/evaluation for the trusted embedded seed may be bypassed.

Affected Files

Path Change
Phantom.Workspaces/ViewModels/MainWindowViewModel.cs Replace constructor-started entityBrokerTask with lazy, single-flight factory invocation in InitializeAsync.
Phantom.Workspaces/ViewModels/MainWindowViewModelInitializationOptions.cs Add the internal required-init property record and optional broker factory seam.
Phantom.Workspaces/EntityBroker.cs Add an internal initialization overload that accepts repository initialization options; preserve the public API as a full-validation delegator.
Phantom.Workspaces/EntityRepository.cs Split seed-time and runtime pipeline construction, remove nested validation, enforce the in-memory-only fast-mode guard, and rebuild fresh wrappers after seed.
Phantom.Workspaces/EntityRepositoryInitializationOptions.cs Add the internal required-init property record and EmbeddedSeedValidationMode enum, defaulting to Full.
Phantom.Workspaces.Data.Core/ReferentialIntegrityDataAccessLayer.cs Reuse its existing rewrite/reference/discriminator behavior and inherited validation seam; no public bypass is added.
Phantom.Workspaces.Data.Core/SchemaValidatingDataAccessLayer.cs Reuse the protected ValidateChangeAsync seam for the private trusted-seed specialization; normal runtime behavior stays unchanged.
Phantom.Workspaces.Tests/EntityRepositoryTests.cs Add seed-equivalence, post-seed validation/cache, source guard, and session-bootstrap coverage.
Phantom.Workspaces.Tests/MainWindowViewModelTests.cs Add constructor laziness and concurrent single-flight coverage.
Phantom.Workspaces.Data.Core.Tests/SchemaPopulatorTests.cs Add exhaustive embedded-schema parsing/meta-validation, explicitly covering ancestor.json.

Design / Fix

1. Use required-init option records

Add these internal shapes (names may be placed in files matching current namespace conventions, but do not turn the bypass into public API):

internal sealed record MainWindowViewModelInitializationOptions
{
    public required RepositorySource RepositorySource { get; init; }
    public WorkspacesConfiguration? Configuration { get; init; }
    public ProfileStore? ProfileStore { get; init; }
    public ApplicationServices? ApplicationServices { get; init; }
    public Func<CancellationToken, Task<EntityBroker>>? EntityBrokerFactory { get; init; }
}

internal enum EmbeddedSeedValidationMode
{
    Full,
    TrustedInMemoryFast,
}

internal sealed record EntityRepositoryInitializationOptions
{
    public required RepositorySource RepositorySource { get; init; }
    public string? UserComputerProfileOverride { get; init; }
    public EmbeddedSeedValidationMode EmbeddedSeedValidationMode { get; init; } = EmbeddedSeedValidationMode.Full;
}

Keep existing public constructors/factories as compatibility delegators using named initialization. Their repository default remains Full. The default MainWindowViewModel broker factory may explicitly choose TrustedInMemoryFast only for UnknownRepositorySource, which is the current in-memory source; all other sources use Full.

2. Make MainWindowViewModel initialization lazy and single-flight

The constructor stores the options and factory but does not invoke it. Replace the non-null constructor-created task with a nullable cached task guarded by a lock (or an equivalent Lazy<Task<EntityBroker>>). InitializeAsync calls a private GetOrCreateEntityBrokerTask and awaits the one shared task. Concurrent callers must receive the same operation and invoke EntityBrokerFactory exactly once. The internal EntityBroker.CreateInitializedAsync(EntityRepositoryInitializationOptions, CancellationToken) overload feeds the selected mode to the repository; the existing public overload delegates with Full.

3. Create separate seed and runtime pipelines in exact order

Refactor EntityRepository.CreateAsync into small private helpers such as CreateSeedDataAccessLayer and CreateRuntimeDataAccessLayer, with this ordering:

  1. Validate options first. TrustedInMemoryFast with anything other than the in-memory repository source throws before source creation, migration, or population.
  2. Create the underlying DAL. Web/dev-tunnel behavior remains unchanged.
  3. For local persistent stores, ensure indexes and migrations exactly as today.
  4. Create a seed-only SchemaAccessor over the underlying DAL.
  5. Build the seed stack as MergeProcessingDataAccessLayer -> ReferentialIntegrityDataAccessLayer -> underlying DAL; do not insert another SchemaValidatingDataAccessLayer, because referential integrity already inherits it.
  6. In Full, use the normal referential layer. In TrustedInMemoryFast, use a private/internal sealed specialization (for example TrustedEmbeddedSeedReferentialIntegrityDataAccessLayer) whose constructor requires InMemoryDataAccessLayer and whose only behavioral override is ValidateChangeAsync, returning no schema-composition/evaluation errors for this seed call. Inherited discriminator checks and all referential rewriting/checks remain active.
  7. Run SchemaPopulator.Populate through that seed stack and fail on any returned merge/referential/discriminator error.
  8. Drop every reference to seed-time wrappers and their accessor.
  9. Construct a fresh SchemaAccessor over the now-populated underlying DAL and build the normal runtime stack as MergeProcessingDataAccessLayer -> ReferentialIntegrityDataAccessLayer -> underlying DAL.
  10. Wrap that runtime stack in ScheduleDataAccessLayer, then run WorkspaceEntitySessionBootstrapper.InitializeAsync through it (Phantom.Workspaces/WorkspaceEntitySessionBootstrapper.cs, lines 10-45). All discovery and ViewModel-facing writes therefore use normal validation.

Never construct or prime the runtime accessor before lower-layer seed writes: such writes cannot invalidate an accessor they bypass. The fresh post-seed accessor must discover stored schemas on its first runtime write, then cache and invalidate exactly as it does today.

4. Remove the redundant nested validator everywhere in the production composition

The normal runtime stack needs one validation pass: the pass inherited by ReferentialIntegrityDataAccessLayer. Remove new SchemaValidatingDataAccessLayer(...) from inside it. Continue sharing one accessor with the referential layer so schema resolution, discriminator checks, and cache invalidation stay coherent.

Existing Focused Validation Coverage

Primary coverage already exists and should remain the baseline rather than replacing it with broad timing tests:

  • SchemaPopulatorTests.Populate_LoadsEmbeddedEntities_IntoInMemoryStore, Populate_CreatesExpectedNumberOfDistinctEntities, Populate_WhenRunTwiceThroughRepositoryPipeline_IsIdempotent, and Populate_WhenSeedEntityDiffers_UsesConcurrencyTagAndRestoresCanonicalData cover population and canonical repair.
  • ReferentialIntegrityDataAccessLayerTests.CreateEntity_WithMultiComponentName_CreatesFolderPrefixes, TypedEntityReferences_RequireExistingEntityAndMatchingType, and AddEntityAndRelationship_WithMissingReferencedEntities_Fails cover generated folders and references.
  • SchemaValidatingDataAccessLayerTests.UpdateAsync_BuildsSchemaRegistryOnce_AcrossMultipleNonSchemaUpdates, UpdateAsync_InvalidatesCache_WhenJsonSchemaEntityIsWritten, and UpdateAsync_CachedRegistry_ValidatesEntityCorrectly cover registry caching, invalidation, and enforcement.
  • SchemaAccessorTests.BuildSchemaRegistryAsync_ReturnsCachedRegistryInstance and BuildSchemaRegistryAsync_IsSafeUnderConcurrentAccessAcrossInstances cover accessor caching/concurrency.
  • EntityRepositoryTests.CreateAsync_InitializesWorkspaceEntitySessionDiscoveryEntities already checks discovery, but it must exercise the fast path after this change.

Security / Correctness Invariants

  • No public or general-purpose SkipValidation flag is introduced.
  • Fast mode accepts only the compiled-in embedded corpus and only an in-memory repository source; misuse fails closed before any I/O or seed mutation.
  • Merge processing, reference/discriminator validation, managed relationships, and all 20 generated folders remain active during fast seed.
  • Seed-time wrappers/accessors are never exposed to WorkspaceEntitySessionBootstrapper, EntityRepository.DataAccessLayer, EntityBroker, or a ViewModel.
  • The runtime pipeline has exactly one normal schema-validation pass and a fresh accessor constructed after seed completion.
  • Valid post-seed writes succeed; invalid post-seed writes fail. Schema writes invalidate and rebuild runtime accessor state.
  • No mutable DAL, accessor, schema registry, repository session, or entity collection is shared process-wide.
  • Cancellation/failure of the single-flight broker task is observed consistently by all concurrent InitializeAsync callers; initialization cannot run twice.
  • Full mode remains available as the comparison/reference path and the default for existing repository APIs and all non-in-memory sources.

Considered / Background

  • Process-wide cloned snapshot: previously recommended, but not required for this fix. The measured trusted merge/referential seed path is about 56 ms and preserves generated data without introducing shared canonical state, clone semantics, or cross-repository isolation risk. An immutable validated snapshot may be considered later as an optional optimization only if measurements still justify it; it must include derived folders and must not share accessors/registries or mutable repository state.
  • Direct raw DAL seeding: rejected because it produces 156 rather than 176 entities and omits 20 generated folders plus referential behavior.
  • Infinite/cyclic JSON-schema evaluation: unsupported and not chosen. Earlier stack samples only showed active schema evaluation while the suite continued progressing; they did not prove a stable cycle or deadlock. No cycle-rejection feature or global registry workaround belongs in this fix.
  • Suite context: the affected test assembly intentionally uses one worker, contains roughly 2,410 tests, and has hundreds of MainWindowViewModel construction sites. A prior observed run continued to 466 passing tests before being stopped. This explains the amplified throughput/observability impact but is not itself the production defect.
  • Runner progress diagnostics: useful but orthogonal. Streaming progress can be handled separately and is not a prerequisite for the repository fix.

Expected Tests

Test Name Class Required assertions
CreateAsync_TrustedInMemoryFastSeed_MatchesFullSeedEntityIdsAndData EntityRepositoryTests Full and fast modes produce identical current entity-ID sets and JSON data; the processed corpus includes exactly 20 generated folder entities beyond the 156 embedded entities.
CreateAsync_TrustedInMemoryFastSeed_FreshRuntimeValidationAcceptsValidAndRejectsInvalidUpdates EntityRepositoryTests A valid post-seed update succeeds and a schema-invalid update fails through the fresh runtime wrapper.
CreateAsync_TrustedInMemoryFastSeed_PostSeedSchemaUpdateRebuildsRuntimeRegistry EntityRepositoryTests Warm the fresh runtime accessor, write a schema change, then prove the next write rebuilds/uses the updated schema rather than stale cached state.
CreateAsync_TrustedInMemoryFastSeed_InitializesWorkspaceEntitySessionDiscoveryEntities EntityRepositoryTests User, computer, and profile discovery entities are present and their IDs populate WorkspaceEntitySession.
CreateAsync_TrustedInMemoryFastSeedWithNonInMemorySource_ThrowsInvalidOperationException EntityRepositoryTests Every non-in-memory source is rejected before DAL creation/population; no fallback silently bypasses validation.
MainWindowViewModel_Constructor_DoesNotInvokeEntityBrokerFactory MainWindowViewModelTests Construction leaves the injected factory invocation count at zero.
MainWindowViewModel_InitializeAsync_WhenCalledConcurrently_InvokesEntityBrokerFactoryOnce MainWindowViewModelTests Concurrent calls share one task, one broker result, and one factory invocation.
EmbeddedSchemas_EveryResource_ParsesAndPassesDraft202012MetaSchemaValidation SchemaPopulatorTests Enumerate every embedded JsonSchemas/*.json resource, assert the set explicitly includes ancestor.json, build with the repository dialect/registry, and meta-validate every schema with resource-specific failure output.

Avoid a tight elapsed-time assertion. The entity-equivalence and factory-count tests are deterministic contracts; the measured medians are diagnostic evidence.

Commit Plan

  1. Repository seed/runtime composition — change EntityRepository and add EntityRepositoryInitializationOptions/EmbeddedSeedValidationMode; remove the nested validator, add the in-memory-only trusted seed specialization, rebuild fresh runtime wrappers, and add the focused EntityRepositoryTests plus exhaustive SchemaPopulatorTests coverage above.
  2. Lazy ViewModel broker initialization — change MainWindowViewModel and EntityBroker, add MainWindowViewModelInitializationOptions, and add the constructor/no-call and concurrent single-flight MainWindowViewModelTests.

Each commit is independently reviewable, changes at most three production types, and carries the focused tests for its behavior.

Activity

  1. changed the title [-]Bug: Phantom.Workspaces.Tests can hang in Json.Schema seed validation[/-] [+]Bug: Serialized test suite slowed by repeated eager repository/schema initialization[/+] on Sep 14, 2026
  2. changed the title [-]Bug: Serialized test suite slowed by repeated eager repository/schema initialization[/-] [+]Bug: Eager in-memory repository startup validates trusted embedded seed twice[/+] on Sep 14, 2026
  3. JoshuaRowePhantom commented on Sep 16, 2026

    @JoshuaRowePhantom
    OwnerAuthor

    Design

    The duplicated work is caused by composition, not by a schema cycle: MergeProcessingDataAccessLayer -> ReferentialIntegrityDataAccessLayer already owns schema validation through inheritance, but repository and web-server startup place a second SchemaValidatingDataAccessLayer beneath it. I will establish one composition factory whose single validation owner is the referential layer, and route eager in-memory startup, server startup, and focused test pipelines through it. Embedded seed data will remain a distinct trusted source contract, but it will still pass through that authoritative validator once; no skip-validation flag or cache/snapshot sharing will be introduced.

    Tests will be written first and use per-pipeline instrumentation (not elapsed time) to prove one seed validation pass, one pass for ordinary/malformed external writes, per-instance behavior under concurrent startup, and cancellation/failure without publication. Seed-loading tests will cover malformed resources, schema/dialect mismatches, resource disposal, and the prior ancestor.json path. Cancellation will be threaded through repository creation/population so it is neither delayed nor masked. Merge processing, referential rewriting, generated folders, migrations, runtime cache invalidation, and eager ViewModel behavior remain unchanged.

    Expected production files: SchemaValidatingDataAccessLayer.cs, a central validated-pipeline factory/typed seed contract in Data.Core, SchemaPopulator.cs, EntityRepository.cs, EntityBroker.cs, and WebServerDataAccessLayerFactory.cs; focused tests in the corresponding Data.Core, repository, and web-server test projects.

  4. JoshuaRowePhantom commented on Sep 16, 2026

    @JoshuaRowePhantom
    OwnerAuthor

    Implementation notes

    Commit: 068406a93b06736e933bbc53f60a7ce8689b85ed

    Deviations from proposed design

    The diagnosis was narrowed to the requested release-blocker scope: eager startup remains eager, while the redundant nested validator is removed. No validation bypass or fast-mode flag was added. A central ValidatedDataAccessLayerFactory makes ReferentialIntegrityDataAccessLayer the sole schema-validation owner; a typed internal validation-pass kind identifies trusted embedded population for deterministic instrumentation only and grants no reduced validation behavior.

    Scope changes

    • Propagated cancellation through repository creation, embedded loading, validation, and web-server population.
    • Replaced duplicate-validator composition in production and focused test pipelines.
    • Added exhaustive embedded schema Draft 2020-12/meta-schema coverage including ancestor.json; malformed trusted/external, detached/disposed JSON, concurrent-instance, cancellation, atomic-publication, and exact-count coverage.
    • Targeted tests: 42/42 passed; targeted stability: 10/10 runs passed; fast suite: 7,411/7,411 passed.
    • Full suite executed three times. Each reached 7,642/7,643 and failed only the unrelated existing Release gate: policy envelope owner is not pinned to current user #1578 PolicyEnvelope_ConcurrentCreation_RestrictsAndConsumesEveryEnvelope ACL race; its isolated reruns pass 18/18. No Bug: Eager in-memory repository startup validates trusted embedded seed twice #1576 test failed and no push occurred.

    Files changed

    • Phantom.Workspaces.Data.Core.Tests/AgentManifestAgentDefinitionToolSchemaTests.cs
    • Phantom.Workspaces.Data.Core.Tests/AgentOptionsDocumentationTests.cs
    • Phantom.Workspaces.Data.Core.Tests/AzureDevOpsEntitySchemaTests.cs
    • Phantom.Workspaces.Data.Core.Tests/BaseEntitySchemaTests.cs
    • Phantom.Workspaces.Data.Core.Tests/DocumentationEntitySchemaTests.cs
    • Phantom.Workspaces.Data.Core.Tests/EntitySchemaComposerTests.cs
    • Phantom.Workspaces.Data.Core.Tests/FieldTypeResolverTests.cs
    • Phantom.Workspaces.Data.Core.Tests/GitEntitySchemaTests.cs
    • Phantom.Workspaces.Data.Core.Tests/GitHubEntitySchemaTests.cs
    • Phantom.Workspaces.Data.Core.Tests/SchemaAccessorTests.cs
    • Phantom.Workspaces.Data.Core.Tests/SchemaPopulatorTests.cs
    • Phantom.Workspaces.Data.Core/IDataAccessLayer.cs
    • Phantom.Workspaces.Data.Core/Phantom.Workspaces.Data.Core.csproj
    • Phantom.Workspaces.Data.Core/ReferentialIntegrityDataAccessLayer.cs
    • Phantom.Workspaces.Data.Core/SchemaPopulator.cs
    • Phantom.Workspaces.Data.Core/SchemaValidatingDataAccessLayer.cs
    • Phantom.Workspaces.Data.Core/ValidatedDataAccessLayerFactory.cs
    • Phantom.Workspaces.Data.Web.Server.Tests/WebServerDataAccessLayerFactoryTests.cs
    • Phantom.Workspaces.Data.Web.Server/Phantom.Workspaces.Data.Web.Server.csproj
    • Phantom.Workspaces.Data.Web.Server/WebServerDataAccessLayerFactory.cs
    • Phantom.Workspaces.Tests/EntityRepositoryTests.cs
    • Phantom.Workspaces.Tests/GitWorkspaceUpdateToolTests.cs
    • Phantom.Workspaces.Tests/JsonValidationViewModelTests.cs
    • Phantom.Workspaces/EntityBroker.cs
    • Phantom.Workspaces/EntityRepository.cs
    • Phantom.Workspaces/EntityRepositoryInitializationOptions.cs
  5. JoshuaRowePhantom commented on Sep 16, 2026

    @JoshuaRowePhantom
    OwnerAuthor

    Verification result

    Status: ✅ Verified locally — independent, fresh verification (no reliance on implementation notes)
    Commit checked: 068406a9 Fix #1576: validate embedded seed once
    Branch/worktree: fix/1576-single-seed-validation (worktree 9), full 26-file diff traced end-to-end.

    Scope reconciliation (design deviation is real and intentional)

    The delivered fix does not use the issue's proposed TrustedInMemoryFast bypass mode or lazy/single-flight broker creation. Instead it removes the redundant nested SchemaValidatingDataAccessLayer so ReferentialIntegrityDataAccessLayer (which inherits schema validation) is the single validation owner. This eliminates the double pass while keeping eager startup and one full validation. The deviation is transparently documented in the implementation comment. The core release-blocker (embedded seed validated twice) is fixed. Because there is no bypass mode and no laziness, the issue's CreateAsync_TrustedInMemoryFast*, non-in-memory source-guard, and MainWindowViewModel_* laziness tests are intentionally absent and moot; the laziness optimization is deferred (worth a follow-up if still desired).

    Single-validation invariant — verified by data-flow trace

    • Production composition: EntityRepository.CreateAsync (non-web) and WebServerDataAccessLayerFactory.CreateDefaultAsync both route through ValidatedDataAccessLayerFactory.Create, which composes MergeProcessing → ReferentialIntegrity(underlying, sharedAccessor) → raw store. The old nested SchemaValidatingDataAccessLayer(underlying, accessor) is gone. git grep confirms the only production construction of the validating layers is inside the factory.
    • Seed pass: SchemaPopulator.PopulateAsync sends one UpdateRequest; it traverses the single validating owner once. The referential layer rewrites first (generated folders), then the inherited SchemaValidatingDataAccessLayer.UpdateAsync validates the rewritten set (incl. the 20 folders) once, then forwards to the raw store. Registry visibility for in-request schemas is via the pre-existing CreateOverlayForRequest.
    • Accessor coherence (no fresh post-seed accessor needed): after any schema-bearing write, if (requestHasSchemas) { _schemaAccessor.InvalidateOnSchemaUpdate(); _cachedRegisteredEntityTypeNames = null; } runs, so the reused accessor rebuilds from the populated store on the first runtime write. Proven end-to-end by CreateAsync_InitializesWorkspaceEntitySessionDiscoveryEntities and the post-seed upsert tests passing.

    Security / correctness invariants — all hold

    • No caller-controlled bypass / no alreadyValidated semantics. The pass kind is instrumentation only: SchemaValidatingDataAccessLayer.UpdateAsync fires the validationPassStarted callback then runs the full validation unconditionally — no branch on the kind reduces validation. Confirmed by reading the method body and by grep (kind is only set in SchemaPopulator and asserted in tests).
    • Trusted label is unforgeable / tightly internal. UpdateMetadata.ValidationPassKind is internal + [JsonIgnore] (defaults to ExternalUpdate), so JSON/web clients cannot set it; SchemaValidationPass(Kind), the enum, and the callback overloads are internal, exposed via InternalsVisibleTo only to first-party solution assemblies. Even a forged TrustedEmbeddedSeed is fully validated and rejected — ValidatedPipeline_MalformedTrustedSeed_ValidatesOnceAndDoesNotPublish.
    • No weakening for external/user/web/server seeds. Malformed external and malformed trusted-seed writes both fail and are not published; web-server startup uses the same single validator; web/dev-tunnel client sources add no local seed (unchanged).
    • Decorators forward trust; no type confusion. MergeProcessingDataAccessLayer forwards UpdateMetadata unchanged (empirically confirmed — the Assert.Single(TrustedEmbeddedSeed) seed assertions pass through the full MergeProcessing → Referential stack). SchemaValidationPass is a readonly record struct; factory returns IDataAccessLayer; no unsafe casts.
    • Migration/ordering, partial publication, concurrency, cancellation/disposal, exception preservation, cross-repo isolation: MongoDB EnsureIndexes/Migrate still run before seed; the seed is one atomic request (malformed → nothing published; pre-cancel → nothing entered/published); each CreateAsync builds a fresh store+accessor+pipeline (CreateAsync_ConcurrentInMemoryInstances_ValidateSeedsIndependently asserts NotSame + independent single passes); cancellation threaded through create/load/validate/populate with OperationCanceledException propagated; seed failures surface as InvalidOperationException; disposed/detached source document still validates (ValidatedPipeline_DisposedSourceDocument_ExternalUpdateValidatesOnce).
    • Public compatibility: public ValidatedDataAccessLayerFactory.Create(underlying), EntityBroker.CreateInitializedAsync, and EntityRepository.CreateAsync(RepositorySource,…) preserved as delegators; only internal overloads add the instrumentation seam.

    Tests count actual validation and reproduce the double-spin — not timing

    • Deterministic reproduction of the prior double-spin: WebServerDataAccessLayerFactoryTests.CreateDefaultAsync_ComposesServerValidationPipeline asserts MergeProcessing → ReferentialIntegrity → InMemory (the referential layer's underlying is the raw store, not a nested validator) — this fails if the second validator is reintroduced. SchemaPopulatorTests.Populate_ValidatedPipeline_PerformsExactlyOneTrustedSeedValidationPass, EntityRepositoryTests.CreateAsync_InMemoryEagerStartup_ValidatesEmbeddedSeedExactlyOnce, and WebServer…ValidatesEmbeddedSeedExactlyOnce assert exactly one instrumented pass; Populate_LoadsEmbeddedEntities_IntoInMemoryStore asserts CountingDataAccessLayer.UpdateCallCount == 1.
    • Test churn did not bypass validation: the ~11 churned helpers changed a double-validating composition to ValidatedDataAccessLayerFactory.Create(...) / bare ReferentialIntegrityDataAccessLayer(...) — both still validate once and still assert Assert.Empty(errors). No constructor swap removed validation.
    • Embedded-schema coverage: EmbeddedSchemas_EveryResource_ParsesAndPassesDraft202012MetaSchemaValidation enumerates every JsonSchemas/*.json, explicitly asserts ancestor.json, builds with the repo dialect/registry, and meta-validates each with per-resource failure output.
    • No quarantined/timing/sync-blocking tests: no [Skip], Task.Delay, Thread.Sleep, fixed timeouts, .Result/.Wait()/GetResult(), or fixed dispatcher pumping in any changed file (CS0618 suppression is only for expensive ExportAsync enumeration, unrelated to validation).

    CODE-VERIFICATION criteria

    Feature coverage ✅ (single-authoritative-validation implemented) · described/implied cases for the delivered design ✅ · non-trivial logic & branch coverage (both pass kinds, cancel, malformed×2, concurrent, disposed) ✅ · no disabled tests ✅ · no unbacked TODOs ✅ · no timing-dependent tests ✅ · no fixed dispatcher pumping ✅ · no UI-thread/lock sync-blocking ✅ · data-flow trace ✅ (produced→validated-once→persisted→reloaded→forwarded). Note: issue-listed laziness/fast-mode tests are N/A to the delivered design (scope deviation above).

    Tests run (this verification)

    Targeted union (-Mode fast, 17 classes: SchemaPopulatorTests, EntityRepositoryTests, WebServerDataAccessLayerFactoryTests, SchemaValidatingDataAccessLayerTests, ReferentialIntegrityDataAccessLayerTests, SchemaAccessorTests, BaseEntitySchemaTests, EntitySchemaComposerTests, FieldTypeResolverTests, GitEntitySchemaTests, GitHubEntitySchemaTests, AzureDevOpsEntitySchemaTests, DocumentationEntitySchemaTests, AgentManifestAgentDefinitionToolSchemaTests, AgentOptionsDocumentationTests, GitWorkspaceUpdateToolTests, JsonValidationViewModelTests):

    Executed : 237 | Passed : 237 | Failed : 0
    

    10-run stability (same union, -NoBuild): 10/10 runs → 237/237, Failed: 0 each.

    Conclusion: Verified locally (issues: 0). No validation weakening, no forgeable trust, no bypass, no failing/flaky tests. One documented, non-blocking scope deviation: the lazy/single-flight broker optimization from the issue was deferred; eager startup is retained with a single authoritative validation pass.

  6. added a commit that references this issue on Sep 16, 2026
    068406a
  7. JoshuaRowePhantom commented on Sep 16, 2026

    @JoshuaRowePhantom
    OwnerAuthor

    Checked in

    Commit: 068406a
    Branch fast-forwarded: features → 068406a

  8. removed
    queuedIn the active work queue (tracked in work-queue.md)
    on Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't workingdiagnosedRoot cause identifiedverified-locallyImplementation has been verified locally

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions