Darling and Lite: a daily collector runs at a time of day you set, in every surface, and a schedule Reset clears it (#4938)

[erikdarlingdata]

Closes #4938.

A daily collector can have a run time, a fixed time of day such as 02:00 on the monitored server's clock. The table that holds it, config.config_collector_run_times (V160), the shared run-time rules, and the detached daily runs with their fleet-wide cap (#4999) are already in dev. This change moves every surface onto that table. It makes the worker and Lite start a daily collector at its slot, and it adds one rule for Reset.

What changes

Every Darling surface below reads or writes config.config_collector_run_times. None of them puts a run time on a schedule row. The viewer's schedule Save deletes a scope's rows and inserts them again with a fixed column list. A viewer that does not know such a column clears it on every Save.

• The viewer's Collector Schedules window. The window has a Run at column (Use default, None on a server row, or a HH:MM time). A line says what the time means in server time and in UTC, and a server row also gets a next-run line. The window reads the table next to the schedule rows. A store below V160 has no table. The read answers that with no run times, so the window still opens.
• One-transaction Save. SaveCollectorScheduleAsync writes the run-time changes (one statement each) and then the scope's schedule rows on one connection in one transaction. A schedule write that fails leaves the run time unsaved, and a run-time write that fails leaves the schedule rows as they were. A Save that changed no run time sends no run-time statement, so it still saves its schedules on a store below V160. The schedule statements are the ones a released viewer runs and never name the run time.
• --set-collector-run-at. The service's CLI verb sets a time (HH:MM), stops a fleet-wide time on one server (none), or clears the row (default). It works fleet-wide or with --server. It writes only the run time and keeps the enabled flag, cadence and retention on the row. The service's command executor writes the same table for the matching command.
• The MCP and web health reads. get_collection_health shows each collector's run time and next run, and a note when a day was skipped. The web Collection Health tab shows the run time beside Every (min). Its Collectors table also has Run at, Next run and Run time columns.
• The Darling worker. A daily collector that has a run time starts at its slot. The slot is the run time plus a spread of under 60 minutes taken from the server id. The pass hands the collector off from the slot until 60 minutes after it. A day it cannot hand off in that hour is skipped, not replayed. The skip counts as a skipped slot only when the sweep loop was running through it. The worker holds each server's clock in memory. It reads the clock once at connect, and again after a run of server_properties (SQL Server) or pg_server_config (PostgreSQL), never on a tick. A slot is computed again only when the run time, the interval or the clock changes. The detached run, its cap, its permits and its watch are Every daily collector runs detached from its server's pass, behind a fleet-wide cap sized to the store's pool (#4938) #4999's. This change only decides when a collector with a run time is handed to them.
• Lite. The schedule file takes an optional run_at (HH:MM on the monitored server's clock). The schedule editor sets it, checks it and shows the next run. The sweep starts a collector with a run time once a day, from its slot until 60 minutes after it. A missed day is skipped. At start-up Lite reads each daily collector's last run from collection_log. It also reads the server's clock from the newest server_properties row. A restart or a tab open then no longer re-runs a daily collector that ran today. A failed attempt of a collector that runs once a day or less often ends its period, with or without a run time. Lite's get_collection_health carries run_at, next_run_utc and run_time_note for its own schedule, so the two products answer alike.
• The docs. Darling/README.md has what the run time sets, what stays fixed, and the --set-collector-run-at verb. docs/how-collection-works.md has a section, "When the daily collectors run". Its one sentence on detached runs gives Every daily collector runs detached from its server's pass, behind a fleet-wide cap sized to the store's pool (#4938) #4999's cap. The cap is the smaller of 16 and what the store pool leaves after the sweep and 8 kept free for reads. It is never below 1. The top-level README.md gets a line on run_at.
• The Reset rule. A schedule Reset also deletes that scope's run times, in the same transaction as the schedule rows. Reset means back to the shipped defaults, and they have no fixed time. Three paths in the viewer reset schedule rows, and each follows the rule:
  • "Reset to Defaults" in the window. Before this change the Save deleted only the run-time rows the window had loaded for collectors this build defines. Now the window remembers the Reset, and its Save passes clearScopeRunTimes. The scope's run-time rows are deleted by scope, first, in the Save's transaction. That covers a row for a collector this build does not define and a row another writer added while the window was open. A time typed into the grid after the Reset is inserted after the clear and kept.
  • A server on "Use default schedule". Same call, same clear.
  • "Apply Default to All Servers". ResetAllServerSchedulesAsync is now ONE call. In one transaction it deletes every server's schedule rows and then every server's run times. It keeps the fleet's rows in both tables. A failure in the run-time delete takes the schedule delete back with it. The 42501 error still maps to the read-only message, and 42P01 or 42703 to the schema-skew message. It returns the rows removed from each table, counted apart, and the status line names each. ResetAllServerRunTimesAsync, the separate statement the window used to call after it, is gone.
  • On a store below V160 the run-time half is skipped, so a Reset still works there. The table is looked up by name first, because a failed statement aborts the transaction.
  • No other code path resets schedule rows. The search covered the viewer, the service CLI and command executor, the MCP tools and the web files. It looked for a DELETE or TRUNCATE on config_collector_schedules and for reset verbs. Only the viewer's data service deletes them. The service's --enable-collector and --disable-collector verbs and its enable and disable commands upsert one enabled value. Lite keeps its schedule in its own file, not in this store.
  • A viewer released before the table existed resets the schedule rows only. It never touches the run-time table. So a run time set by this viewer survives a Reset done in an older one.
• The serial-loop census. StartupCommandTimeoutTests now lists ReadNotificationRoutesAsync, which the service's config load already runs on the serial loop. 11 members hold 12 commands, and 9 of the 12 also run once at start-up. The prose in ServiceCommandDeadlines.cs says the same.
• Three names that Every daily collector runs detached from its server's pass, behind a fleet-wide cap sized to the store's pool (#4938) #4999 left saying "daily". _dailyRuns, TrackDailyRun and InFlightDailyRuns hold every detached run, the daily ones and the three detached by name. They are now _detachedRuns, TrackDetachedRun and InFlightDetachedRuns, with the tests and comments that name them. Behavior does not change.

Test plan

Each RED below was run against a live store, with the code as it stood before the Reset change.

• Apply Default to All Servers removes both tables' server rows in one call. ApplyDefaultToAllServers_RemovesEveryServersScheduleRowsAndRunTimes_AndKeepsTheFleets. RED: Expected: 5, Actual: 2. GREEN after.
• A failure after the schedule delete leaves the schedule rows and the run times in place. A trigger refuses the run-time delete, and the same call without the trigger removes both. ApplyDefaultToAllServers_ThatFailsAfterTheScheduleDelete_LeavesTheScheduleRowsAndTheRunTimesInPlace. RED: No exception was thrown. GREEN after.
• The updated schedule-statement test: the replace statements still leave run times alone, and the all-servers reset removes the server run times. AScheduleSave_ThroughTheViewersRealSql_LeavesEveryRunTimeAlone_AndApplyDefaultToAllRemovesOnlyTheServerRunTimes. RED: Expected: 2, Actual: 0. GREEN after.
• Reset to Defaults, then Save, leaves no run-time row for the scope, and no other scope's rows change. It covers a row for a collector this build lacks and a row added while the window was open. It also covers a time typed after the Reset, a server on Use default schedule, and the fleet scope. ASaveAfterAReset_LeavesNoRunTimeRowForTheScope_AndKeepsEveryOtherScopes. RED: server 7's rows survived. GREEN after.
• Source pin. The window arms the clear in ResetDefaults_Click and passes it to its one Save call. It calls ResetAllServerSchedulesAsync once and has no run-time reset of its own. That method opens one transaction and deletes the schedule rows before the run times. AScheduleReset_DeletesTheScopesRunTimes_InTheSameTransactionAsItsScheduleRows. Written with the fix, and not run against the old code.
• A store below V160 (table dropped). The all-servers reset still removes the server's schedule rows and returns their count. A Reset's Save (clear flag set, no run-time change) still saves its schedules. Test: AStoreWithoutTheRunTimeTable_ReadsNoRunTimes_AndASaveSaysTheStoreIsOlder. The reset half passes before and after. The Reset's Save check was added with the fix.
• These were RED earlier on this branch. They were not re-run for the RED here, and they pass now.
  • The CLI verb: CollectorRunTimeCliVerbTests and CollectorRunTimeCliVerbLiveTests.
  • The 42P01 handling on the read and on the save: the store-below-V160 test above.
  • A Save that leaves run-time rows alone: TheViewerScheduleSave_LeavesEveryRunTimeRowAlone_ForAFleetScopeAndAServerScope and TheViewersScheduleStatements_NeverNameTheRunTime.
  • The one-transaction Save: AScheduleWriteThatFails_LeavesTheRunTimeUnsaved_BecauseTheSaveIsOneTransaction and TheWindowsSave_IsOneCallInOneTransaction_AndNeverCommitsTheRunTimesOnItsOwn.
• The merge with dev conflicted in DarlingWorker.cs, DarlingMcpDataTools.cs, ViewerDataService.CollectorSchedules.cs and DailyCollectorDetachTests.cs. Each took dev's (Every daily collector runs detached from its server's pass, behind a fleet-wide cap sized to the store's pool (#4938) #4999's) detached-run code and kept this branch's run-time code on top. git diff origin/dev...HEAD holds only run-time work and the three renames. SweepBodyDetachPolicyTests.cs is dev's unchanged, and DailyCollectorDetachTests.cs is dev's apart from the renames.
• The targeted Darling classes without a store: Total 827, Failed 0, Skipped 61 (the live facts). The same classes live: Total 827, Failed 0, Skipped 0. BothSkusToolDescriptions_StayByteIdentical: 1 of 1.

  every class that names config_collector_run_times, CollectorRunTime, ResetAllServer, SaveCollectorScheduleAsync,
  RanDetached, BeginDailyRunWatch, MaxConcurrentDailyRuns or DailyRun, and by name:
  ViewerCollectorSchedules*, CollectorRunTimeCliVerb*, CollectionHealth*, DailyCollectorDetachTests,
  SweepBodyDetachPolicyTests, DetachedByNameWatchdogTests, DailyRunCapAndWatchdogTests, StartupCommandTimeoutTests,
  StragglerCommandTimeoutTests, ViewerCommandTimeoutTests, SerialLoopStoreSizeSourceTests, DarlingSweepSchedulingTests,
  ViewerDataServiceTests, McpPayloadContractCensusTests, RepoFileAdoptionTests, DocCommentHygieneTests,
  CommentFilterAdoptionTests
  

• Lite.Tests CollectorRunTime*, CollectionHealth* and CollectorFrequency*: Total 112, Failed 0.
• Darling.Tests, Lite.Tests, the viewer and the service build with 0 warnings and 0 errors.
• The full Darling.Tests suite runs in CI's PostgreSQL test jobs, green at 3f3fbd66e. An earlier local live run on a throwaway cluster did not complete: it had 24 command timeouts under the parallel load, and then the cluster crashed (could not reserve shared memory region). Every class that timed out passed when re-run alone on a fresh database: CollectorRunTimeCliVerbLiveTests 6 of 6, and the other 15 classes 57 of 57.
• The window itself was not opened, and neither was Lite's editor. The data-service tests, the pure ToRunTimeChanges and the source pin cover the Reset and Save, not a click-through.

Fixes from review

What changes (Lite)

• A run_at in collection_schedule.json that is not a JSON string, such as "run_at": 120, true, an object or an array, now costs only that collector's run time. Before, the whole file failed to load, Lite fell back to the backup or the defaults, and every other edit in the file was lost. The value now reads as its own text, is refused the same way "25:00" is (same warning, naming the collector), and the collector runs on its interval. The file is written back with the value as text. No other property this PR added has this shape.
• The shipped Lite builds its schedule manager with the app's logger, as the other services built beside it do. Before, the warning for an ignored run time ("25:00", or "02:00" on an hourly collector) was never written.
• One floor for a daily collector's last run, used by both Lite paths: the collector's own interval plus a day (ScheduleManager.LastRunFloor). A run older than that reads as never run. get_collection_health applies it to the health row's run. The start-up read applies it to each collector it seeds, so a weekly collector on the same server no longer widens the floor of a 3-day one. For a collector on a 2-day-or-longer interval, the tool and the scheduler now name the same next run.

Test plan

• RED on the PR head (5c71816), green now: a file with "run_at": 120 (also 1.5, true, false, an object, an array) on one collector and a 7-minute interval on another. The 7-minute interval and the other collector's retention survive, no error is logged, one warning names the collector, the collector runs on its interval, and a reload reads the same. 6 cases failed before.
• Guard, passes before and after: "run_at": "02:30", "25:00" and null read as they always did.
• RED, green now: the app's MainWindow builds exactly one schedule manager and passes new AppLoggerAdapter<ScheduleManager>(). A second test sends "25:00" (daily) and "02:00" (hourly) through that same adapter and finds the warning in the app log. This second test passes before the fix too, because the adapter already existed: the first one is the RED.
• RED, green now: a 3-day collector with a run time whose last run is 6 slot dates back (older than interval plus a day, inside the health read's 7 days). The scheduler's first run slot equals the tool's next_run_utc (they differed). Cases for 2, 3 and 5 slot dates back pass before and after: 2 and 3 are inside the floor, and at 5 the old grid happens to land on the same slot.
• RED, green now: a server with a 3-day and a weekly collector, the 3-day one last run 6 slot dates back. The scheduler treats it as never run and runs it at the next daily slot (before, the weekly neighbour's wider read kept the old run and put it on a later day).
• Floor helper: interval plus a day, inclusive at the edge, per interval.
• Lite.Tests, 854 tests in the classes that name ScheduleManager, CollectorSchedule, McpCollectorRunTimes, the run-time part of RemoteCollectorService, or read MainWindow.xaml.cs: 0 failed. After the final merge, the run-time, health-tool, delta, grid and install-id classes: 152 tests, 0 failed.
• Darling.Tests, no store, final: 198 tests (AlertMasterSwitchSurface, AlertReadFailureSurface, CollectorMeasurementSeam, CollectorStateContract, FleetCardCollectionStale..., LiteJobHistorySource, MuteRuleReloadRetention, RemoteCollectorServiceCancellationCensus, TsqlConventionGuard, RepoFileAdoption, DocCommentHygiene, CommentFilterAdoption), 0 failed, 1 skipped. The member-scan guard failed once on the first version of the floor helper (an expression body it read short), so the helper is a block body.
• Lite and Lite.Tests build with 0 warnings; Darling.Tests builds with 0 warnings.
• The full Lite suite runs in CI's Lite tests jobs, green at 3f3fbd66e.

Known limit

The health read covers seven days. The floor for an interval of seven days or more (8 days or more) is wider than that, so a run between 7 days and the floor is seen by the scheduler and not by the tool. This was already true before the floor, and closing it needs a wider read for those collectors.

What changes (Darling viewer, Collector Schedules window)

• A failed read can no longer make Save delete run times. When the run-time read failed, the window held no run times, so a server whose only override was a run time opened on "Use default schedule", and a plain Save cleared that run time. Now, when the run-time read fails, its warning stays at the top of the status line (the next status text no longer replaces it) and Save sends no run-time change at all: no new time, no delete, no clear. The schedule itself still saves. If the user asked for a run-time change that cannot be written safely (a time typed into a Run at cell, or Reset to Defaults, which clears them), Save is refused with a message that says why, instead of quietly dropping it. Reopening the window reads the run times again.
• The schedule read has the same guard. When the schedule read fails, the window shows the built-in defaults, and a Save would have replaced the stored rows with them. Save is now refused until the schedules are read, and the warning stays visible.
• "Apply Default to All Servers" names what it reset. It used to read "Reset 2 per-server schedule override(s)" even when only run times were removed. The call now counts the two kinds apart, and the status names each, singular and plural: "Reset 2 per-server run times", "Reset 1 per-server schedule override", or "Reset 1 per-server schedule override and 2 per-server run times".
• "Use default schedule" shows the fleet-wide run time. With the box ticked, the line under the grid treated the fleet's time as one the server had of its own. It now says "Uses the fleet-wide run time (02:00)." and then the server's own slot, the same wording as a server row set to "Use default". Display only; Save is unchanged.

Test plan (each check was written first and fails to build on the branch as it stood)

• Failed run-time read, then a plain Save with nothing changed: no run-time delete, upsert or clear is sent (fails before: the save plan and its guard do not exist). The same Save after a read that worked still clears the server's run times.
• Failed run-time read, schedule edited: the schedule rows are saved and no run-time change goes with them.
• Failed run-time read, a time typed or Reset to Defaults: the Save is refused with the run-time message and nothing is sent.
• Failed schedule read: the Save is refused and nothing is sent, for the fleet scope and for a server, ticked or not.
• The window writes its status line in one place and builds its Save from the plan (the old Save lines are pinned out).
• A reset of run times alone reads "Reset 2 per-server run times"; seven count combinations assert the exact status text, and a live check on a throwaway store confirms the counts come back apart (0 and 2 for run times only; 2 and 3, 2 and 2, 1 and 0, 0 and 0 for the other cases).
• "Use default schedule" with a fleet-wide run time of 02:00 starts with "Uses the fleet-wide run time (02:00)."; with no fleet-wide time it says there is no fixed run time; a fleet row and a server with its own schedule are unchanged.
• Targeted classes without a store: 200 run, 0 failed, 13 skipped (the live ones). Same classes against a throwaway store: 117 run, 0 failed, 0 skipped. Build: 0 warnings.
• The full Darling.Tests suite runs in CI, green at 3f3fbd66e.
• Not run: the window itself on screen.

What changes (Darling service, MCP and docs)

• A day is served only when its run starts. A daily collector with a run time counted its day as served at hand-off. When yesterday's run was still going, or still waiting for a permit, at today's slot, today's run was then refused at once: the pass had counted it as run and moved the next run to tomorrow, so that day's run never happened. Now the day counts as served, and the next run moves, only once today's run takes its slot. Until then the day stays due and the pass asks again on the next tick, inside the hour after the slot. Past that hour the day counts as one skipped slot, as before.
• An on-load collector with a run time counts today's slot as served at connect only when its run at connect succeeded. A failed run leaves the slot due inside its hour, so it is tried again that day.
• get_collection_health no longer fails as a whole when the MCP role cannot read the run-time table. That happens on a store you provisioned yourself before this table existed. The tool shows no run times and logs one warning saying to re-run provision-roles.sql, which now grants the MCP role read on config.config_collector_run_times. The managed store's MCP role already reads every config table.
• Commands store the catalog's spelling of the collector name. set_collector_run_at, enable_collector and disable_collector, from the command plane and from the CLI, stored the name as typed. The unique indexes compare names exactly, so a typed Wait_Stats and the viewer's wait_stats became two rows for one collector.
• A PostgreSQL server whose TimeZone this host cannot resolve gets one warning. Examples are localtime, or a POSIX string such as EST5EDT,M3.2.0,M11.1.0. The warning is logged once per server and zone, names the zone, and says that server's run times use UTC. Before, they moved to UTC with no message.
• The docs:
  • a run held by the fleet's concurrency cap starts when a slot frees, which can be after the hour (Darling/README.md, docs/how-collection-works.md);
  • Lite's "does not catch up for a day it was closed" applies only to a collector with a run time;
  • the top-level README.md says a run can start nearly 2 hours after run_at.
• Not fixed here: Darling: a run-time edit that lands in the same tick as a daily run's start can run that collector a second time the same day #5033. A run-time edit that a reload applies just as that collector's daily run starts can run it a second time that day.

Test plan (Darling service and MCP)

Each check below fails when its fix is taken out. The second mutation for each was written separately from the fix.

• Pass_WhileYesterdaysRunStillHoldsTheSlot_DoesNotCountTodayAsRun_AndRetriesInsideTheGrace and its past-the-grace sibling.
  • RED: recording the slot at hand-off again fails both.
  • RED: a run that finds its slot held recording the day anyway fails the first.
• OnLoadSeed_AfterAFailedRunAtConnect_LeavesTheSlotDueInsideTheGrace.
  • RED: seeding from "now" whatever the outcome fails it.
  • RED: reporting the run at connect as succeeded fails it.
• AnInsufficientPrivilegeReadAnswersNoRunTimesWithOneWarning and AMissingTableAnswersNoRunTimesAndAnythingElsePropagates (other errors still propagate). RED: not matching 42501 fails the first.
• ProvisionRolesSql_GrantsMcpReadOnTheCollectorRunTimes. RED: removing the grant line fails it.
• ResolvePlan_CollectorWrites_BindTheCatalogsSpellingNotTheTypedOne, for every verb, scoped and fleet-wide. RED: binding the typed name for set_collector_run_at fails it.
• AnUnresolvedZone_WarnsOncePerServer_NamingTheZone_AndReadsAsUtc and a resolvable zone (including UTC) logging nothing.
  • RED: lowering the warning to trace fails it.
  • RED: warning on every read fails it.
• The viewer's save plan: ASave_AfterAFailedRunTimeRead_ThatChangedNothing_SendsNoRunTimeDeleteOrClear. RED: clearing the run times after a failed read fails it.
• Builds: Darling.Tests and Lite.Tests, --no-incremental, 0 warnings and 0 errors at the head.
• At the head without a store, every class 0 failed: DarlingRunTimeSchedulingTests 22, DarlingCollectorRunTimeReaderTests 18, DarlingCommandExecutorTests 34 (8 live skipped), ProvisionRolesAclDriftTests 9, DarlingPgUnresolvedZoneWarningTests 2, CollectorRunTimeCliVerbTests 21, CollectorRunTimeRungTests 19, CollectorScheduleEditorSaveGuardTests 8, CollectorScheduleResetStatusTests 8, CollectorScheduleUseDefaultRunAtLineTests 4, CollectionLogDrainForensicsStoreTests 13, DailyCollectorDetachTests 30, and the guards (TsqlConventionGuardTests, DocCommentHygieneTests, RepoFileAdoptionTests, CommandDeadlineScannerAdoptionTests, FleetIdentifierScrubTests, QueryStoreSliceTieBreakSourceTests).
• Lite: the REDs for the three Lite fixes above are in CollectorRunTimeScheduleTests and CollectorRunTimeWarningLogTests. CI's Lite tests jobs run them green at 3f3fbd66e.
• CI is green at 3f3fbd66e, including the full Darling.Tests run in the PostgreSQL test jobs.

Not done here, and notes

• Not done: a click-through of the Collector Schedules window and of Lite's schedule editor. In the window, that means Reset to Defaults, Use default schedule and Apply Default to All Servers.
• The Apply Default to All Servers confirmation now says it removes per-server run times too. It also says the fleet-wide schedule and run times are not changed. The "Reset to Defaults" prompt already said it clears every run time, and now it clears them by scope.
• The Apply Default to All Servers status line names the schedule overrides and the run times it removed apart (see "Fixes from review").
• A Reset keeps the fleet-wide rows when it runs for servers, and clears the fleet's rows only when the fleet scope itself is reset.
• This change does not touch the table definition or the migration.

CHANGELOG

No entry in this PR. The release owner adds it under [Unreleased].

Proposed entry:

SECTION: Added
ENTRY:
- **A daily collector can run at a time you choose** ([#5018]) - A collector that runs once a day, or every whole number of days, such as `index_object_stats`, can have a run time, a fixed time of day on the monitored server's own clock, so a heavy collector runs in a quiet hour instead of whenever the service happened to start. In Darling, set it in the Viewer's Collector Schedules window (the new Run at column) or with `--set-collector-run-at`, fleet-wide or per server; the table `config.config_collector_run_times`, added in store version 160 ([#5007]), holds it. In Lite, set it in the schedule editor, or as `run_at` in the schedule file. Each server starts at the run time plus a fixed spread of under 60 minutes, a run may start up to 60 minutes after its slot (later when the fleet's daily-run cap holds it), and a missed day is skipped, not replayed. `get_collection_health` and the web Collection Health tab show each collector's run time and next run. Reset to Defaults, a server's Use default schedule and Apply Default to All Servers also delete the run times of the schedule they reset; a Viewer released before this change resets the schedule rows only, so a run time set here survives a Reset done in an older Viewer. On a store whose roles you provisioned yourself, re-run `provision-roles.sql` after the upgrade so the MCP role can read the run times.
REF:
[#5018]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5018
[#5007]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5007