feat(prime): impact of human-message priority (#2334) and bash continuation holds (#2465) on Pylon occurrence model #727

Activity

1. rynfar commented on Sep 24, 2026

   

   rynfar

   on Sep 24, 2026

   CollaboratorAuthor

   More actions

   Current-source/status correction for this impact issue:

   • Upstream human-priority PR #2334 merged 2026-09-14 and is an ancestor of the v0.9.5 release (published 2026-09-16). It is already a stock v0.9.5 behavior, not a future hypothetical.
   • Upstream bash-hold PR #2465 merged 2026-09-21, after v0.9.5; it needs a separately proved Pylon managed/stock artifact before assigning it to any running session.
   • Pylon's current PrimeAgentDaemonSessionRuntime records native queue count and consumes native event cursors; its removeOnlyInputQueueItem path only mutates a queue lane when it contains exactly one entry, using expectedText on an isolated mutation connection. This source does not by itself prove that priority insertion breaks Pylon event occurrence order or that an abort must cancel all background shell work.

   The next acceptance step is an isolated stock/managed-version trace with mixed user/agent/cron queued messages and a running background bash handle: record native event cursor order, queue snapshots, prompt-admission receipts, turn settlement, and abort latency. Keep this as a proof task until a concrete mismatch is shown; do not relax occurrence or recovery fences based solely on the scheduling change.
2. rynfar commented on Sep 26, 2026

   

   rynfar

   on Sep 26, 2026

   CollaboratorAuthor

   More actions

   Characterization Trace Results & Adversarial Invariant Analysis

   Completed the isolated stock v0.9.5 characterization trace and independent adversarial invariant analysis per the acceptance criteria above.

   ---

   1. Empirical Characterization Trace (Stock v0.9.5 Daemon)

   • Harness Setup: Tested against /Users/rynfar/.local/bin/prime-agent (v0.9.5, schema revision 28, protocol 7) over an isolated domain socket with a deterministic local streaming OpenAI completion mock provider.
   • Queue Snapshots Under Mixed Traffic:
     • Turn 1 was dispatched and held in the mock backend stream.
     • While busy, a background agent message (source: "internal", agentMessageId: "agent-sub-1") and a scheduled cron prompt (source: "internal") were enqueued in followUp.
     • An interactive human user prompt (source: "interactive") was then submitted.
     • Snapshot confirmed PR #2334 priority insertion: the human message jumped ahead to index 0 of the followUp lane:

       {
         "steering": [],
         "followUp": [
           "MESSAGE_FROM_HUMAN_USER",
           "MESSAGE_FROM_AGENT",
           "MESSAGE_FROM_CRON"
         ]
       }

   • Turn Settlement & Dispatch Order:
     • Upon Turn 1 stream completion, the daemon dispatched Turn 2 (MESSAGE_FROM_HUMAN_USER), followed by Turn 3 (MESSAGE_FROM_AGENT), and Turn 4 (MESSAGE_FROM_CRON). Execution strictly mirrored priority queue ordering.
   • Native Event Cursor Monotonicity:
     • All 62 events captured progressed strictly monotonically (seq = 1, 2, 3, ..., 61) with zero sequence inversions or gaps. Monotonicity holds because cursors (meta.cursor: { generation, sequence }) are assigned at execution dispatch time, not queue admission time.
   • Abort Latency:
     • abort_and_clear_queue acknowledged in 2 ms, atomically emptying both steering and followUp.

   ---

   2. Concrete Architectural Mismatches in Pylon

   While cursor sequence numbers are monotonic in isolation, an adversarial review against Pylon's dual-state runtime (PrimeAgentDaemonSessionRuntime.ts) and adapter (PrimeAgentDaemonAdapter.ts) uncovered concrete invariant violations if PR #2334 or PR #2465 are adopted into Pylon's managed fork without adapter changes:

   1. Transcript Resync Failure on Reconnect (snapshotRecoversSubmittedUser):

      • PrimeAgentDaemonAdapter.ts:2799-2830 computes submittedUserIndex = activeTurn.nativeTranscriptBaselineMessageCount and asserts matchesSubmittedUserMessage(activeTurn, missingMessages[submittedUserIndex]).
      • If Pylon disconnects while a background task turn is queued, and during the disconnect window a human message jumps to index 0 and completes, missingMessages[submittedUserIndex] is the human message, whereas activeTurn.submittedUserMessage was the background task.
      • matchesSubmittedUserMessage fails, setting reconnectRecoveryFailed = true and force-terminating the entire session via stopSessionInternal.

   2. Native Event Attribution Blackout (activeTurnForNativeEvent):

      • PrimeAgentDaemonAdapter.ts:2600-2615 only attributes events to context.activeTurn if attribution.correlationId === activeTurn.correlationId.
      • Pylon maintains a single active turn pointer. When a preempting human turn executes, its events carry correlationId = "corr-human". Because activeTurn.correlationId points to the background turn, activeTurnForNativeEvent returns undefined.
      • Impact: All streaming tokens (MessageStarted, MessageDelta, RunCompleted) for the user turn are silently dropped, and the UI appears completely frozen.

   3. Queue Mutation TOCTOU Race (removeOnlyInputQueueItem):

      • PrimeAgentDaemonSessionRuntime.ts:8096-8200 validates entries.length === 1 and issues mutate_queued_message(lane, index: 0, expectedText).
      • If a human prompt jumps into index 0 while the mutation RPC is in flight, the daemon rejects the mutation due to text mismatch (expectedText !== item.text). The targeted background item is shifted to index 1 and remains undeleted.

   4. Admission Guard Contradiction (sendTurn):

      • PrimeAgentDaemonAdapter.ts:6075-6095 checks nativeInputQueuedCount > 0 and rejects subsequent turn submissions with HTTP 400 Validation Error (reason: "busy"). Human preemption in the daemon is currently blocked at the Pylon adapter boundary.

   5. IPython Background Process Leaks on Abort:

      • The 2ms latency of abort_and_clear_queue is a synchronous fire-and-forget signal (requestAbort()).
      • requestAbort() only signals _bashAbortControllers; it never signals ReplKernelManager background handles. Background processes running inside the IPython kernel continue executing after the abort, leaking file locks, ports, and child processes.

   ---

   3. Conclusion & Disposition

   • Maintainer directive confirmed: Do NOT relax occurrence or recovery fences.
   • Do NOT adopt upstream PR #2334 or PR #2465 into Pylon's fork (pylon-code/prime-agent) until Pylon's adapter is re-architected to support:
     1. Multi-correlation concurrent turn event dispatching (retiring the single activeTurn pointer constraint).
     2. Non-FIFO fingerprint-based transcript reconciliation on reconnect.
     3. Correlation-ID-targeted queue mutations rather than array index targeting.
     4. Synchronous process reaping in daemon abort handlers.