Skip to content

[Bug][Windows] synchronous ACL hardening still stalls /healthz and poisons response spill writes after #612 #3011

Closed
Closed
[Bug][Windows] synchronous ACL hardening still stalls /healthz and poisons response spill writes after #612#3011
Assignees
Ingwannu
Labels
bugSomething isn't workingplatformOS/service/tray/ACL (Windows-heavy, not Windows-only)

Description

@chano-gpt
chano-gpt
opened on Aug 30, 2026

Client or integration

Codex App

Area

Proxy and routing / Platform (Windows) / Service lifecycle

Summary

On Windows, the proxy can keep port 10100 in LISTEN while /healthz, /readyz, and normal Codex requests stop responding for 10–47 seconds. The same process later recovers without a restart.

A timestamped health probe correlated one 40.956-second /healthz delay exactly with:

[opencodex] ACL hardening timed out (ETIMEDOUT) — transient icacls stall; the volume may still support per-user NTFS ACLs; ACL state unverified (budget exhausted) — continuing without NTFS ACL harden

Issue #612 moved the response-state snapshot writer to an asynchronous ACL path. However, the released 2.36.0 source still has synchronous runtime paths:

  • src/config/paths.ts: hardenConfigDir() calls hardenSecretDir().
  • src/responses/spill-store.ts: writeResponseSpillDurably() synchronously hardens the spill directory and temporary file.
  • src/lib/windows-secret-acl.ts: the synchronous runner uses Bun.spawnSync(...).

Because /healthz only constructs a local JSON response, a timeout while the port remains listening indicates that the Bun event loop itself is blocked.

There is also a functional consequence beyond liveness. Before restart, GET /api/system/memory reported:

responseState.totalBytes:       66,982,525
responseState.count:            216
responseState.spillWrites:      849
responseState.spillWriteFailures: 2,032
responseState.tombstoneCount:   46

The spill directory contained no live spill files at inspection time. In state.ts, a spill-write exception increments spillWriteFailures and replaces the resident continuation with a tombstone. This can degrade previous_response_id replay even after the health endpoint recovers.

Expected behavior:

  • ACL hardening must remain fail-closed where required.
  • Runtime ACL subprocess waits must not block /healthz or unrelated proxy requests.
  • A transient ACL timeout must not leave response spilling effectively failed for the remaining process lifetime.

Reproduction

The failure is intermittent rather than deterministic on this machine.

  1. Install OpenCodex globally on Windows and run it through the Task Scheduler service backend.
  2. Keep the proxy running during normal Codex App usage.
  3. Probe http://127.0.0.1:10100/healthz once per minute with a 10-second timeout while recording process memory and response-state metrics.
  4. Observe that the port remains in LISTEN, but /healthz sometimes takes more than 10 seconds or times out.
  5. Compare the probe timestamp with service.log.

Observed across 2,125 samples:

failed probes:       6
latency >= 1 second: 58
latency >= 5 seconds: 17
maximum latency:     47,139 ms

Strongest correlation:

2026-08-24T09:33:34+09:00  /healthz latency=40,956ms
2026-08-24T09:33:34+09:00  ACL hardening timed out (ETIMEDOUT)

The OpenCodex data directory was small (30 files, one subdirectory). A read-only icacls query against it completed in 48–92ms across ten runs. Windows Defender real-time protection was enabled, so transient process-spawn/scanner contention may be the trigger, but the service-wide outage results from waiting synchronously on the event loop.

After collecting the evidence, the installation was upgraded to 2.36.0 and the service restarted. The new process is currently healthy. Source inspection of the installed 2.36.0 package confirmed the synchronous config-directory and spill paths listed above remain.

Version

Observed at runtime on 2.31.0. Residual synchronous paths confirmed in the installed 2.36.0 stable package.

Operating system

Windows 11 Home x64, version 10.0.26200, build 26200.

Provider and model

Not provider-specific. /healthz is local and unauthenticated.

Logs or error output

# Port remained in LISTEN, but CLI probes failed:
{"ok":false,"pid":null,"port":null}
{"ready":false,"status":"unreachable","pid":null,"port":null}

# Service log at a correlated stall:
[opencodex] ACL hardening timed out (ETIMEDOUT) — transient icacls stall; the volume may still support per-user NTFS ACLs; ACL state unverified (budget exhausted) — continuing without NTFS ACL harden

Suggested safe direction

  1. Add asynchronous variants for runtime hardenConfigDir / existing-secret checks.
  2. Convert writeResponseSpillDurably to an awaited asynchronous writer.
  3. Serialize spill publications with a single-flight queue, preserving publication order and current fail-closed ACL semantics.
  4. Add a regression test with an injected slow ACL runner: /healthz should remain responsive while the hardening promise is pending.
  5. Add a spill regression asserting that a transient ACL delay does not produce process-lifetime spillWriteFailures growth.

Increasing the ACL timeout or weakening ACL checks would not address the event-loop blocking.

Redacted configuration

{
  "serviceBackend": "Task Scheduler",
  "hostname": "127.0.0.1",
  "port": 10100,
  "providerSpecific": false
}

Checks

Activity

  1. github-actions
    added
    bugSomething isn't working
    platformOS/service/tray/ACL (Windows-heavy, not Windows-only)
    on Aug 30, 2026

  2. Ingwannu commented on Aug 30, 2026

    @Ingwannu
    Ingwannu
    on Aug 30, 2026
    Owner

    Confirmed against current dev@df8b3882f221b3d68eddcfc34cc3a6edccbb32b3.

    The reported boundary still exists. src/responses/spill-store.ts:325-335 synchronously hardens both the spill directory and each new temporary file, and the exclusive-copy fallback synchronously hardens the destination at lines 243-250. On Windows those calls reach hardenSecretDir / hardenSecretPath and the Bun.spawnSync(icacls) runner. They execute directly from rememberResponseState demotion/admission paths (src/responses/state.ts:311, 393, 1060, and 1161), so a slow ACL subprocess blocks the same Bun event loop that serves /healthz. The reported LISTEN-without-response behavior is therefore consistent with the current source, not only the older runtime.

    I will keep the fix bounded to the runtime write path:

    1. make optional config-directory hardening asynchronous/single-flight on Windows so ordinary config reads cannot stall the server;
    2. add an awaited asynchronous spill publication path using hardenSecretDirAsync / hardenSecretPathAsync, preserving the existing temp-write, fsync, no-replace publication, identity memo, and fail-closed ACL contracts;
    3. serialize spill publications and preserve compare-before-swap ordering, so a transient timeout does not permanently replace a still-available resident continuation with spill-failed;
    4. add a deterministic slow-ACL regression proving unrelated event-loop work remains responsive and a retry can later publish the spill;
    5. leave startup/installation paths that intentionally require synchronous completion unchanged unless they are also reached after the server starts.

    This needs Windows CI for the real icacls path; Linux focused tests can prove queue ordering, failure recovery, and liveness with an injected async runner. I will not weaken required ACL checks or merely increase the timeout.

  3. Ingwannu
    self-assigned this
    on Aug 30, 2026

  4. lidge-jun commented on Aug 30, 2026

    @lidge-jun
    lidge-jun
    on Aug 30, 2026
    Owner

    리뷰 · 우선순위 71 / 80

    이 이슈가 말하는 구멍은 지금 dev HEAD f5a625cf3 에도 그대로 있다. 윈도우에서 프록시 포트는 LISTEN 인데 /healthz 가 10초에서 47초까지 안 받는다. 프로세스는 안 죽고 나중에 혼자 돌아온다. 작성자가 분 단위로 2125번 찔러 본 숫자다. 실패한 탐침 6번, 1초 이상 58번, 5초 이상 17번, 제일 긴 지연 47139ms. 그 중 40956ms 짜리 한 번은 서비스 로그의 ACL 타임아웃과 초가 같다. /healthz 는 로컬 JSON만 만든다. 포트가 열려 있는데도 그 응답이 멈춘다는 것은, 다른 일이 이벤트 루프를 붙잡고 있다는 뜻이다.

    그 다른 일이 동기 ACL 이다. #612 가 비동기 icacls 실행기를 넣었다. 지금 파일 src/lib/windows-secret-acl.ts 맨 위 주석도 그렇게 적는다. hardenSecretPathAsync / hardenSecretDirAsync 는 자식이 끝날 때까지 루프를 붙잡지 말라고 #612 가 만든 API 다. src/codex/native-profile-manager.ts 는 그 비동기 API 를 쓴다. src/config/atomic-write.ts 도 hardenSecretPathAsync 를 들여온다. 그런데 런타임에 남는 동기 길이 이슈가 적은 그대로다. src/config/paths.ts 의 hardenConfigDir() 는 win32 에서 hardenSecretDir(dir, { required: false }) 를 부른다. src/responses/spill-store.ts 의 harden() 은 디렉터리에 required: true 로 hardenSecretDir, 파일에 hardenSecretPath 를 부른다. writeResponseSpillDurably() 는 디렉터리를 만든 뒤와 임시 파일을 쓴 뒤에 그 harden() 을 그대로 기다린다. 동기 실행기는 defaultIcaclsRunner 이고, 그 안은 Bun.spawnSync 다. 기본 예산은 30000ms, 환경 변수로 최대 60000ms 다. 타임아웃이 나도 그때까지 루프는 서 있다.

    hardenConfigDir 는 required: false 라서 타임아웃 뒤에 그냥 넘어간다. 로그에 나온 그 문장, 예산이 끝나서 ACL 을 건너뛴다는 말이 그 길이다. 실패해도 설정은 남는다. 그래도 기다리는 동안 /healthz 는 죽는다. 스필 쪽은 더 세다. required: true 라서 타임아웃이 예외가 된다. writeResponseSpillDurably 는 그걸 Response spill write failed 로 다시 던진다. src/responses/state.ts 는 그 예외를 받아 spillCounters.writeFailures 를 올리고 replaceWithSpillFailure 로 톰스톤을 넣는다. 작성자가 재시작 전에 본 숫자, 스필 쓰기 849번에 실패 2032번, 톰스톤 46개는 그 길이와 맞다. 스필 폴더에 살아 있는 파일이 없던 것도 이상하지 않다. 실패하면 파일이 안 남고 톰스톤만 남는다. previousResponseProviderState 는 kind === "spill-failed" 이면 providers 를 안 돌려준다. 그래서 previous_response_id 재현이 그 프로세스 안에서 죽는다. 헬스만 잠깐 아픈 것이 아니다.

    이 이슈는 #612 의 중복이 아니다. #612 는 비동기 API 와 일부 호출자를 옮겼다. 남은 것은 설정 디렉터리 경화와 응답 스필 쓰기다. #2963 도 이 구멍을 닫지 않는다. 그 고침은 이미 맞는 ACL 이면 변이를 건너뛰는 선택이다. 건너뛰면 icacls 를 덜 부른다. 그래도 부를 때는 아직 동기 대기다. 타임아웃을 늘리거나 ACL 검사를 약하게 만들자는 말은 이 이슈에 없다. 그 말은 맞다. 루프를 붙잡는 것이 장애이고, 예산을 늘리면 멈춤만 길어진다. types.ts/config.ts 분할과도 안 겹친다. 미리보기 배포도 없다.

    재현은 이 머신에서 매번 터지지 않는다. 디펜더가 켜져 있었고, 읽기 전용 icacls 는 48에서 92ms 로 끝났다. 데이터 폴더는 파일 30개였다. 그래서 평소 디스크가 느린 문제는 아니다. 자식 프로세스와 스캐너가 가끔 싸우면 동기 대기가 서비스 전체를 세운다는 설명이 지금 코드와 맞다. 런타임 관찰은 2.31.0 이고, 설치된 2.36.0 패키지에서 동기 길이 확인했다. 지금 HEAD 도 그 길이가 있다. 올린 뒤 새 프로세스가 건강하다는 것은, 지금 안 터진다는 뜻이지 길이 없어졌다는 뜻이 아니다.

    src/lib/windows-secret-acl.ts 라인 311 - defaultIcaclsRunner 가 아직 Bun.spawnSync 다. 같은 파일에 비동기 Bun.spawn 실행기가 있는데, 설정 디렉터리와 스필 쓰기는 동기 쪽을 부른다. 타임아웃이 나도 그 시간 동안 이벤트 루프가 선다.
    src/config/paths.ts 라인 38 - hardenConfigDir 가 hardenSecretDir(..., { required: false }) 를 동기로 부른다. 실패는 넘어가지만 /healthz 는 그 기다림을 같이 먹는다.
    src/responses/spill-store.ts 라인 186 - 스필 harden() 이 required: true 동기 경화다. 타임아웃이 예외가 되고, 같은 파일 라인 326과 334에서 디렉터리와 임시 파일마다 그 대기를 한다.
    src/responses/state.ts 라인 1068 - writeResponseSpillDurably 예외를 spillWriteFailures 로 세고 라인 1070에서 톰스톤으로 바꾼다. 라인 1331에서 톰스톤은 previous_response_id 의 제공자 상태를 안 돌려준다. 잠깐 선 헬스 뒤에 이어가기까지 죽는다.

    메인테이너의 판단이 필요한 지점

    너의 추천
    열어 두고 이어서 고친다. #612 가 만든 비동기 API 로 hardenConfigDir 와 writeResponseSpillDurably 의 런타임 경화를 옮긴다. 타임아웃을 늘리지 않는다. ACL 실패 닫힘은 유지하되, 일시 ETIMEDOUT 은 그 프로세스 평생 톰스톤으로 바꾸지 않는다. 이슈가 적은 테스트 두 개가 바로 증명이다. 느린 ACL 실행기를 넣었을 때 /healthz 가 살아 있는지, 일시 지연이 spillWriteFailures 를 평생 쌓지 않는지. 중복으로 닫지 않는다.

    이 댓글은 grok-bot이 작성했습니다

  5. Ingwannu commented on Aug 30, 2026

    @Ingwannu
    Ingwannu
    on Aug 30, 2026
    Owner

    Implemented the bounded fix in #3018 at head 400a6a3849d4cbf3a5f95f9f86e55bfe4ad4e831.

    The PR keeps Linux/macOS publication unchanged and moves only the Windows ACL-dependent runtime path onto a serialized async publisher. While icacls is pending, the exact resident continuation remains replayable; the spill stub is installed only after hardening, fsync, no-replace publication, and compare-before-swap. Pending payloads are pinned under a 256 MiB cap, one real timeout gets one bounded recovery attempt, and a stale same-id publication is deleted instead of replacing newer state. Optional config-directory ACL hardening is now a per-directory async single-flight.

    Local exact-head validation:

    • focused config/response/ACL/memory suite: 469 pass, 0 fail
    • typecheck: pass
    • privacy scan: pass

    I have requested @lidge-jun review. The issue remains open until exact-head CI, especially the real Windows icacls jobs, is green and the PR is merged.

  6. lidge-jun
    added 2 commits that reference this issue on Aug 31, 2026

    docs(devlog): plan the priority-70+ entitlement and spill train

    9e79dfe

    fix(codex): restore GPT-5.6 for entitled accounts by raising the gate…

    4bdc0f6

  7. lidge-jun commented on Aug 31, 2026

    @lidge-jun
    lidge-jun
    on Aug 31, 2026
    Owner

    Fixed on dev in e5d5886 (PR #3044), built on @Ingwannu's base commit 25ce4d5f5 from #3018.

    What was wrong. Moving Windows ACL hardening off the response event loop was the right call — that is what removed the up-to-47s /healthz stall you reported — but it introduced a shutdown race: a spill publication that had been made asynchronous could still be in flight when the process exited, so the response was lost even though the client had been told it was durable.

    What landed.

    • A fixed-point shutdown drain: shutdown repeatedly drains pending publications until the queue reaches a fixed point, so a publication that enqueues another one cannot escape.
    • Supersession is decided by the writer, not by observed state, so an abandoned writer can neither publish nor orphan its temp file.
    • Every ACL harden in the async path draws from one caller-owned budget per write attempt (SpillAclBudget), and budget exhaustion throws ETIMEDOUT instead of reopening a default 30s window. A single stalled queue head can no longer push unrelated responses over the pending-byte cap.
    • src/lib/bounded-subprocess.ts: every subprocess await in this path is bounded. Killing a wedged icacls is not the same as settling it — before this, a killed runner was still awaited forever, which made all the budgets advisory.
    • A rejected drain now exits nonzero even when a replacement process spawns successfully.

    Deliberate trade, documented so it is not a surprise. If the fallback budget is exhausted, the payload is destroyed, a spill-failed tombstone is persisted, and the process exits nonzero. A replay then returns previous_response_not_found/spill_failed and the client resends. Losing one response is preferable to wedging shutdown indefinitely.

    Residual gap, stated honestly. NTFS unlink semantics and icacls behavior are verified only as far as CI models them; this has not been exercised on a real Windows host beyond the Windows CI jobs. Verification at the merged head: 30 checks green (1 shard skipped), and a full Linux suite run at 16526 pass / 0 fail / 16 skip.

    This will ship in the next release. Closing now that it is on dev — GitHub only auto-closes on merges into main, and PRs here target dev.

    Thanks @Ingwannu — your two review rounds caught the missing async budget and the wrong exit code after five internal review rounds had missed both.

  8. lidge-jun
    closed this as completedon Aug 31, 2026
  9. tarunravi
    added a commit that references this issue on Sep 14, 2026

    fix(codex): restore GPT-5.6 for entitled accounts by raising the gate…

    6ff3263
  10. agentHits
    added a commit that references this issue on Sep 17, 2026

    fix(codex): restore GPT-5.6 for entitled accounts by raising the gate…

    28d7dac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingplatformOS/service/tray/ACL (Windows-heavy, not Windows-only)

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions