Skip to content

test: stabilize OPA timeout regression - #3848

Merged
MohammadHaroonAbuomar merged 1 commit into
microsoft:mainfrom
dpaul0501:codex/stabilize-opa-timeout-test
Sep 13, 2026
Merged

MohammadHaroonAbuomar merged 1 commit into
microsoft:mainfrom
dpaul0501:codex/stabilize-opa-timeout-test

Conversation

@dpaul0501

@dpaul0501 dpaul0501 commented Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • increase the fake OPA timeout test budget from 30 ms to 500 ms
  • avoid racing the fake OPA script before it writes the PID file on slower local environments

Repro

Before this change, running the core package tests on this machine failed in opa_timeout_kills_child_process_without_lingering because core/tests/opa.rs attempted to read a PID file that had not been written yet.

Test

  • cargo test -p agent_control_specification_core --test opa --quiet
  • cargo test -p agent_control_specification_core --quiet

Signed-off-by: Debjyoti Paul debjyotipaul@users.noreply.github.com

Signed-off-by: dpaul0501 <deb.jyoti93.paul@gmail.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added tests size/XS Extra small PR (< 10 lines) labels Aug 29, 2026
@github-actions

Copy link
Copy Markdown

Welcome to the Agent Governance Toolkit! Thanks for your first pull request.
Please ensure tests pass, code follows style (ruff check), and you have signed the CLA.
See our Contributing Guide.

@github-actions

Copy link
Copy Markdown

PR Review Summary

Check Status Details
🔍 Code Review ⚠️ Missing No current-run comment
🛡️ Security Scan ⚠️ Missing No current-run comment
🔄 Breaking Changes ⚠️ Missing No current-run comment
📝 Docs Sync ⚠️ Missing No current-run comment
🧪 Test Coverage ⚠️ Missing No current-run comment

Verdict: ⚠️ AI review incomplete; ready for human review

AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims.

San-Hsien (SanHsien) added a commit to SanHsien/agent-governance-toolkit that referenced this pull request Aug 30, 2026
在本 fork 實測重現:已註冊的 server 用 http:// 被 entry.require_tls 擋下,但同一個
URL 換成未註冊的名字,落到 fallback 路徑就完全沒有 TLS 檢查、直接放行。也就是說
allowlist 的鍵打錯一個字,傳輸層下限就等於關掉——治理工具本身的護欄失效。

落地照上游:McpAuthPolicy 新增 default_require_tls=True(fail-closed)、fallback
分支補上與已註冊條目相同的 scheme 檢查(只認 https/wss)、from_yaml 讀該欄位。
只在真的有給 URL 時才擋,與已註冊條目一致,所以不誤擋從不傳 URL 的呼叫端。
測試四條(含上游沒有的「未註冊+不給 URL 仍放行」),21 passed。

同輪另三筆:
- microsoft#3848(OPA timeout 測試 30ms 太緊)採用:假 opa sleep 30 秒,30ms 預算可能在
  行程還沒 spawn 完就到期,失敗的是 spawn 不是逾時,測到的不是這條要測的東西。
  本 fork 的 ci.yml 與 policy-engine-ci.yml 都跑 cargo test --workspace,opa.rs:254
  就是同一行。驗證限制照實說:本機沒裝 cargo,這一行沒實跑過,靠 CI。
- microsoft#3846(土耳其文翻譯)不引用:本 fork 沒有 docs/i18n/,維護一份讀不懂也無法審校的
  翻譯只會變成長期漂移的死文件。
- microsoft#3850(清理 python-app.yml 的 flake8)不引用:本 fork 沒有那支 workflow。

水位 PR 3845 → 3850。順帶把 test_fork_overlay 釘字面日期與字面水位的斷言改成釘性質
(第五個 repo 出現同一條)。

驗證:compileall / ruff / fork tests / 連結檢查 / mcp_auth 測試全部 exit 0。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the test rather than the number, and the diagnosis in your PR body is exactly right. Worth
spelling out on the thread, because a one-line timeout bump looks like the usual "make the flake go
away" change and this one is not.

The fake OPA script is:

echo $$ > "$ACS_FAKE_OPA_PID_FILE"
exec sleep 30

and the assertion that actually matters is the last one, that
/proc/<pid> no longer exists after the timeout. That assertion needs the PID file to have been
written. With a 30 ms budget the runner can reach its timeout and kill the process group before
/bin/sh has spawned, written the file and reached exec, at which point
fs::read_to_string(&pid_file).unwrap() panics on a missing file. So the failure was never in the
timeout behaviour being tested, it was the harness racing its own fixture, which is why it presents
as a mysterious flake on slower machines and never on a warm CI runner.

500 ms is the right value for the same reason. It is far enough above shell startup to make the race
disappear and still two orders of magnitude below the fixture's sleep 30, so
error.detail().contains("OPA eval exceeded timeout") still asserts a real timeout rather than a
completed run. Nothing about what the test proves changes.

Nothing blocking. The cost is up to half a second on one #[cfg(unix)] test, which is a good trade
for an assertion that is currently unreliable.

@dpaul0501

Copy link
Copy Markdown
Contributor Author

Thanks, that framing is exactly what I was trying to preserve here.

To make the intent explicit: the timeout value is not the behavior under test. The behavior under test is that an OPA child process which has entered the fake long-running command is killed after the evaluator timeout, and the final assertion verifies that /proc/<pid> no longer exists.

The old 30 ms value let the harness race its own fixture: on a slower host, the timeout could fire before /bin/sh had written ACS_FAKE_OPA_PID_FILE and reached exec sleep 30, so the test could panic while trying to read the missing PID file. In that failure mode, the test has not actually reached the condition it is supposed to verify.

The 500 ms budget gives the fixture enough time to publish the PID deterministically, while remaining far below the fake command's sleep 30. So the test still exercises the timeout path and still verifies child-process cleanup; it just removes the setup race before the real assertion.

@MohammadHaroonAbuomar MohammadHaroonAbuomar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after group integration review (tests, gates and adversarial pass on the combined change).

@MohammadHaroonAbuomar
MohammadHaroonAbuomar merged commit 8d8ba07 into microsoft:main Sep 13, 2026
133 checks passed
Karim Mehalebi (karimad) pushed a commit to karimad/agent-governance-toolkit that referenced this pull request Sep 14, 2026
Signed-off-by: dpaul0501 <deb.jyoti93.paul@gmail.com>
Yuvraj Singh (yuvrajsingh2428) pushed a commit to yuvrajsingh2428/agent-governance-toolkit that referenced this pull request Oct 1, 2026
Signed-off-by: dpaul0501 <deb.jyoti93.paul@gmail.com>
Signed-off-by: yuvrajsingh2428 <offcyuvi2428@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XS Extra small PR (< 10 lines) tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants