Skip to content

[Bug]: Frequent 502 upstream_server_error after the first failure in a conversation. #186

Closed

Description

@LimLex

Summary

I frequently encounter 502 errors when using OCX with Codex.
Error Details
Model: gpt-5.6-sol
Provider: openai
Error: upstream_server_error
Request duration: 13,465 ms
Provider unreachable: The socket connection was closed unexpectedly.
For more information, pass verbose: true in the second argument to fetch()
Observed Behavior
The issue appears to be related to individual Codex sessions. Once this error occurs for the first time in a particular Codex session, subsequent requests within that same Codex session tend to encounter the same error much more frequently.
I am confident that this is not caused by my local network connection. When I stop using OCX, Codex returns to normal and the connectivity issue disappears.
This suggests that OCX may not be correctly recovering the upstream connection after the first socket failure within a Codex session.

Reproduction

1.Start 5 Codex sessions simultaneously.
2.Let all sessions work normally.
3.Wait until one of the sessions encounters the first 502 upstream_server_error.
4.Continue sending requests in that same Codex session.
5.Observe that the same session starts encountering 502 errors much more frequently than before.

Logs and screenshots

Area

Proxy runtime

Version

No response

OS

No response

Config shape

Checks

  • I searched existing issues and docs first.
  • I removed secrets, tokens, and personal data from logs and screenshots.

Activity

  1. Wibias commented on Jul 20, 2026

    @Wibias
    Contributor

    Can u please post the full logs? Thanks.

  2. LimLex commented on Jul 20, 2026

    @LimLex
    Author

    Can u please post the full logs? Thanks.

    Image Unfortunately, I had just run `ocx stop` so I could continue working in Codex. When I later tried to restart it with `ocx start`, I found that all of the logs from the previous OCX session had disappeared. This may be a separate bug. From what I remember, the failed requests were all reported as `upstream_server_error`, with the following message:
    Provider unreachable: The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()
    

    I hope this information is still useful.

    One additional detail: I was using two OpenAI accounts at the time—one Pro 20x account and one Pro 5x account—with account-pool round-robin mode enabled.

    I had five Codex sessions running concurrently. The errors only occurred in some of the sessions, not all of them.

    Once a particular session encountered this error for the first time, the issue tended to persist in that same session. Even when I repeatedly replied with “continue” to resume the task, subsequent requests would continue to fail with the same upstream_server_error.

  3. Wibias commented on Jul 20, 2026

    @Wibias
    Contributor

    Alright, thanks. Gonna have a look on it tomorrow if jun didn't fix it in the meantime.

  4. Wibias commented on Jul 20, 2026

    @Wibias
    Contributor

    Root cause was Codex thread affinity staying pinned after connect_error / timeout while failover only rotated activeCodexAccountId for new threads - so continue in the same session kept hitting the bad account (502 sticky). Soft keep-alive resets were a secondary amplifier.

    Full fix in #194 :

    • 30s soft-avoid (not hard 429 cooldown) after transient failures so affinity can rebind
    • clear all affinities on failover streak (parity with 429)
    • Connection: close on connection-reset retries
  5. Wibias commented on Jul 20, 2026

    @Wibias
    Contributor

    Once this land it would be good, if u could test it, since I only have one account!

  6. Wibias commented on Jul 20, 2026

    @Wibias
    Contributor

    Separate bug (logs gone after ocx stop / ocx start): GUI /api/logs was memory-only while usage.jsonl already persisted. Fix in #195 - unrelated to the sticky-502 / affinity fix in #194.

  7. LimLex commented on Jul 21, 2026

    @LimLex
    Author

    Once this land it would be good, if u could test it, since I only have one account!

    Thank you very much for working on this issue.

    I updated OCX to the latest version and enabled it again. I then repeated the test using account-pool round-robin mode with five Codex sessions running concurrently.

    Unfortunately, I am still seeing the same issue as yesterday. Two out of the five sessions repeatedly encountered stream disconnection errors.

    I suspect the issue may still be related to account-pool rotation. Based on what I observed, the sessions that did not encounter the problem may have been using the primary account, while the affected sessions may have been assigned to the additionally added account. However, I cannot confirm the account-to-session mapping, so this is only a hypothesis.

    Below is the detail for one of the 502 errors. The other errors had the same details, so I have omitted the duplicates.

    {
      "requestId": "ocx-*****-7b",
      "timestamp": 1784617805726,
      "model": "gpt-5.6-sol",
      "provider": "openai",
      "requestedModel": "gpt-5.6-sol",
      "requestedEffort": "xhigh",
      "configuredServiceTier": "default",
      "modelSupportsServiceTier": false,
      "status": 502,
      "durationMs": 1635,
      "errorCode": "upstream_server_error",
      "closeReason": "non_stream",
      "upstreamError": "Provider unreachable: The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
      "usageStatus": "unreported",
      "displayMetrics": {
        "tokPerSecond": {
          "kind": "unavailable",
          "reason": "usage_missing"
        },
        "cost": {
          "kind": "unavailable",
          "reason": "usage_missing"
        }
      }
    }

    Is there currently any way to determine which account was assigned to each Codex session? That would help confirm whether the failures are specific to the additional account.

    One more detail: the log persistence issue still appears to be present.

    I tested it again by running ocx stop and then restarting OCX with ocx start. After the restart, the previous logs were gone.

    I updated OCX using ocx update, and the installed version is v2.7.28.

    So, at least in my environment, the fix for the log persistence issue does not appear to be working in v2.7.28.

  8. LimLex commented on Jul 21, 2026

    @LimLex
    Author

    如果这块陆地真的建好了就太好了,如果你能帮忙测试一下就更好了,因为我只有一个账号!

    非常感谢您处理这个问题。

    我将 OCX 更新到最新版本并重新启用。然后,我使用账户池轮询模式重复了测试,同时运行了五个 Codex 会话。

    很遗憾,我仍然遇到和昨天一样的问题。五次会话中有两次反复出现流媒体断开连接的错误。

    我怀疑问题可能仍然与账户池轮换有关。根据我的观察,未遇到问题的会话可能使用的是主账户,而受影响的会话可能被分配到了新增的账户。但是,我无法确认账户与会话之间的对应关系,所以这只是一个假设。

    以下是其中一个502错误的详细信息。其他错误的详细信息相同,因此我省略了重复项。

    {
    "requestId": "ocx-*****-7b",
    "timestamp": 1784617805726,
    "model": "gpt-5.6-sol",
    "provider": "openai",
    "requestedModel": "gpt-5.6-sol",
    "requestedEffort": "xhigh",
    "configuredServiceTier": "default",
    "modelSupportsServiceTier": false,
    "status": 502,
    "durationMs": 1635,
    "errorCode": "upstream_server_error",
    "closeReason": "non_stream",
    "upstreamError": "Provider unreachable: The socket connection was closed unexpectedly. For more information, pass verbose: true in the second argument to fetch()",
    "usageStatus": "unreported",
    "displayMetrics": {
    "tokPerSecond": {
    "kind": "unavailable",
    "reason": "usage_missing"
    },
    "cost": {
    "kind": "unavailable",
    "reason": "usage_missing"
    }
    }
    }
    目前是否有办法确定每个 Codex 会话分配给了哪个帐户?这将有助于确认故障是否与额外的帐户有关。

    还有一个细节:日志持久性问题似乎仍然存在。

    我再次运行命令ocx stop并重启 OCX进行了测试ocx start。重启后,之前的日志消失了。

    我使用 更新了 OCX ocx update,已安装的版本是v2.7.28。

    所以,至少在我的环境中,日志持久性问题的修复似乎不起作用v2.7.28。

    Correction: I just realized that v2.7.28 is not the version containing the fixes for these issues. I had mistakenly assumed it was.

    Please disregard my previous comment. I'll retest after updating to the version that actually includes the fixes.

  9. Wibias commented on Jul 21, 2026

    @Wibias
    Contributor

    @LimLex When 2.7.31 comes out, its hopefully fixed!

  10. LimLex commented on Jul 21, 2026

    @LimLex
    Author

    @LimLex When 2.7.31 comes out, its hopefully fixed!

    I currently have 9 issues sitting in my Todoist waiting for fixes. Once v2.7.31 is released, I'll go through all of them and report back. 😄

  11. lidge-jun commented on Jul 21, 2026

    @lidge-jun
    Owner

    v2.7.31에 수정 반영했습니다. mid-stream reset을 transient failed 502로 재분류하고 계정 cooldown과 affinity 진단을 보강했습니다. 반영 커밋: 87be0d8

  12. LimLex commented on Jul 22, 2026

    @LimLex
    Author

    @lidge-jun @Wibias

    I updated to v2.7.31.

    The log persistence issue is now clearly fixed: after restarting OCX, I can still see the logs from the previous session.

    I also repeated the test with five Codex sessions running concurrently. This time, only one session encountered the same issue as before, and the first failure occurred later than in my previous tests.

    This suggests that the fix may have reduced the frequency of the issue, although I cannot rule out local network fluctuations as the reason for the difference.

    Since I first encountered this problem, I have run this test three times. In every test, the failures were concentrated in only a subset of the Codex sessions, while the other sessions remained completely unaffected.

    Because of this pattern, I strongly suspect that the issue may be related to the account pool—for example, the primary account may be working normally while an additional account is failing. However, the current logs do not show which account handled each request, so I cannot verify this hypothesis.

    Below is one of the repeated errors from the affected session. The other errors were identical, so I have omitted them.

    {
      "requestId": "ocx-****-kg",
      "timestamp": 1784696473397,
      "model": "gpt-5.6-sol",
      "provider": "openai",
      "requestedModel": "gpt-5.6-sol",
      "requestedEffort": "xhigh",
      "configuredServiceTier": "default",
      "modelSupportsServiceTier": false,
      "responseServiceTier": "auto",
      "resolvedModel": "gpt-5.6-sol",
      "status": 502,
      "durationMs": 90294,
      "errorCode": "upstream_server_error",
      "terminalStatus": "failed",
      "closeReason": "terminal",
      "usageStatus": "unreported",
      "transportPhase": "mid_stream",
      "terminalSource": "synthetic",
      "displayMetrics": {
        "tokPerSecond": {
          "kind": "unavailable",
          "reason": "usage_missing"
        },
        "cost": {
          "kind": "unavailable",
          "reason": "usage_missing"
        }
      }
    }

    After the issue occurred, I temporarily switched the OpenAI provider from account-pool round-robin mode to direct-connection mode, and then tried to continue the affected Codex session. However, the session still could not continue and kept failing.

    Would it be possible to include the selected account identifier in the request logs? That would make it much easier to confirm whether the failures are tied to a specific account in the pool.

  13. LimLex commented on Jul 22, 2026

    @LimLex
    Author

    @lidge-jun @Wibias

    I updated to v2.7.31.

    The log persistence issue is now clearly fixed: after restarting OCX, I can still see the logs from the previous session.

    I also repeated the test with five Codex sessions running concurrently. This time, only one session encountered the same issue as before, and the first failure occurred later than in my previous tests.

    This suggests that the fix may have reduced the frequency of the issue, although I cannot rule out local network fluctuations as the reason for the difference.

    Since I first encountered this problem, I have run this test three times. In every test, the failures were concentrated in only a subset of the Codex sessions, while the other sessions remained completely unaffected.

    Because of this pattern, I strongly suspect that the issue may be related to the account pool—for example, the primary account may be working normally while an additional account is failing. However, the current logs do not show which account handled each request, so I cannot verify this hypothesis.

    Below is one of the repeated errors from the affected session. The other errors were identical, so I have omitted them.

    {
    "requestId": "ocx-****-kg",
    "timestamp": 1784696473397,
    "model": "gpt-5.6-sol",
    "provider": "openai",
    "requestedModel": "gpt-5.6-sol",
    "requestedEffort": "xhigh",
    "configuredServiceTier": "default",
    "modelSupportsServiceTier": false,
    "responseServiceTier": "auto",
    "resolvedModel": "gpt-5.6-sol",
    "status": 502,
    "durationMs": 90294,
    "errorCode": "upstream_server_error",
    "terminalStatus": "failed",
    "closeReason": "terminal",
    "usageStatus": "unreported",
    "transportPhase": "mid_stream",
    "terminalSource": "synthetic",
    "displayMetrics": {
    "tokPerSecond": {
    "kind": "unavailable",
    "reason": "usage_missing"
    },
    "cost": {
    "kind": "unavailable",
    "reason": "usage_missing"
    }
    }
    }
    After the issue occurred, I temporarily switched the OpenAI provider from account-pool round-robin mode to direct-connection mode, and then tried to continue the affected Codex session. However, the session still could not continue and kept failing.

    Would it be possible to include the selected account identifier in the request logs? That would make it much easier to confirm whether the failures are tied to a specific account in the pool.

    Image Image
  14. bumasoft commented on Aug 23, 2026

    @bumasoft

    I am getting the same issue now with the latest version in Claude Code...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions