Skip to content

Fix WebSocket exception leaks after kill -9 termination #60

Open
@yhl-cs

Description

@yhl-cs

Describe the Bug

When establishing a WebSocket connection over SSL, if the server process is abruptly terminated via kill -9, the reverse proxy cannot gracefully handle the exception.

To Reproduce

  1. Run the server:
# ws-server.py
import asyncio

from fastapi import FastAPI, WebSocket

app = FastAPI()


@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    print("Client connected")

    response = await websocket.receive()
    print(f"Received: {response}")

    await asyncio.sleep(100)

    await websocket.send_text("Success: The request completed after 100 seconds.")
    print("Response sent to client")


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(
        app, host="0.0.0.0", port=8766, ssl_certfile="cert.pem", ssl_keyfile="key.pem"
    )
$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=localhost"
$ python ./ws-server.py 
INFO:     Started server process [32042]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on https://0.0.0.0:8766 (Press CTRL+C to quit)
  1. Run the reverse proxy:
# ws-rs.py
from fastapi_proxy_lib.fastapi.app import reverse_ws_app
from httpx import AsyncClient


app = reverse_ws_app(AsyncClient(verify=False), base_url="https://127.0.0.1:8766/")

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8765)
$ python ./ws-rs.py 
INFO:     Started server process [31689]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8765 (Press CTRL+C to quit)
  1. Run the Client:
# ws-client.py
import asyncio
import httpx
import httpx_ws

async def websocket_client():
    url = "ws://127.0.0.1:8765/ws"
    async with httpx.AsyncClient() as client:
        async with httpx_ws.aconnect_ws(url, client) as websocket:
            print("Connected to WebSocket server")

            message = "Hello, WebSocket Server!"
            await websocket.send_text(message)
            print(f"Sent: {message}")

            response = await websocket.receive_text()
            print(f"Received: {response}")


async def main():
    await websocket_client()


if __name__ == "__main__":
    asyncio.run(main())
$ python ./ws-client.py 
Connected to WebSocket server
Sent: Hello, WebSocket Server!
  1. Terminate the server process:
$ ps aux | grep python
root        1560  0.0  0.6 481600 21056 ?        Ssl  15:54   0:01 /usr/bin/python3 -Es /usr/sbin/tuned -l -P
root       31689  1.2  1.7 277760 53376 pts/0    S+   18:17   0:02 python ./ws-rs.py
root       32042  3.0  1.6 274880 49984 pts/1    S+   18:19   0:02 python ./ws-server.py
root       32122  1.0  1.1 256320 35776 pts/3    S+   18:19   0:00 python ./ws-client.py
root       32267  0.0  0.0 214080  1536 pts/2    S+   18:20   0:00 grep python
$ kill -9 32042
  1. Check the reverse proxy status(throw out unexcepted error):
$ python ./ws-rs.py 
INFO:     Started server process [31689]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8765 (Press CTRL+C to quit)
INFO:     ('127.0.0.1', 56556) - "WebSocket /ws" [accepted]
Exception is not set. when close ws connection. client: <Task pending name='client_to_server_task' coro=<_wait_client_then_send_to_server() running at /root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py:281> wait_for=<Future pending cb=[Task.task_wakeup()]>>, server:<Task pending name='server_to_client_task' coro=<_wait_server_then_send_to_client() running at /root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py:306> wait_for=<Future pending cb=[Task.task_wakeup()]>>
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/root/venv/lib/python3.12/site-packages/uvicorn/protocols/websockets/wsproto_impl.py", line 235, in run_asgi
  |     result = await self.app(self.scope, self.receive, self.send)  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 152, in __call__
  |     await self.app(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 362, in handle
  |     await self.app(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 95, in app
  |     await wrap_app_handling_exceptions(app, session)(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 93, in app
  |     await func(session)
  |   File "/root/venv/lib/python3.12/site-packages/fastapi/routing.py", line 383, in app
  |     await dependant.call(**solved_result.values)
  |   File "/root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/fastapi/router.py", line 112, in ws_proxy
  |     return await proxy.proxy(websocket=websocket, path=path)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py", line 806, in proxy
  |     return await self.send_request_to_target(
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py", line 567, in send_request_to_target
  |     async with stack:
  |   File "/opt/python/lib/python3.12/contextlib.py", line 754, in __aexit__
  |     raise exc_details[1]
  |   File "/opt/python/lib/python3.12/contextlib.py", line 737, in __aexit__
  |     cb_suppress = await cb(*exc_details)
  |                   ^^^^^^^^^^^^^^^^^^^^^^
  |   File "/opt/python/lib/python3.12/contextlib.py", line 231, in __aexit__
  |     await self.gen.athrow(value)
  |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 1349, in aconnect_ws
  |     async with _aconnect_ws(
  |   File "/opt/python/lib/python3.12/contextlib.py", line 231, in __aexit__
  |     await self.gen.athrow(value)
  |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 1254, in _aconnect_ws
  |     async with session:
  |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 658, in __aexit__
  |     await self._exit_stack.aclose()
  |   File "/opt/python/lib/python3.12/contextlib.py", line 696, in aclose
  |     await self.__aexit__(None, None, None)
  |   File "/opt/python/lib/python3.12/contextlib.py", line 754, in __aexit__
  |     raise exc_details[1]
  |   File "/opt/python/lib/python3.12/contextlib.py", line 737, in __aexit__
  |     cb_suppress = await cb(*exc_details)
  |                   ^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 1041, in _background_receive
    |     await self.close(CloseReason.INTERNAL_ERROR, "Stream error")
    |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 985, in close
    |     await self.stream.write(data)
    |   File "/root/venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 365, in write
    |     await self._stream.write(buffer, timeout)
    |   File "/root/venv/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 50, in write
    |     await self._stream.send(item=buffer)
    |   File "/root/venv/lib/python3.12/site-packages/anyio/streams/tls.py", line 212, in send
    |     await self._call_sslobject_method(self._ssl_object.write, item)
    |   File "/root/venv/lib/python3.12/site-packages/anyio/streams/tls.py", line 172, in _call_sslobject_method
    |     raise EndOfStream from None
    | anyio.EndOfStream
    +------------------------------------

Expected Behavior

Exceptions should be gracefully handled, not thrown out (or exposed).

Configuration

  • Python 3.12.3
  • httpx 0.28.1
  • httpx-ws 0.7.2
  • FastAPI 0.115.6
  • fastAPI-proxy-lib 0.3.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions