Skip to content

HTTP Listener Becomes Unresponsive Under Load and Unable to Recover #1830

@spectrum248

Description

@spectrum248

Issue:
Under high load conditions, the HTTP listener in cpprestsdk becomes unresponsive. It stops accepting new GET/POST requests, only completing the in-flight ones. The listener cannot be restarted; it requires a full application restart to recover.


Steps to Reproduce:

  1. Set up an HTTP listener using cpprestsdk.
  2. Simulate high concurrency by sending a large number of GET/POST requests.
  3. After reaching a certain load threshold, observe that the listener stops accepting new requests.
  4. Only in-flight requests are completed, and no new requests are processed.
  5. Attempting to restart the listener does not resolve the issue; a full application restart is required.

Expected Behavior:

  • The HTTP listener should continue accepting and processing requests even under high concurrency.
  • The listener should handle high-load scenarios without requiring a full restart of the application or any other recovery method.

Analysis:

  • Thread Stuck: All threads are observed stuck on .wait(), indicating possible thread exhaustion or a deadlock condition.
  • Resource Limits: It is likely that the system is hitting resource limits, such as threads or sockets, under high concurrency.
  • Non-cpprest Features Unaffected: Other features (e.g., healthStatus) remain unaffected and continue to function as expected under load.
  • Lack of Async Handling or Throttling: cpprestsdk does not appear to have built-in asynchronous handling or throttling mechanisms to manage load effectively under high concurrency.

Environment:

  • cpprestsdk version: 2.10.18-1build2
  • Operating System: Ubuntu 22.04.5 LTS
  • Compiler: g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
  • C++ Standard: C++14
  • Network Configuration: Default

Additional Context:

This issue has been observed in a production environment where the application serves high-traffic endpoints. The HTTP listener, built using cpprestsdk, handles a mix of lightweight GET requests and heavier POST requests. During periods of high load (e.g., when many simultaneous requests are being made), the listener becomes unresponsive after a certain threshold of concurrent connections is reached.


Request for Guidance:

If anyone has encountered a similar issue or knows of a potential recovery strategy or workaround, any insights would be appreciated. Is there a way to recover from this deadlock or improve concurrency handling in cpprestsdk without needing a full restart of the application?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions