The socket is stuck in HTTP long-polling with over 100 concurrent connections

[timurkhakhalev]

Hello!
Im using Artillery.io and trying to test performance on my aws instance.
My artillery scenario is:

config:
  target: "https://aws-instance"
  phases:
    - duration: 120
      arrivalRate: 15
  socketio:
    transports: [ "polling", "websocket" ]
  engines:
    socketio-v3: {}
scenarios:
  - name: "My sample scenario"
    engine: "socketio-v3"
    flow:
      # wait for the WebSocket upgrade (optional)
      - think: 20
      - emit:
          channel: "echo"
          data: "Hello from Artillery"
          response:
            channel: "echoResponse"
            data: "Hello from Artillery"
      - think: 5

nginx.conf:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 30000;
        debug_connection *.*.*.*;
        multi_accept on;
}

revers-proxy config

location /socket {
    proxy_pass http://localhost:PORT;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
  }

Report is

--------------------------------
Summary report @ 14:24:51(+0300)
--------------------------------

engine.socketio.emit: .......................................................... 763
engine.socketio.emit_rate: ..................................................... 11/sec
errors.Error: timeout: ......................................................... 1418
vusers.completed: .............................................................. 763
vusers.created: ................................................................ 2181
vusers.created_by_name.My sample scenario: ..................................... 2181
vusers.failed: ................................................................. 1418
vusers.session_length:
  min: ......................................................................... 26520.4
  max: ......................................................................... 80463
  median: ...................................................................... 46644.4
  p95: ......................................................................... 76904.9
  p99: ......................................................................... 80043.6

During this test I watch whats going on in Admin UI and I see (in "Namespaces" block) that after ~100 concurrent connection, next connections is stucked in long polling and not able to connect as websocket. At the "top" in instance I see that node uses maximum 6% of cpu (2 core) and 2-4% of RAM (4GB total + 4GB as swap).

At my local PC it works good, even with arrival rate 55 I get ~1300 concurrent connection which stay at websocket transport.

Also tried this script and after ~150 concurrent sockets I get "disconnect due to ping timeout" message.

Can you help me please what I did wrong?

[darrachequesne]

Hi! Please check the documentation here: https://socket.io/docs/v4/performance-tuning/#at-the-os-level

[timurkhakhalev]

Okay I didnt see about maximum number of available local ports, opened it

cat /proc/sys/net/ipv4/ip_local_port_range
10000   65535

but after 430 concurrent sockets got disconnects:
disconnect due to transport error

[darrachequesne]

Are there any error in the nginx logs? Could you please try to reach the Socket.IO server directly, without the nginx proxy in between?

[timurkhakhalev]

Yeah thank you alot! I added this configs in nginx but now stucked at 1777 connections as said in Admin UI

main:
worker_rlimit_nofile 1048576;

events:
 worker_connections 30000;

But in nginx error logs no errors
In nginx access logs I see new connections.

At 'top' util I see that nginx + node takes 37% cpu and 10% ram.

Any thoughts?

[darrachequesne]

Are you using the admin UI in "development" mode? It can be a bit CPU and memory hungry.

instrument(io, {
  mode: "production"
});

[timurkhakhalev]

Oh sorry it was my fault, jwt token got expired. Now I work good. Thank you again!