Memory leak while proxying data through persistent connection through NJS

[coolkingh]

Describe the bug

When a persistent ( websocket ) connection is proxied through NJS through single server, every time when packet comes to nginx stream server and then njs proxy it to tomcat server without offloading the upstream event, Nginx process memory increases every time which is handling this connection and does not free the memory even when the packet is sent to tomcat. When the connection disconnects then it frees.

When a persistent connection is proxied through NJS through multiple stream server, it keeps on increasing the memory and even when connection closes it doesn't free the memory.

• I have shared the basic configuration to reproduce

• [yes ] The bug is reproducible with the latest version of njs.

• [ yes ] I minimized the code and NGINX configuration to the smallest
  possible to reproduce the issue.

To reproduce

Steps to reproduce the behavior:

• JS script

var isFirst = false;
var upstream = "172.28.9.90:3389";

function upstream_type() {
    return upstream_t;
}
function cupstream_type(s) {
        return upstream;
}
function detect_http(s) {

	s.on('upload', function (data, flags) {
			
		var n = data.indexOf('connect');
		if (n != -1) {
			upstream_t = "tcpstream";
		}
		if (data.length || flags.last) {
				s.done();
		}
	});
}
function validate(s) {
    s.on('upload', async function (data, flags) {
        var n = data.indexOf('\r\n');
        if (n != -1) {
			
			/* Do request procesing */
			
			if (data.length || flags.last){
				s.done();
            }
        }
    });
}

function connect(s) {

	s.on("upstream", async (data, flags) => {
		if(isFirst){
			/* Memory leak if this step is included 
			 * Do something each time packet comes  */
			s.sendUpstream(data, flags);
		}
		else{
			var n = data.indexOf('\r\n\r\n');
			isFirst = false;	
			if (n != -1) {
				s.sendDownstream("200 OK", flags);
			}
		}
	});
}

Your NGINX configuration here

js_set $upstream stream.upstream_type;
js_set $cupstream stream.cupstream_type ;

upstream tcpstream {
        server 127.0.0.1:9002;
}

server {
        listen 127.0.0.1:9002 ;
        js_preread stream.validate;
        js_filter stream.connect;
        proxy_pass $cupstream;
}

server {
        listen 443 ssl;
        ssl_protocols TLSv1.3;
        ssl_certificate "sslcert.cer";
        ssl_certificate_key "sslcert.pem";
        js_preread stream.detect_http;
        proxy_pass $upstream;
}

Your NGINX logs here

[root@node32 res]# ps aux | awk '{print $2 " " $6/1024 " MB\t\t" $11}' | sort -n | grep nginx
236457 15.6484 MB nginx:
236458 28.9727 MB nginx:
236459 28.1094 MB nginx:
236460 26.8906 MB nginx:
236461 26.8711 MB nginx:

[root@node32 res]# netstat -natplu | grep 3389
tcp 0 0 172.28.8.32:36894 172.28.9.90:3389 ESTABLISHED 236458/nginx: worke

[root@node32 res]# ps aux | awk '{print $2 " " $6/1024 " MB\t\t" $11}' | sort -n | grep nginx
236457 15.6484 MB nginx:
236458 28.9727 MB nginx:
236459 28.1094 MB nginx:
236460 26.8906 MB nginx:
236461 26.8711 MB nginx:
[root@node32 res]# ps aux | awk '{print $2 " " $6/1024 " MB\t\t" $11}' | sort -n | grep nginx
236457 15.6484 MB nginx:
236458 29.7461 MB nginx:
236459 28.1094 MB nginx:
236460 26.8906 MB nginx:
236461 26.8711 MB nginx:
[root@node32 res]# ps aux | awk '{print $2 " " $6/1024 " MB\t\t" $11}' | sort -n | grep nginx
236457 15.6484 MB nginx:
236458 30.5195 MB nginx:
236459 28.1094 MB nginx:
236460 26.8906 MB nginx:
236461 26.8711 MB nginx:
[root@node32 res]# netstat -natplu | grep 3389 // No connection as it is closed
[root@node32 res]# ps aux | awk '{print $2 " " $6/1024 " MB\t\t" $11}' | sort -n | grep nginx
236457 15.6484 MB nginx:
236458 30.7773 MB nginx: // memory not freed.. waited till more than 1 hour
236459 28.1094 MB nginx:
236460 26.8906 MB nginx:
236461 26.8711 MB nginx:

• Output of the nginx -V command if applicable.

nginx version: nginx/1.28.0
built by gcc 11.4.1 20231218 (Red Hat 11.4.1-3.0.1) (GCC)
built with OpenSSL 3.2.2 4 Jun 2024
TLS SNI support enabled
configure arguments: --with-debug --with-cc-opt='-O0 -g' --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --modules-path=/usr/lib64/nginx/modules --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-stream_ssl_preread_module --with-stream_ssl_module --with-pcre --with-stream --with-threads --with-file-aio --add-module=../../njs/njs-master/nginx   

• Exact steps to reproduce the behavior

Expected behavior

Memory must free if the connection is closed. Moreover, it must also free the memory when packet is sent to the tomcat server. it increase memory to 4 GB per worker process and 8 worker process takes like 32 GB memory

Your environment

• Version of njs or specific commit
  code compiled with njs-master
• Version of NGINX if applicable
  1.28.0
• List of other enabled nginx modules if applicable
• OS: [e.g. Ubuntu 20.04]
  -Oracle

Additional context

Add any other context about the problem here.

[xeioex]

@coolkingh

Context: js_filter will be called each times until the end of the connection. If the connection is long, this code will grow memory for njs engine. NJS does not have a garbage collection, it frees all the allocated memory at the moment JS context is destroyed. QJS engine has a GC.

But specifically in your case, I suggest to use s.off() after you done processing in JS. When there are no registered event handlers the data will be passed by nginx without extra allocations.

[coolkingh]

@xeioex I cannot offload the event as I have to update the timestamp till the connection is live. If njs does not have garbage collection can we free the memory explicitly. When JS context is actually destroyed. memory is allocated by malloc how can we not deallocate it. What can be done if I want to free the memory in NJS

[xeioex]

@coolkingh

If you cannot offload the event I suggest to try qjs engine. It has garbage collector. Unfortunately, njs does not have a manual function to free an object.

[coolkingh]

@xeioex I tried with qjs engine. persistent connection was not established. I got no error. It seems like it got stuck somewhere.

NJS is much better than this. If memory is allocated it can be freed. Is it impossible to free memory for log persistenet connection in NJS itself.

moreover, it also stuck in normal rest api.

code used is same

function connect(s) {

	s.on("upstream", async (data, flags) => {
		if(isFirst){
			/* Memory leak if this step is included 
			 * Do something each time packet comes  */
			s.sendUpstream(data, flags);
		}
		else{
			var n = data.indexOf('\r\n\r\n');
			isFirst = false;	
			if (n != -1) {
				s.sendDownstream("200 OK", flags);
			}
		}
	});
}

[coolkingh]

@xeioex I also tried using event offload but it still doesn't work. I restarted nginx and then simple API worked and long persistent connection doesn't worked then again I tried simple API then it stucked.

This is simple code I used.
function connect(s) {

s.on("upstream", async (data, flags) => {

		var n = data.indexOf('\r\n\r\n');
		if (n != -1) {
			s.sendDownstream("200 OK", flags);
                           s.off("upstream");
		}
	
});

}

[coolkingh]

@xeioex anything on this .

[xeioex]

Hi @coolkingh,

I identified and fixed memory consumption issues with qjs engine. Feel free to test.
It works only in qjs, because, as I already said, njs has no GC. This means njs engine consumes memory as it processes more and more data.

[coolkingh]

@xeioex Thanks. QJS is working after rewriting the whole code. I have not used this patch but found memory leaks in QJS also. I will test this patch and update the same.

[xeioex]

Hi @coolkingh,

Please, let me know when and whether you are going to try the patch.

[coolkingh]

@xeioex I have checked the patch and found no leak in long connections in QJS engine.