Memory usage ever grows - data leak

[r4f4ss]

EDIT: Execution with parameter (3 networks) --netrorks "history,beacon,state"

The figure below is the dashboard, where one can see in the right-top corner the graph showing the memory that ever growing.

The same happens with metrics disables, the memory usage can be viewed with top and show the same behavior:

Eventually the PC freeze by lack of memory.

The cause is related with github.com/optimism-java/utp-go/buffers.NewSyncBuffer, as shown by pprof:

[GrapeBaBa]

@thinkAfCod It seems that the memory leak issue has reappeared

[r4f4ss]

Thanks for the explanation @GrapeBaBa , I believe you are correct.

I managed to reduce the leak by:

1. Setting a self delete after 180 seconds of t.activeCallByAuth[nonce] here
2. Setting a self close after 20 seconds of created utpConn at utp-go lib here
3. Setting a self close after 20 seconds of created utpConn droped of backlog (full) here

It is not a solution since it breake the properly workig of shisui, but serves to prove places where the leak happens. There is still a slow memory usage grow, not solved yet.

Looking forward to see @thinkAfCod approach.

[thinkAfCod]

I'm trying this solution and will know by tomorrow if it works: #219 .

And, how can I reproduce this more quickly? I currently have 10 nodes running on the server, but their memory growth is very slow. It's not like what you showed, where the memory can grow to around 4GB in a hour.

[GrapeBaBa]

I'm trying this solution and will know by tomorrow if it works: #219 .

And, how can I reproduce this more quickly? I currently have 10 nodes running on the server, but their memory growth is very slow. It's not like what you showed, where the memory can grow to around 4GB in a hour.

We could setup influxdb+grafana locally

[r4f4ss]

And, how can I reproduce this more quickly? I currently have 10 nodes running on the server, but their memory growth is very slow. It's not like what you showed, where the memory can grow to around 4GB in a hour.

I have been using locally pprof + grafana and influxdb

As a qick rule, the plot of memory in grafana should become constant under 1 hour, but a day test may be more precise becuse it deal with high and low trafic periods. Basically, if the memory only grows, even after the garbage collector pattern in the plot (it is like a triangle), there is some leak.

Btw, debug with pprof implemented in the pr #219 is a good idea, it should be incorporated in shisui.

[r4f4ss]

@thinkAfCod I really can not explain why my measures are diferent from the memory usage you have in your server, I will let it runnig for several hours (with and without grafana), at least, to have a more conclusive answer.

[thinkAfCod]

@thinkAfCod I really can not explain why my measures are diferent from the memory usage you have in your server, I will let it runnig for several hours (with and without grafana), at least, to have a more conclusive answer.

Don't worry, this PR( #219 ) has no effect. The memory leak occurs in the dispatch method of Udp_v5, where some cached packets are not being cleared. I need to think about how to clean up these caches.

[thinkAfCod]

in heap of pprof, this part's memory usage keeps increasing:

[r4f4ss]

Just noted that runnig the three subnets (--networks "history,beacon,state") increase considerably the memory growth , I guess this bug related to execution with three networks.

I confirm that executing with default configuration (only history) gives me a slow memory growth and a pprof reported above by @thinkAfCod .

[thinkAfCod]

The memory leak is likely occurring at the point where new UTP connections are accepted.

There should be a large number of connections timing out here, but they are not being cleaned up. A cleanup task needs to be added.

func (l *Listener) AcceptUTPContext(ctx context.Context, connIdSeed uint32) (*Conn, error) {
	l.recordRequireConnId(connIdSeed)
	ticker := time.NewTicker(time.Millisecond * 17)
	defer ticker.Stop()
	for {
		select {
		case newConn, ok := <-l.acceptChan:
			if ok {
				if conn := l.filterConn(newConn, connIdSeed); conn != nil {
					return conn, nil
				}
				continue
			}
			err := l.encounteredError
			if err == nil {
				err = l.makeOpError("accept", net.ErrClosed)
			}
			return nil, err
		case <-ticker.C:
			if conn := l.removeAndGetCommingConn(connIdSeed); conn != nil {
				return conn, nil
			}
		case <-ctx.Done():
			return nil, ctx.Err()
		}
	}
}

[r4f4ss]

I tried to close every connection (newConn and conn) after 20 seconds with a goroutine.
After run for about 1:30 hours it does not solves

May be conn is not been closed properly or somewhere is holding the conn.

[thinkAfCod]

Yes, it is necessary to modify the code of the utp-go library.

Simply modifying the code of shisui will not have any effect.