Potential memory leak for repeated creation of otelgrpc Server Unary Interceptor

[ihtkas]

Description

Calling otelgrpc.UnaryServerInterceptor or otelgrpc.NewServerHandler repeatedly in the program leads to a memory leak.

More context

We use the interceptor in an internal library implementation to process messages consumed from a queue service. We were creating the interceptor for each message received. This was working fine until we upgraded the library from v0.30.0 to v0.48.0. After a recent golang version upgrade from 1.18 to 1.21, we also upgraded the otelgrpc libraries. This leads to a constant increase in the memory of production servers eventually crashing with OOM. We have now resolved the issue by initializing the interceptor once in the program's lifetime to avoid the memory leak.

Environment

• OS: darwin
• Architecture: arm64
• Go Version: 1.21.6
• otelgrpc version: 0.48.0

Note: I Reproduced the issue in the above local environment. However, the initial issue was encountered in a different production environment.

Steps To Reproduce

-> main.go

package main

import (
	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
	"log"
	"math/rand"
	"net/http"
	_ "net/http/pprof"
	"time"
)

func main() {
	go func() {
		log.Println(http.ListenAndServe("localhost:6060", nil))
	}()
	for i := 0; i <= 16; i++ {
		go func() {
			for {
				otelgrpc.UnaryServerInterceptor()
				time.Sleep(time.Millisecond * time.Duration(rand.Intn(6)))
			}
		}()
	}
	select {}
}

Run the below commands
go build -o otelgrpc main.go ./otelgrpc

Run the below commands to debug the heap profile. I prefer web mode
go tool pprof http://localhost:6060/debug/pprof/heap

Expected behavior

No memory leaks.
One of the below options will help others avoid the issue

• Completely avoid memory leaks by changing the implementation of otelgrpc
• Return a cleanup function or way to perform a cleanup operation and free the object references from the methods UnaryServerInterceptor and NewServerHandler

[hthuwal]

@ihtkas I was facing a similar issue. On further investigation it seems to be resolved with: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.30.0