How can we troubleshoot OutOfMemoryError in Pinpoint 3.0 Collector Component? #11983
Open
Description
What version of pinpoint are you using?
Pinpoint 3.0
Describe your problem
I have deployed Pinpoint 3.0 using the official pinpoint-docker repository with docker-compose. The deployment seems successful, as all containers are running without errors. However, no trace data is visible in the Web UI after attempting to trace applications with the agent.
What have you done?
This issue does not occur 100% of the time. After I restart the collector service, it can recover briefly, but then the problem will reappear.
Configuration Details
- Host OS: CentOS 7.9
- Host Specs: 8-core CPU, 32 GB RAM
- Docker Deployment: Official
pinpoint-dockerrepository - Docker Compose File: Default configuration from the repository (no manual adjustments made).
Screenshots
Logs
This is pinpoint-collector logs
01-20 07:28:15.015 [el-Worker(16-2)] INFO i.g.n.N.connections : Transport failed
java.lang.OutOfMemoryError: Cannot reserve 32768 bytes of direct buffer memory (allocated: 10485032, limit: 10485760)
at java.nio.Bits.reserveMemory(Unknown Source) ~[?:?]
at java.nio.DirectByteBuffer.<init>(Unknown Source) ~[?:?]
at java.nio.ByteBuffer.allocateDirect(Unknown Source) ~[?:?]
at io.netty.buffer.UnpooledDirectByteBuf.allocateDirect(UnpooledDirectByteBuf.java:104) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:634) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:398) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) ~[netty-transport-native-unix-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) ~[netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
01-20 07:28:15.015 [verExecutor4377] WARN grpcAgentServerCallExecutor : The executor uses finite bounds for both maximum threads and work queue capacity, and is saturated. Check the maxPoolSize, queueCapacity, and HBase options in the configuration. maxPoolSize=8, rejectedCount=6010
01-20 07:28:15.015 [ecycleFlusher-1] WARN c.n.p.c.r.g.s.KeepAliveService : Failed to update state. closeState:false lifeCycle=PingSession{id=4290, header=Header{name='agent', agentId='172.20.6.89', agentName='null', applicationName='***-record', agentStartTime=1737037610457, socketId=589, serviceType=1210, supportCommandCodeList=null, properties={}}, eventIdAllocator=2, serviceType=-1, updated=false, lastPingTimeMillis=1737355851800} Running/Agent ping
com.navercorp.pinpoint.common.hbase.HbaseSystemException: callTimeout=10000, callDuration=10433: Failed to get user region lock in10000 ms. for accessing meta region server. row '172.20.6.89\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFEk}c\xBF\xA4' on table 'AgentEvent' at null
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.rethrowHbaseException(HbaseTemplate.java:886) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:854) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.put(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao.insert(HbaseAgentEventDao.java:85) ~[classes/:3.0.0]
at jdk.internal.reflect.GeneratedMethodAccessor65.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717) ~[spring-aop-6.1.5.jar:6.1.5]
at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao$$SpringCGLIB$$0.insert(<generated>) ~[classes/:3.0.0]
at com.navercorp.pinpoint.collector.service.AgentEventService.insert(AgentEventService.java:44) ~[classes/:3.0.0]
at jdk.internal.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174) ~[spring-context-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717) ~[spring-aop-6.1.5.jar:6.1.5]
at com.navercorp.pinpoint.collector.service.AgentEventService$$SpringCGLIB$$0.insert(<generated>) ~[classes/:3.0.0]
at com.navercorp.pinpoint.collector.service.async.AgentEventAsyncTaskService.handleEvent(AgentEventAsyncTaskService.java:51) ~[classes/:3.0.0]
at com.navercorp.pinpoint.collector.receiver.grpc.service.KeepAliveService.updateState(KeepAliveService.java:98) ~[classes/:3.0.0]
at com.navercorp.pinpoint.collector.receiver.grpc.service.KeepAliveService.updateState(KeepAliveService.java:59) ~[classes/:3.0.0]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130) ~[spring-context-6.1.5.jar:6.1.5]
at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124) ~[spring-context-6.1.5.jar:6.1.5]
at io.micrometer.observation.Observation.observe(Observation.java:499) [micrometer-observation-1.12.4.jar:1.12.4]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124) [spring-context-6.1.5.jar:6.1.5]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.1.5.jar:6.1.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.SocketTimeoutException: callTimeout=10000, callDuration=10433: Failed to get user region lock in10000 ms. for accessing meta region server. row '172.20.6.89\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFEk}c\xBF\xA4' on table 'AgentEvent' at null
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:157) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:566) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:554) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:243) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:852) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
... 45 more
Caused by: org.apache.hadoop.hbase.client.LockTimeoutException: Failed to get user region lock in10000 ms. for accessing meta region server.
at org.apache.hadoop.hbase.client.ConnectionImplementation.takeUserRegionLock(ConnectionImplementation.java:1133) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:996) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:897) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HRegionLocator.lambda$getRegionLocation$0(HRegionLocator.java:80) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HRegionLocator.tracedLocationFuture(HRegionLocator.java:162) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:79) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:71) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:60) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:219) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:566) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:554) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:243) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:852) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
... 45 more
01-20 07:28:15.015 [CallExecutor445] WARN c.n.p.c.r.g.s.KeepAliveService : Failed to update state. closeState:false lifeCycle=PingSession{id=12817, header=Header{name='agent', agentId='***-ami-03', agentName='null', applicationName='***-ami', agentStartTime=1736870946684, socketId=1379, serviceType=1210, supportCommandCodeList=null, properties={}}, eventIdAllocator=1, serviceType=-1, updated=false, lastPingTimeMillis=1737358035714} Running/Agent connected
com.navercorp.pinpoint.common.hbase.HbaseSystemException: callTimeout=10000, callDuration=10433: Failed to get user region lock in10000 ms. for accessing meta region server. row '***-ami-03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFEk}cx\xB5' on table 'AgentEvent' at null
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.rethrowHbaseException(HbaseTemplate.java:886) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:854) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.put(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao.insert(HbaseAgentEventDao.java:85) ~[classes/:3.0.0]
at jdk.internal.reflect.GeneratedMethodAccessor65.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717) ~[spring-aop-6.1.5.jar:6.1.5]
at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao$$SpringCGLIB$$0.insert(<generated>) ~[classes/:3.0.0]
at com.navercorp.pinpoint.collector.service.AgentEventService.insert(AgentEventService.java:44) ~[classes/:3.0.0]
at jdk.internal.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174) ~[spring-context-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717) ~[spring-aop-6.1.5.jar:6.1.5]
at com.navercorp.pinpoint.collector.service.AgentEventService$$SpringCGLIB$$0.insert(<generated>) ~[classes/:3.0.0]
at com.navercorp.pinpoint.collector.service.async.AgentEventAsyncTaskService.handleEvent(AgentEventAsyncTaskService.java:51) ~[classes/:3.0.0]
at com.navercorp.pinpoint.collector.receiver.grpc.service.KeepAliveService.updateState(KeepAliveService.java:98) [classes/:3.0.0]
at com.navercorp.pinpoint.collector.receiver.grpc.service.KeepAliveService.updateState(KeepAliveService.java:74) [classes/:3.0.0]
at com.navercorp.pinpoint.collector.receiver.grpc.service.AgentLifecycleListener.connect(AgentLifecycleListener.java:61) [classes/:3.0.0]
at com.navercorp.pinpoint.grpc.server.lifecycle.DefaultPingEventHandler.connect(DefaultPingEventHandler.java:59) [pinpoint-grpc-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.collector.receiver.grpc.service.AgentService$2.onNext(AgentService.java:106) [classes/:3.0.0]
at com.navercorp.pinpoint.collector.receiver.grpc.service.AgentService$2.onNext(AgentService.java:94) [classes/:3.0.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262) [grpc-stub-1.62.2.jar:1.62.2]
at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) [grpc-api-1.62.2.jar:1.62.2]
at io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) [grpc-api-1.62.2.jar:1.62.2]
at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) [grpc-api-1.62.2.jar:1.62.2]
at io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) [grpc-api-1.62.2.jar:1.62.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:329) [grpc-core-1.62.2.jar:1.62.2]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:314) [grpc-core-1.62.2.jar:1.62.2]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:834) [grpc-core-1.62.2.jar:1.62.2]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.62.2.jar:1.62.2]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.62.2.jar:1.62.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.SocketTimeoutException: callTimeout=10000, callDuration=10433: Failed to get user region lock in10000 ms. for accessing meta region server. row '***-ami-03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFEk}cx\xB5' on table 'AgentEvent' at null
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:157) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:566) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:554) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:243) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:852) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
... 47 more
Caused by: org.apache.hadoop.hbase.client.LockTimeoutException: Failed to get user region lock in10000 ms. for accessing meta region server.
at org.apache.hadoop.hbase.client.ConnectionImplementation.takeUserRegionLock(ConnectionImplementation.java:1133) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:996) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:897) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HRegionLocator.lambda$getRegionLocation$0(HRegionLocator.java:80) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HRegionLocator.tracedLocationFuture(HRegionLocator.java:162) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:79) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:71) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:60) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:219) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:566) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:554) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:243) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:852) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
... 47 more
Metadata
Assignees
Labels
No labels