Souce connector can not be created successfully on pulsar cluster that installed by the offical helm chart

[xiemeilong]

Describe the bug
I install pulsar on kubernetes using official helm chart . All my connector can not created successfully, even the builtin( for example debezium-mysql ).
I found the OutOfDirectMemoryError of direct memory in the log of broker. But the exception still exists after increate direct memory from 256MB to 2GB.

But the connector runs in local mode normally.

To Reproduce

1. helm upgrade --install pulsar apache/pulsar --timeout 10m --set initialize=true
   2

 bin/pulsar-admin source create --tenant iot --namespace pipe  \
--name xxxxx --destination-topic-name xxx/xxx/xxx --parallelism 1 --archive builtin://debezium-mysql \
--source-config  '{ "database.hostname": "mysql", "database.port": "3306", "database.user": "xxxx", "database.password": "xxxxx","database.server.id": "1","database.server.name": "xxxxx","database.whitelist": "xxxx", "table.whitelist": "xxxxxx","schemas.enable":false,"pulsar.service.url": "pulsar://pulsar-proxy:6650" }'

3. the broker log

06:24:14.597 [DL-io-0] ERROR org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl - Unable to allocate memory
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 251658247, max: 268435456)
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:742) ~[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:697) ~[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:758) ~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:734) ~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245) ~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:227) ~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:147) ~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]

Expected behavior
The source connector created successfully.

Additional context
helm chart version： 2.7.0 and 2.6.3-2
kubernetes：v1.18.10

[yufan022]

You can add configuration in helm:

bookkeeper:
  configData:
    nettyMaxFrameSizeBytes: "104857600"
    PULSAR_MEM: "-Xms128m -Xmx1024m -XX:MaxDirectMemorySize=1024m"
broker:
  configData:
    maxMessageSize: "104857600"
    PULSAR_MEM: "-Xms128m -Xmx1024m -XX:MaxDirectMemorySize=1024m"

These can change the configuration and JVM env of Broker and BooKeeper.

[xiemeilong]

@yufan022 I had done that. It not works.

[xiemeilong]

This is my chart values

initialize: true
clusterDomain: yunmotec-dev.local
proxy:
  replicaCount: 1
broker:
  replicaCount: 1
  resources:
    requests:
      memory: 2Gi
      cpu: 2
  configData:
    nettyMaxFrameSizeBytes: "104857600"
    PULSAR_MEM: >
      -Xms256m -Xmx2048m -XX:MaxDirectMemorySize=2048m
zookeeper:
  replicaCount: 1
bookkeeper:
  replicaCount: 4
  resources:
    requests:
      memory: 1Gi
      cpu: 1
  configData:
    nettyMaxFrameSizeBytes: "104857600"
    PULSAR_MEM: >
      -Xms256m -Xmx1024m -XX:MaxDirectMemorySize=1024m

[xiemeilong]

It seems cased by KubernetesRuntimeFactory， this issue not exists after set functionRuntimeFactoryClassName to ThreadRuntimeFactory.

[xiemeilong]

@codelipenghui why did you close this?

[armangurkan]

This should be open we have the same issue in 2.8.2