Open
Description
Describe the bug
Trying to use apm on elasticsearch jvm to get jvm metrics.
APM agent finds APM server, but fails to send any data with java.net.SocketPermission access denied.
Steps to reproduce
elasticapm.properties:
service_name=elasticsearch
server_urls=http://apm:8200
dockercompose.yml:
elasticsearch:
image: elasticsearch:7.9.3
mem_limit: 768m
expose:
- "9200"
environment:
ES_JAVA_OPTS: -Xmx768m
HTTP_PORT: 9200
discovery.type: single-node
ELASTIC_APM_SERVER_URLS: http://apm:8200
labels:
co.elastic.logs/module: elasticsearch
app: elasticsearch
volumes:
- es-data:/usr/share/elasticsearch/data
- ./config/es/jvm.options:/usr/share/elasticsearch/config/jvm.options:ro
- ./config/es/apm:/usr/share/elasticsearch/config/apm:ro
apm:
image: store/elastic/apm-server:7.9.3
mem_limit: 64m
depends_on:
- elasticsearch
expose:
- "8200"
labels:
app: apm
environment:
apm-server.host: localhost:8200
output.elasticsearch.hosts: http://elasticsearch:9200
volumes:
- apm-data:/usr/share/apm-server/data
Expected behavior
Data sent to apm server.
Debug logs
docker logs docker_elasticsearch_1 | grep apm
2021-01-20 18:17:39,569 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.20.0 as elasticsearch on Java 15 Runtime version: 15+36-1562 VM version: 15+36-1562 (Oracle Corporation) Linux 4.19.128-microsoft-standard
2021-01-20 18:17:39,570 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - VM Arguments: [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-11682935632529430251, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -javaagent:/usr/share/elasticsearch/config/apm/elastic-apm-agent.jar, -Des.cgroups.hierarchy.override=/, -Xmx768m, -XX:MaxDirectMemorySize=402653184, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]
2021-01-20 18:17:39,571 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2021-01-20 18:17:41,275 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: { "build_date": "2020-10-16T08:05:05Z", "build_sha": "af7b123f289d79c34c9da11471803f457eafa5c0", "version": "7.9.3"}
2021-01-20 18:17:41,301 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
{"type": "server", "timestamp": "2021-01-20T18:17:43,849Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "daa16e1cfd08", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-11682935632529430251, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -javaagent:/usr/share/elasticsearch/config/apm/elastic-apm-agent.jar, -Des.cgroups.hierarchy.override=/, -Xmx768m, -XX:MaxDirectMemorySize=402653184, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
2021-01-20 18:18:11,514 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type JSON_WRITER with this error: access denied ("java.net.SocketPermission" "apm:8200" "connect,resolve")