[CI] Windows azure repository tests fail with AccessControlException in AzureRepositoryPlugin

[williamrandolph]

Build scan: https://gradle-enterprise.elastic.co/s/3n2pftsr72wdm and many others

Repro line: None generated, but should be something like:

./gradlew ':x-pack:plugin:repository-encrypted:qa:azure:internalClusterTest' on a Windows machine.

Reproduces locally?: yes, on Windows

Applicable branches: master, 7.x

Failure history:

https://gradle-enterprise.elastic.co/scans?search.relativeStartTime=P7D&search.tags=WINDOWS&search.tags=master&search.tasks=checkPart1&search.timeZoneId=America/New_York

I believe it's been failing since this PR was merged: #72995

Failure excerpt:

> Task :x-pack:plugin:repository-encrypted:qa:azure:internalClusterTest
OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.


access denied ("java.io.FilePermission" "C:\Windows\system32\drivers\etc\hosts" "read")
java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Windows\system32\drivers\etc\hosts" "read")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
	at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
	at java.base/java.io.File.exists(File.java:826)
	at io.netty.resolver.HostsFileParser.locateHostsFile(HostsFileParser.java:59)
	at io.netty.resolver.HostsFileParser.parseSilently(HostsFileParser.java:85)
	at io.netty.resolver.DefaultHostsFileEntriesResolver.parseEntries(DefaultHostsFileEntriesResolver.java:75)
	at io.netty.resolver.DefaultHostsFileEntriesResolver.<init>(DefaultHostsFileEntriesResolver.java:37)
	at io.netty.resolver.HostsFileEntriesResolver.<clinit>(HostsFileEntriesResolver.java:28)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:375)
	at org.elasticsearch.repositories.azure.AzureRepositoryPlugin.lambda$static$0(AzureRepositoryPlugin.java:63)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
	at org.elasticsearch.repositories.azure.AzureRepositoryPlugin.<clinit>(AzureRepositoryPlugin.java:61)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:375)
	at org.elasticsearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:169)
	at org.elasticsearch.test.ESTestCase.<clinit>(ESTestCase.java:229)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:466)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:631)

Whatever fix we find for this, it is probably relevant to the work needed for issue #73479

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[iverase]

Here is a case that seems related although a bit different: https://gradle-enterprise.elastic.co/s/yp55eggepsbv6

error:

»    ↓ errors and warnings from C:\Users\jenkins\workspace\platform-support\59\qa\smoke-test-plugins\build\testclusters\integTest-0\logs\integTest.log ↓
» [2021-06-02T11:23:48,451][WARN ][o.a.h.u.Shell            ] [integTest-0] Did not find winutils.exe: java.io.FileNotFoundException: Hadoop bin directory does not exist: C:\Users\jenkins\workspace\platform-support\59\qa\smoke-test-plugins\build\testclusters\integTest-0\tmp\hadoop18381456726111058927\bin -see https://wiki.apache.org/hadoop/WindowsProblems
» [2021-06-02T11:23:49,124][WARN ][stderr                   ] [integTest-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
» [2021-06-02T11:23:49,126][WARN ][stderr                   ] [integTest-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
» [2021-06-02T11:23:49,126][WARN ][stderr                   ] [integTest-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
» [2021-06-02T11:23:49,342][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [integTest-0] fatal error in thread [main], exiting
»  java.lang.ExceptionInInitializerError: null
»  	at java.lang.Class.forName0(Native Method) ~[?:?]
»  	at java.lang.Class.forName(Class.java:315) ~[?:?]
»  	at org.elasticsearch.repositories.azure.AzureRepositoryPlugin.lambda$static$0(AzureRepositoryPlugin.java:63) ~[?:?]
»  	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
»  	at org.elasticsearch.repositories.azure.AzureRepositoryPlugin.<clinit>(AzureRepositoryPlugin.java:61) ~[?:?]
»  	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
»  	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
»  	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
»  	at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
»  	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:671) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:614) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:434) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:153) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.node.Node.<init>(Node.java:325) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:391) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Windows\system32\drivers\etc\hosts" "read")
»  	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
»  	at java.security.AccessController.checkPermission(AccessController.java:895) ~[?:?]
»  	at java.lang.SecurityManager.checkPermission(SecurityManager.java:322) ~[?:?]
»  	at java.lang.SecurityManager.checkRead(SecurityManager.java:661) ~[?:?]
»  	at java.io.File.exists(File.java:815) ~[?:?]
»  	at io.netty.resolver.HostsFileParser.locateHostsFile(HostsFileParser.java:59) ~[?:?]
»  	at io.netty.resolver.HostsFileParser.parseSilently(HostsFileParser.java:85) ~[?:?]
»  	at io.netty.resolver.DefaultHostsFileEntriesResolver.parseEntries(DefaultHostsFileEntriesResolver.java:75) ~[?:?]
»  	at io.netty.resolver.DefaultHostsFileEntriesResolver.<init>(DefaultHostsFileEntriesResolver.java:37) ~[?:?]
»  	at io.netty.resolver.HostsFileEntriesResolver.<clinit>(HostsFileEntriesResolver.java:28) ~[?:?]

[ywangd]

Happened again:

• https://gradle-enterprise.elastic.co/s/wgk3kq7wbzgwg
• https://gradle-enterprise.elastic.co/s/j7xtr5g7tepj4

[benwtrent]

https://gradle-enterprise.elastic.co/s/oclz34bo62jno

Very similar failure. I am not sure we should mute all the :x-pack:qa:smoke-test-plugins running on windows...

== Log output of node `node{:x-pack:qa:smoke-test-plugins:integTest-0}` ===

»    ↓ errors and warnings from C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\logs\integTest.log ↓
» [2021-06-07T11:44:44,327][WARN ][o.a.h.u.Shell            ] [integTest-0] Did not find winutils.exe: java.io.FileNotFoundException: Hadoop bin directory does not exist: C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\tmp\hadoop4217403915129519326\bin -see https://wiki.apache.org/hadoop/WindowsProblems
» [2021-06-07T11:44:45,047][WARN ][stderr                   ] [integTest-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
» [2021-06-07T11:44:45,048][WARN ][stderr                   ] [integTest-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
» [2021-06-07T11:44:45,049][WARN ][stderr                   ] [integTest-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
» [2021-06-07T11:44:45,244][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [integTest-0] fatal error in thread [main], exiting
»  java.lang.ExceptionInInitializerError: null
»  	at java.lang.Class.forName0(Native Method) ~[?:?]
»  	at java.lang.Class.forName(Class.java:315) ~[?:?]
»  	at org.elasticsearch.repositories.azure.AzureRepositoryPlugin.lambda$static$0(AzureRepositoryPlugin.java:63) ~[?:?]
»  	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
»  	at org.elasticsearch.repositories.azure.AzureRepositoryPlugin.<clinit>(AzureRepositoryPlugin.java:61) ~[?:?]
»  	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
»  	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
»  	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
»  	at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
»  	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:671) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:614) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:434) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:153) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.node.Node.<init>(Node.java:326) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.node.Node.<init>(Node.java:259) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:225) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:391) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Windows\system32\drivers\etc\hosts" "read")
»  	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
»  	at java.security.AccessController.checkPermission(AccessController.java:895) ~[?:?]
»  	at java.lang.SecurityManager.checkPermission(SecurityManager.java:322) ~[?:?]
»  	at java.lang.SecurityManager.checkRead(SecurityManager.java:661) ~[?:?]
»  	at java.io.File.exists(File.java:815) ~[?:?]
»  	at io.netty.resolver.HostsFileParser.locateHostsFile(HostsFileParser.java:59) ~[?:?]
»  	at io.netty.resolver.HostsFileParser.parseSilently(HostsFileParser.java:85) ~[?:?]
»  	at io.netty.resolver.DefaultHostsFileEntriesResolver.parseEntries(DefaultHostsFileEntriesResolver.java:75) ~[?:?]
»  	at io.netty.resolver.DefaultHostsFileEntriesResolver.<init>(DefaultHostsFileEntriesResolver.java:37) ~[?:?]
»  	at io.netty.resolver.HostsFileEntriesResolver.<clinit>(HostsFileEntriesResolver.java:28) ~[?:?]
»  	... 25 more
»   ↓ last 40 non error or warning messages from C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\logs\integTest.log ↓
» [2021-06-07T11:44:01.598388100Z] [BUILD] Configuring custom cluster specific distro directory: C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT
» [2021-06-07T11:44:02.794662800Z] [BUILD] Copying additional config files from distro [C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT\config\elasticsearch.yml, C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT\config\jvm.options, C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT\config\jvm.options.d, C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT\config\log4j2.properties, C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT\config\roles.yml, C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT\config\role_mapping.yml, C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT\config\users, C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT\config\users_roles]
» [2021-06-07T11:44:02.799556800Z] [BUILD] installing 20 plugins in a single transaction
» [2021-06-07T11:44:31.062192800Z] [BUILD] installed plugins
» [2021-06-07T11:44:31.063176400Z] [BUILD] Creating elasticsearch keystore with password set to []
» [2021-06-07T11:44:32.219394500Z] [BUILD] Installing 0 modules
» [2021-06-07T11:44:32.219394500Z] [BUILD] Setting up 1 users
» [2021-06-07T11:44:34.124664800Z] [BUILD] Starting Elasticsearch process
» [2021-06-07T11:44:38,557][INFO ][o.e.n.Node               ] [integTest-0] version[8.0.0-SNAPSHOT], pid[7884], build[default/zip/9146606e49b23cbd555fd6603f45d5c666ef047f/2021-06-07T10:29:43.405073200Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.2/11.0.2+7]
» [2021-06-07T11:44:38,562][INFO ][o.e.n.Node               ] [integTest-0] JVM home [C:\Users\jenkins\.java\openjdk11], using bundled JDK [false]
» [2021-06-07T11:44:38,562][INFO ][o.e.n.Node               ] [integTest-0] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=logs, -XX:ErrorFile=logs\hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs\gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms512m, -Xmx512m, -ea, -esa, -Dingest.geoip.downloader.enabled.default=false, -Dio.netty.leakDetection.level=paranoid, -XX:MaxDirectMemorySize=268435456, -Delasticsearch, -Des.path.home=C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\distro\8.0.0-DEFAULT, -Des.path.conf=C:\Users\jenkins\workspace\platform-support\69\x-pack\qa\smoke-test-plugins\build\testclusters\integTest-0\config, -Des.distribution.flavor=default, -Des.distribution.type=zip, -Des.bundled_jdk=true]
» [2021-06-07T11:44:45.670563700Z] [BUILD] Stopping node
build complete, generating: C:\Users\jenkins\workspace\platform-support\69\build\69.tar.bz2