[CI] AzureStorageCleanupThirdPartyTests AzureStorageRepositoryClientYamlTestSuiteIT

[albertzaharovits]

Two test suites AzureStorageCleanupThirdPartyTests and AzureStorageRepositoryClientYamlTestSuiteIT fail with the below two exceptions:

            com.microsoft.azure.storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
                at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
                at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
                at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:196)
                at com.microsoft.azure.storage.blob.BlobBatchOperation.execute(BlobBatchOperation.java:24)
                at com.microsoft.azure.storage.blob.CloudBlobClient.executeBatch(CloudBlobClient.java:796)
                at com.microsoft.azure.storage.blob.CloudBlobClient.executeBatch(CloudBlobClient.java:762)
                at org.elasticsearch.repositories.azure.AzureStorageService.lambda$deleteBlobsIgnoringIfNotExists$3(AzureStorageService.java:210)
                at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:69)
                at java.security.AccessController.doPrivileged(Native Method)
                at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:68)
                at org.elasticsearch.repositories.azure.AzureStorageService.deleteBlobsIgnoringIfNotExists(AzureStorageService.java:210)
                at org.elasticsearch.repositories.azure.AzureStorageService.deleteBlobDirectory(AzureStorageService.java:247)
                at org.elasticsearch.repositories.azure.AzureBlobStore.deleteBlobDirectory(AzureBlobStore.java:96)
                at org.elasticsearch.repositories.azure.AzureBlobContainer.delete(AzureBlobContainer.java:108)
                ... 6 more

            java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "setFactory")
                at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
                at java.security.AccessController.checkPermission(AccessController.java:886)
                at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
                at java.lang.SecurityManager.checkSetFactory(SecurityManager.java:1664)
                at javax.net.ssl.HttpsURLConnection.setSSLSocketFactory(HttpsURLConnection.java:363)
                at com.microsoft.azure.storage.core.BaseRequest.createURLConnection(BaseRequest.java:233)
                at com.microsoft.azure.storage.blob.BlobRequest.createURLConnection(BlobRequest.java:481)
                at com.microsoft.azure.storage.blob.BlobRequest.listBlobs(BlobRequest.java:1325)
                at com.microsoft.azure.storage.blob.CloudBlobContainer$7.buildRequest(CloudBlobContainer.java:1424)
                at com.microsoft.azure.storage.blob.CloudBlobContainer$7.buildRequest(CloudBlobContainer.java:1412)
                at com.microsoft.azure.storage.core.ExecutionEngine.setupStorageRequest(ExecutionEngine.java:324)

The former one happens more recently but the second one happened before it more often.

The tests in both of these two test suites fail concomitantly.

Sample build failure: https://gradle-enterprise.elastic.co/s/psa3duwnangzs

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[original-brownbear]

The second one should now be fixed but the first one looks new :( Looking into it

[original-brownbear]

This is an Azure service/SDK bug still: Azure/azure-storage-java#538

I tried fixing this by moving our code to using service level SAS tokens as suggested in by the Azure team before but that didn't work so I'm reverting Azure bulk deletes to fix this because they only work with account + key but not SAS token auth.

[mark-vieira]

This seems to still be failing along with AzureStorageRepositoryClientYamlTestSuiteIT, which I'm not sure is related. If not I can open a separate issue.

https://gradle-enterprise.elastic.co/s/owceoyfc43k6o/tests/failed