Azure CLI: refreshing the token fails

[tombuildsstuff]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

---

Porting this over from hashicorp/terraform-provider-azurerm#2602 where it was originally reported by @btai24 and @torumakabe:

Error: Error applying plan:

1 error(s) occurred:

* azurerm_resource_group.shared (destroy): 1 error(s) occurred:

* azurerm_resource_group.shared: Error deleting Resource Group "myrg": azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to
https://management.azure.com/subscriptions/myid/operationresults/myresult?api-version=2018-05-01: StatusCode=0 -- Original Error: Manually created ServicePrincipalToken does not contain secret material to retrieve a new access token

[usrbinpikachu]

FWIW this is reliably reproducible when building VM Scale Sets. It fails a few seconds past the 58m31s mark every time I run the plan I'm currently working on.

Original Error: Manually created ServicePrincipalToken does not contain secret material to retrieve a new access token

[gvilarino]

Happening to me as well every time I create a VM with attached disks

Is there a provided version downgrade that could help? Currently using 1.22

[alert101]

I get this error while trying to create an AKS cluster using azurerm_kubernetes_cluster.
Terraform v0.11.11
provider.azurerm v1.21.0 and v1.22.1.

[AhmedAZafar]

I get this error while trying to add an azure virtual machine extension for my VMs.

I am adding the Microsoft's Azure Active Directory Linux SSH to allow users on the azure portal to access the VMs I create with Terraform. Details of the extension are:-

resource "azurerm_virtual_machine_extension" "AADLogin" {
...
  publisher = "Microsoft.Azure.ActiveDirectory.LinuxSSH"
  type = "AADLoginForLinux"

  type_handler_version = "1.0"
}

Versions:

Terraform: v0.11.11
azurerrm: v1.22.0

[obikay200]

Thanks very much for the release of 0.5.0 provider has some useful feature for stack.

However, im getting this issue when running terraform plan against 1811 release of stack, the stack will be updated but i dont think thats the problem.

terraform validate passes - ok.
terraform plan / apply - fails.
disabled provider registration - still fails

terraform - v0.11.13
azurestack provider - v0.5.0
stack - v1811 (disconnected with ADFS)
auth method - service principle client ID/Secret ( has the correct permissions for API / Subscription)

Error

Error: Error running plan: 1 error(s) occurred:

* provider.azurestack: Unable to list provider registration status, 
it is possible that this is due to invalid credentials or the service principal does not have permission to use Resource Manager API,
Azure error: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the token for the request to https://management.stack.com/subscriptions/ { subscription id }/providers?api-version=2016-02-01: Status code=404
-- Original Error: adal: Refresh request failed. Status code = '404' response body: { html }

This is stopping us using terraform all together.

[obikay200]

Update to above

i have tried all methods of authentication now,

terraform validate passes - ok.
terraform plan / apply - fails.
disabled provider registration - still fails

terraform - v0.11.13
azurestack provider - v0.5.0
stack - v1811 (disconnected with ADFS)
auth method - service principle client certificate/password ( has the correct permissions for API / Subscription)

Error

Error: Error running plan: 1 error(s) occurred:

* provider.azurestack: adal: Refresh request failed. Status code = '404'. response body { html }

This is stopping us using terraform all together.

Using the Azure CLI by itself works, but it cant seem to integrate with terrafom. so sounds like an underlying library problem for the token service.

[erick-thompson]

I'm running into this issue as well, during both provisioning and destruction. Retrying the operation usually works, but it is troublesome.

[scott1138]

Also seeing this problem with Virtual Network Gateway creation after 50 minutes or so:

• azurerm_virtual_network_gateway.vng: Error waiting for completion of AzureRM Virtual Network Gateway "VNG-FreemanHT-SouthCentralUS-Core-Prod" (Resource Group "RG-VNetTest"): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the
  Token for request to https://management.azure.com/subscriptions/REDACTED-GUID/providers/Microsoft.Network/locations/southcentralus/operations/REDACTED-GUID?api-version=2018-08-01: StatusCode=0 -- Original Error: Manually created ServicePrincipalToken does not contain secret material to retrieve a new access token

@tombuildsstuff - I know you guys are super busy and we are totally appreciative of the hard work, but any chance there is an ETA on when this will be resolved?

[CptQuint]

I'm getting this error:

provider.azurerm: Unable to list provider registration status, it is possible that this is due to invalid credentials or the service principal does not have permission to use the Resource Manager API, Azure error: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request

When trying to create an AKS cluster using azurerm_kubernetes_cluster. Terraform v0.11.7 provider.azurerm v1.20 and v1.24.

This is using a service principal which works perfectly via the azure cli.

[obikay200]

Update to above

i have tried all methods of authentication now,

terraform validate passes - ok.
terraform plan / apply - fails.
disabled provider registration - still fails

terraform - v0.11.13
azurestack provider - v0.5.0
stack - v1811 (disconnected with ADFS)
auth method - service principle client certificate/password ( has the correct permissions for API / Subscription)

Error

Error: Error running plan: 1 error(s) occurred:

* provider.azurestack: adal: Refresh request failed. Status code = '404'. response body { html }

This is stopping us using terraform all together.

Using the Azure CLI by itself works, but it cant seem to integrate with terrafom. so sounds like an underlying library problem for the token service.

managed to modify the azure adal library to resolve my issue, it is using the tenantid in the request for a token, but with adfs you need to replace the tenantid with "adfs" to hit the correct endpoint

[tombuildsstuff]

@obikay200 oh interesting, is that just for ADFS/disconnected Azure Stack?

[tombuildsstuff]

@CptQuint unfortunately Terraform doesn't support using a Service Principal with the Azure CLI - you can either use the Azure CLI as a user or the Service Principal via credentials (either specified as Environment Variables/in-line) - but unfortunately the Azure CLI doesn't expose all of the information we need when using a Service Principal.

[obikay200]

yeah it seem so because i presume you developed against an SDK stack that does have AD. disconnected only has ADFS, so the underlying adal azure library had to be changed to not provide the tenantid in the request. unless microsoft decide to change the stack api of course.. which is a possiblity!

so i was thinking maybe you guys could have another label in the provider block that defines ADFS true or false maybe that could call a different function in adal during the authentication, or something similar that you know of a better way..

or connected / disconnected

[CptQuint]

@CptQuint unfortunately Terraform doesn't support using a Service Principal with the Azure CLI - you can either use the Azure CLI as a user or the Service Principal via credentials (either specified as Environment Variables/in-line) - but unfortunately the Azure CLI doesn't expose all of the information we need when using a Service Principal.

Sorry, yes this is what I'm doing. I'm passing service provider credentials via a .tfvars file (one per workspace), I just meant these credentials work fine via the azure CLI.

This setup had been working perfectly the last time I used it around December time.

[Moeser]

Is this a bug in the upstream go-autorest library? They throw this error if the type matches ServicePrincipalNoSecret, but they pass the ServicePrincipalNoSecret type when setting up the credential: https://github.com/Azure/go-autorest/blob/da8db3a19ec5aba5d01f6032407abf1bb1cc15d3/autorest/adal/token.go#L465 maybe the check should be whether refresh_token is actually present, instead of whether the type is ServicePrincipalNoSecret?

[florianrusch]

I had the same problem.

A workaround which fix my problem was to relogin with the cli:

$ az logout
$ az login

[Moeser]

So here's the spot in the upstream repo where that exact error message is coming from: https://github.com/Azure/go-autorest/blob/2913f263500c4a5b23dada1b46ccd22ac972315f/autorest/adal/token.go#L173
That's coming from ServicePrincipalNoSecret, which is brought in from the upstream's NewServicePrincipalTokenFromManualToken func here: https://github.com/Azure/go-autorest/blob/2913f263500c4a5b23dada1b46ccd22ac972315f/autorest/adal/token.go#L471
The code here this local repo calls NewServicePrincipalTokenFromManualToken as part of the CLI auth flow here:

go-azure-helpers/authentication/auth_method_azure_cli_token.go

Line 83 in e1c07bf

spt, err := adal.NewServicePrincipalTokenFromManualToken(*oauthConfig, a.profile.clientId, endpoint, adalToken)

I opened an issue in the upstream go-autorest repo asking whether this is a bug on their side, since contrary to the error message, there is a refresh token available to retrieve a new access token.

In the meantime: I looked at how go-autorest authenticates with cli credentials, and it's very similar except they skip the step of creating a ServicePrincipalToken from the adalToken. Since this error seems to come from the code brought in by NewServicePrincipalTokenFromManualToken (and from there, ServicePrincipalNoSecret) maybe dropping that NewServicePrincipalToken[...] step would work as a fix to this issue. Here's a link to the similar cli auth helper in go-autorest:
https://github.com/Azure/go-autorest/blob/2913f263500c4a5b23dada1b46ccd22ac972315f/autorest/azure/auth/auth.go#L291

I'm not sure if there's code to auto refresh the token outside of the ServicePrincipalToken flow though, so maybe that's why this extra step was introduced to terraform? If that's the case, maybe there's a way to override the token type or the callback to avoid that code path in go-autorest, but still get the benefits of auto refreshing tokens?

[Moeser]

One of the go-autorest devs responded and pointed out that terraform isn't receiving the refreshToken when looking up the accessToken via the az cli. Without a refreshToken, the refresh probably wasn't going to work anyway. If the refreshToken is set, the code path for this error response should be skipped over. The only place I know of where this is cached is in the az cli's accessTokens.json.

Maybe the fix here is to look up the refreshToken after obtaining the accessToken, and make sure that gets set in the token before creating the ServicePrincipalToken?

[sam-cogan]

Any update on this? Seeing the same issue with an AKS deployment

[choovick]

Also getting this on long running operations like creating/deleting AKS cluster:
Error: Error waiting for the deletion of Managed Kubernetes Cluster "xxxxx" (Resource Group "xxxxx"): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/xxxxx/providers/Microsoft.ContainerService/locations/eastus2/operations/57c8184c-1c37-41b4-8626-83d64d3fce65?api-version=2017-08-31: StatusCode=0 -- Original Error: Manually created ServicePrincipalToken does not contain secret material to retrieve a new access token

[Moeser]

I think at this point we just need someone to write up code to read the refreshToken from az cli's accessTokens.json. I can do some simple changes in golang, but this change would be significant enough that someone more experienced than me should probably take a stab at it. Any volunteers?

[mikhailshilkov]

@Moeser I did this in #39, any comments are appreciated.

[Moeser]

@mikhailshilkov #39 looks good to me. Might need @tombuildsstuff to take a look at it too ^

[srinathrangaramanujam]

I am not sure how this repo works. But why is this issue closed?.. getting the issue for long running modules like APIM and WAF. Any advise how to resolve?

[cicorias]

btw, this looks fixed: hashicorp/terraform-provider-azurerm#2602 (comment)

[easkay-castlight]

@Tenseiga I was having the same issue until I upgraded all of the providers I was using that do anything with Azure (in my case azurerm and azuread). It's up to the consuming plugins such as azurerm to upgrade the helpers library that they depend on. The issue has been fixed in this repo and so this issue is closed, but other issues elsewhere may be closed as well or still open.

I believe that version 1.37.0 of azurerm includes the fixes which closed this issue, so perhaps give that provider version a try and see whether it solves the token refresh issue.

[dl888888]

We upgraded to 1.44.0 version of azurerm and now I'm seeing this problem first time. Anyone else experiencing this?

this is what I have now:
Terraform v0.12.21

• provider.azuread v0.7.0
• provider.azurerm v1.44.0
• provider.random v2.2.1

[alexyakunin]

Same for me.

[alexyakunin]

I figured out what was the cause in my case:

• I use auxiliary_tenant_ids = [...] in azurerm provider section
• But the token used for auxiliary tenant isn't for the same app / user as for the primary one, even though https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/authenticate-multi-tenant says it should originate from the same app.

[markokole]

Same problem when I try to create a databricks cluster.
Error:

Error: adal: Refresh request failed. Status Code = '404'. Response body: 


  on databricks.tf line 30, in resource "databricks_cluster" "test":
  30: resource "databricks_cluster" "test" {

Environment:

Terraform v0.12.24
+ provider.azurerm v2.20.0
+ provider.databricks v0.2.3
+ provider.null v2.1.2

[anoncam]

I had the same problem.

A workaround which fix my problem was to relogin with the cli:

$ az logout
$ az login

before going too deep in a rabbit hole. do this

[JoshuaVanDaalen]

I had the same problem.
A workaround which fix my problem was to relogin with the cli:

$ az logout
$ az login

before going too deep in a rabbit hole. do this

This is not something I would be doing with using Azure DevOps YAML pipelines, so there has to be more to this.

I'm getting this creating a Resource Group.

[tombuildsstuff]

Locking since this issue is 2.5 years old - but feel free to ask questions in the Community Discuss forum where someone should be able to take a look.

Thanks!