Error when deploying the version 2.8 via Microsoft.Azure.IIoT.Deployment

[emulic]

Deployment aborts during the deployment step jumpbox-vm-setup with the following error:
'Long running operation failed with status 'Failed'. Additional Info:'At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.''
I found following error description in the Azure portal for this deployment step:
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"VMExtensionProvisioningError","message":"VM has reported a failure when processing extension 'scriptextensions_setup'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\nt.com/repos/azure-cli bionic/main amd64 azure-cli all 2.26.1-1~bionic [62.7 MB]\nFetched 62.7 MB in 1s (51.8 MB/s)\nSelecting previously unselected package azure-cli.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 76955 files and directories currently installed.)\r\nPreparing to unpack .../azure-cli_2.26.1-1~bionic_all.deb ...\r\nUnpacking azure-cli (2.26.1-1~bionic) ...\r\nSetting up azure-cli (2.26.1-1~bionic) ...\r\n[\n {\n \"environmentName\": \"AzureCloud\",\n \"homeTenantId\": \"f76c822d-4d63-4207-ab41-39817e0d7b5d\",\n \"id\": \"56460eb4-5e4d-454b-bf43-5c9884692933\",\n \"isDefault\": true,\n \"managedByTenants\": [],\n \"name\": \"Windows Azure MSDN ? Visual Studio Premium\",\n \"state\": \"Enabled\",\n \"tenantId\": \"f76c822d-4d63-4207-ab41-39817e0d7b5d\",\n \"user\": {\n \"assignedIdentityInfo\": \"MSI\",\n \"name\": \"systemAssignedIdentity\",\n \"type\": \"servicePrincipal\"\n }\n }\n]\nMerged \"akscluster-87776-admin\" as current context in /root/.kube/config\nconfigmap/container-azm-ms-agentconfig created\n\"ingress-nginx\" has been added to your repositories\n\"jetstack\" has been added to your repositories\n\"aiiot\" has been added to your repositories\nHang tight while we grab the latest from your chart repositories...\n...Successfully got an update from the \"aiiot\" chart repository\n...Successfully got an update from the \"ingress-nginx\" chart repository\n...Successfully got an update from the \"jetstack\" chart repository\nUpdate Complete. ⎈Happy Helming!⎈\nnamespace/ingress-nginx created\nNAME: ingress-nginx\nLAST DEPLOYED: Fri Jul 23 13:33:55 2021\nNAMESPACE: ingress-nginx\nSTATUS: deployed\nREVISION: 1\nTEST SUITE: None\nNOTES:\nThe ingress-nginx controller has been installed.\nIt may take a few minutes for the LoadBalancer IP to be available.\nYou can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'\n\nAn example Ingress that makes use of the controller:\n\n apiVersion: networking.k8s.io/v1beta1\n kind: Ingress\n metadata:\n annotations:\n kubernetes.io/ingress.class: nginx\n name: example\n namespace: foo\n spec:\n rules:\n - host: www.example.com\n http:\n paths:\n - backend:\n serviceName: exampleService\n servicePort: 80\n path: /\n # This section is only required if TLS is to be enabled for the Ingress\n tls:\n - hosts:\n - www.example.com\n secretName: example-tls\n\nIf TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:\n\n apiVersion: v1\n kind: Secret\n metadata:\n name: example-tls\n namespace: foo\n data:\n tls.crt: <base64 encoded cert>\n tls.key: <base64 encoded key>\n type: kubernetes.io/tls\nnamespace/cert-manager created\nNAME: cert-manager\nLAST DEPLOYED: Fri Jul 23 13:34:35 2021\nNAMESPACE: cert-manager\nSTATUS: deployed\nREVISION: 1\nTEST SUITE: None\nNOTES:\ncert-manager has been deployed successfully!\n\nIn order to begin issuing certificates, you will need to set up a ClusterIssuer\nor Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).\n\nMore information on the different types of issuers and how to configure them\ncan be found in our documentation:\n\nhttps://cert-manager.io/docs/configuration/\n\nFor information on how to configure cert-manager to automatically provision\nCertificates for Ingress resources, take a look at the ingress-shim\ndocumentation:\n\nhttps://cert-manager.io/docs/usage/ingress/\nclusterissuer.cert-manager.io/letsencrypt-prod created\nnamespace/azure-industrial-iot created\n\n[stderr]\nd support levels: https://aka.ms/CLI_refstatus\nWARNING: Please note that \"az acr helm\" commands do not work with Helm 3, but you can still push Helm chart to ACR using a different command flow. For more information, please check out https://docs.microsoft.com/en-us/azure/container-registry/container-registry-helm-repos\nWARNING: Downloading client from https://get.helm.sh/helm-v3.3.4-linux-amd64.tar.gz, it may take a long time...\nWARNING: Successfully installed helm to /usr/local/bin.\nWARNING: Please ensure that /usr/local/bin is in your search PATH, so the helmcommand can be found.\n+ az login --identity\n+ [[ AzureKubernetesServiceClusterAdminRole -eq AzureKubernetesServiceClusterAdminRole ]]\n+ az aks get-credentials --resource-group iiot-test-edmu --name akscluster-87776 --admin\n+ kubectl apply -f /var/lib/waagent/custom-script/download/0/04_oms_agent_configmap.yaml\n+ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\n+ helm repo add jetstack https://charts.jetstack.io\n+ helm repo add aiiot https://microsoft.github.io/charts/repo\n+ helm repo update\n+ kubectl create namespace ingress-nginx\n+ helm install --atomic ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --version 3.12.0 --timeout 30m0s --set controller.replicaCount=2 --set 'controller.nodeSelector.beta\\.kubernetes\\.io\\/os=linux' --set controller.service.loadBalancerIP=20.103.128.19 --set 'controller.service.annotations.service\\.beta\\.kubernetes\\.io\\/azure-dns-label-name=iiot-test-edmu' --set 'controller.config.compute-full-forward-for=\"true\"' --set 'controller.config.use-forward-headers=\"true\"' --set 'controller.config.proxy-buffer-size=\"32k\"' --set 'controller.config.client-header-buffer-size=\"32k\"' --set controller.metrics.enabled=true --set defaultBackend.enabled=true --set 'defaultBackend.nodeSelector.beta\\.kubernetes\\.io\\/os=linux'\n+ kubectl create namespace cert-manager\n+ helm install --atomic cert-manager jetstack/cert-manager --namespace cert-manager --version v1.1.0 --timeout 30m0s --set installCRDs=true\n+ n=0\n+ iterations=20\n+ [[ 0 -ge 20 ]]\n+ kubectl apply -f /var/lib/waagent/custom-script/download/0/90_letsencrypt_cluster_issuer.yaml\n+ break\n+ [[ 0 -eq 20 ]]\n+ kubectl create namespace azure-industrial-iot\n+ '[' '' = true ']'\n+ helm install --atomic azure-industrial-iot aiiot/azure-industrial-iot --namespace azure-industrial-iot --version --aiiot_image_tag --timeout 30m0s --set image.tag=--aiiot_image_namespace --set loadConfFromKeyVault=true --set azure.tenantId=f76c822d-4d63-4207-ab41-39817e0d7b5d --set azure.keyVault.uri=https://keyvault-11400.vault.azure.net/ --set azure.auth.servicesApp.appId=ec97099b-9cea-4b39-8988-f19621d2cf11 --set azure.auth.servicesApp.secret=8_9_A1D0.0un1F_-v4~3W23Laz25GtO~UO --set externalServiceUrl=https://iiot-test-edmu.westeurope.cloudapp.azure.com --set deployment.microServices.engineeringTool.enabled=true --set deployment.microServices.telemetryCdmProcessor.enabled=true --set deployment.ingress.enabled=true --set 'deployment.ingress.annotations.kubernetes\\.io\\/ingress\\.class=nginx' --set 'deployment.ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/affinity=cookie' --set 'deployment.ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/session-cookie-name=affinity' --set-string 'deployment.ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/session-cookie-expires=14400' --set-string 'deployment.ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/session-cookie-max-age=14400' --set-string 'deployment.ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/proxy-read-timeout=3600' --set-string 'deployment.ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/proxy-send-timeout=3600' --set 'deployment.ingress.annotations.cert-manager\\.io\\/cluster-issuer=letsencrypt-prod' --set 'deployment.ingress.tls[0].hosts[0]=iiot-test-edmu.westeurope.cloudapp.azure.com' --set 'deployment.ingress.tls[0].secretName=tls-secret' --set deployment.ingress.hostName=iiot-test-edmu.westeurope.cloudapp.azure.com\nError: failed to download \"aiiot/azure-industrial-iot\" (hint: runninghelm repo update may help)\n\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "}]}

To Reproduce
Steps to reproduce the behavior:

1. Prepare service prinsipal to be used by deployment y following instructions here.
2. Execute Microsoft.Azure.IIoT.Deployment with having the following appsettings,json file in the execution folder (security sensitive data has been replaced by "XXX"):
   `{
   "Logging": {
   "LogLevel": {
   "Default": "Debug",
   "Microsoft": "Debug",
   "Microsoft.Hosting.Lifetime": "Debug"
   },
   "Console": {
   "LogLevel": {
   "Default": "Debug"
   }
   }
   },

// RunMode determines which steps of Industrial IoT solution deployment will
// be executed. Valid values are:
//
// - Full
// Performs application registration, deployment of Azure resources
// and deployment of microservices into AKS cluster.
//
// - ApplicationRegistration
// Performs only application registration and output JSON for
// ApplicationRegistration property bellow.
//
// - ResourceDeployment
// Performs deployment of Azure resources and deployment of
// microservices into AKS cluster.
//
"RunMode": "Full",

// Defines authentication details. Different authentication flows will be
// used based on which configuration parameters are specified:
//
// - If ClientId and ClientSecret are provided then Client credentials
// authentication flow will be used. This flow requires a Service
// Principal.
// https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-authentication-flows#client-credentials
//
// - If ClientId and ClientSecret are not provided then Interactive
// authentication flow will be used on Windows.
// https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-authentication-flows#interactive
//
// - If ClientId and ClientSecret are not provided then Device code
// authentication flow will be used on Linux and macOS.
// https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-authentication-flows#device-code
//
"Auth": {

// Defines which Azure cloud to use. Valid values are:
//
//    - AzureGlobalCloud
//    - AzureChinaCloud
//    - AzureUSGovernment
//    - AzureGermanCloud
//
"AzureEnvironment": "AzureGlobalCloud",

// Id of the tenant to be used. Should be Guid.
"TenantId": "XXX",

// Details of Service Principal that should be used for authentication.
//
//    - ClientId
//        ClientId of Service Principal. Other aliases of ClientId are AppId
//        and ApplicationId.
//
//    - ClientSecret
//        ClientSecret of Service Principal. It is also sometimes referred
//        to as password.
//
"ClientId": "XXX",
"ClientSecret": "XXX

},

// Id of Azure Subscription within tenant. Should be Guid.
"SubscriptionId": "XXX",

// Name of the application deployment. Should be globally unique name.
//
// Note: This name will be used as name of App Service resource, thus
// determining its URL as .azurewebsites.net. As a result, it
// should be a globally unique name.
//
"ApplicationName": "iiot-test-edmu",

// Base URL that will be used for generating RedirectUris for client
// application. This is required for enabling client authentication for
// use of exposed APIs (including access to Swagger).
//
// Usually this would be .azurewebsites.net
//
// This parameter is used only in ApplicationRegistration run mode.
//
"ApplicationUrl": "iiot-test-edmu.azurewebsites.net",

"ResourceGroup": {
// name of the resource group where azure resources will be created.
"name": "iiot-test-edmu",

// Determines whether an existing Resource Group should be used or a new one
// should be created.
"UseExisting": false,

// Region where new Resource Group should be created. The following regions
// are supported:
//
//    - USEast
//    - USEast2
//    - USWest
//    - USWest2
//    - USCentral
//    - EuropeNorth
//    - EuropeWest
//    - AsiaSouthEast
//    - AustraliaEast
//
"Region": "EuropeWest"

},

// azure-industrial-iot Helm chart details
"Helm": {
// Helm repository URL
"RepoUrl": "https://microsoft.github.io/charts/repo",
// azure-industrial-iot Helm chart version
"ChartVersion": null, //"0.4",
// Azure IIoT components image tag
"ImageTag": null, //"2.8",
// Default docker container registry
"ContainerRegistryServer": "mcr.microsoft.com"
},

// Provides definitions of applications and Service Principals to be used.
// Those definitions will be used instead of creating new application
// registrations and Service Principals for deployment of Azure resources.
//
// This is useful in ResourceDeployment mode. Execution in
// ApplicationRegistration run mode will output JSON object for this property.
//
// Properties correspond to that of application registration and Service
// Principal manifests. Definition of application properties can be found
// here:
//
// https://docs.microsoft.com/en-us/azure/active-directory/develop/reference-app-manifest
//
// Application objects should contain the following properties:
//
// {
// "Id": "",
// "DisplayName": "",
// "IdentifierUris": [
// ""
// ],
// "AppId": ""
// }
//
// Service Principal objects should contain the following
// properties:
//
// {
// "Id": "",
// "DisplayName": "",
// }
//
// ApplicationSecret is client secret (password) of an Application.
//
//"ApplicationRegistration": {
// "ServiceApplication": { ... Application objects ... },
// "ServiceApplicationSP": { ... Application Service Principal objects ... },
// "ServiceApplicationSecret": "",
//
// "ClientApplication": { ... Application objects ... },
// "ClientApplicationSP": { ... Application Service Principal objects ... },
// "ClientApplicationSecret": "",
//
// "AksApplication": { ... Application objects ... },
// "AksApplicationSP": { ... Application Service Principal objects ... },
// "AksApplicationSecret": ""
//},

// Defines whether to create .env file after deployment or not.
//
// .env file will contain environment variables necessary for connecting to
// microservices of Industrial IoT solution for degugging purposes or for
// access via cli client.
//
"SaveEnvFile": true,

// Defines whether to perform cleanup of registered applications and Azure
// resources if an error occurs during deployment of Industrial IoT solution.
"NoCleanup": false
}
`
2. Wait for about 10-15 minutes for error to ocure.

Expected behavior
Deployment should be executed without any errors.

Desktop (please complete the following information):

• OS: Windows 10
• Browser: MS Edge, required and used for Azure Portal only

Additional context

1. I'm executing the deployment in Service Principal Credentials mode.
2. The documentation here is saying, the used Helm version is by default 0.4 and the IIoT version is 2.8.
   So I set the both settings to null in my appsettings.json. Maybe is this wrong?
3. Maybe I'm wrong but this kind of deployment appeared for me very straight-forward. Please let me know if there is a better or more reliable way to deploy the v2.8.

[marcschier]

Hi @emulic, we have documented 0.4.0 helm chart but due to a delay have not yet released it, a faux pas on our side, apologies.

Please use version 0.3.2 which is the current version that works and was tested with 2.8.0 images until we release 0.4.0 in September.

[emulic]

Hi Marc,
I can confirm, deployment works fine when using helm Chart v 0.3.2.
Thanks for your support.
Best Regards
Edin Mulic

[marcschier]

Re-opening to track publishing 0.4.0 to charts.io or update documentation.

[karok2m]

@emulic, the root cause of your problem was that in your appsettings.json file you had "ChartVersion": null, which then resulted in empty string being selected as the chart version and passed to the script that deploys azure-industrial-iot chart. With #1383 we have fixed that bug. We have also released 0.4.0 version of azure-industrial-iot Helm chart. So this issue should be resolved now.

[emulicmedia]

@emulic, the root cause of your problem was that in your appsettings.json file you had "ChartVersion": null, which then resulted in empty string being selected as the chart version and passed to the script that deploys azure-industrial-iot chart. With #1383 we have fixed that bug. We have also released 0.4.0 version of azure-industrial-iot Helm chart. So this issue should be resolved now.

Thanks!