[draft] Azure Batch worker pool supports managed identity

[adamrtalbot]

This feature adds an option to tell Azure Batch the worker pool has a managed identity attached.

azcopy needs to be updated, since the current version isn't recent enough.

When using this feature, you can tell Nextflow that a node pool has a managed identity attached:

azure {

    storage {
        accountName = 'seqeralabs'
    }

    batch {
        location = 'eastus'
        accountName = 'seqeralabs'
        copyToolInstallMode = "off"
        pools {
            "hello-world-entra-mi" {
                vmType = "standard_e2ds_v5"
                managedIdentityId = "myManagedIdentityId"
            }

        }
    }
}

it will then use the native azcopy managed identity authentication to download and upload files.

I haven't handled all the old code yet, so Nextflow will generate a SAS which it only uses sometimes.

Importantly, this 'fixes' #5669, because it doesn't need a SAS token to access the files (tested).

To use:

1. Create a node pool in Azure Batch and attach a user-assigned managed identity. Follow the instructions here or try this if you want to use Terraform
2. Change the start task to have the following settings to update azcopy:
   • Command line: bash -c "tar -xzvf azcopy.tar.gz && chmod +x azcopy*/azcopy && mkdir -p $AZ_BATCH_NODE_SHARED_DIR/bin/ && cp azcopy*/azcopy $AZ_BATCH_NODE_SHARED_DIR/bin/"
   • Resource file URL: https://aka.ms/downloadazcopy-v10-linux, path azcopy.tar.gz
3. Retrieve the managed identity client ID
4. Use the following Nextflow config, where the pool name is my-pool:

process.queue = 'my-pool'
azure {

    storage {
        accountName = 'seqeralabs'
    }

    batch {
        location = 'eastus'
        accountName = 'seqeralabs'
        copyToolInstallMode = "off"
        pools {
            "my-pool" {
                managedIdentityId = "myManagedIdentityId"
            }

        }
    }
}

6. Launch pipeline and 🤞

To do:

• See if we can apply the managed ID to pools created by Nextflow. This might need an update from Microsoft
• Documentation
• Remove creation of SAS when using managed identity to make everything more secure and allow this to work if we switch off SAS tokens

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	c93dc78
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67e59e2e60dfd500083c6449

[Takadonet]

Hello,
This new feature will be very helpful for our project to avoid long duration SaS token.
Just wondering if a timeline or we can provide some assistance.

[adamrtalbot]

This breaks Fusion for everyone who isn't using managed identities. Ideally we take the pool.opts when creating the launcher but I've ran out of time.

[pditommaso]

What's the rationale? remove AZURE_STORAGE_SAS_TOKEN when pool.opts.managedIdentityId is provided?

[pditommaso]

These looks unrelated

[pditommaso]

This should not be affected

[pditommaso]

What's the rationale? remove AZURE_STORAGE_SAS_TOKEN when pool.opts.managedIdentityId is provided?