feat(AzBatchService): Allow Azure Batch tasks to be submitted to different pools

[adamrtalbot]

context: https://nextflow.slack.com/archives/C02T98A23U7/p1738748591243169

Processes could not be submitted to different queues on Azure Batch because the logic looked for an existing job/pool before submitting and didn't check the queue details. This includes the pool ID so that it will now submit to different queues based on user configuration.

You can test it with the following main.nf:

process ECHO {
    queue { if (a == "a") "my_test_pool_1" else if (a == "b") "my_test_pool_2" else "my_test_pool_3" }
    container "ubuntu:2204"
    input:
        val a
    output:
        tuple val("sampleId"), val(a)

    script:
    """
    echo "a: ${a}"
    """
}


workflow {
    Channel.of("a", "b", "c")
    | ECHO
    | view
}

and nextflow.config (with credentials and details redacted):

azure {
    batch {
        copyToolInstallMode = 'node'
        deleteJobsOnCompletion = false
        deletePoolsOnCompletion = false
        autoPoolMode = false
        allowPoolCreation = true
        pools {
            my_test_pool_1 {
                vmType = 'Standard_e2d_v5'
                autoScale = true
                vmCount = 1
                maxVmCount = 4
            }
            my_test_pool_2 {
                vmType = 'Standard_e2d_v5'
                autoScale = true
                vmCount = 1
                maxVmCount = 4
            }
            my_test_pool_3 {
                vmType = 'Standard_e2d_v5'
                autoScale = true
                vmCount = 1
                maxVmCount = 4
            }
        }
   }
}

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	66b0987
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67a7a152ada50f0008da6e7e