Config parsing error in 24.01.0-edge when using includeConfig of assignment of function value

[mahesh-panchal]

Bug report

Expected behavior and actual behavior

The expected behaviour is that nextflow should run and not throw an error.
The actual behaviour is that nextflow throws an error.

Steps to reproduce the problem

main.nf:

workflow {
    TASK() | view
}

process TASK {
    input:

    script:
    """
    echo ${task.ext.args?:'World'}
    """

    output:
    stdout   
}

nextflow.config:

includeConfig 'test.config'

test.config:

def sayHello(){
    return "Hello"
}

def value = sayHello()

process {
    withName: 'TASK' {
        ext.args = { value }
    }
}

Program output

$ NXF_VER=24.01.0-edge nextflow run main.nf
N E X T F L O W  ~  version 24.01.0-edge                             
ERROR ~ Unknown method invocation `sayHello` on _parse_closure5 type

 -- Check '.nextflow.log' file for details

Environment

• Nextflow version: 24.01.0-edge
• Java version: openjdk 17.0.10-internal 2024-01-16
• Operating system: Linux
• Bash version: GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

Additional context

If the function is moved to the nextflow.config there is no problem, however when included via includeConfig and assigned to a variable outside a scope, it produces the above error.

The related Slack thread where this was discovered: https://nextflow.slack.com/archives/C02T98A23U7/p1707390130895379

[bentsherman]

This kind of thing is not really supported in config files. If it ever works it is only by chance. I would try to find a different approach.

One idea I had was that we could import the lib directory into config files as we do for scripts, so that helper functions defined in the lib directory can be used in the config file.

[mahesh-panchal]

This is the affected code: https://github.com/nf-core/pangenome/blob/9fc8ccd2eebb78ad93d14b52744d2e20967494dc/conf/modules.config#L14-L48
@subwaystation

[bentsherman]

You can always embed the code directly where you are using it. For example, rewriting the second function as an expression:

    [
        "asm5": "-P 1,19,39,3,81,1",
        "asm10": "-P 1,9,16,2,41,1",
        "asm15": "-P 1,7,11,2,33,1",
        "asm20": "-P 1,4,6,2,26,1"
    ][params.smoothxg_poa_params] ?: "-P ${params.smoothxg_poa_params}"

[subwaystation]

While this may work for this simple function I would not have any idea how to do it for the more complex one.

[bentsherman]

I think you can inline the closure in the function call

{
    def wfmash_sparse_map_cmd = ""
    if (params.wfmash_sparse_map == "auto") {
        def n = params.n_haplotypes
        def x = Math.log(n)/n * 10
        wfmash_sparse_map_frac = 1
        if (x < 1) {
            def wfmash_sparse_map_frac = x
        }
        wfmash_sparse_map_cmd = "-x ${wfmash_sparse_map_frac}"
    } else {
        if (params.wfmash_sparse_map != null) {
            wfmash_sparse_map_cmd = "-x ${params.wfmash_sparse_map}"
        }
    }
    return wfmash_sparse_map_cmd
}()

[mahesh-panchal]

What may be simpler for users is that functions can be replaced with closures and be called in a similar way.

workflow {
    TASK( [ id: 'Susi' ] )
    | view
}

process TASK {
    input:
    val meta

    script:
    def args = task.ext.args?: 'No message'
    """
    echo $args
    """

    output:
    stdout
}

functions {
    sayHello = { String name -> "Hello $name" }
}

process {
    withName: 'TASK' {
        ext.args = { functions.sayHello(meta.id) }
    }
}

Using a custom scope could also help group the closures, but it's not necessary.
One could also abuse this slightly and use the params scope to make it visible in the workflow as well as config.

[bentsherman]

I think the best solution is still to define these functions in the lib directory and allow config files to use them

[mahesh-panchal]

From a best practice point of view I agree.