Enabling Yarn 2-4, Yarn and corepack interaction on Windows causes yarn binary not to be found, when it could be

[doctorpangloss]

Core issue

    private fun computeProductBinDir(productDirProvider: Provider<Directory>, platform: Property<Platform>) =
            if (platform.get().isWindows()) productDirProvider else productDirProvider.map { it.dir("bin") }

This isn't true. On Windows, corepack also installs yarn.cmd (along with yarn the bash script and yarn.ps1, etc.) to it.dir("bin") too.

This line should be

            productDirProvider.map { it.dir("bin") }

related to #176 , this enables Yarn 2-4 and works around the posted issue:

import groovy.json.JsonSlurper

def packageJson = new JsonSlurper().parseText(rootProject.file('package.json').text)
def yarnVersionStr = (packageJson.packageManager as String).split("@")[1]

node {
  download = false
  yarnVersion = yarnVersionStr
  yarnWorkDir.set(rootProject.layout.projectDirectory.dir(".gradle/yarn"))
}

def isWindows = org.gradle.internal.os.OperatingSystem.current().isWindows()
def binExt = isWindows ? ".cmd" : ""
tasks.register("corepack") {
  def dirPath = ".gradle/yarn/yarn-v${yarnVersionStr}/${isWindows ? "" : "/bin"}"
  outputs.dir(dirPath)
  doLast {
    mkdir(dirPath)
    exec {
      commandLine "corepack${binExt}", "enable", "--install-directory", dirPath
    }
  }
}

tasks.named("yarnSetup").configure {
  enabled = false
  dependsOn "corepack"
}

// as an alternative to yarn install, which has a lot going on, and also supports workspaces
tasks.register('yarnWorkspace', YarnTask) {
  // todo: you don't want to inspect workspaces if you don't use it
  description("Installs all the necessary modules to develop ${packageJson.workspaces.join(', ')}")
  inputs.files("package.json", "yarn.lock")
          .withPropertyName("yarnLock")
          .withPathSensitivity(PathSensitivity.RELATIVE)
  inputs.files(packageJson.workspaces.collect { "$it/package.json" })
          .withPropertyName("workspacePackageJsons")
          .withPathSensitivity(PathSensitivity.RELATIVE)
  // todo: this only makes sense for node-modules resolution and will need to inspect the .yarnrc.yml file
  outputs.file("node_modules/.yarn-state.yml")
  outputs.cacheIf { false }
  args = ['install', '--immutable']
}

[deepy]

This is an interesting quirk, because depending on whether we use --global --prefix foo or not we get different results

C:\dev\test> npm install yarn

added 1 package in 600ms

C:\dev\test> ls .\node_modules\yarn\bin\


    Directory: C:\dev\test\node_modules\yarn\bin


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        19/12/2023     18:06           1025 yarn
-a----        19/12/2023     18:06             34 yarn.cmd
-a----        19/12/2023     18:06           1015 yarn.js
-a----        19/12/2023     18:06             42 yarnpkg
-a----        19/12/2023     18:06             30 yarnpkg.cmd

C:\dev\gradle-node-plugin> ls .\examples\simple-node\yarn\.gradle\yarn\yarn-latest\


    Directory: C:\dev\gradle-node-plugin\examples\simple-node\yarn\.gradle\yarn\yarn-latest


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        17/08/2023     17:16                lib
d-----        19/12/2023     18:12                node_modules
-a----        19/12/2023     18:12            320 yarn
-a----        19/12/2023     18:12            331 yarn.cmd
-a----        19/12/2023     18:12            829 yarn.ps1
-a----        19/12/2023     18:12            320 yarnpkg
-a----        19/12/2023     18:12            331 yarnpkg.cmd
-a----        19/12/2023     18:12            829 yarnpkg.ps1

C:\dev\test> .\node_modules\yarn\bin\yarn.cmd --version
1.22.21

C:\dev\gradle-node-plugin> .\examples\simple-node\yarn\.gradle\yarn\yarn-latest\yarn.cmd --version
1.22.21

And for comparison, here's the same command the plugin uses:

C:\dev\test> npm install --global --no-save --prefix . yarn

added 1 package in 453ms

C:\dev\test> ls


    Directory: C:\dev\test


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        19/12/2023     18:18                node_modules
-a----        19/12/2023     18:18            320 yarn
-a----        19/12/2023     18:18            331 yarn.cmd
-a----        19/12/2023     18:18            829 yarn.ps1
-a----        19/12/2023     18:18            320 yarnpkg
-a----        19/12/2023     18:18            331 yarnpkg.cmd
-a----        19/12/2023     18:18            829 yarnpkg.ps1

# Luckily we have both!

C:\dev\test> ls .\node_modules\yarn\bin\


    Directory: C:\dev\test\node_modules\yarn\bin


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        19/12/2023     18:18           1025 yarn
-a----        19/12/2023     18:18             34 yarn.cmd
-a----        19/12/2023     18:18           1015 yarn.js
-a----        19/12/2023     18:18             42 yarnpkg
-a----        19/12/2023     18:18             30 yarnpkg.cmd

I need to look into this again, and read up a bit more on --global, --prefix, and yarn 1 vs 2+

[doctorpangloss]

thanks for investigating. Everything seems robust enough for Yarn 4 and cache support which is great.

[solonovamax]

After some effort, here are some tasks which are a bit more 'correct', in that they support all the features that the normal node-gradle plugin supports, and the code is essentially identical to what node-gradle does, aside from the fact it's using corepack:

tasks {
    withType<NodeSetupTask> {
        val variantComputer by lazy {
            VariantComputer()
        }

        doLast {
            if (nodeArchiveFile.get().asFile.extension != "zip") {
                // Fix broken symlink
                val nodeBinDirPath = variantComputer.computeNodeBinDir(node.resolvedNodeDir, node.resolvedPlatform).get().asFile.toPath()
                val script = nodeBinDirPath.resolve("corepack")
                val scriptFile = node.resolvedNodeDir.map { nodeDir ->
                    if (node.resolvedPlatform.get().isWindows())
                        nodeDir.file("node_modules/corepack/dist/corepack.js")
                    else
                        nodeDir.file("lib/node_modules/corepack/dist/corepack.js")
                }.get().asFile.toPath()
                script.deleteIfExists()
                script.createSymbolicLinkPointingTo(nodeBinDirPath.relativize(scriptFile))
            }
        }
    }

    val corepackYarnSetup by registering(CorepackYarnSetupTask::class)

    val corepackYarnInstall by registering(CorepackYarnInstallTask::class) {
        dependsOn(corepackYarnSetup)
    }

    yarnSetup {
        dependsOn(corepackYarnSetup)
        enabled = false
    }

    // example task
    val yarnBuild by registering(YarnTask::class) {
        dependsOn(corepackYarnInstall)

        args = listOf("buildDev")
        ignoreExitValue = true

        inputs.dir("src")
        inputs.files(buildInputs)
        outputs.dir(buildDir("dev"))
    }
}

fun buildDir(name: String) = layout.buildDirectory.dir(name)

abstract class CorepackYarnSetupTask : DefaultTask() {
    init {
        group = NodePlugin.YARN_GROUP
        description = "Setup a specific version of Yarn to be used by the build."
        dependsOn(NodeSetupTask.NAME)
    }

    @get:Inject
    abstract val execOperations: ExecOperations

    @get:Internal
    val nodeExtension = NodeExtension[project]

    @get:Input
    val download: Boolean by nodeExtension.download

    @get:Internal
    var result: ExecResult? = null

    @get:Internal
    var platform by nodeExtension.resolvedPlatform

    @get:Internal
    internal val variantComputer by lazy {
        VariantComputer()
    }

    @TaskAction
    fun exec() {
        val command = computeCommand()
        // this is a lot of stuff inlined & simplified from NpmExecRunner.executeNpmCommand, however with "corepack" instead of "npm"
        result = run {
            val execCommand = "corepack"

            val nodeDirProvider = nodeExtension.resolvedNodeDir
            val nodeBinDirProvider = variantComputer.computeNodeBinDir(nodeDirProvider, nodeExtension.resolvedPlatform)
            val nodeBinDir = nodeBinDirProvider.get()
            val execCommandFile = if (platform.isWindows()) "$execCommand.cmd" else execCommand

            val corepackScriptFileProvider = nodeDirProvider.map { nodeDir ->
                if (platform.isWindows())
                    nodeDir
                else
                    nodeDir.dir("lib")
            }.map { it.file("node_modules/corepack/dist/$execCommand.js").asFile.absolutePath }

            val env = if (NpmProxy.shouldConfigureProxy(System.getenv(), nodeExtension.nodeProxySettings.get())) {
                val npmProxyEnv = NpmProxy.computeNpmProxyEnvironmentVariables()
                if (npmProxyEnv.isNotEmpty())
                    npmProxyEnv
                else
                    mapOf()
            } else {
                mapOf()
            }

            execOperations.exec {
                val execConfiguration = when {
                    !download -> ExecConfiguration(execCommandFile, command, listOf(), env)

                    nodeBinDir.dir(execCommandFile).asFile.exists() -> ExecConfiguration(
                        nodeBinDir.dir(execCommandFile).asFile.absolutePath,
                        command,
                        listOf(nodeBinDir.asFile.absolutePath),
                        env
                    )

                    else -> ExecConfiguration(
                        computeNodeExec(nodeExtension, nodeBinDirProvider).get(),
                        command + corepackScriptFileProvider.get(),
                        listOf(nodeBinDir.asFile.absolutePath),
                        env
                    )
                }

                executable = execConfiguration.executable
                args = execConfiguration.args
                environment = computeEnvironment(execConfiguration)
                isIgnoreExitValue = execConfiguration.ignoreExitValue
                workingDir = computeWorkingDir(nodeExtension.nodeProjectDir, execConfiguration)
            }
        }
    }

    @Input
    fun getVersion(): Provider<String> {
        return nodeExtension.yarnVersion
    }

    @get:OutputDirectory
    val yarnBinDir by lazy {
        val yarnDir = variantComputer.computeYarnDir(nodeExtension)
        variantComputer.computeYarnBinDir(yarnDir, nodeExtension.resolvedPlatform)
    }

    fun computeCommand(): List<String> {
        val yarnDir = yarnBinDir.get().asFile
        yarnDir.mkdirs()
        return listOf("enable", "--install-directory", yarnDir.absolutePath)
    }
}

abstract class CorepackYarnInstallTask : YarnTask() {
    init {
        group = NodePlugin.YARN_GROUP
        description = "Install node packages using Yarn."
        yarnCommand = nodeExtension.npmInstallCommand.map { listOf(it) }
    }

    @Optional
    @InputFile
    @PathSensitive(PathSensitivity.RELATIVE)
    fun getPackageJsonFile() = projectFileIfExists("package.json")

    @Optional
    @InputFile
    @PathSensitive(PathSensitivity.RELATIVE)
    fun getYarnLockFile() = projectFileIfExists("yarn.lock")

    @Optional
    @OutputFile
    fun getYarnLockFileAsOutput() = projectFileIfExists("yarn.lock")

    @Optional
    @InputFile
    @PathSensitive(PathSensitivity.RELATIVE)
    fun getYarnInstallStateFile() = projectFileIfExists(".yarn/.install-state.gz")

    @Optional
    @OutputFile
    fun getYarnInstallStateFileAsOutput() = projectFileIfExists(".yarn/install-state.gz")

    private fun projectFileIfExists(name: String): Provider<File> {
        return nodeExtension.nodeProjectDir.map { it.file(name).asFile }
            .flatMap { if (it.exists()) providers.provider { it } else providers.provider { null } }
    }

    @Optional
    @OutputDirectory
    fun getNodeModulesDirectory() = nodeExtension.nodeProjectDir.dir("node_modules")
}