shim's cause issues and should be replaced by symlinks

[timotheecour]

shims are the wrapper processes generated by choosenim via proxyexe.nim instead of the conventional symlinks. They cause a number of issues, as shown here.

Example 1: breaks testament use in nimble

git clone https://github.com/planety/prologue
nimble tests # which uses `exec "testament all"`

Current Output

nimble tests
  Executing task tests in /Users/timothee/git_clone/nim/prologue/prologue.nimble
progress[all]: i: 0 / 15 cat: benchmark
'' --nim:/Users/timothee/.nimble/bin/nim pcat benchmark
progress[all]: i: 1 / 15 cat: unit
'' --nim:/Users/timothee/.nimble/bin/nim pcat unit
...
/bin/sh: : command not found

Expected Output

works

Notes:

• which testament prints: /Users/timothee/.nimble//bin/testament
• it works if we: nim c -o:/Users/timothee/.nimble//bin/testament testament/testament.nim (works meaning it doesn't have the above mentioned issue; note that testament: fix #16829, fix partially #16830 #16831 is also needed after that, but for unrelated reasons)

Example 2: breaks debugging with lldb

lldb -o r -o quit /Users/timothee/.nimble//bin/testament fails with:
Error: Spawning of process failed. (Error was: Interrupted system call)
Info: If unexpected, please report this error to https://github.com/dom96/choosenim

note that the same command works if /Users/timothee/.nimble//bin/testament is a symlink to a testament binary or if it's a testament binary itself (eg as generated by nim c -o:/Users/timothee/.nimble//bin/testament testament/testament.nim)

Example 3

same as example 2 but with /Users/timothee/.nimble//bin/nim

Example 4

shim doesn't provide you with the path to the target binary (whereas with a symlink it's easily obtainable via ls -l for example); in particular, it's not at all clear what is the real binary /Users/timothee/.nimble//bin/testament points to

note that this is the main reason for adding nimExe:

/Users/timothee/.nimble//bin/nim dump . | jq -r .nimExe
/Users/timothee/git_clone/nim/Nim_devel/bin/nim

but this doesn't help for other binaries wrapped in a shim.

on OSX I figured it what it pointed to via dtruss as follows:
sudo dtruss -a -t stat64 /Users/timothee/.nimble//bin/testament 2>&1 | grep testament

  testament [options] command [arguments]
On Azure Pipelines, testament will also publish test results via Azure Pipelines' Test Management API
41872/0x1d77561:      2106      12      8 stat64("/Users/timothee/git_clone/nim/Nim_devel/bin/testament\0", 0x7FFEE1106230, 0x0)                 = 0 0
41873/0x1d77565:       561       8      5 stat64("/Users/timothee/git_clone/nim/Nim_devel/bin/testament\0", 0x7FFEE7C9AEF0, 0x0)                 = 0 0

=> it resolved to /Users/timothee/git_clone/nim/Nim_devel/bin/testament

obviously, this isn't portable nor desirable. At the very least, choosenim (which is responsible for generating shims / binary wrappers), and in particular choosenim/src/choosenimpkg/proxyexe.nim, should allow an easy way to reveal the wrapped binary, eg via an environment variable CHOOSENIM_DEBUG or a cmdline testament --choosenim:debug that'd be intercepted in proxyexe.parseCliParams. This would be an improvement but still wouldn't replace the other issues.

Example 5: breaks nimeval.findNimStdLib

see comment there:

    let nimexe = os.findExe("nim")
      # this can't work with choosenim shims, refs https://github.com/dom96/choosenim/issues/189
      # it'd need `nim dump --dump.format:json . | jq -r .libpath`
      # which we should simplify as `nim dump --key:libpath`

Example 6:

regression since 1.4.2: vm crash with lists.SinglyLinkedRing · Issue #16384 · nim-lang/Nim
difference in behavior:

crashes with no output if running via choosenim nim shim
crashes with segmentation fault (no output) if running without nim shim on devel nim binary or 1.4.2,

Additional Information

• nim 1.5.1 25745ad
• which testament prints: /Users/timothee/.nimble//bin/testament
• it works if we: nim c -o:/Users/timothee/.nimble//bin/testament testament/testament.nim (works it doesn't have the above mentioned issue; note that testament: fix #16829, fix partially #16830 #16831 is also needed but for unrelated reasons)
• see also linked (closed) issues: Error: Spawning of process failed caused by choosenim using a wrapper process (shim) instead of symbolic link dom96/choosenim#126, use symlinks instead of shims where symlinks are supported dom96/choosenim#189; even though shims are generated in https://github.com/dom96/choosenim instead of nim repo, this is a nim ecosystem issue that affects tooling in general as shown in above example (and some of those issues were marked as wontfix in choosenim even though the issue is still there)/

the argument that "windows doesn't windows symlinks therefore we should use shims instead of symlinks on all platforms" IMO is moot, given that windows does support symlinks and we shouldn't limit other platforms based on lowest common denominator; see also #16784 which discusses improving symlink support on windows.

note

nimble already uses symlinks so there's a precedent for that:

lrwxr-xr-x 1 timothee 49 Feb 22 22:50 /Users/timothee/.nimble/bin/nimble -> /Users/timothee/.nimble/pkgs/nimble-0.12.0/nimble
lrwxr-xr-x 1 timothee 44 Sep 23 12:24 /Users/timothee/.nimble/bin/inim -> /Users/timothee/.nimble/pkgs/inim-0.6.0/inim

[Araq]

This is not a Nim issue but a choosenim issue.

[timotheecour]

yes, but it affects nim ecosystem (including testament use in nimble packages, see example 1), and existing issues in choosenim were closed:

see also linked (closed) issues: dom96/choosenim#126, dom96/choosenim#189; even though shims are generated in dom96/choosenim instead of nim repo, this is a nim ecosystem issue that affects tooling in general as shown in above example (and some of those issues were marked as wontfix in choosenim even though the issue is still there).

that means there is 0 open issue tracking this, anywhere.

[Araq]

I think @dom96 won't mind patches to choosenim.