v0.5.5 breaks ReadStat.jl on windows

[davidanthoff]

The recent v0.5.5 release breaks the ReadStat.jl build.jl on Windows. With any version of BinaryProvider.jl pre v0.5.5 everything works, with v0.5.5 on Windows I get this:

 Building StringEncodings → `C:\Users\david\.julia\packages\StringEncodings\ICuGB\deps\build.log`
  Building ReadStat ───────→ `C:\Users\david\.julia\dev\ReadStat\deps\build.log`
 Resolving package versions...
┌ Error: Error building `ReadStat`:
│ ┌ Warning: platform_key() is deprecated, use platform_key_abi() from now on
│ │   caller = ip:0x0
│ └ @ Core :-1
│ [ Info: Downloading https://github.com/davidanthoff/ReadStatBuilder/releases/download/v0.1.1-build.3/ReadStat.v0.1.1-dev.x86_64-w64-mingw32.tar.gz to C:\Users\david\.julia\dev\ReadStat\deps\usr\downloads\ReadStat.v0.1.1-dev.x86_64-w64-mingw32.tar.gz...
│ ERROR: LoadError: LoadError: LibraryProduct(nothing, ["libreadstat"], :libreadstat, "Prefix(C:\\Users\\david\\.julia\\dev\\ReadStat\\deps\\usr)") is not satisfied, 
cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at .\error.jl:33
│  [2] #write_deps_file#169(::Bool, ::Bool, ::Function, ::String, ::Array{Product,1}) at C:\Users\david\.julia\packages\BinaryProvider\A0sDa\src\Products.jl:419      
│  [3] write_deps_file(::String, ::Array{Product,1}) at C:\Users\david\.julia\packages\BinaryProvider\A0sDa\src\Products.jl:400
│  [4] top-level scope at none:0
│  [5] include at .\boot.jl:326 [inlined]
│  [6] include_relative(::Module, ::String) at .\loading.jl:1038
│  [7] include(::Module, ::String) at .\sysimg.jl:29
│  [8] include(::String) at .\client.jl:403
│  [9] top-level scope at none:0
│  [10] include at .\boot.jl:326 [inlined]
│  [11] include_relative(::Module, ::String) at .\loading.jl:1038
│  [12] include(::Module, ::String) at .\sysimg.jl:29
│  [13] include(::String) at .\client.jl:403
│  [14] top-level scope at none:0
│ in expression starting at C:\Users\david\.julia\dev\ReadStat\deps\build_ReadStat.v0.1.1-dev.jl:47
│ in expression starting at C:\Users\david\.julia\dev\ReadStat\deps\build.jl:3
└ @ Pkg.Operations C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Pkg\src\Operations.jl:1075

Could that release be reverted until this is fixed?

[thofma]

I think the issue reported at Nemocas/Nemo.jl#600 is the same. I changed nothing in the build script but suddenly GMPBuilder cannot be installed anymore. Note that there, the error was reported for arch linux.

[hhaensel]

This topic is already heavily discussed (#170)
It seems that deleting .julia "solves" the issue. It would be good to hear whether that solves the issue for you as well.
If you try, pleae only rename .julia so that you can revert to the non-functional behaviour.
@zgornel reported that

success(`$(Base.julia_cmd()) -e "import Libdl; Libdl.dlopen(\"$(dl_esc_path)\")"`)

in line 169 of Products.jl returns false in the context of package building but true at the REPL.
I realised that the ENV variable has a different LOAD_PATH depending on where it was called from.
["@v#.#", "@stdlib"] are missing when called from Pkg.build.
Could you try, whether

success(`$(Base.julia_cmd()) -e "import Libdl; splice!(LOAD_PATH, 2:1, ["@v#.#", "@stdlib"]); Libdl.dlopen(\"$(dl_esc_path)\")"`)

solves the issue? I have no failing system, so I can't do it ...
@staticfloat :
Shall we revert to isolate=false as default until the issue is solved?

[thofma]

I don't have a failing system, so I cannot try this.

[staticfloat]

Isolation is quite important; there are certain .DLL's that will crash julia when attempting to be dlopen()'ed, which is a very bad experience for the user. It's important for us to keep isolation as the default.

@davidanthoff do you have a local example of this failure? Are you saying that if you pin BP to v0.5.4, it works? What does the output say if you modify C:\Users\david\.julia\dev\ReadStat\deps\build.jl to have verbose set unconditionally to true? (e.g. modify this file to say simply verbose=true)

[davidanthoff]

Locally, I can easily reproduce this, and when I locally pin BinaryProvider to v0.5.4 it all works again. I'm not working with deved versions of BinaryProvider to reproduce this.

I also run a nightly automated huge integration test for all queryverse packages, and that started failing with exactly this error on the day v0.5.5. was released. That whole process starts with a fresh environment every time it runs, so I think that very clearly pins it down to some change that was introduced by v0.5.5. You can see the latest log of those nightly builds here.

I think the right process here is to revert very quickly, and not try to diagnose what is going on and fix. That can happen after things have reverted, but right now things are broken widely and that needs to be mitigated ASAP, IMHO.

[zgornel]

I agree with @davidanthoff this breaks too much stuff.

[hhaensel]

@davidanthoff :
would you mind running

julia -E "import Libdl; Libdl.open(\"C:\\Users\\david\\.julia\\dev\\ReadStat\\deps\\usr\\bin\\libreadstat-0.dll\")"
at a cmd prompt and report whether you receive an error?

I obtain:

IOStream(<file C:\Users\hh\.julia\dev\ReadStat\deps\usr\bin\libreadstat-0.dll>)

[davidanthoff]

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.1.1 (2019-05-16)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> import Libdljulia> Libdl.open("C:\\Users\\david\\.julia\\dev\\ReadStat\\deps\\usr\\bin\\libreadstat-0.dll")
IOStream(<file C:\Users\david\.julia\dev\ReadStat\deps\usr\bin\libreadstat-0.dll>)

julia>

With BinaryProvider#master, and with a failing build step for ReadStat.jl.

[hhaensel]

That is reproducing the isolate=false case. Good to see rhat this is working.
Could you also run the above command a a bash prompt (not at the REPL)? That is closer to isolate=true.

[davidanthoff]

This:

 david@DANTHOFF-5820 > julia -E "import Libdl; Libdl.open(ARGS[1])" C:\Users\david\.julia\dev\ReadStat\deps\usr\bin\libreadstat-0.dll
IOStream(<file C:\Users\david\.julia\dev\ReadStat\deps\usr\bin\libreadstat-0.dll>)

I had to slightly change things because the escaping didn't work, but I think that code should do what you want, right?

[hhaensel]

Maybe then escaping is the point?
Are you working in a WSL or native windows?
In the latter case could you try from a regular cmd and from a powershell?

[davidanthoff]

This is from regular cmd:

C:\Users\david>\Users\david\AppData\Local\julia-1.1.1\bin\julia.exe -E "import Libdl; Libdl.open(ARGS[1])" C:\Users\david\.julia\dev\ReadStat\deps\usr\bin\libreadstat-0.dll
IOStream(<file C:\Users\david\.julia\dev\ReadStat\deps\usr\bin\libreadstat-0.dll>)

C:\Users\david>

This from Powershell:

 david@DANTHOFF-5820 > julia -E "import Libdl; Libdl.open(\"C:\\Users\\david\\.julia\\dev\\ReadStat\\deps\\usr\\bin\\libreadstat-0.dll\")"
ERROR: syntax: incomplete: invalid string syntax
 david@DANTHOFF-5820 >

No WSL involved, this is all native windows.

[hhaensel]

Very interesting, maybe the error depends on the platform that is used for issuing commands. I'm currently on my phone and don't have access to a computer.
I like your idea with the ARGS syntax. You might try entering that in the isolate==true case.

[davidanthoff]

I don't really follow, to be honest :) What should I do next? Also, I don't really understand why the issues about escaping from the command line are relevant for this question in general?