Broken precompilation under Julia 1.3-alpha1/win64 #110

henry-eshbaugh · 2019-09-12T20:22:38Z

Hi all,

FFTW precompilation when directly imported breaks once, then seems OK (I have replicated this a few times with different Julia instances). It seems that the DLL libfftw3-3.dll does not export fftw_threads_set_callback() which is required for the new PARTR scheduling scheme under 1.3. Curious if the DLL needs updating or I'm just pulling a wonky one from the build script.

julia> using FFTW
ERROR: InitError: ccall: could not find function fftw_threads_set_callback in library C:\Users\henry\.julia\packages\FFTW\xi4tZ\deps\usr\bin\libfftw3-3.dll
Stacktrace:
 [1] __init__() at C:\Users\henry\.julia\packages\FFTW\xi4tZ\src\FFTW.jl:60
 [2] _include_from_serialized(::String, ::Array{Any,1}) at .\loading.jl:692
 [3] _require_search_from_serialized(::Base.PkgId, ::String) at .\loading.jl:776
 [4] _require(::Base.PkgId) at .\loading.jl:1001
 [5] require(::Base.PkgId) at .\loading.jl:922
 [6] require(::Module, ::Symbol) at .\loading.jl:917
during initialization of module FFTW

julia> using FFTW

julia> fft([1 -1])
1×2 Array{Complex{Float64},2}:
 0.0+0.0im  2.0+0.0im

Repeating "using" until the library imports does not seem to work in the context of other libraries (e.g. DSP.jl, Wavelets.jl):

julia> using DSP
[ Info: Precompiling DSP [717857b8-e6f2-59f4-9121-6e50c889abd2]
ERROR: LoadError: InitError: ccall: could not find function fftw_threads_set_callback in library C:\Users\henry\.julia\packages\FFTW\xi4tZ\deps\usr\bin\libfftw3-3.dll
Stacktrace:
 [1] __init__() at C:\Users\henry\.julia\packages\FFTW\xi4tZ\src\FFTW.jl:60
 [2] _include_from_serialized(::String, ::Array{Any,1}) at .\loading.jl:692
 [3] _require_search_from_serialized(::Base.PkgId, ::String) at .\loading.jl:776
 [4] _require(::Base.PkgId) at .\loading.jl:1001
 [5] require(::Base.PkgId) at .\loading.jl:922
 [6] require(::Module, ::Symbol) at .\loading.jl:917
 [7] include at .\boot.jl:328 [inlined]
 [8] include_relative(::Module, ::String) at .\loading.jl:1105
 [9] include(::Module, ::String) at .\Base.jl:31
 [10] top-level scope at none:2
 [11] eval at .\boot.jl:330 [inlined]
 [12] eval(::Expr) at .\client.jl:433
 [13] top-level scope at .\none:3
during initialization of module FFTW
in expression starting at C:\Users\henry\.julia\packages\DSP\wwKNu\src\DSP.jl:3
ERROR: Failed to precompile DSP [717857b8-e6f2-59f4-9121-6e50c889abd2] to C:\Users\henry\.julia\compiled\v1.3\DSP\OtML7_PQCoN.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1274
 [3] _require(::Base.PkgId) at .\loading.jl:1024
 [4] require(::Base.PkgId) at .\loading.jl:922
 [5] require(::Module, ::Symbol) at .\loading.jl:917

The library lives here:

julia> pathof(FFTW)
"C:\\Users\\henry\\.julia\\packages\\FFTW\\xi4tZ\\src\\FFTW.jl"

Finally, version info:


julia> versioninfo()
Julia Version 1.3.0-rc1.0
Commit 768b25f6a8 (2019-08-18 00:04 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, sandybridge)
Environment:
  JULIA_NUM_THREADS = 8

Help would be appreciated :). Do I just need updated DLLs?

The text was updated successfully, but these errors were encountered:

giordano · 2019-09-12T21:31:35Z

Did you build FFTW recently? If not, run ]build FFTW, restart Julia and try again.

henry-eshbaugh · 2019-09-12T21:39:09Z

Did you build FFTW recently? If not, run ]build FFTW, restart Julia and try again.

No dice, sorry!

giordano · 2019-09-12T21:52:03Z

Just to be sure, what's the path of your libfftw3?

using FFTW
FFTW.libfftw3

Also, what happens if you do the following?

julia> using Libdl, FFTW

julia> Libdl.dlsym(Libdl.dlopen(FFTW.libfftw3), :fftw_threads_set_callback)
Ptr{Nothing} @0x00007fa8f8b58540

I can see the symbol fftw_threads_set_callback in bin/libfftw3-3.dll at https://github.com/JuliaBinaryWrappers/FFTW_jll.jl/releases/download/FFTW-v3.3.9%2B0/FFTW.v3.3.9.x86_64-w64-mingw32.tar.gz:

% nm bin/libfftw3-3.dll|grep set_callback
00000000711648a2 T fftw_threads_set_callback

I suspect you're not pulling/using the right library

stevengj · 2019-09-13T02:35:43Z

Maybe FFTW is missing some kind of Windows __declspec(export) thing for the new fftw_threads_set_callback function, in which case I'll have to fix it upstream and generate new binaries. (Might take me a few days to get to it if this is the problem, but it will happen in time for 1.3.)

stevengj · 2019-09-13T03:02:39Z

Though CI is passing on Windows nightly for FFTW.jl: https://ci.appveyor.com/project/ararslan/fftw-jl/branch/master

giordano · 2019-09-13T08:46:57Z

Though CI is passing on Windows nightly for FFTW.jl

Yes, that's why I suspect for some reasons the OP is not using the correct library version and wanted to double check that.

henry-eshbaugh · 2019-09-13T15:23:46Z

Hi all,

Many thanks again!

julia> using Libdl, FFTW

julia> Libdl.dlsym(Libdl.dlopen(FFTW.libfftw3), :fftw_threads_set_callback)
ERROR: could not load symbol "fftw_threads_set_callback":
The specified procedure could not be found.

Stacktrace:
 [1] dlsym(::Ptr{Nothing}, ::Symbol) at C:\cygwin\home\henry\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Libdl\src\Libdl.jl:56
 [2] top-level scope at none:1

julia> FFTW.libfftw3
"C:\\Users\\henry\\.julia\\dev\\FFTW\\deps\\usr\\bin\\libfftw3-3.dll"

I'll pull the DLLs from the nightly and see if a fix is had on my box, stay tuned!

giordano · 2019-09-13T15:28:22Z

Wait, you said that your package is in

julia> pathof(FFTW)
"C:\\Users\\henry\\.julia\\packages\\FFTW\\xi4tZ\\src\\FFTW.jl"

but the library is in another place:

julia> FFTW.libfftw3
"C:\\Users\\henry\\.julia\\dev\\FFTW\\deps\\usr\\bin\\libfftw3-3.dll"

You have something mixed up

giordano · 2019-09-13T15:31:11Z

My guess is that this should work for you:

Libdl.dlsym(Libdl.dlopen("C:\\Users\\henry\\.julia\\packages\\FFTW\\xi4tZ\\deps\\usr\\bin\\libfftw3-3.dll"), :fftw_threads_set_callback)

henry-eshbaugh · 2019-09-13T15:35:25Z

Newbie to Julia, do alpha releases hide the packages you install in the \dev path? Could explain the mismatch in paths; I've not tried to do anything outrageously funky :).

My guess is that this should work for you:

Libdl.dlsym(Libdl.dlopen("C:\\Users\\henry\\.julia\\packages\\FFTW\\xi4tZ\\deps\\usr\\bin\\libfftw3-3.dll"), :fftw_threads_set_callback)

Unfortunately, again no dice :(.


julia> Libdl.dlsym(Libdl.dlopen("C:\\Users\\henry\\.julia\\packages\\FFTW\\xi4tZ\\deps\\usr\\bin\\libfftw3-3.dll"), :fftw_threads_set_callback)
ERROR: could not load symbol "fftw_threads_set_callback":
The specified procedure could not be found.

Stacktrace:
 [1] dlsym(::Ptr{Nothing}, ::Symbol) at C:\cygwin\home\henry\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Libdl\src\Libdl.jl:56
 [2] top-level scope at none:1

Could you link me those DLLs more directly? I can move them in by hand to see if the problem is mitigated in this build.

Edit: should also mention I've used a DLL explorer to check the exported symbol table of the xi4tZ path and there's definitely no function like it there. Not tried it on the dev path but expect the same. Libdl is some nifty stuff!

giordano · 2019-09-13T15:54:56Z

I'd recommend to remove FFTW in Julia:

]rm FFTW

and manually remove the directory in "C:\Users\henry\.julia\dev\FFTW" (unless you have particular reason to keep the repository on your computer).

Then, restart Julia and reinstall FFTW:

]add FFTW
build FFTW

(note: both commands must be run in the Pkg manager mode, I put the opening ] only once). Now in the Julia prompt you should hopefully be able to do

using FFTW

giordano · 2019-09-13T16:12:53Z

If this is still failing let's check that you got the right library:

I expect FFTW.libfftw3 to be:

"C:\\Users\\henry\\.julia\\packages\\FFTW\\xi4tZ\\deps\\usr\\bin\\libfftw3-3.dll"

check the SHA256 hash of the library. You can do this within Julia with the package SHA:

julia> using SHA, FFTW

julia> bytes2hex(open(FFTW.libfftw3) do f; sha2_256(f); end)
"af1c862f036b3d302e837a1725dc7fba596ace3b00522091cbaeca5b6d6deebe"

If you get different results, you either still have something wrong or I miscomputed the hash (:see_no_evil:)

henry-eshbaugh · 2019-09-13T17:01:22Z

Deleted the dev path after removing FFTW, then did add and build. Again, no success, though FFTW.libfftw3 ponts in the right direction:

julia> using FFTW
julia> FFTW.libfftw3
"C:\\Users\\henry\\.julia\\packages\\FFTW\\xi4tZ\\deps\\usr\\bin\\libfftw3-3.dll"

Very unfortunately:

julia> using SHA
julia> bytes2hex(open(FFTW.libfftw3) do f; sha2_256(f); end)
"af1c862f036b3d302e837a1725dc7fba596ace3b00522091cbaeca5b6d6deebe"

stevengj · 2019-09-14T13:58:02Z

Though CI is passing on Windows nightly for FFTW.jl

Ah, but I just realized that AppVeyor does not exercise this feature — only the Travis script (which is currently just for linux and osx) tries setting JULIA_NUM_THREADS=2

henry-eshbaugh · 2019-09-16T17:44:35Z

That's promising; does that mean there's simply a missing _declspec?

stevengj · 2019-09-16T18:50:10Z

Yes.

stevengj · 2019-09-16T19:49:27Z

I updated the FFTW source code at FFTW/fftw3@8a9236c … now we need a new build.

stevengj · 2019-09-18T17:32:51Z

Should be fixed on the master branch now, if you can try that (pkg> dev FFTW) and confirm it would be appreciated.

henry-eshbaugh · 2019-09-18T19:46:29Z

Happy to report all OK! Thanks so much!

ashwani-rathee · 2021-05-10T05:42:49Z

@giordano I am facing this issue, I have provided the information that has been used to solve this issue about above.
It's local issue to my system and works fine for another contributor there.

(@v1.6) pkg> activate .
  Activating environment at `~/julia-related/MusicProcessing.jl/Project.toml`

julia> using Revise

julia> using MusicProcessing

julia> using FFTW

julia> pathof(FFTW)
"/home/ashwani/.julia/packages/FFTW/Iu2GG/src/FFTW.jl"

julia> FFTW.libfftw3
"/home/ashwani/.julia/artifacts/81791030d1dcd08bf0c67b3e8224cb573d0f5a0a/lib/libfftw3.so"

julia> FFTW.libfftw3f
"/home/ashwani/.julia/artifacts/81791030d1dcd08bf0c67b3e8224cb573d0f5a0a/lib/libfftw3f.so"

julia> using Libdl

julia> Libdl.dlsym(Libdl.dlopen(FFTW.libfftw3), :fftw_threads_set_callback)
Ptr{Nothing} @0x00007fb0086c2c40

# this is where issue is in my system
julia> Libdl.dlsym(Libdl.dlopen(FFTW.libfftw3f), :fftw_threads_set_callback)
ERROR: could not load symbol "fftw_threads_set_callback":
/home/ashwani/.julia/artifacts/81791030d1dcd08bf0c67b3e8224cb573d0f5a0a/lib/libfftw3f.so: undefined symbol: fftw_threads_set_callback

julia> audio_one_channel = SampleBuf(rand(1000), 10)
1000-element SampleBuf{Float64, 1}:
 0.7188992365329432
 0.9291267382769475
 0.9235711252208105
 0.7298652018343716
 ⋮
 0.7817233868027189
 0.9691375556064439
 0.009171005643992869

julia> speedup(audio_one_channel, 1.5)
# also this doesn't make much sense yet to me
ERROR: could not load library "libfftw3"
libfftw3.so: cannot open shared object file: No such file or directory
Stacktrace:
 [1] plan_dft_c2r_1d!
   @ ~/julia-related/MusicProcessing.jl/src/complex.jl:22 [inlined]
 [2] irfft!(dest::Vector{Float64}, src::Ptr{ComplexF64}, nfft::Int64)
   @ MusicProcessing ~/julia-related/MusicProcessing.jl/src/complex.jl:35
 [3] istft(stft::Matrix{ComplexF64}, samplerate::Float64, windowsize::Int64, hopsize::Int64; nfft::Int64, window::typeof(DSP.Windows.hanning))
   @ MusicProcessing ~/julia-related/MusicProcessing.jl/src/TFR.jl:117
 [4] istft
   @ ~/julia-related/MusicProcessing.jl/src/TFR.jl:101 [inlined]
 [5] #speedup#3
   @ ~/julia-related/MusicProcessing.jl/src/audio.jl:99 [inlined]
 [6] speedup (repeats 3 times)
   @ ~/julia-related/MusicProcessing.jl/src/audio.jl:97 [inlined]
 [7] top-level scope
   @ REPL[15]:1

version info:

julia> versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

henry-eshbaugh mentioned this issue Sep 12, 2019

partr threads backend FFTW/fftw3#175

Closed

giordano mentioned this issue Sep 17, 2019

[FFTW] Upgrade to latest version on master JuliaPackaging/Yggdrasil#83

Closed

This was referenced Sep 17, 2019

new FFTW upstream tarball JuliaPackaging/Yggdrasil#84

Merged

updated FFTW 3.3.9 (prerelease) build #112

Merged

stevengj closed this as completed in #112 Sep 18, 2019

stevengj mentioned this issue Sep 18, 2019

JULIA_NUM_THREADS not working with 1.3-rc2 #113

Closed

ashwani-rathee mentioned this issue May 9, 2021

Trying to work on functionality for the display and other updates JuliaMusic/MusicProcessing.jl#15

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Broken precompilation under Julia 1.3-alpha1/win64 #110

Broken precompilation under Julia 1.3-alpha1/win64 #110

henry-eshbaugh commented Sep 12, 2019 •

edited

Loading

giordano commented Sep 12, 2019 •

edited

Loading

henry-eshbaugh commented Sep 12, 2019

giordano commented Sep 12, 2019 •

edited

Loading

stevengj commented Sep 13, 2019 •

edited

Loading

stevengj commented Sep 13, 2019

giordano commented Sep 13, 2019

henry-eshbaugh commented Sep 13, 2019

giordano commented Sep 13, 2019

giordano commented Sep 13, 2019 •

edited

Loading

henry-eshbaugh commented Sep 13, 2019 •

edited

Loading

giordano commented Sep 13, 2019

giordano commented Sep 13, 2019

henry-eshbaugh commented Sep 13, 2019

stevengj commented Sep 14, 2019

henry-eshbaugh commented Sep 16, 2019

stevengj commented Sep 16, 2019

stevengj commented Sep 16, 2019

stevengj commented Sep 18, 2019

henry-eshbaugh commented Sep 18, 2019

ashwani-rathee commented May 10, 2021

Broken precompilation under Julia 1.3-alpha1/win64 #110

Broken precompilation under Julia 1.3-alpha1/win64 #110

Comments

henry-eshbaugh commented Sep 12, 2019 • edited Loading

giordano commented Sep 12, 2019 • edited Loading

henry-eshbaugh commented Sep 12, 2019

giordano commented Sep 12, 2019 • edited Loading

stevengj commented Sep 13, 2019 • edited Loading

stevengj commented Sep 13, 2019

giordano commented Sep 13, 2019

henry-eshbaugh commented Sep 13, 2019

giordano commented Sep 13, 2019

giordano commented Sep 13, 2019 • edited Loading

henry-eshbaugh commented Sep 13, 2019 • edited Loading

giordano commented Sep 13, 2019

giordano commented Sep 13, 2019

henry-eshbaugh commented Sep 13, 2019

stevengj commented Sep 14, 2019

henry-eshbaugh commented Sep 16, 2019

stevengj commented Sep 16, 2019

stevengj commented Sep 16, 2019

stevengj commented Sep 18, 2019

henry-eshbaugh commented Sep 18, 2019

ashwani-rathee commented May 10, 2021

henry-eshbaugh commented Sep 12, 2019 •

edited

Loading

giordano commented Sep 12, 2019 •

edited

Loading

giordano commented Sep 12, 2019 •

edited

Loading

stevengj commented Sep 13, 2019 •

edited

Loading

giordano commented Sep 13, 2019 •

edited

Loading

henry-eshbaugh commented Sep 13, 2019 •

edited

Loading