Description
I'm trying everything to get startup down, and now:
https://github.com/petvana/AutoSysimages.jl
It works for me. First the good news:
julia> @time using PythonCall
0.010378 seconds (8.97 k allocations: 565.888 KiB)
and with -O0:
julia> @time using PythonCall
0.003670 seconds (8.97 k allocations: 565.888 KiB)
Some (but not all, since 5.7 sec otherwise) of the startup-cost has been shifted to startup of Julia itself with the sysimage:
$ hyperfine '~/julia-1.9-DEV-b01fd3367b/bin/julia -J /home/pharaldsson/.julia/asysimg/1.9.0-DEV.1694/30JkAN/asysimg-2022-10-30T22-32-01.991.so -e ""'
Benchmark 1: ~/julia-1.9-DEV-b01fd3367b/bin/julia -J /home/pharaldsson/.julia/asysimg/1.9.0-DEV.1694/30JkAN/asysimg-2022-10-30T22-32-01.991.so -e ""
Time (mean ± σ): 3.116 s ± 0.031 s [User: 3.791 s, System: 1.184 s]
Range (min … max): 3.067 s … 3.151 s 10 runs
And 1.91 sec. for -O0.
I believe some work on the package could improve this, with or without a sysimage. The sysimage is 185 MB vs 209 MB for Julia's own 1.9 sysimage (sys.so, not sure why it's larger) vs a 75 MB sysimage I made for other reasons (to cut startup of Julia in half for benchmarking/scripting reasons).
AutoSysimages is really easy to use (at first I used build_sysimage() directly which likely is incorrect), just follow the docs, then for me this worked:
$ PATH=/home/pharaldsson/.local/bin/:$PATH asysimg
Julia 1.9 alone seems a bit faster than 1.8, at least julia -O0:
julia> @time using PythonCall
3.970856 seconds (2.55 M allocations: 170.295 MiB, 3.47% gc time, 81.35% compilation time: <1% of which was recompilation)
A question related to it, I got 1.9-DEV to try, and it should also work with PythonCall.jl (and does, on its own), but is this expected:
Prefix: /home/pharaldsson/.julia/environments/v1.9/.CondaPkg/env
Updating specs:
- conda-forge::libstdcxx-ng[version='>=3.4,<11.4']
- conda-forge::python[version='>=3.7,<4',build=*cpython*]
+ _libgcc_mutex 0.1 conda_forge conda-forge/linux-64 Cached
+ _openmp_mutex 4.5 2_gnu conda-forge/linux-64 Cached
+ bzip2 1.0.8 h7f98852_4 conda-forge/linux-64 Cached
+ ca-certificates 2022.9.24 ha878542_0 conda-forge/linux-64 Cached
+ ld_impl_linux-64 2.39 hc81fddc_0 conda-forge/linux-64 778kB
+ libffi 3.4.2 h7f98852_5 conda-forge/linux-64 Cached
+ libgcc-ng 12.2.0 h65d4601_19 conda-forge/linux-64 954kB
+ libgomp 12.2.0 h65d4601_19 conda-forge/linux-64 466kB
+ libnsl 2.0.0 h7f98852_0 conda-forge/linux-64 Cached
+ libsqlite 3.39.4 h753d276_0 conda-forge/linux-64 822kB
+ libstdcxx-ng 11.3.0 h239ccf8_19 conda-forge/linux-64 4MB
+ libuuid 2.32.1 h7f98852_1000 conda-forge/linux-64 Cached
+ libzlib 1.2.13 h166bdaf_4 conda-forge/linux-64 66kB
+ ncurses 6.3 h27087fc_1 conda-forge/linux-64 1MB
+ openssl 3.0.5 h166bdaf_2 conda-forge/linux-64 Cached
+ pip 22.3 pyhd8ed1ab_0 conda-forge/noarch 2MB
+ python 3.11.0 ha86cf86_0_cpython conda-forge/linux-64 38MB
+ readline 8.1.2 h0f457ee_0 conda-forge/linux-64 298kB
+ setuptools 65.5.0 pyhd8ed1ab_0 conda-forge/noarch 787kB
+ tk 8.6.12 h27826a3_0 conda-forge/linux-64 3MB
+ tzdata 2022e h191b570_0 conda-forge/noarch 121kB
+ wheel 0.37.1 pyhd8ed1ab_0 conda-forge/noarch 32kB
+ xz 5.2.6 h166bdaf_0 conda-forge/linux-64 Cached
I see issue here related to libstdcxx-ng, and I believe maybe you don't need to download it, at least soon (work almost done on julia master).
Good to see Python 3.11.0 is done and getting it with your package. It's not a huge deal to trim the list above, but is e.g. tk, readline, ncurses, libgcc-ng, libsqlite really needed or basically all of it? Is pip redundant with conda/mamba, i.e. as you use it?
Have you looked at CircuitPython/MicroPython? I suppose that can't work with your package (JuliaCall), though maybe StaticCompiler.jl.