Plots don't show automatically when PyPlot compiled with PackageCompiler

[marius311]

Not sure if this is supposed to work (in any case it seems like its really close to working). I compiled PyPlot v2.8.2 with PackageCompiler v1.1.0 on julia 1.4,

create_sysimage([:PyPlot], sysimage_path="pyplot.so")

then load it with

$ julia -J pyplot.so 

julia> using PyPlot

# no plot gets shown shown here (but there should be)
julia> plot([1,2,3])  
1-element Array{PyCall.PyObject,1}:
 PyObject <matplotlib.lines.Line2D object at 0x7f71cca43d68>

# this shows the plot, but the following error pops up after closing the window
julia> show() 
ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/marius/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError('signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object')
  File "/home/marius/miniconda3/lib/python3.7/site-packages/matplotlib/pyplot.py", line 269, in show
    return _show(*args, **kw)
  File "/home/marius/miniconda3/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py", line 413, in wrapper
    return func(*args, **kwargs)
  File "/home/marius/miniconda3/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 3302, in show
    cls.mainloop()
  File "/home/marius/miniconda3/lib/python3.7/site-packages/matplotlib/backends/backend_qt5.py", line 1099, in mainloop
    signal.signal(signal.SIGINT, old_signal)
  File "/home/marius/miniconda3/lib/python3.7/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))

Stacktrace:
 [1] pyerr_check at /home/marius/.julia/packages/PyCall/zqDXB/src/exception.jl:60 [inlined]
 [2] pyerr_check at /home/marius/.julia/packages/PyCall/zqDXB/src/exception.jl:64 [inlined]
 [3] _handle_error(::String) at /home/marius/.julia/packages/PyCall/zqDXB/src/exception.jl:81
 [4] macro expansion at /home/marius/.julia/packages/PyCall/zqDXB/src/exception.jl:95 [inlined]
 [5] #110 at /home/marius/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:43 [inlined]
 [6] disable_sigint at ./c.jl:446 [inlined]
 [7] __pycall! at /home/marius/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:42 [inlined]
 [8] _pycall!(::PyCall.PyObject, ::PyCall.PyObject, ::Tuple{}, ::Int64, ::Ptr{Nothing}) at /home/marius/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:29
 [9] _pycall! at /home/marius/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:11 [inlined]
 [10] #pycall#115 at /home/marius/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:80 [inlined]
 [11] pycall at /home/marius/.julia/packages/PyCall/zqDXB/src/pyfncall.jl:80 [inlined]
 [12] #show#148 at /home/marius/.julia/packages/PyPlot/4wzW1/src/PyPlot.jl:183 [inlined]
 [13] show() at /home/marius/.julia/packages/PyPlot/4wzW1/src/PyPlot.jl:183
 [14] top-level scope at REPL[3]:1

Any ideas if this can work and if so how? Thanks.

[marius311]

Looks like the error goes away after upgrading matplotlib 3.1.1 -> 3.2.1 (as per #459 (comment)).

The plot still is not showing automatically though. Any way to reset whatever has gone wrong to make them show?

[xzackli]

Hey @marius311, this happens because PyPlot.display_figs(), the function called at the end of an IJulia cell, thinks there is no available display. You can see this in the notebook, i.e.

using PyPlot
PyPlot.isjulia_display

1-element Array{Bool,1}:
 0

PyPlot sets this global when it runs its init, which I suppose has now been compiled into your sysimage.

PyPlot.jl/src/init.jl

Line 178 in c451315

global isjulia_display = Bool[isdisplayok()]

This kind of thing should happen to any PackageCompiler-ed package which relies on logic during package loading. You can manually show the figure for now, i.e.

fig = plt.figure()
plt.plot()
fig

I'll see if I can make a PR.

[xzackli]

Probably all it takes is just to check if there are displays whenever a cell is requested. For example, this works in a sysimage,

using PyCall
using PyPlot

function display_figs() # called after IJulia cell executes
    if PyPlot.isdisplayok()  # CHANGED FROM QUERYING GLOBAL
        for manager in PyPlot.Gcf."get_all_fig_managers"()
            f = manager."canvas"."figure"
            if f.number ∉ PyPlot.withfig_fignums
                fig = Figure(f)
                isempty(fig) || display(fig)
                pycall(plt."close", PyAny, f)
            end
        end
    end
end

plt.plot()
display_figs()

One also needs to push the pre-executes (which don't seem to make it in the sysimage)

Main.IJulia.push_preexecute_hook(PyPlot.force_new_fig)
Main.IJulia.push_postexecute_hook(PyPlot.display_figs)
Main.IJulia.push_posterror_hook(PyPlot.close_figs)

[marius311]

Thanks, yea I think you're on the right track. The thing is that PackageCompiler still runs PyPlot.__init__() when you open a new terminal rather than when its being compiled, but I think whenever its running it must be too early because PyPlot.isdisplayok() must be returning false, as evidenced by as show that PyPlot.isjulia_display is false. Its only later that PyPlot.isdisplayok() starts returning true. So you end up with this when you use the system image:

[stevengj]

I'm afraid that's going to mess up other aspects of the initialization, too.

In particular, if a Julia display is detected when PyPlot is initialized, then it starts up Matplotlib in non-interactive mode, so that figures are only displayed when the cell finishes executing and the post-execute hook is called. Whereas if displayok() == false at the __init__ call, then it will start up the GUI event loop and display plots in the GUI by default (assuming you have a working Matplotlib GUI backend).

Why is __init__ not being called when the user runsimport PyPlot?

[xzackli]

Why is init not being called when the user runsimport PyPlot?

As you described in the PR, the best way to fix this is probably to do things in a lazy fashion within PyPlot. Here's an example of the order of init calls changing when using sysimages. Consider the following test where I just stick a print statement in the PyPlot init.

normal order

julia -e 'println("Julia started"); flush(stdout); sleep(1); import PyPlot;'

Julia started
INIT IS CALLED

with sysimage

julia --sysimage=mysys.dylib -e 'println("Julia started"); flush(stdout); sleep(1); import PyPlot;'

INIT IS CALLED
Julia started

With the sysimage, PyPlot's init is called before the first line of Julia code is started. Presumably PyPlot wakes up and runs init, only to find a barren land without any IJulia loaded.

I'd be interested in reading how Julia starts up, maybe it's in the docs.

[tfiers]

My current solution is calling a custom function autodisplay_figs() (see below) at the start of the notebook.
You can call it manually; I call it in the __init__ of a custom package that gets using'd in the notebook, i.e. after IJulia has been initialized. I suppose you could call it in startup_ijulia.jl as well.

This function simply sets PyPlot.isjulia_display[] manually to true, and adds the three IJulia hooks, as in xzackli's proposal above.

This doesn't fix the fact that a GUI event loop is, presumably, unnecesarily spun up.
But that is not noticeable (I suspect because the startup cost of that is in the sysimg?).

using PyPlot is instant, and plots display automatically under cells again.

`autodisplay_figs()`:

"""
When using a sysimg with PyPlot in it, PyPlot's `__init__` gets called before IJulia is
initialized. As a result, figures do not get automatically displayed in the notebook.
(https://github.com/JuliaPy/PyPlot.jl/issues/476).
Calling `autodisplay_figs()` after IJulia is initialized fixes that.
"""
function autodisplay_figs()
    if (isdefined(Main, :PyPlot) && isdefined(Main, :IJulia) && Main.IJulia.inited
        && (Main.PyPlot.isjulia_display[] == false)
    )
        Main.PyPlot.isjulia_display[] = true
        Main.IJulia.push_preexecute_hook(Main.PyPlot.force_new_fig)
        Main.IJulia.push_postexecute_hook(Main.PyPlot.display_figs)
        Main.IJulia.push_posterror_hook(Main.PyPlot.close_figs)
    end
end

[stevengj]

See also my #480 (comment) on how I think this should be fixed.

[ufechner7]

The following command works for me as a workaround when using the REPL:

PyPlot.show(block=false)