Building for Python 3

[Vaakapallo]

Hi,

I've been trying to get PD working with Python 3 and even though I got this external to build with Python 3.6, PD says "can't load library". (A 2.7. build worked fine).

I heard that it is likely due to differences in the C API between Python 2.7 and Python 3, but I'm not too familiar with this area and lost on the topic.

Anyone have experience at this or any good resources? I found this, but don't really know where to go to from here.

[grrrr]

Hi Lassi, my impression is that the Python 3.6 headers are not used, but rather the 2.7 ones, which of course works until the linking stage. Then, linking with Pd used lazy linking (-undefined dynamic_lookup option), shadowing missing linker symbols. I am pretty sure the change to the Python 3 API is not as effortless as one might wish.

[grrrr]

I have made a python3 branch which makes tiny changes to the build system. It should still work with 2.7 but now use the proper paths to the Python headers.
Using it with 3.6 throws compilations errors immediately because of drastic changes in the Python3 API. These need to be fixed.

[Vaakapallo]

I went through the errors and fixed the ones I could. I have a commit in my fork that tries to fix the easiest changes, but had to comment out or inadequately fix some. f807ad6

It builds, but this version crashes PD when loaded (no surprises there). I was wondering if there's any way to get information on what is going wrong or some way to test that the code behaves like it did before the changes.

[grrrr]

Thanks, i have had a quick look but fail to understand a few replacements of old versus new functions. In any case, all adaptations of the code should be conditional so that compilation works for both python2 and python3 APIs.

[Vaakapallo]

I responded to the comments in the commit. And made a new commit with Unicode checks: 3908b2b

Yeah, compatibility with Python 2 and 3 would be ideal, but unfortunately that is probably beyond the scope of my abilities. I currently just need something that works with Python3, which would be a good milestone to getting something that works for both.

Though it seems I've hit a dead end. Without a really good understanding of the system (or comprehensive way to test it), replacing functions in the code will mostly lead to nonsensical changes. Gaining the understanding solely through reading code with my level in C++ would take a very long time. Currently I only see that PD crashes when I try to load py with the changes.

As it stands, my fork points out parts of the code that need some sort of update for Python3 compatibility. Don't really know how to move forward from here.

[Vaakapallo]

Found another quick resource for porting

[Vaakapallo]

module_obj = Py_InitModule(const_cast<char *>(PYEXT_MODULE), func_tbl);
module_dict = PyModule_GetDict(module_obj); // borrowed reference

The replacement for this part in pybase.cpp is the thing that is crashing. And it is related to the new module initialization system, but I don't know how to make it work correctly on the new system.

[ktatar]

Hi all,

I am trying to build the source using flext. I couldn't get rid of the following error:

source/pyprefix.h:20:10: fatal error: 'Python/Python.h' file not found
#include <Python/Python.h>

any ideas?

[umlaeute]

Python2 is reaching EOL in about 100 days from now.

Debian is currently fading out all packages that still require Python2 (including py).

Maybe it's time :-)

[grrrr]

Yes, absolutely. I have been working on a branch for python3 development. https://github.com/grrrr/py/tree/python3
However, module management has fundamentally changed and i am proceeding slower than i would want.

[umlaeute]

i became aware of the python3 branch after i posted.
but looking at it (and amending it to linux) i notice that porting is indeed not trivial... (so i decided to stay silent)

[ahihi]

hello,

i forked @grrrr's python3 branch and am working on it here https://github.com/SopiMlab/py/tree/python3 (also includes some build tweaks for Conda support on Mac)

i can build the code for Python 3.7 and 2.7, but i have some runtime issues...

on 3.7, the external initializes successfully. when i open simple-1.pd, the Python module loads (verified by putting a print at the end), but creating the pyext Pd object fails with no other information than "couldn't create".

on 2.7, i get a crash on init like this:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000008
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [44060]

VM Regions Near 0x8:
--> 
    __TEXT                 0000000100000000-0000000100162000 [ 1416K] r-x/rwx SM=COW  /var/folders/*/Pd-0.50-0.app/Contents/Resources/bin/pd

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libpython2.7.dylib            	0x0000000104a360db PyDict_SetItemString + 59
1   py.pd_darwin                  	0x00000001006f5592 pyext::Setup(flext_class*) + 1266
2   py.pd_darwin                  	0x00000001007025d2 flext_obj_multi::obj_add(bool, bool, bool, bool, char const*, char const*, void (*)(flext_class*), flext_obj_multi* (*)(int, _atom*), void (*)(flext_hdr*), int, ...) + 946
3   py.pd_darwin                  	0x00000001006f5044 pyext_setup() + 84
4   py.pd_darwin                  	0x00000001006fc74d pybase::lib_setup() + 429
5   py.pd_darwin                  	0x0000000100702147 flext_obj_multi::lib_init(char const*, void (*)()) + 55
6   pd                            	0x00000001000a7bfa sys_do_load_lib + 618
7   ???                           	0x612f73726573552f 0 + 7002942880749933871

i guess the next step is to get a debugging environment set up :) i found the instructions on the Pd site, but if anyone has experience doing this with pyext specifically, any tips are appreciated!

[umlaeute]

for me, it has been crashing with python2.7 for a while. given that python2 is dead now, i wouldn't invest time in fixing it