Problems with static python builds.

[Neumann-A]

There is a problem with static python builds that cannot import extensions since the extension has not been statically linked into the python executable. However you could build an application which has more extensions and links more then the static python library which is apparent from https://www.kitware.com/static-python-and-numpy-with-paraview/

The question is how to achieve something similar for vcpkg? Basically using host python + extensions might not be enough since if the host triplet is a static python build the extensions become unusable and the code cannot be run from the host.

[m-kuhn]

My immediate plan of action involves pushing microsoft/vcpkg#36556 to improve support for dynamic mac builds and ignoring the problems that come with static python builds.

[Neumann-A]

that would require for the dynamic triplets to become the default one which will probably not happen.

[m-kuhn]

that would require for the dynamic triplets to become the default one which will probably not happen.

I might be accepting that for my immediate use cases.

But back to your question, is host python with [numpy] support a requirement?
I imagine static support boils down to a user application linking python and various modules, i.e. code that is maintained outside vcpkg (or to a limited scope within the module itself just to make the import tests work). Or we have a dedicated port with features python3-static[numpy,pyqt] but this offers very little flexibility and high maintenance.

[Neumann-A]

is host python with [numpy] support a requirement?

If a port wants to run a python script using numpy for whatever reason yes it is a requirement.

[m-kuhn]

In this case this port will need to link (host) python itself or have a helper port that does it instead, I don't think there's a way around this using static builds.

[Neumann-A]

I think if microsoft/vcpkg#39449 is merged this can be closed due to the extension feature in python3

[m-kuhn]

@Neumann-A does microsoft/vcpkg#39449 work for every scenario, if we have common dependencies in the main app and a python extension, which is linked (statically) into both binaries, won't we run into ODR violations?

[Neumann-A]

won't we run into ODR violations?

I don't fully understand the question, so let me ask another question: How do you think this violation could occurs?

• common dependencies in the main app and a python extension
• common dependencies, which is linked (statically) into both binaries,

Case 1) Everything dynamic -> no problem
Case 2) Everything static -> no problem
Case 3) Mixed linkage, e.g. common dependency is static but python is dynamic. In this case the python extension has its own version of the dependency just like the main app. This is only a problem if there is shared state. There is no ODR violation since linking the python executable does not pull in the symbols from the common dependency.

[m-kuhn]

What I'm thinking is: qt being linked statically to the main app (executable, might not be python.exe but another application initializing a python interpreter) and a dynamically loaded shared lib as part of a python module (e.g. pyqt) also linking qt statically.
I experienced crashes when the qt lib was loaded twice from two identical files by the application. Does this fall into the "shared state" category?

[Neumann-A]

I experienced crashes when the qt lib was loaded twice from two identical files by the application.

That is a different kind of error, however you shouldn't see that in static linkage since the dynamic loader shouldn't be involved (and make sure normal symbol visibility is hidden)?
Are you trying to pass Qt Objects between the two? That is probably not going to work, especially if ownership is getting transferred.
Dies PyQt even support static qt?

[m-kuhn]

I experienced crashes when the qt lib was loaded twice from two identical files by the application.

That is a different kind of error, however you shouldn't see that in static linkage since the dynamic loader shouldn't be involved (and make sure normal symbol visibility is hidden)?

I didn't try, so I cannot tell

Are you trying to pass Qt Objects between the two?

Yes, pyqt is used to write plugins for the main app which interact with Qt objects / widgets from the main app

That is probably not going to work, especially if ownership is getting transferred. Dies PyQt even support static qt?

I did not try this yet, as I was always under the impression going with all dynamic will be less risky (despite not being an officially supported vcpkg triplet)

[Neumann-A]

I think going dynamic is the only correct approach here. Has the error gone away after fixing the symlinks?

[m-kuhn]

Thanks for confirming.

Yes, it has gone away. Macdeployqt is patched in vcpkg, upstream merge pending.
What's still not fixed is nuget, it doesn't understand symlinks, so files from nuget cached packages need to be deduplicated before being usable.