Sources for windows AIO

[jmichault]

Currently, the gramps repository lacks sources to create the Windows AIO installer.

This pull request adds:

• all necessary files in the aio directory, including a script that automates all steps.
• a workflow to verify that AIO can be built.

[Nick-Hall]

Excellent idea!

We should get our Windows maintainer @prculley to check this.

Perhaps we could even automatically generate the AIO as part of the release process?

[PQYPLZXHGF]

Great idea.

@jmichault Any chance the share.tgz could be created on the fly, looks like it contains existing icons etc.

Next would be to automate the Apple Build

[emyoulation]

Maybe a good test of installer generators would be a scheduled broad 5.2 Beta test? (Only for GitHub distribution, not for inclusion on the Wiki download page or wiki rollover.)

[prculley]

I've looked this over and it appears to work. I had to modify it slightly to utilize my directory layout, but that should not affect the functionality. Might be a good idea to document the expected and resulting dir layout.

I was initially a bit concerned that the various patches I had applied to the older MINGW files were not getting used. But as near as I can tell the things that I had patched seem to have been fixed over that last several years.

I generated an experimental AIO for the 5.1.6 Gramps. To do this I had to fix the gramps/setup.py slightly to deal with bdist_wheel which was fixed in master, but not in maintenance/gramps51. It appears to install correctly and runs the limited tests I made correctly. I will probably spend some more time with this in the near future.

If/when this PR is accepted, we will want to create a newer updated wiki article to help lead folks in running it.

[emyoulation]

Could there be a item moved in the Windows install? The Examples would be FAR better located in the Gramps User Directory. This would also allow the Examples to be updated via the Addon Manager

And could the installer also set the GRAMPSHOME and GRAMPSDIR

https://www.gramps-project.org/wiki/index.php/Gramps_5.1_Wiki_Manual_-_Command_Line#GRAMPSHOME
https://www.gramps-project.org/wiki/index.php/Gramps_5.1_Wiki_Manual_-_Command_Line#GRAMPSDIR

[Nick-Hall]

Don't set GRAMPSHOME or GRAMPSDIR. We can change the defaults on Windows if necessary, but they should be fine as they are.

[Nick-Hall]

@jmichault The Windows build is failing.

I have merged the aio directory, but not the workflow for now.

[Nick-Hall]

Rebased.

[prculley]

Regarding build failure;
I looked at this a little, and it appears that something has changed over at the MSYS2 package repositoryies. There used to be https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-python3-... type of files in the list, now there are not. I note that the 'python3' packages have been something called 'Virtual packages' for some time now. I frankly don't know exactly what this means, but I suspect some connection.

Perhaps someone who knows more about package management can chime in.

[jralls]

@prculley The latest python3 is at https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-python-3.11.4-4-any.pkg.tar.zst and there are packages going back to 3.9.7. Dunno what you mean by virtual: 23M compressed seems like it would be pretty complete.

That has nothing to do with the build errors:

error: target not found: mingw-w64-x86_64-python3-bsddb3
error: target not found: mingw-w64-x86_64-python-bsddb3

That breaks the pacman -S command that installs our environment. All of the other errors are due to that failure.

What I don't get is why they fail: https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-python-bsddb3-6.2.9-4-any.pkg.tar.zst exists, but on my VM:

John Ralls@Win32VM MSYS ~
$ pacman -Qi mingw-w64-x86_64-python3-bsddb3
Name            : mingw-w64-x86_64-python-bsddb3
Version         : 6.2.9-4
Description     : Python bindings for Oracle Berkeley DB (mingw-w64)
Architecture    : any
URL             : https://www.jcea.es/programacion/pybsddb.htm
Licenses        : BSD
Groups          : None
Provides        : mingw-w64-x86_64-python3-bsddb3
Depends On      : mingw-w64-x86_64-python  mingw-w64-x86_64-db
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : mingw-w64-x86_64-python3-bsddb3
Replaces        : mingw-w64-x86_64-python3-bsddb3
Installed Size  : 1077.77 KiB
Packager        : CI (msys2/msys2-autobuild/ea46306e/2483942923)
Build Date      : Sun Jun 12 08:58:43 2022
Install Date    : Sun Jul 16 10:52:50 2023
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

John Ralls@Win32VM MSYS ~
$ pacman -Si mingw-w64-x86_64-python-bsddb3
error: package 'mingw-w64-x86_64-python-bsddb3' was not found

John Ralls@Win32VM MSYS ~
$ pacman -Sl python-bsddb3
error: repository "python-bsddb3" was not found.

So it worked two weeks ago, but doesn't now. I suspect that's due to my having done pacman -Syuu last Sunday that updated Python3 to 3.10.12. The bsddb3 PyPi page has had a deprecation warning for some time claiming that it doesn't support Python > 3.9. That's not true, it's in the macOS 5.1.6 and 5.2.0b1 packages with Python3.11.4 and it manages to load bsddb databases without complaint, but maybe somebody at MSYS2 took it seriously and disabled it. It has been replaced by berkeleydb. That's not packaged by MSYS2 but we could install either it or bsddb3 with PyPi. Using berkelydb instead requires some mods to gramps/plugins/db/bsddb. I had a patch for 5.1 but that's obsolete now.

[Nick-Hall]

Virtual packages in pacman are just placeholders. A package could require a "python3" package, but the dependency could actually be fulfilled by a "python" package.

I expect that they have finally completed a transition where "python" packages have moved from "python2" to "python3".
We will probably have to remove the "3" from other packages as well.

Also, we may not need all the dependencies in v5.2 since we can now attempt to install addon prerequisites using PyPI. I have been unable to test this for Windows though.

[emyoulation]

While looking at the Windows with MSYS2 wiki page, the thought occurred that maybe inviting more people (like everyone who edited that wiki page) to this conversation might help?

Changing the participation impetus from 'pull' to 'push' might be necessary.

https://www.gramps-project.org/wiki/index.php/Gramps_for_Windows_with_MSYS2

[Nick-Hall]

I've updated the package names and made a few changes to get it to build. There are still a few issues that need further investigation:

• bsddb3 is not installing properly. I had to comment out a bsddb3 line.
• I removed a gtkspell dll. Is there a gspell equivalent?
• There is a version number problem with pre-releases. I just did a quick edit to get things working.

[jralls]

Is there a gspell equivalent?

There is

[jralls]

bsddb3 is not installing properly. I had to comment out a bsddb3 line.

That's OK for CI but if you or @prculley are using aio/build.sh to build the AIO you need to get bsddb3 from PyPI instead (just like asyncio) and uncomment the line or users won't be able to load their old BerkeleyDB databases.

[Nick-Hall]

@jralls I loaded the gspell package, but I didn't know what entry to put in the missing dll list.

I'll give PyPI a try for bsddb3.

[hgohel]

I loaded the gspell package, but I didn't know what entry to put in the missing dll list.

@Nick-Hall Jumping in the conversation here: Based on contents of the gspell pacakge being installed, the DLL should be libgspell-1-2.dll

[Nick-Hall]

I can't get bsddb3 installed using pip. The setup.py is looking for the Berkeley DB build files to be in a directory called db below the working directory. Unfortunately, they are not found when I put them there. I get the error:

File "D:/a/_temp/msys64/tmp/pip-install-bgti2odo/bsddb3_ee5d4a5ff6c64f0d906c273002eaf9ad/setup3.py", line 385, in <module>
    with open(os.path.join(incdir, 'db.h'), 'r') as f :
FileNotFoundError: [Errno 2] No such file or directory: 'db/include/db.h'

An ls confirms that the db/include/db.h file exists prior to the pip install.

[prculley]

I just looked into this a bit, and I found that bsddb3 doesn't work for Python v10. See https://www.jcea.es/programacion/pybsddb.htm

So for upgrade purposes we may have to switch to the berkeleydb. Quoting the author "Take note that upgrading to berkeleydb is easy but not transparent. Notably, keys and values are bytes in berkeleydb lib, while in ‘bsddb3’ they are strings."
I suspect that this won't matter too much as the conversions seem to be in one module...

[Nick-Hall]

I'm using python-bsddb 6.2.9 with python 3.11 without any problems.

[Nick-Hall]

So for upgrade purposes we may have to switch to the berkeleydb.

I get the same error with berkeleydb as I do with bsddb3.

[jralls]

You need os.environ["BERKELEYDB_DIR"] = "/mingw64" before pip install foo for either of them.

[Nick-Hall]

@jralls The environment variables only seem the be read for a posix install. For Windows, it expects everything in a fixed directory.

[jralls]

That's annoying. It doesn't look like there's a good solution that will make it work with pip.

How about

pacman -U https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-python-bsddb3-6.2.9-4-any.pkg.tar.zst

bypassing the removal of pybsddb3 from the package management system? Maybe for long term safety we should copy that tarball to sourceforge and install it from there instead of repo.msys2.org.

[Nick-Hall]

With the pacman -U it appears to install OK, but I get:

ImportError: No module named 'bsddb3'

[jralls]

Implying that it didn't end up on sys.path. Did it perhaps install in the wrong python version (i.e. /mingw64/lib/python3.10 instead of /mingw64/lib/python3.11)?

[Nick-Hall]

The build now works and the GrampsAIO is available as an artifact.

@jralls It was installing to 3.10 rather than 3.11, so I just copied over the files and it seems to work.

I'm not using any icons from the share.tgz file, because I can find most of them elsewhere. The difference is that I'm using the standard Adwaita icon theme installation which has a reduced set of icon sizes.

The only thing that I am still extracting from share.tgz are the dictionaries. These are quite large. Do we really want to include all of them in the AIO?

It would be useful if someone with a copy of Windows could install the AIO to check that it actually works.

[emyoulation]

error returned: "Fatal error in launcher: U"

[Nick-Hall]

It looks like pip wasn't installed properly.

[emyoulation]

Net search says that is a partial message. Should be "Fatal error in launcher: Unable to create process using pip"

[Nick-Hall]

Try python -m pip install svgwrite from the command line.

[emyoulation]

hmmm. python isn't in the path either. Searching again. But it will probably need a full path to pip

[prculley]

Since Windows is pretty particular about modifying anything in the "Program Files" without admin rights, I tried running Gramps with admin rights. I now get a different "Module installation failed"
Error encountered in argument parsing: Error parsing the arguments Details: option -m not recognized Error parsing the arguments: [ -m pip install life_line_chart ] Type gramps --help for an overview of commands, or read the manual pages.

We may need to rethink the whole idea of using pip to install, and go back to making the installation in the addons directory (where admin rights are not needed). This is what was done in the 5.1.x versions of the lifeline charts.

[Nick-Hall]

@prculley Did you remember to change "sys.executable" to "python.exe"?

[Nick-Hall]

We may need to rethink the whole idea of using pip to install, and go back to making the installation in the addons directory

That maybe OK for Windows, but we probably wouldn't want to do that for Mac or Linux.

[wroldwiedbwe]

@Nick-Hall Looks like the aio is using the embedded python for Windows? If so the following might be relevant:

• The embeddable package which states: Third-party packages should be installed by the application installer alongside the embedded distribution. Using pip to manage dependencies as for a regular Python installation is not supported with this distribution, though with some care it may be possible to include and use pip for automatic updates. In general, third-party packages should be treated as part of the application (“vendoring”) so that the developer can ensure compatibility with newer versions before providing updates to users.

Some stackoverflow answers seems to indicate various ways to include pip but are not definitive!

[prculley]

I see that we are actually trying to run "python -m pip install lifeline_chart" which cannot work on Windows since we don't have python.exe present.

[Nick-Hall]

OK. So we do have to take a different approach on Windows.

I'm actually happy to disable the auto-install functionality in Linux. It is easy to click the "Requires" button and install any requirements manually.

I'm not sure what we want to do with the Mac bundle.

[emyoulation]

I just installed Python 3.8.1 and re-tried python -m pip install svgwrite

it installed. but warned about using pip 19.2.3 when 23.2.1 is available

[emyoulation]

However, Gramps does not recognize svgwrite in the installed location under Python38

[prculley]

I tried patching the code to run pip.exe instead of python.exe. The error I get is Fatal error in launcher: U which I don't have a clue about. Doesn't change if Gramps is run in admin mode.

[emyoulation]

I tried patching the code to run pip.exe instead of python.exe. The error I get is Fatal error in launcher: U which I don't have a clue about. Doesn't change if Gramps is run in admin mode.

it is a partial message: https://bobbyhadz.com/blog/python-fatal-error-in-launcher-unable-to-create-process-using-pip-exe

"Fatal error in launcher: U"...nable to create process using pip

The Python "Fatal error in launcher: Unable to create process using pip" occurs for multiple reasons:

Not having the path to pip in your user's PATH environment variable.
Not having pip installed on your machine.
Having a corrupted Python installation.
Having an outdated version of pip.

[prculley]

It looks like pip.exe is trying to open python.exe and failing... (found with Process monitor).

[jralls]

I'm not sure what we want to do with the Mac bundle.

IIUC you're trying to make it feasible for addons contributors to have any random dependency and still allow it to work with an AIO bundle. Neither provides the python executable directly, so shelling out to pip won't work. Importing pip--after all, it's a python module--and calling its API from the add-on manager might work.

Adding stuff to C:\Program Files outside of an installer is probably not a great idea, and adding stuff to a macOS app bundle voids its signature and may cause it to not launch. The best solution to that is to tell pip to do a user install next to the plugins and to add that location to sys.path so that libpython can find it.

If that doesn't work then you'll have to fall back to the pre-5.2 approach of providing a reasonable set of dependencies in the AIO and making it clear to add-on contributors that if their addons have other dependencies it will mean that they're available only to Linux users and the very few users on Windows and macOS capable of and willing to manage a from-scratch build of Gramps.

[ennoborg]

Addons Manager was not available without a tree loaded

What do you mean by "not available"? Is the icon visible? If it is, what happens when you click on it?

I meant that it's not in the drop down menu, when no tree is loaded. That menu only has one entry, named something like Preferences (Voorkeuren in Dutch).

The icon is visible, and works OK.

[ennoborg]

Try python -m pip install svgwrite from the command line.

When I do that, it works for the Python version that I installed in Windows:

C:\Users\enno>python -m pip install svgwrite
Collecting svgwrite
  Downloading svgwrite-1.4.3-py3-none-any.whl (67 kB)
     ---------------------------------------- 67.1/67.1 kB 729.3 kB/s eta 0:00:00
Installing collected packages: svgwrite
Successfully installed svgwrite-1.4.3

C:\Users\enno>python -m pip install life_line_chart
Collecting life_line_chart
  Downloading life_line_chart-1.7.7-py3-none-any.whl (107 kB)
     ---------------------------------------- 107.2/107.2 kB 1.2 MB/s eta 0:00:00
Requirement already satisfied: svgwrite in c:\users\enno\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from life_line_chart) (1.4.3)
Installing collected packages: life_line_chart
Successfully installed life_line_chart-1.7.7

Gramps doesn't see it, because it has no idea about this package location, and only cares about the one in its own installation folder.

PullRequest# 1569 : Make the Addon Manager available without a tree loaded
#1569

[ennoborg]

We may need to rethink the whole idea of using pip to install, and go back to making the installation in the addons directory (where admin rights are not needed). This is what was done in the 5.1.x versions of the lifeline charts.

I ran some tests in W10, installing svgwrite in Python for Windows, in Python for MSYS, and by letting the 5.1.6 LifeLines plug-in have its way, and saw that, apart from Python for MSYS, they all installed svgwrite inside AppData, meaning in user space, which is the proper way to do this, IMHO. Even when I run pip in Linux, without sudo, it does that, by putting it inside ~/.local/lib/python-3.9/site-packages.

To me, this suggests that we should do the same in Gramps 5.2, meaning that we create a folder named GrampsAIO64-5.2.0 or something similar inside AppData\Local\Packages, and let Gramps pip store all packages there.

[Nick-Hall]

The best solution to that is to tell pip to do a user install next to the plugins and to add that location to sys.path so that libpython can find it.

I defined LIB_PATH as USER_PLUGINS/lib and appended it to sys.path. Then I ran:

pip.main(["install", "--target", LIB_PATH, package])

It installs the modules, but doesn't recognise them until I restart Gramps.

[prculley]

Using pip in the AIO seems like a non-starter. pip requires python.exe, which doesn't exist in the embedded python created by cx_freeze. I looked into importing pip, the developers say you should NOT do this, they even changed some things recently to break the import hack some folks were doing. Attempts to install Python and pip to the system from some other source, doesn't work for several reasons, like not caring unless it is in its own installation folder, and having to be the exact pip/python as the version used to create the AIO .

[prculley]

https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program

[prculley]

I'm thinking that if we want to try to support prerequisites installation, we are going to have to do it like the 5.1.x lifelinecharts gpr code did it. I think it would make some sense to provide a couple of utility calls in Gramps, perhaps with appropriately adjusted code from the lifelinecharts, and storing the loaded modules in the correct user space (instead of with each addon). Not as good a pip, as the addon would be required to deal with extended prerequisites.

[Nick-Hall]

Here is a Windows AIO for testing. As Paul points out it may not work.

It is really doing the same thing as the old Life line charts code. The module is downloaded to a directory under USER_PLUGINS/lib which I have appended to sys.path.

We could replace the pip code with something similar to download a PyPI package and extract it into a target directory.

[ennoborg]

And Paul is right ...

147454: ERROR: grampsapp.py: line 188: Unhandled exception
Traceback (most recent call last):
  File "C:\Program Files\GrampsAIO64-5.2.0\gramps\gui\plug\_windows.py", line 287, in __on_install_clicked
    result = pip.main(["install", "--target", LIB_PATH, package])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "AIO/pip/__init__.py", line 11, in main
  File "AIO/pip/_internal/utils/entrypoints.py", line 7, in <module>
  File "AIO/pip/_internal/cli/main.py", line 10, in <module>
  File "AIO/pip/_internal/cli/autocompletion.py", line 10, in <module>
  File "AIO/pip/_internal/cli/main_parser.py", line 9, in <module>
  File "AIO/pip/_internal/build_env.py", line 19, in <module>
  File "AIO/pip/_internal/cli/spinners.py", line 9, in <module>
  File "AIO/pip/_internal/utils/logging.py", line 29, in <module>
  File "AIO/pip/_internal/utils/misc.py", line 40, in <module>
  File "AIO/pip/_internal/exceptions.py", line 18, in <module>
  File "AIO/pip/_vendor/requests/__init__.py", line 149, in <module>
  File "AIO/pip/_vendor/requests/utils.py", line 63, in <module>
  File "AIO/pip/_vendor/certifi/core.py", line 37, in where
  File "AIO/contextlib.py", line 137, in __enter__
  File "AIO/importlib/resources/_common.py", line 80, in _tempfile
  File "AIO/zipfile.py", line 2452, in read_bytes
  File "AIO/zipfile.py", line 2413, in open
FileNotFoundError: C:\Program Files\GrampsAIO64-5.2.0\lib\library.zip/pip/_vendor/certifi/cacert.pem

I checked the ZIP file, to be sure, and the certificate wasn't there.

[emyoulation]

But on the other hand, it properly posted that error to the Error Report dialog instead of just posting to the console.

[Nick-Hall]

The link that Paul gave suggests using "python -m pip" which was my first approach. I was aware that my last attempt was not recommended.

The pip.main(["install", "--target", LIB_PATH, package]) command just downloads a package and extracts it into a target directory.

If someone can write a function download_pypi(module, target_dir) it should work. It looks like that's all that the Life Line Chart code was doing. I'm not going to do it myself, because I have no way of testing it on Windows.