Better instructions / support for offline building

[nh2]

Description

I'm trying to package pypdfium2 for NixOS / nixpkgs (the Linux distribution with currently by far the most packages in).

Like many other distros, the build system has the requirement that builds are hermetic, without network access.

For the beginning I'm trying to use the pre-built pdfium binaries provided.

But the build system (paths down from setup.py) still tries to connect to the network in various places:

• downloading pdfium binaries
• fetching tags with git
• urlretrieve() in build_pdfium_bindings(); it has a headers_dir=None default argument, but no path in the code allows to set it to something not-None

I noticed the prepared! concept from the README, but not sure that's what's needed, as that seems to skip ctypesgen.

Would it be possible to provide setup.py with pre-downloaded files of whatever is needed to build a single pypdfium2 version?

For example, from what I've read so far, for version 6097 it seems it would be sufficient to provide:

• pdfium binary: https://github.com/bblanchon/pdfium-binaries/releases/download/chromium%2F6097/pdfium-linux-x64.tgz
• headers: https://pdfium.googlesource.com/pdfium/+archive/refs/heads/chromium/6097/public.tar.gz

If one could just provide those to setup.py with flags or environment variables, it would solve the hermetic distro packaging issue.

[nh2]

Another thing that's not clear:

https://github.com/pypdfium2-team/pypdfium2/tree/a397ff7efa71c2fbb2200548bd9be7f30de84d7c#incompatibility-with-cpython-376-and-381

pypdfium2 built with mainstream ctypesgen cannot be used with releases 3.7.6 and 3.8.1 of the CPython interpreter due to a python/cpython#16799 (comment) that ctypesgen/ctypesgen#77 ctypesgen-created string handling code.

Since version 4, pypdfium2 is built with a patched fork of ctypesgen

The linked issues are fixed, and we're way past CPython 3.8.1, yet there's still

    assert getattr(ctypesgen, "PYPDFIUM2_SPECIFIC", False)

in the code -- is this intended, or should building with normal ctypesgen be supported again?

If no, the fork (https://github.com/pypdfium2-team/ctypesgen) should be linked in the README.

[nh2]

Another point of confusion:

The README says

update_pdfium.py downloads binaries and generates the bindings.

and its parse_args() says

    parser = argparse.ArgumentParser(
        description = "Download pre-built PDFium packages and generate bindings.",
    )

But its main() seems to only download() and extract(), not generate any bindings.

[mara004]

Thanks for your inquiry. Since a lot of questions are raised in this thread, it may take some time to answer it all.

First, note that there were some heavy changes to the setup infrastructure recently with the addition of conda packaging (PR #268), and the changes yet need some consolidation.
Particularly, I still intend to add means of plugging in custom pdfium headers (and setting custom runtime libdirs) with the system target.
Also, what you mention in #272 (comment) (the incorrect desc) is an oversight from the change; bindings generation was externalized. I'll fix that soon.

I understand users may want an offline installation, and we have the wheel packages for that, but I'm afraid offline setup.py wasn't really an intended use case here.
However, I believe it should already be doable with the prepared! prefix indeed (see new comment below).
(We can also consider adding a new setup target to take a caller-provided binary+headers from a custom location if that helps.)

Concerning ctypesgen, the upstream issue is closed because CPython reverted the change, but ctypesgen's problematic bloated string code is still there. Upstream development is pretty stuck, and there are a lot more improvements in the fork, like the preventing of invalid struct fields (this can be very confusing, see #221 (reply in thread)) and a leaner and more controllable library loader. (See e.g. the commit message of pypdfium2-team/ctypesgen@569dc4b for some of the technical debt in the old library loader.)
We don't currently have a mind of adding back a compatibility layer for upstream ctypesgen.
Note that the fork is in the Readme already - the link points to the fork, and the dependency section mentions ctypesgen (pypdfium2-team fork).

[mara004]

What you could do now is to just copy in the offline-provided bindings and binary into src/pypdfium2_raw/.
For convenience, you can use the reference bindings at autorelease/bindings.py to begin with, so you don't have to worry about ctypesgen and headers yet.
The version file will have to be written manually (take a look at a generated example for the format).
Then prepared!{platform_name}, e.g. prepared!linux_x64 should hopefully work.

(Update: No need to do this manually, I figured you can just use the emplace tooling, as long as we separate data files generation and the actual install.)

[mara004]

I also took a quick look at your patch, but this is not an acceptable solution. If changes, then we should rather look at the higher-level setup.py/emplace.py/packaging_base.py instead of messing up update_pdfium.py.

[mara004]

Actually, I believe you could just call ./run emplace {platform} in the "online allowed" stage of the package building to create the files, and then use prepared!{platform} in the offline-only stage.

You have to get the data files at some point, and instead of doing it manually you can just use the emplace tooling for that, while prepared! allows you to be hermetic on the actual install time.

[nh2]

@mara004 You looked a bit too early, while I was still iterating on it. Please have a look now: PR #273