build: Fix libpq for Docker multi-arch builds and Bookworm upgrade

[s373nZ]

This PR adds upon the excellent initial work done by @arowser in #7827 on cross-compiling libpq through the multi-architecture Docker release builds. Exhaustive testing exposed some local Docker build errors which have been addressed in additional commits.

Upon running into some local QEMU segmentation faults while cross compiling, we noticed that qemu-user-static was on a rather old version of 5.2 under Debian Bullseye. Therefore, we also update the Docker release base images to Bookworm, pushing QEMU to 7.2. Bullseye is "old stable" and Bookworm is now "stable", so there is a case to be made for upgrading.

Additionally, Python is upgraded to 3.11 as a corollary. Currently setting pip flags to break system packages, mainly in the interest of time, but this could be addressed to use virtual environments if required.

• Fixed wget.
• Added Postgres build dependencies libicu-dev, bison and flex.
• Removed the existing libpq Debian package installs.
• Defined POSTGRES_CONFIG declaration for each cross compile target.
• Defined PG_CONFIG for all targets.
• Used pg_config to run ldconfig, making compiled libraries available to the system.
• Copied the libpq libraries from the build stage to the final stage using a bind mount RUN command.
• Upgraded Debian base image to Bookworm.
• Upgraded Python to 3.11.
• Clean up Dockerfile formatting and standardize the use of ENV directives when the values are not interdependent.

The full multi-arch build has been tested locally by building on amd64. Still need to figure out a way to get the arm target image onto a Mac to test libpq from inside a container.

Fixes #7649
Relates to and duplicates #7827

Respectfully, these changes have also been submitted in a PR against @arowser's branch here. As there is a bounty posted for #7649, I yield to the CLN core teams determination on who should win the payout or on whether / how to divide it. 🙏

Checklist

Before submitting the PR, ensure the following tasks are completed. If an item is not applicable to your PR, please mark it as checked:

• The changelog has been updated in the relevant commit(s) according to the guidelines.
• Tests have been added or modified to reflect the changes.
• Documentation has been reviewed and updated as needed.
• Related issues have been listed and linked, including any that this PR closes.

[s373nZ]

Sooo... after some more rigorous testing, the first iteration actually broke libpq availability for all targets. Although libpq was installed correctly, the libraries needed to also be added to the linker for configurator to find them to signal yes for Postgres. Also needed to be done in the final build stage. The recent changes:

• Take care to set PG_CONFIG for the compilation correctly for all targets as they drop build artifacts in different places.
• Uses pg_config to run ldconfig on the compiled libpq libraries during the build phase.
• Copies the libpq libraries to another location and uses a bind mount to copy them from the builder image into the final image at /usr/local/pgsql/lib and then runs ldconfig on that directory to make them available.
• Cleans up Dockerfile formatting and standardizes the use of multi-line ENV directives when the values don't depend on each other.

This has been tested by running a cross-platform build on an amd64 machine and then running the target images directly and executing the following:

root@25584da367b1:/# lightningd --wallet=postgres://test:test@localhost:5432/lightning
2024-12-11T13:22:41.967Z INFO    lightningd: v24.11-3-g7dbe807
2024-12-11T13:22:44.641Z UNUSUAL plugin-bookkeeper: topic 'utxo_deposit' is not a known notification topic
2024-12-11T13:22:44.641Z UNUSUAL plugin-bookkeeper: topic 'utxo_spend' is not a known notification topic
2024-12-11T13:22:44.826Z **BROKEN** lightningd: Error calling DB setup: Could not connect to postgres://test:test@localhost:5432/lightning: connection to server at "localhost" (::1), port 5432 failed: Connection refused??Is the server running on that host and accepting TCP/IP connections??connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused??Is the server running on that host and accepting TCP/IP connections??
Error calling DB setup: Could not connect to postgres://test:test@localhost:5432/lightning: connection to server at "localhost" (::1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?

This error indicates libpq is available, without having to configure the whole node. Tested the arm64 and armv7 from the amd64 machine to the same success. Still haven't ported the actual cross platform images to a Mac to test natively.

The learning from testing this has prompted considerations on trying to add static linking support for libpq in the Makefile, which might be tried in a separate prototype.

[s373nZ]

Managed to test the cross-compiled images on an M2 Mac and Raspberry Pi 4, both arm64, with successful results per the above methodology. 🎉

Don't have an armv7 device to test with Docker.

[s373nZ]

Rebased against master.

[s373nZ]

Thanks for the review! So the only TODO is to look into installing the pip packages alternatively, w/o PIP_BREAK_SYSTEM_PACKAGES=1, right? If so, I'll start looking at that and resolve the other comments.

[ShahanaFarooqui]

If so, I'll start looking at that and resolve the other comments.

Yes, that should be good enough.

[s373nZ]

• Manage Python dependencies in virtual environments.
• Put libpq library location definition in /etc/ld.so.conf.d/libpq.conf in the final image to be more "Debian-like".
• Rebased against master.

[ShahanaFarooqui]

@s373nZ I am unable to build images for arm64/v8 and arm32/v7 archs. I initially attempted this using the docker buildx command locally, but all attempts failed. I then created a new branch, docker-test-s373nZ-7827-libpq-multiarch-fixes, adding your updates in commit, to test the changes via GitHub Actions. Both local builds and GitHub Actions tests have consistently failed. Failed Github Actions (Timed Out): 12672684253 & 12664785827

In preliminary testing, I suggest adding --host to the POSTGRES_CONFIG too, like: POSTGRES_CONFIG="--without-readline --host=${target_host} --prefix=${QEMU_LD_PREFIX}". However, the builder also seems to be missing other dependencies, such as libicu-dev.

Is there a specific reason for explicitly running make install in separate directories (e.g., src/include, src/interfaces/libpq) instead of performing a single make install from the root directory? For example:

RUN mkdir postgres && tar xvf postgres.tar.gz -C postgres --strip-components=1
WORKDIR /postgres
RUN ./configure ${POSTGRES_CONFIG}
RUN make && make install

WORKDIR /
RUN rm postgres.tar.gz && rm -rf postgres && ldconfig "$(${PG_CONFIG} --libdir)"

[s373nZ]

• Added fixes for implicit upgrade of poetry to release 2.0.0.
• Rebased against master

I am unable to build images for arm64/v8 and arm32/v7 archs. I initially attempted this using the docker buildx command locally, but all attempts failed.

For local testing, I'm just running docker build --platform=linux/arm64 -t libpq-fixes-arm64 . while changing the --platform flag manually to linux/arm64 and linux/arm/v7 to test those platforms independently. Host machine is amd64 and given the first comment in the Dockerfile:

# This Dockerfile is used by buildx to build ARM64, AMD64, and ARM32 Docker images from an AMD64 host.

this is a legitimate testing scenario, right?

My docker context is pointed to Docker Desktop, which I'm pretty sure is using buildx under the hood.

docker buildx ls
NAME/NODE           DRIVER/ENDPOINT                   STATUS     BUILDKIT   PLATFORMS
default             docker
 \_ default          \_ default                       running    v0.17.3    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (6 more)
desktop-linux*      docker
 \_ desktop-linux    \_ desktop-linux                 running    v0.17.3    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (2 more)

After adding an additional commit to update for Poetry 2.0.0, this method is building fine for me. Can you try again, or share the exact commands you're running and error you see?

I then created a new branch, docker-test-s373nZ-7827-libpq-multiarch-fixes, adding your updates in commit, to test the changes via GitHub Actions. Both local builds and GitHub Actions tests have consistently failed. Failed Github Actions (Timed Out)

Not sure what's going on with this approach. The cross-compiled builds take a long time for me, but not five hours -- maybe one? Given they timeout compiling the Python wheels, maybe it has something to do with using venvs? My suggestion would be to sort out the failures on your local and align between ourselves on testing methodology before debugging the pipeline.

In preliminary testing, I suggest adding --host to the POSTGRES_CONFIG too, like: POSTGRES_CONFIG="--without-readline --host=${target_host} --prefix=${QEMU_LD_PREFIX}". However, the builder also seems to be missing other dependencies, such as libicu-dev.

Is this contributing to the error you're seeing locally? My understanding is that this doesn't need to be done given my testing approach, but maybe I'm misunderstanding.

Is there a specific reason for explicitly running make install in separate directories (e.g., src/include, src/interfaces/libpq) instead of performing a single make install from the root directory?

This was part of @arowser's initial implementation. My understanding is that there are a lot of other components to build in the Postgres source code and this just builds libpq, which is all we need.

[ShahanaFarooqui]

For local testing, I'm just running docker build --platform=linux/arm64 -t libpq-fixes-arm64 . while changing the --platform flag manually to linux/arm64 and linux/arm/v7 to test those platforms independently. Host machine is amd64 and given the first comment in the Dockerfile:

Could you please test using the docker buildx build command instead? For building Core Lightning Docker images, we typically use either the tools/build-release.sh script (which internally uses docker buildx build) or the GitHub workflow .github/workflows/docker-release.yml, where docker/build-push-action@v5 also uses docker buildx build under the hood.

For local testing, I used a command like:

docker buildx build --load --platform linux/arm64 -t shahanafarooqui/lightningd:v25.01-postgress .

With QEMU emulation set up for cross-platform builds.

Optionally, I prefer using the tools/build-release.sh script with the following adjustments:

Change DOCKER_USER="elementsproject" to DOCKER_USER="shahanafarooqui" & modify DOCKER_OPTS from --push --platform linux/amd64,linux/arm64,linux/arm/v7 to --load --platform linux/arm64. Example command:

tools/build-release.sh --force-version=v25.01-postgress --force-unclean --force-mtime=2025-01-10 docker

Not sure what's going on with this approach. The cross-compiled builds take a long time for me, but not five hours -- maybe one? Given they timeout compiling the Python wheels, maybe it has something to do with using venvs? My suggestion would be to sort out the failures on your local and align between ourselves on testing methodology before debugging the pipeline.

In some cases, the GitHub workflow may not fail or exit as expected. The purpose of testing these changes on GitHub Actions was to confirm that the failures observed locally also occur in the workflow. Once the issue is resolved locally, the GitHub Actions workflow should also pass successfully.

Is this contributing to the error you're seeing locally? My understanding is that this doesn't need to be done given my testing approach, but maybe I'm misunderstanding.

I had to add --host while testing locally. After doing so, I encountered a missing libicu-dev error, and then ran into another issue after installing libicu-dev. At that point, I had to pause the testing due to other priorities. It was just a suggestion—if the image is building successfully without it, we can leave it out.

This was part of @arowser's initial implementation. My understanding is that there are a lot of other components to build in the Postgres source code and this just builds libpq, which is all we need.

Since we are only copying libpq from the builder, the root compilation will not increase the image size. Therefore, if building the entire Postgres does not significantly impact the build time, performing the root-level make would be preferable for simplicity and future-proofing.

[s373nZ]

• Small fix for Poetry 2.0.0 update due to merge conflict resolution.

@ShahanaFarooqui I just tested the build again locally using docker buildx build --load --platform linux/arm64 -t libpq-fixes-arm64 . on both my desktop-linux and default contexts, and it worked successfully. Also, on the docker buildx build documentation, docker build is listed as an alias to this command, FYI.

Before we debug too much further, can you share which version of Docker you're using? Mine is:

> docker version
Client: Docker Engine - Community
 Version:           27.4.1
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        b9d17ea
 Built:             Tue Dec 17 15:45:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       c710b88
  Built:            Tue Dec 17 15:45:46 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[ShahanaFarooqui]

Hi @s373nZ,

My local docker version matches yours, but I am unsure how you were able to build the image with the current Dockerfile. I retested it using docker buildx, docker build, and in the GitHub workflow, but all attempts continued to fail.

Issues Encountered and Resolutions:

1: Initially, the Python installation failed with the error:

pyproject.toml changed significantly since poetry.lock was last generated. Run `poetry lock` to fix the lock file.

To address this, I updated poetry.lock for Core Lightning and added poetry lock before all poetry export commands to ensure consistency between pyproject.toml and the lock file.

2: Then I encountered the error externally-managed-environment along with a suggestion to pass --break-system-packages. Despite using a virtual environment (/root/.venvs/cln), the issue persisted because each RUN instruction in a Dockerfile is executed in a new shell. As a result, the virtual environment activation did not persist across RUN commands. So I explicitly activated the venv in every RUN instruction.

3: Finally I was able to successfully build the image but both python plugins (clnrest & wss-proxy) were not working due to missing libraries. I discovered that the libraries were being copied from the global Python location instead of the virtual environment. To fix this, I updated the Dockerfile to copy libraries from the virtual environment (/root/.venvs/cln/lib/python3.11/site-packages) in the final stage.

Current Status:

• The images are now build successfully on my local setup and work correctly with both SQLite and Postgres.
• The GitHub workflow is still running: Workflow Link.
• The relevant updates can be found in the last four commits of the branch: docker-test-s373nZ-7827-libpq-multiarch-fixes.

[s373nZ]

Hi @ShahanaFarooqui, this work has been really sensitive to develop and even more time consuming to test due to the long build process, cross-compilation variants and their performance multiplier. I thank you for your help and patience with the review and testing -- and the ideas you're contributing toward fixing the discrepancies 🙏

1: Initially, the Python installation failed with the error: pyproject.toml changed significantly since poetry.lock was last generated. Run poetry lock to fix the lock file.

I managed to reproduce this by running my testing build locally with the --no-cache flag to ensure a completely clean process:

docker buildx build --load --no-cache --platform linux/arm64 -t libpq-fixes-arm64:latest .

A good reminder for me to always test a clean build before submission to avoid being tricked by the layer caching. After cherry-picking b3c97 with your updates to the poetry.lock file, I still encountered the error. Upon further inspection, it seems the new lock file was generated with Poetry 1.8.2 and still needs an update? (# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.) Perhaps it's working inconsistently due to layer caching? After regenerating it using Poetry 2.0.1, the error still occurred, likely because we were modifying the lock file directly in Docker and Poetry 2 doesn't like that. I've made a small change to add poetry lock to that command chain in the base-builder image, which I think is a more robust fix and seems to work. Better yet would be to restructure the Poetry dependencies into groups and use the --without flag of the export, but that is outside the scope of this PR. After encountering the same error for the clnrest and wss-proxy poetry export commands, I regenerated their lock files as well. This is probably a larger issue for the repo after the Poetry 2 upgrade.

Then I encountered the error externally-managed-environment along with a suggestion to pass --break-system-packages. Despite using a virtual environment (/root/.venvs/cln), the issue persisted because each RUN instruction in a Dockerfile is executed in a new shell. As a result, the virtual environment activation did not persist across RUN commands. So I explicitly activated the venv in every RUN instruction.

After making the above change and running with --no-cache, I'm not experiencing this particular issue locally. For context, the strategy for making the virtual environment available "globally" in the Docker build context is to add it's bin directory to the $PATH where necessary. I've done this in the images where it was necessary, but we should be able to rely on this method to avoid the --break-system-packages issue and avoid the need to activate it explicitly on each RUN command. Would you mind trying kicking off a new local build with --no-cache against the updated PR branch to be sure it's still an issue?

both python plugins (clnrest & wss-proxy) were not working

Taking a look at your commit, it makes complete sense to me that copying the packages from the venv is necessary. Could you please share how you tested clnrest and wss-proxy against the Docker image to discover the error, for my own knowledge and so I can try to reproduce it locally and know for the future?

The GitHub workflow is still running: Workflow Link.

It looks like it completed successfully 🎉 this is encouraging!

Lastly, I have a question about the errors you reported earlier:

In preliminary testing, I suggest adding --host to the POSTGRES_CONFIG too, like: POSTGRES_CONFIG="--without-readline --host=${target_host} --prefix=${QEMU_LD_PREFIX}". However, the builder also seems to be missing other dependencies, such as libicu-dev.

Are these prior issues with the cross-compilation still present, or did they get resolved? I sort of remember encountering challenges like this when I first began the work, but I think a Docker upgrade helped to fix it. Did the same work for you, or was it something else?

The changes referenced in this comment have been edited into their corresponding commits in this branch. Please let me know if the venvs still break for you, or you encounter other errors. I think we're close! 🤞

P.S. Do you think setting the runs-on property here to a higher version of Ubuntu LTS, or even a latest Docker image version might speed the CI up and avoid Docker versioning issues? Could be worth a try.

[ShahanaFarooqui]

I have made a few changes to your PR related to this update (hope it is okay). Here is a summary:

• The poetry.lock version used is 1.8.2.
• Updated the checkout action to ensure the branch triggering the workflow is checked out."

Everything is working as expected in my local build. I am just running the final test on GitHub Actions here. Once it passes, the PR should be ready to merge.

This work has been particularly challenging to develop and even more time-consuming to test, given the long build process, cross-compilation variants, and their performance impact. I sincerely thank you for your help and patience throughout the review and testing, as well as for the valuable ideas you've contributed to resolving the discrepancies.

It is a team effort, and I appreciate your contributions too. While testing Docker updates can be time-intensive, adding postgres support to the image will be well worth the effort :).

It seems the new lock file was generated with Poetry 1.8.2 and still needs an update?

Let us keep the poetry.lock files at v1.8.2 for now, as they are used for standalone setups also. Poetry v2 is just 10 days old, and we can wait until it is more mature and stable before upgrading. If using the same Poetry version in the Docker images aligns better with the project, we can lock Poetry to v1.8.2 in the Dockerfile as well.

After making the above change and running with --no-cache, I'm not experiencing this particular issue locally. For context, the strategy for making the virtual environment available "globally" in the Docker build context is to add its bin directory to the $PATH where necessary.

That makes sense. I might have tested it in an inconsistent environment, as I recall accidentally changing the $PATH from /root/.venvs/cln/bin to /root/.venvs/cln/lib.

I've made a small change to add the Poetry lock to that command chain in the base-builder image, which I think is a more robust fix and seems to work.

Great.

Taking a look at your commit, it makes complete sense to me that copying the packages from the venv is necessary. Could you please share how you tested clnrest and wss-proxy against the Docker image to discover the error? This would help me reproduce it locally and understand it better.

I usually test Docker images as described in the Core Lightning documentation. However, I think the missing piece in your setup is configuring these plugins in your CLN config using clnrest-port and wss-bind-addr params. For example, adding the following configuration:

clnrest-port=3010
wss-bind-addr=127.0.0.1:5020

should produce the following logs in CLN:

plugin-clnrest: REST server running at https://127.0.0.1:3010
...
plugin-wss-proxy: Websocket Secure Server Started

Are the prior issues with cross-compilation still present, or have they been resolved? I remember encountering similar challenges early on, but I believe a Docker upgrade resolved them. Did you take a similar approach, or was it something else?

No, those issues no longer persist, and I did not upgrade Docker either. However, I guess you can still reproduce them by building the image up to the builder stage and then running the ./configure ${POSTGRES_CONFIG} command.

P.S. Do you think setting the runs-on property to a higher version of Ubuntu LTS or using the latest Docker image might speed up CI and mitigate Docker versioning issues? Could be worth exploring.

Currently project-wide GitHub workflows run on ubuntu-22.04 to maintain consistency. We can revisit and consider upgrading to a newer version in the next 4-6 months if needed.

This was part of @arowser's initial implementation. My understanding is that there are a lot of other components to build in the Postgres source code and this just builds libpq, which is all we need.

I also attempted to modify the make install command for Postgres to run only from the root directory, but it encountered multiple errors, such as:

337.5 /usr/lib/gcc-cross/arm-linux-gnueabihf/12/../../../../arm-linux-gnueabihf/bin/ld: cannot find -licui18n: No such file or directory
337.5 /usr/lib/gcc-cross/arm-linux-gnueabihf/12/../../../../arm-linux-gnueabihf/bin/ld: cannot find -licuuc: No such file or directory
337.5 /usr/lib/gcc-cross/arm-linux-gnueabihf/12/../../../../arm-linux-gnueabihf/bin/ld: cannot find -licudata: No such file or directory

Given these issues, we can leave it as is for now and revisit it later when we have more time to focus on making it cleaner.

[ShahanaFarooqui]

Tested both Docker images, generated locally and by the workflow, using SQLite and PostgreSQL across all three architectures. Everything is working as expected.

@s373nZ The PR looks ready for merge to me. Let me know if you are okay with it or if you would like any changes.

ACK 3286bab.

[s373nZ]

The poetry.lock version used is 1.8.2.

I originally changed this due to some build errors indicating the .lock files were out of date. Also noticed the CHANGELOG for the Poetry 2.0 release listed some changes to the lock file format, so it seemed like the errors could be legitimate. However, it's building for me locally as well, after you reverted the changes to 1.8.2, and it passed CI, so looks fine now. Updating all the Poetry configs for the whole repo should probably be done in tandem, too. Thanks for the edit!

I also attempted to modify the make install command for Postgres to run only from the root directory, but it encountered multiple errors

I agree with the original approach of building just libpq because:

• We would have to add additional package dependencies to support build components we won't need.
• Build time would be even longer for all platforms and possibly even more complicated, especially with the touchy cross-compiling.

There might be a way to prettify the command, though. One thought is to download the Postgres tarball and start reading through the various Makefiles and doing some experiments to see if there is a magic combination of targets for the libpq artifacts from the root of the archive. I won't have time to look into this an re-test the whole build for a bit, though.

The PR looks ready for merge to me. Let me know if you are okay with it or if you would like any changes.

🎉 Local build for arm64 passed for me today, CI/CD looks good, and you've already ACK'd it. Everything LGTM also! Thanks again!