Compositing PNG with tile=true where height > 512px, "pngload_buffer: out of order read at line"

[jortuck]

Question about an existing feature

What are you trying to achieve?

I am currently trying to overlay text onto an image. I am doing this by generating the text using canvas, converting it to a png, then placing it on the source image. When running my code on my Mac outside of docker, everything works as intended, but as soon as I build the project into a docker container, it gives me this error when trying to overlay the text. The weird thing is, it works when overlaying text the first time, but it errors on the second time. I am not sure why. I would assume it might have to do with the image being corrupted, but if it was corrupted it should fail to work in both environments.

triggerUncaughtException(err, true /* fromPromise */);
[Error: pngload_buffer: out of order read at line 673]

For additional context, here is my docker build file:

FROM node:20-bookworm
RUN apt-get update
RUN apt-get install -y build-essential libcairo2-dev libpango1.0-dev libjpeg-dev libgif-dev librsvg2-dev libvips libvips-tools
COPY . .

RUN yarn install

ENTRYPOINT ["yarn","run","start"]

When you searched for similar issues, what did you find that might be related?

#3600 Had a similar error to mine. The proposed solution there was to set sequentialRead: false when creating a new sharp instance, however trying this gave me a segment fault error on both my Mac environment and docker environment.

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question

It's kind of hard to recreate this issue in a standalone example since it's very specific, but here's the link to the code where I am performing the image operations.
https://github.com/DriedSponge/chica/blob/main/src/commands/image/Caption.ts#L24-L77

Here is a quick summary of what is happening

               const captionBackdrop = sharp(buffer, {animated:true})

		const textCanvas: Canvas = new Canvas(width, height);

               // Perform operations on the canvas to create a text overlay.
		

		// Overlay the text
		captionBackdrop.composite([
			{
				input: await textCanvas.toBuffer(),
				tile: true,
				gravity: "northwest"
			}
		]);

Please keep in mind that this error only occurs inside of my docker container, it works perfectly fine on the outside. Also that is the entire error. It did not give me a full stack trace or specific file/line number where the error was occurring.

Please let me know if you need more context, I am happy to provide it. Also even if you can't provide a direct solution, I'm happy to hear any ideas as to why you think this may be happening.

The images were too big to upload to github so here's an imgur link https://imgur.com/a/00dM1iV

Thank you!

[lovell]

Are you using the latest version of sharp? Do you see this problem when using the prebuilt binaries (that provide libspng) instead of a globally-installed libvips (probably using libpng)?

[jortuck]

Are you using the latest version of sharp? Do you see this problem when using the prebuilt binaries (that provide libspng) instead of a globally-installed libvips (probably using libpng)?

I am using the latest version of sharp. I just tried it using the prebuilt binaries in my docker container and it still gave me the same error.

node:internal/process/promises:288
 triggerUncaughtException(err, true /* fromPromise */);
            ^
[Error: pngload_buffer: out of order read at line 636]

I did the same on my Mac too. I uninstalled my installation of libvips and opted for the prebuilt binaries, but the operations completed without any errors.

I would try to wrap it in a try catch block, however I have no idea where I would put it since the error is not giving me any context as to where the error is occurring.

[lovell]

Thank you. Given you're using a globally-installed libvips and compiling sharp from source, please can you try applying the following patch:

sharp/src/pipeline.cc

Line 656 in 44a0ee3

compositeImage = compositeImage.replicate(across, down);

- compositeImage = compositeImage.replicate(across, down);
+ compositeImage = compositeImage.copy_memory().replicate(across, down);

[jortuck]

Thank you. Given you're using a globally-installed libvips and compiling sharp from source, please can you try applying the following patch:

sharp/src/pipeline.cc

Line 656 in 44a0ee3

compositeImage = compositeImage.replicate(across, down);

- compositeImage = compositeImage.replicate(across, down);
+ compositeImage = compositeImage.copy_memory().replicate(across, down);

Thanks for the reply! I am not actually building sharp myself from source, I am just allowing it to automatically build from the npm installation command. How can I go about building it myself? Maybe I am not looking hard enough but I can't find any instructions in the documentation. Thanks.

[lovell]

I am not actually building sharp myself from source

If you're using a globally-installed version of libvips then sharp will attempt to re-compile itself from source at npm install time to link against it. This info is written to the installation log (you may need to add the --foreground-scripts flag).

https://sharp.pixelplumbing.com/install#building-from-source

[ansoncsh]

sharp/src/pipeline.cc

Line 656 in 44a0ee3

compositeImage = compositeImage.replicate(across, down);

- compositeImage = compositeImage.replicate(across, down);
+ compositeImage = compositeImage.copy_memory().replicate(across, down);

@lovell
I have got the same error on AWS Lambda. I am going to composite png images with animated images (e.g. gif or animated webp).
However, not all images will produce the error. It works fine with images in smaller size.

I try to apply the patch and compile sharp from source on a M1 Mac for AWS Lambda.
npm install --arch=x64 --platform=linux --libc=glibc --build-from-source

But I got fatal error: 'vips/vips8' file not found. Is there anything I am missing? Sorry that I am not familiar with it.

Thank you!

> sharp@0.32.5 install
> (node install/libvips && node install/dll-copy && prebuild-install) || (node install/can-compile && node-gyp rebuild && node install/dll-copy)

sharp: Using existing vendored libvips v8.14.4
gyp info it worked if it ends with ok
gyp info using node-gyp@9.4.0
gyp info using node@16.19.0 | darwin | arm64
gyp info find Python using Python version 3.9.6 found at "/Applications/Xcode.app/Contents/Developer/usr/bin/python3"
gyp info spawn /Applications/Xcode.app/Contents/Developer/usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/Users/ac/workspace/sharp/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/ac/workspace/sharp/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/ac/workspace/sharp/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/ac/Library/Caches/node-gyp/16.19.0/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/Users/ac/Library/Caches/node-gyp/16.19.0',
gyp info spawn args   '-Dnode_gyp_dir=/Users/ac/workspace/sharp/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/Users/ac/Library/Caches/node-gyp/16.19.0/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/Users/ac/workspace/sharp',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
  TOUCH Release/obj.target/libvips-cpp.stamp
  CC(target) Release/obj.target/nothing/node_modules/node-addon-api/nothing.o
  LIBTOOL-STATIC Release/nothing.a
warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: archive library: Release/nothing.a the table of contents is empty (no object file members in the library define global symbols)
  CXX(target) Release/obj.target/sharp-linux-x64/src/common.o
../src/common.cc:13:10: fatal error: 'vips/vips8' file not found
#include <vips/vips8>
         ^~~~~~~~~~~~
1 error generated.
make: *** [Release/obj.target/sharp-linux-x64/src/common.o] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/ac/workspace/sharp/node_modules/node-gyp/lib/build.js:203:23)
gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)
gyp ERR! System Darwin 22.4.0
gyp ERR! command "/Users/ac/.nvm/versions/node/v16.19.0/bin/node" "/Users/ac/workspace/sharp/node_modules/.bin/node-gyp" "rebuild"
gyp ERR! cwd /Users/ac/workspace/sharp
gyp ERR! node -v v16.19.0
gyp ERR! node-gyp -v v9.4.0
gyp ERR! not ok 

[jortuck]

I am not actually building sharp myself from source

If you're using a globally-installed version of libvips then sharp will attempt to re-compile itself from source at npm install time to link against it. This info is written to the installation log (you may need to add the --foreground-scripts flag).

https://sharp.pixelplumbing.com/install#building-from-source

@lovell Thanks again for the reply. I understand now that it is built at install, but I'm still a bit confused on how I would apply that patch. Wouldn't I need to install the package first before making any changes? Would it be easier for me just to make a fork? The problem is, my docker container runs yarn install on every start up, so I'm not quite sure how to make that patch before the process is fully running.

[lovell]

I've now been able to reproduce this locally and can confirm the suggested fix works.

The problem occurs when compositing a PNG with tile: true where the height of the image to composite is > 512px.

Commit 4340d60 adds the fix and a test that would previously have failed to help prevent regression.

[t-mdo]

Hey @lovell,
Just passing by to thank you for your responsiveness and your work!
I've encountered a few problems using sharp, but each time I found a solution in this gh's issues.

Also, I just had this bug when I flipped my gif to have a vertical resolution, and it works perfectly fine now with the fix you shipped yesterday.
I really had a great experience using this package, amazing work!

[lovell]

v0.32.6 now available, thanks for reporting.