libvips Lambda Layer Builder

Builds a Lambda Layer containing libvips 8.17.1 and all native dependencies for Ruby 3.2 Lambda functions on Amazon Linux 2.

Overview

This project builds libvips 8.17.1 as an AWS Lambda Layer with support for:

• Image formats: JPEG, PNG, WebP, GIF, SVG
• Text rendering: Pango, Cairo, HarfBuzz, FreeType with full Unicode (including Thai)
• Performance: FFTW3 for fast Fourier transforms, libimagequant for palette quantization
• Architecture: x86_64 (compatible with Lambda Ruby 3.2 runtime on Amazon Linux 2)

Current Version

• libvips: 8.17.1
• Ruby: 3.2 (Amazon Linux 2)
• Default Layer Name: libvips817

Prerequisites

1. Docker - Must be installed and running
2. AWS CLI - Configured with credentials (aws configure)
3. AWS ECR Repository - Must exist at <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/marpipe/libvips
4. AWS Permissions - IAM permissions for:
   • ECR: ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:PutImage, ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload
   • Lambda: lambda:PublishLayerVersion

Setting up ECR Repository

If the ECR repository doesn't exist yet, create it:

export AWS_ACCOUNT_ID="your-account-id"
export AWS_REGION="us-east-1"

aws ecr create-repository \
  --repository-name marpipe/libvips \
  --region $AWS_REGION

Quick Start

Build and publish libvips 8.17.1 as a Lambda Layer:

# Set required environment variables
export AWS_ACCOUNT_ID="your-account-id"
export VIPS_VERSION="8.17.1"
export PACKAGE_VERSION="1"

# Optional: customize layer name and region
export LAYER_NAME="libvips817"     # default: libvips817
export AWS_REGION="us-east-1"      # default: us-east-1

# Build the layer (takes ~10 minutes first time, ~30 seconds for incremental changes)
./bin/build

# Publish to AWS Lambda
./bin/publish-layer

The scripts will output the Layer ARN which you can use in your Lambda functions.

Build Process

The build happens in three phases:

Phase 1: Dependency Installation (~3 minutes)

Installs and builds all dependencies libvips needs:

• Build tools: Meson, Ninja, CMake
• Image libraries: expat, libpng, giflib, libjpeg-turbo, libwebp, libimagequant
• System libraries: glib, libmount, fftw
• Text rendering: Pango, Cairo, HarfBuzz, FreeType, Fontconfig, librsvg2

Phase 2: Build libvips (~5 minutes)

Compiles libvips 8.17.1 using Meson build system with all features enabled.

Key optimization: This step is cached by Docker. When you add missing runtime libraries in Phase 3, Docker reuses this cached build, saving ~5 minutes per iteration.

Phase 3: Package Runtime Libraries (~10 seconds)

Copies all shared libraries needed at runtime to /opt/lib structure for Lambda Layer.

Testing Locally

Before publishing to Lambda, test the layer locally to verify all dependencies are present:

# Build test Docker image
docker build --platform=linux/amd64 -f Dockerfile-test -t ruby-vips-test:8.17.1 .

# Check for missing libraries
docker run --platform=linux/amd64 --rm --entrypoint="" \
  ruby-vips-test:8.17.1 \
  bash -c "ldd /opt/lib/libvips.so.42 | grep 'not found'"

# If no output, all libraries are present. Run the full test:
docker run --platform=linux/amd64 --rm ruby-vips-test:8.17.1

The test script loads ruby-vips, prints version info, and performs basic image operations.

Dockerfile Structure

The Dockerfile is organized to optimize Docker layer caching:

# Lines 15-124: Install all system packages and build dependencies
# - Cached unless you change dependency versions
# - Rebuilds all dependencies (takes ~3 minutes)

# Lines 135-146: Build libvips
# - Cached unless you change libvips version or build config
# - This is the slowest step (~5 minutes)
# - Moving library copying to Phase 3 means this layer stays cached

# Lines 156-212: Copy runtime libraries to layer
# - Fast step (~10 seconds)
# - Adding missing libraries only rebuilds from here
# - Does not trigger libvips rebuild

This structure means adding a missing library is quick - just add the copy command to lines 156-212 and rebuild.

Environment Variables

Required for Building:

• AWS_ACCOUNT_ID - Your AWS account ID (no default)
• VIPS_VERSION - libvips version to build (no default, e.g., "8.17.1")
• PACKAGE_VERSION - Build iteration number (no default, e.g., "1")

Optional:

• LAYER_NAME - Lambda layer name (default: "libvips817")
• AWS_REGION - AWS region (default: "us-east-1")

Using the Layer in Lambda

After publishing, add the layer to your Lambda function in serverless.yml:

functions:
  myFunction:
    handler: handler.process
    runtime: ruby3.2
    layers:
      - arn:aws:lambda:us-east-1:123456789012:layer:libvips817:1

Or update an existing function:

aws lambda update-function-configuration \
  --function-name my-function \
  --layers arn:aws:lambda:us-east-1:123456789012:layer:libvips817:1

Project Structure

ruby-vips-lambda/
├── Dockerfile              # Main build configuration
├── Dockerfile-test         # Local testing environment
├── test_layer.rb          # Test script for layer verification
├── bin/
│   ├── build              # Build Docker image and extract layer
│   └── publish-layer      # Publish layer to AWS Lambda
├── share/                 # Generated: contains libvips.zip after build
└── README.md             # This file

Troubleshooting

"cannot open shared object file" errors in Lambda

A library is missing from the layer. To debug:

1. Test locally to identify the missing library:

   docker run --platform=linux/amd64 --rm --entrypoint="" \
     ruby-vips-test:8.17.1 \
     bash -c "ldd /opt/lib/libvips.so.42 | grep 'not found'"

2. Find where the library exists in the build image:

   docker run --platform=linux/amd64 --rm --entrypoint="" \
     <ECR_IMAGE> \
     bash -c "ls -la /usr/lib64/libmissing*"

3. Add copy command to Dockerfile (lines 156-212):

   RUN cp -a /usr/lib64/libmissing* /build/share/lib

4. Rebuild (fast due to caching):

   PACKAGE_VERSION=$((PACKAGE_VERSION + 1)) ./bin/build

"Invalid ELF header" or architecture errors

The layer was built for the wrong architecture. Ensure Dockerfile line 1 has:

FROM --platform=linux/amd64 public.ecr.aws/sam/build-ruby3.2

This forces x86_64 builds even on ARM Macs (M1/M2/M3).

Build fails with "Could not detect Ninja v1.8.2"

The Dockerfile installs Ninja 1.11.1 from source. If this fails, check your internet connection or the GitHub releases URL.

Docker build is slow

First build takes ~10 minutes. Subsequent builds:

• Changing dependencies: ~3 minutes (Phase 1 rebuilds)
• Adding runtime libraries: ~30 seconds (only Phase 3 rebuilds)
• No changes: ~10 seconds (all cached)

Key Changes from 8.9.2

This version (8.17.1) includes:

1. Meson build system - Replaces autotools, requires Ninja ≥ 1.8.2
2. Optimized Dockerfile - Library copying moved after libvips build for better caching
3. Updated dependencies:
   • libpng 1.6.50 (was 1.6.37)
   • libjpeg-turbo 3.1.2 (was 2.0.4)
   • libwebp 1.6.0 with libsharpyuv (was 1.1.0)
   • expat 2.7.3 (was 2.2.9)
4. Additional text support - Thai language (libthai), advanced fonts (libgraphite2)
5. Ruby 3.2 compatible - Fixed libffi conflicts with Ruby FFI gem

Dependencies

All dependencies use latest stable versions:

Core Libraries

• expat 2.7.3 - XML parser with security fixes
• libpng 1.6.50 - PNG support
• giflib - System package from Amazon Linux 2
• libjpeg-turbo 3.1.2 - JPEG encoder
• libwebp 1.6.0 - WebP support with libsharpyuv
• libimagequant 2.12.6 - Palette quantization
• glib 2.56.1 - System package (libvips requires ≥ 2.52)
• fftw - System package for fast Fourier transforms

Text Rendering Stack (System Packages)

• Pango, Cairo, Fontconfig, FreeType, HarfBuzz
• Fribidi, Pixman, gdk-pixbuf, librsvg2
• X11 libraries (libX11, libXrender, libxcb)
• libthai (Thai language), libgraphite2 (advanced fonts)
• OpenGL libraries (libGL, libEGL)

License

This build configuration is maintained by Marpipe. libvips itself is licensed under LGPL 2.1+.

Related Projects

• libvips: https://github.com/libvips/libvips
• ruby-vips: https://github.com/libvips/ruby-vips
• vipsGems layer: Companion layer at ../ruby-lambda-layers/layers/vipsGems containing ruby-vips gem and FFI