How to build custom `datadog/agent` image

[mleonhard-notion]

Hi DataDog Agent Team,
We need to make a small change to the datadog agent code. It's been challenging figuring out how to build a custom version of the datadog/agent Docker image. I found several conflicting sets of instructions for building for development, but none of them work for a prod image.

1. https://datadoghq.dev/datadog-agent/setup/
2. https://github.com/DataDog/datadog-agent/blob/7.61.0/docs/dev/agent_build.md
3. https://github.com/DataDog/datadog-agent/blob/7.61.0/README.md#getting-started
4. https://github.com/DataDog/datadog-agent/blob/7.61.0/docs/dev/agent_omnibus.md
5. https://github.com/DataDog/datadog-agent-buildimages/blob/14a41bca/README.md

I tried manually evaluating the docker_build_agent7 GitLab CI config, through about 15 levels of inheritance and dependencies, and came up with a simple Dockerfile that will build the agent into /src/bin/agent/.

Then I tried to use Dockerfiles/agent/Dockerfile to build the image, but it expects a datadog-agent*.tar.xz file and I couldn't find the code that creates this file to see what goes into it.

So I tried replacing the binary in the existing image:

# Dockerfile

# This is adapted from
# https://github.com/DataDog/datadog-agent/blob/fc8851b47920afe200f134b23b890fa2173fe9aa/.gitlab/container_build/docker_linux.yml

# To build on your ARM (Apple Silicon) laptop, switch to the arm64 image.
FROM datadog/agent-buildimages-deb_x64:v53759313-14a41bca AS build
#FROM datadog/agent-buildimages-deb_arm64:v53759313-14a41bca AS build

ADD --keep-git-dir=true "https://github.com/DataDog/datadog-agent.git#7.61.0" /src
# TODO: Apply our patch.

WORKDIR /src
ENV CI_PROJECT_DIR=/src
ENV PATH=/usr/local/rvm/gems/ruby-2.7.2/bin:/usr/local/rvm/gems/ruby-2.7.2@global/bin:/usr/local/rvm/rubies/ruby-2.7.2/bin:/root/miniforge3/envs/ddpy3/bin:/root/miniforge3/condabin:/root/.cargo/bin:/opt/clang/bin:/go/bin:/usr/local/go/bin:/root/miniforge3/condabin:~/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/rvm/bin
RUN --mount=type=cache,target=/go/pkg/mod/cache inv -e agent.build

FROM datadog/agent:7.61.0
COPY --from=build /src/bin/agent/agent /opt/datadog-agent/bin/agent/agent

Unfortunately, the system does not build a statically-linked binary and the binary fails when trying to import libs on macOS ARM:

% docker build .
...
 => => writing image sha256:69119de5d5e57b9ff8140ad6d31dd276e9f134da94f27a549fdaaff9d32798bc                                                                                                                                               0.0s
...
mleonhard@Mac datadog-agent % docker run -it sha256:69119de5d5e57b9ff8140ad6d31dd276e9f134da94f27a549fdaaff9d32798bc /usr/bin/env /opt/datadog-agent/bin/agent/agent --version 
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

And also fails on Linux AMD64 in our production-like environment:

agent: error while loading shared libraries: libdatadog-agent-rtloader.so.2: cannot open shared object file: No such file or directory

Would you please provide a Dockerfile that will build the agent from source to run on Linux AMD64?

[mleonhard-notion]

Alternatively, it would be helpful to know:

1. How to make a release build of the agent
2. What to include in the datadog-agent*.tar.xz that datadog-agent/Dockerfiles/agent/Dockerfile wants?

[chouquette]

Hi, and thanks for reaching out!

As you noticed, things are not easy enough when attempting to build the agent locally.
We are working internally to make things better, but for the time being, I opened a PR to clarify things in the docs/dev/agent_omnibus.md doc: #33654

TL;DR: run inv -e omnibus.build which will generate the .tar.xz the Dockerfile wants.

[mleonhard-notion]

Thanks for sharing the inv -e omnibus.build command. It's not working for me.

Build failure

mleonhard@Michaels-MacBook-Pro % docker run -it --rm --mount=type=bind,source=/Users/mleonhard/datadog-agent,target=/src datadog/agent-buildimages-deb_arm64:v53759313-14a41bca /usr/bin/env bash
(ddpy3) root@70d6c87007e6:/# cd /src
(ddpy3) root@70d6c87007e6:/src# git rev-parse HEAD
202f54bcf5a5b72a0d2a4a3e1a7456204a180bd6
(ddpy3) root@70d6c87007e6:/src# export PATH=/usr/local/rvm/gems/ruby-2.7.2/bin:/usr/local/rvm/gems/ruby-2.7.2@global/bin:/usr/local/rvm/rubies/ruby-2.7.2/bin:/root/miniforge3/envs/ddpy3/bin:/root/miniforge3/condabin:/root/.cargo/bin:/opt/clang/bin:/go/bin:/usr/local/go/bin:/root/miniforge3/condabin:~/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/rvm/bin
(ddpy3) root@70d6c87007e6:/src# export CI_PROJECT_DIR=/src
(ddpy3) root@70d6c87007e6:/src# inv -e omnibus.build
...

The following shell command exited with status 1:

    $ CFLAGS=-I/opt/datadog-agent/embedded/include -O2 CPPFLAGS=-I/opt/datadog-agent/embedded/include -O2 CXXFLAGS=-I/opt/datadog-agent/embedded/include -O2 LDFLAGS=-Wl,-rpath,/opt/datadog-agent/embedded/lib,-z,origin -L/opt/datadog-agent/embedded/lib -Wl,-rpath-link=/opt/datadog-agent/embedded/lib LD_RUN_PATH=/opt/datadog-agent/embedded/lib OPT=-DNDEBUG -fwrapv PATH=/opt/datadog-agent/bin:/opt/datadog-agent/embedded/bin:/gems/ruby/2.7.0/bin:/usr/local/rvm/gems/ruby-2.7.2/bin:/usr/local/rvm/gems/ruby-2.7.2@global/bin:/usr/local/rvm/rubies/ruby-2.7.2/bin:/root/miniforge3/envs/ddpy3/bin:/root/miniforge3/condabin:/root/.cargo/bin:/opt/clang/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/rvm/bin PKG_CONFIG_PATH=/opt/datadog-agent/embedded/lib/pkgconfig sed -i "s/${CC}/gcc/g" /opt/datadog-agent/embedded/lib/python3.12/_sysconfigdata__linux_*-linux-gnu.py

Output:

    (nothing)

Error:

    sed: -e expression #1, char 0: no previous regular expression

/gems/ruby/2.7.0/bundler/gems/omnibus-ruby-d365e483ee05/lib/omnibus/util.rb:101:in `rescue in shellout!'
  /gems/ruby/2.7.0/bundler/gems/omnibus-ruby-d365e483ee05/lib/omnibus/util.rb:96:in `shellout!'
  /gems/ruby/2.7.0/bundler/gems/omnibus-ruby-d365e483ee05/lib/omnibus/builder.rb:927:in `shellout!'
...

This builds the Datadog agent binary so it can run from the Docker image:

# This is adapted from https://github.com/DataDog/datadog-agent/blob/fc8851b47920afe200f134b23b890fa2173fe9aa/.gitlab/container_build/docker_linux.yml

# Datadog does not name the images with the Docker $TARGETARCH values 'amd64' and 'arm/v7'.
# To build on an ARM (Apple Silicon) laptop, switch to the arm64 image.
FROM datadog/agent-buildimages-deb_x64:v53759313-14a41bca AS build
#FROM datadog/agent-buildimages-deb_arm64:v53759313-14a41bca AS build

ADD --keep-git-dir=true "https://github.com/DataDog/datadog-agent.git#7.61.0" /src
# TODO: Apply our patch.

ENV PATH=/usr/local/rvm/gems/ruby-2.7.2/bin:/usr/local/rvm/gems/ruby-2.7.2@global/bin:/usr/local/rvm/rubies/ruby-2.7.2/bin:/root/miniforge3/envs/ddpy3/bin:/root/miniforge3/condabin:/root/.cargo/bin:/opt/clang/bin:/go/bin:/usr/local/go/bin:/root/miniforge3/condabin:~/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/rvm/bin
WORKDIR /src
# The Invoke framework builds a dynamically-linked binary which fails to import libs.
#ENV CI_PROJECT_DIR=/src
#RUN --mount=type=cache,target=/go/pkg/mod/cache inv -e agent.build
# To work around this, we build using the same command-line options used to build the released binary.
# We discovered the options by running https://github.com/sg0hsmt/gobuildinfo.git against the binary.
# Removed tag: sds, likely https://github.com/DataDog/dd-sensitive-data-scanner
# Removed tag: python, likely only needed for https://docs.datadoghq.com/integrations/python/
RUN --mount=type=cache,target=/go/pkg/mod/cache CGO_ENABLED=1 CGO_CFLAGS="-I. -I/opt/datadog-agent/embedded/include -Werror -Wno-deprecated-declarations -I/opt/datadog-agent/embedded/include" CGO_LDFLAGS="-Wl,-rpath,/opt/datadog-agent/embedded/lib -L/opt/datadog-agent/embedded/lib -L/opt/datadog-agent/embedded/lib" go build -buildmode=exe -compiler=gc -ldflags="-X github.com/DataDog/datadog-agent/pkg/version.Commit=202f54bcf5 -X github.com/DataDog/datadog-agent/pkg/version.AgentVersion=7.61.0 -X github.com/DataDog/datadog-agent/pkg/serializer.AgentPayloadVersion=v5.0.137 -X github.com/DataDog/datadog-agent/pkg/config/setup.InstallPath=/opt/datadog-agent -X github.com/DataDog/datadog-agent/pkg/config/setup.ForceDefaultPython=true -X github.com/DataDog/datadog-agent/pkg/config/setup.DefaultPython=3 -r /opt/datadog-agent/embedded/lib '-extldflags=-Wl,-z,lazy ' " -tags=systemd,etcd,cri,consul,jetson,kubelet,containerd,kubeapiserver,orchestrator,trivy,docker,jmx,ec2,datadog.no_waf,zk,podman,oracle,zstd,netcgo,gce,process,crio,zlib,bundle_agent,otlp,apm -o ./bin/agent/agent ./cmd/agent/

FROM datadog/agent:7.61.0
COPY --from=build /src/bin/agent/agent /opt/datadog-agent/bin/agent/agent

[chouquette]

This looks a lot like the bug that was fixed with this commit: 6a7a2be

If updating to the latest release (7.62.x branch) is not an option, you should be able to work around this by providing CC=gcc CXX=g++ as environment variables to your container