Add embedded clang/llvm for runtime compilation of eBPF programs #6978
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`inv deps` did not like it when these files were missing and not generated yet
This figures out the correct path if deployed in a similar repo structure to how it was built. Otherwise, it assumes you are running from the root directory.
xornivore
approved these changes
Jan 4, 2021
xornivore
added
need-change/helm-chart
sunhay
reviewed
Jan 6, 2021
| // defaultProxyPort is the default port used for proxies. | ||
| // This mirrors the configuration for the infrastructure agent. | ||
| defaultProxyPort = 3128 | ||
|
|
||
| // defaultSystemProbeBPFDir is the default path for eBPF programs | ||
| defaultSystemProbeBPFDir = "/opt/datadog-agent/embedded/share/system-probe/ebpf" | ||
|
|
||
| // defaultRuntimeCompilerOutputDir is the default path for output from the system-probe runtime compiler | ||
| defaultRuntimeCompilerOutputDir = "/var/tmp/datadog-agent/system-probe/build" |
There was a problem hiding this comment.
This is also defined separately as a default in pkg/ebpf/config.go. Might be worth cleaning up so its defined once. Fine if this is done in a separate follow-up PR .
ogaca-dd
approved these changes
Jan 6, 2021
sunhay
approved these changes
Jan 6, 2021
This ensures we re-compile assets when config options change and/or we update the default flags.
This doesn't include DNS or HTTP tests at the moment because some refactoring is needed to make that possible.
brycekahle
force-pushed
the
bryce.kahle/runtime-compilation
branch
from
4442771 to
4cbf304
Compare
brycekahle
added a commit
to DataDog/helm-charts
that referenced
this pull request
Jan 21, 2021
4 tasks
brycekahle
added a commit
to DataDog/helm-charts
that referenced
this pull request
Jan 21, 2021
brycekahle
added a commit
to DataDog/helm-charts
that referenced
this pull request
Jan 22, 2021
brycekahle
added a commit
to DataDog/helm-charts
that referenced
this pull request
Jan 22, 2021
brycekahle
added a commit
to DataDog/helm-charts
that referenced
this pull request
Mar 2, 2021
sergei-deliveroo
pushed a commit
to deliveroo/datadog-agent
that referenced
this pull request
Mar 24, 2021
brycekahle
added a commit
to DataDog/helm-charts
that referenced
this pull request
Apr 20, 2021
brycekahle
added a commit
to DataDog/helm-charts
that referenced
this pull request
Apr 22, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a statically-linked, embedded clang/llvm for runtime compilation of eBPF programs. If enabled, these programs are compiled on-demand. The output object file is cached, such that if any of the following changes, it will be re-compiled:
Each runtime-compiled source file is pre-processed during CI build to be a concatenation of the
.cand all user (non-kernel).hfiles. Also during CI build, a hash is computed for the pre-processed file, such that any modifications on the host would result in refusing to compile the program.The pre-processed
.cfile is distributed separately from thesystem-probebinary, like our.ofiles, since it must be GPL-licensed to use certain kernel BPF helper functions.Motivation
Planned features for networks and/or the system-probe require access to kernel structures that are too difficult to access with offset guessing. Runtime compilation allows us to compile on each host so it is targeted for the running kernel.
Additional Notes
g++being killed. I'm investigating why, but it was not happening before I rebased on master.Describe your test plan
The kitchen tests currently pass across a wide range of kernels/distros, but the runtime compilations tests only ensure that the programs can be compiled. The CI tests are run another time with the runtime compiler enabled.
Further testing with
system_probe_config.enable_runtime_compilerturned on would be useful.