IAS created an open source project, dependency-deep-scan-utilities which detects log4j vulnerabilities (CVE-2021-44228 and CVE-2021-45046) in your source code.
Because of the widespread use of log4j, ease of exploit, and ability to perform remote code execution, IAS open sourced this project to help everyone mitigate this exploit.
dependency-deep-scan-utilities is a command line tool which you run against your source code. dependency-deep-scan-utilities goes through every git repository and uses Maven and Gradle to find transitive usage of vulnerable log4j-core in order to detect when your code is susceptible to the log4shell security vulnerability. Then dependency-deep-scan-utilities takes the resulting output of all scans and creates a CSV file including the project, log4j-core version, and Git clone URL so that teams can organize and prioritize mitigation.
- Java and Build Tool Support
- Benchmarks
- Prerequisite setup
- Getting started: Scan dependencies
- How to update your CSV by rerunning scans
- Limitations
- License
This project takes roughly 3 hours to scan 2000 repositories. This includes rerunning the scan which would use dependencies cached to disk as part of scanning.
OpenJDK Java support:
- OpenJDK 1.7
- OpenJDK 1.8 (provided by Amazon Corretto)
- OpenJDK 11 (provided by Amazon Corretto)
- OpenJDK 17 (provided by Amazon Corretto)
Build tool support:
- Gradle (only with Gradle wrapper)
- Maven (assumes Maven 3.8 but can be customized)
All repositores are assumed to be cloned with SSH using an SSH clone URL. This project assumes bare repositories have been cloned already before running scans. This is to enable wider suport for non-GitHub hosted Git repositories.
For automatic repository cloning you'll need to create a GitHub personal access
token with scopes org:read and repo for private repositories.
No scopes are necessary if all projects are public/open source.
Run the following script and provide your corporate proxy as an argument.
./create-maven-dotfiles.sh https://nexus.example.com/repository/maven-public/
It will create the following files.
configs/dotfiles/init.gradle
configs/dotfiles/settings.xml
Feel free to further edit these files for your corporate proxy needs. Later when you Build the Docker image these files will be included as the default Maven and Gradle config files for dependency resolution.
This project scans bare repositories located in the repos/ directory. You
have a couple of options for cloning bare repositories.
- Manually cloning bare repositories
- Automatically clone GitHub repositories
Both of these options are discussed in this section below.
You can clone bare repositories into the repos/ directory. If you're working
with a single organization in GitHub or another source then you can
cd repo/
git clone --mirror git@host:namespace/repo.git
By default this project scans for bare repositories in the repos/ directory
(which is ignored in Git). git clone --mirror URL your git repositories for
scanning into that directory. If you're using GitHub Enterprise or GitHub, then
you can use the cloneable utility to mirror your bare repositories.
If you're using a self-hosted GitHub Enterprise, then set the following environment variable.
export GITHUB_GRAPHQL_URL='https://[hostname]/api/graphql'To run this script create a GitHub personal access token with scopes org:read
and repo. The --owner can be a GitHub user or GitHub organization.
GITHUB_OWNER="<your user or org to clone repos>"
curl -LO https://github.com/samrocketman/cloneable/releases/download/0.8/cloneable.jar
mkdir repos
cd repos/
export GITHUB_TOKEN
read -ersp token: GITHUB_TOKEN
# skip empty repos, skip archived repos, and find compatible projects
java -jar ../cloneable.jar -buae -F pom.xml -F gradlew --owner "${GITHUB_OWNER}" \
| xargs -r -n1 -P16 -- git clone --mirrorBefore running any scans you'll need to build a docker image. To build the default docker image run the following command.
make docker
If you'd like to customize the version of Maven 3.x used by the Docker image you can run the following command.
make docker MAVEN_VERSION=3.5.4
If you need a version of Maven other than 3.x you'll need to modify the
Dockerfile.
If your host is a Mac due to performance limitations we recommend running the scan serially (see Limitations).
make build
If your host is Linux and at least 8 vCPUS with 32GB RAM then run 10 scans in parallel.
make build PARALLEL=10
Feel free to adjust your parallelism for your own hosts needs. Parallelism of
10 means 10 docker containers will be launched at a time to perform scans
until all repositories have been scanned.
This section is for advanced use cases. You can skip this if the prior section for running scans is good enough for your organization.
If you do not wish to use the repos/ directory and you already have a Git
repository backup with bare repositories... you can choose an alternate
directory providing the full path to the alternate bare repository location.
make build PARALLEL=10 REPO_DIR=/alternate/repos/path
If you have multiple alternate repository paths and you don't want the scanning results to conflict you can specify multiple scan scan results directories.
make build PARALLEL=10 REPO_DIR=/alternate/repos/path1 OUTPUT_DIR=path1
make build PARALLEL=10 REPO_DIR=/alternate/repos/path2 OUTPUT_DIR=path2
Once all of the dependency scanning is complete you can generate a CSV file covering the whole org.
make csv
The result of Usage will create a CSV file at the root of the repository.
ls scanned-versions.csv
If you're working from a large Git backup and you'd like to limit what shows up in the spreadsheet then you can run the following script.
./get-unarchived-repositories.sh
# alternately only show GitHub repositories contributed within the last year
./get-unarchived-repositories.sh --after 1yUpload scanned-versions.csv to your favorite spreadsheet software and start
a coordinated approach to fixing log4j issues in your organization.
It makes sense to run a scan multiple times as teams patch their projects with a new version of log4j-core. You can rerun scans by resetting progress metadata and then running another scan.
make reset-progress
make build
make csv
You can get progress updates by running make progress as the scan progresses.
You can update your team spreadsheet with the new scanned-versions.csv.
Docker for Mac does not perform very well with Docker in parallel. Running
make build (defaults to non parallel) is the recomendation when performing a
scan using Mac OS.
We highly recomend using a Linux host. From there scanning can be parallelized to launch separate docker containers per repository. The maximum recommended parallelism is 10 for a machine with 8 vCPUs and 32GB of RAM.
make build PARALEL=10
If you open a second terminal you can see how quickly the scan is progressing by running the following make command.
make progress
We use Maven 3.8 to scan repositories containing pom.xml. If you need a lower
version but still a Maven 3.x version, then you can rebuild the docker image
with an older version of Maven 3.x.
make docker MAVEN_VERSION=3.5.4
If Maven 3.x version of Maven is too new for your projects, then you'll need to
update the Dockerfile. Change the maven installation commands
to whatever Maven version you need.
For gradle repositories, we only scan repositories using Gradle Wraper (if a
repo has a ./gradlew script). Due to incompatible differences across several
versions of Gradle it has been deemed too much work to try to account for all
the nuance. Instead, update all projects missing Gradle wraper to use it.
SBT projects are not included at this time. We may add support for them eventually but for now SBT is excluded from scans.
This assumes a single Git repository contains a single Maven or Gradle project and only source code associated with the project. Multi-module projects are okay but multiple sub directories with independent projects is not supported by this tool.
If you're on GitHub, then you can find which projects are using Gradle without Gradle Wrapper or using SBT.
java -jar cloneable.jar -F build.gradle -F build.sbt -E gradlew --owner "your GitHub org"
You could potentially find projects which may contain subfolders of build tools if it doesn't have any build tools at its root. If your company uses other build tools then you may want to also exclude them in the following command.
java -jar cloneable.jar -E pom.xml -E build.gradle -E build.sbt -E gradlew --owner "your GitHub org"
See java -jar cloneable.jar --help for additional usage and exclusion
capabilities.
- Initial release for log4j-scans
- spring4shell release for scanning multiple dependencies.
- Support for more languages relying on CycloneDX BOM format.
This project is MIT Licensed. See COPYING.txt and
LICENSE.txt for details.