ASH Version 1.0

Author: begimher

.gitignore

@@ -0,0 +1 @@
+utils/cfn-to-cdk/cfn_to_cdk/

CHANGELOG.md

@@ -0,0 +1,14 @@
+<a name="1.0.0-e-23Dec2022"></a>
+# 1.0.0-e-23Dec2022
+### Automated Security Helper
+ASH version 1.0.0-e-23Dec2022 is out!
+
+* Speed - running time is shorter by 40-50%
+* Frameworks support - we support Java, Go and C# code
+* New tool - ASH is running [Semgrep](https://github.com/returntocorp/semgrep) for supported frameworks
+* Force scans for specific frameworks - You can use the `--ext` flag to enforce scan for specific framework  
+For example: `ash --source-dir . --ext py` (Python)
+* Versioning - use `ash --version` to check your current version
+* Bug fixes and improvements
+
+<!-- CHANGELOG SPLIT MARKER -->

README.md

@@ -23,14 +23,26 @@ The security helper supports the following vectors:
         * **[git-secrets](https://github.com/awslabs/git-secrets)** - Find api keys, passwords, AWS keys in the code
     * Python
         * **[bandit](https://github.com/PyCQA/bandit)** - finds common security issues in Python code.
-        * **[Grype](https://github.com/anchore/grype)** - checks for vulnerabilities in Python code.
-        * **[Syft](https://github.com/anchore/grype)** - generates a Software Bill of Materials (SBOM) for Python code.
+        * **[Semgrep](https://github.com/returntocorp/semgrep)** - finds common security issues in Python code.
+        * **[Grype](https://github.com/anchore/grype)** - finds vulnerabilities scanner for Python code.
+        * **[Syft](https://github.com/anchore/grype)** - generating a Software Bill of Materials (SBOM) for Python code.
     * Jupyter Notebook
         * **[nbconvert](https://nbconvert.readthedocs.io/en/latest/)** - converts Jupyter Notebook (ipynb) files into Python executables. Code scan with Bandit.
     * JavaScript; NodeJS
         * **[npm-audit](https://docs.npmjs.com/cli/v8/commands/npm-audit)** - checks for vulnerabilities in Javascript and NodeJS.
-        * **[Grype](https://github.com/anchore/grype)** - checks for vulnerabilities in Javascript and NodeJS.
-        * **[Syft](https://github.com/anchore/grype)** - generates a Software Bill of Materials (SBOM) for Javascript and NodeJS.
+        * **[Semgrep](https://github.com/returntocorp/semgrep)** - finds common security issues in JavaScript code.
+        * **[Grype](https://github.com/anchore/grype)** - finds vulnerabilities scanner for Javascript and NodeJS.
+        * **[Syft](https://github.com/anchore/grype)** - generating a Software Bill of Materials (SBOM) for Javascript and NodeJS.
+    * Go
+        * **[Semgrep](https://github.com/returntocorp/semgrep)** - finds common security issues in Golang code.
+        * **[Grype](https://github.com/anchore/grype)** - finds vulnerabilities scanner for Golang.
+        * **[Syft](https://github.com/anchore/grype)** - generating a Software Bill of Materials (SBOM) for Golang.
+    * C#
+        * **[Semgrep](https://github.com/returntocorp/semgrep)** - finds common security issues in C# code.
+    * Java
+        * **[Semgrep](https://github.com/returntocorp/semgrep)** - finds common security issues in Java code.
+        * **[Grype](https://github.com/anchore/grype)** - finds vulnerabilities scanner for Java.
+        * **[Syft](https://github.com/anchore/grype)** - generating a Software Bill of Materials (SBOM) for Java.
 * Infrastructure
     * Terraform; Cloudformation
         *   **[checkov](https://github.com/bridgecrewio/checkov)**
@@ -68,6 +80,9 @@ ash --output-dir /my/remote/files
 # Force rebuild the entire framework to obtain latests changes and up-to-date database
 ash --force
 
+# Force run scan for Python code
+ash --source-dir . --ext py
+
 * All commands can be used together.
 ```
 
@@ -78,9 +93,11 @@ NAME:
 SYNOPSIS:
         ash [OPTIONS] --source-dir /path/to/dir --output-dir /path/to/dir
 OPTIONS:
+        -v | --version           Prints version number.
         -p | --preserve-report   Add timestamp to the final report file to avoid overriding it after multiple executions
         --source-dir             Path to the directory containing the code/files you wish to scan. Defaults to $(pwd)
         --output-dir             Path to the directory that will contain the report of the scans. Defaults to $(pwd)
+        --ext | -extension       Force a file extension to scan. Defaults to identify files automatically.
         --force                  Rebuild the Docker images of the scanning tools, to make sure software is up-to-date.
          -q | --quiet            Don't print verbose text about the build process.
 

ash

@@ -3,20 +3,22 @@
 # SPDX-License-Identifier: Apache-2.0
 set -e
 START_TIME=$(date +%s)
+VERSION=("1.0.0-e-23Dec2022")
 
 print_usage() {
   echo "NAME:"
   echo -e "\t$(basename $0)"
   echo "SYNOPSIS:"
   echo -e "\t$(basename $0) [OPTIONS] --source-dir /path/to/dir --output-dir /path/to/dir"
   echo "OPTIONS:"
-  echo -e "\t-p | --preserve-report   Add timestamp to the final report file to avoid overriding it after multiple executions"
+  echo -e "\t-v | --version           Prints version number.\n"
+  echo -e "\t-p | --preserve-report   Add timestamp to the final report file to avoid overriding it after multiple executions."
   echo -e "\t--source-dir             Path to the directory containing the code/files you wish to scan. Defaults to \$(pwd)"
   echo -e "\t--output-dir             Path to the directory that will contain the report of the scans. Defaults to \$(pwd)"
+  echo -e "\t--ext | -extension       Force a file extension to scan. Defaults to identify files automatically."
   echo -e "\t--force                  Rebuild the Docker images of the scanning tools, to make sure software is up-to-date."
-  echo -e "\t-q | --quiet             Don't print verbose text about the build process.\n"
-  echo -e "\t-c | --no-color          Don't print colorized output"
-  echo -e "\t-q | --quiet             Don't print verbose text about the build process.\n"
+  echo -e "\t-q | --quiet             Don't print verbose text about the build process."
+  echo -e "\t-c | --no-color          Don't print colorized output."
   echo -e "For more information please visit https://github.com/aws-samples/automated-security-helper"
 }
 
@@ -26,10 +28,7 @@ PY_EXTENSIONS=("py" "pyc" "ipynb")
 INFRA_EXTENSIONS=("yaml" "yml" "tf" "json" "dockerfile")
 CFN_EXTENSIONS=("yaml" "yml" "json")
 JS_EXTENSIONS=("js")
-GRYPE_EXTENSIONS=("js" "py")
-
-# Look for specific files
-CDK_FILENAMES=("cdk.json")
+GRYPE_EXTENSIONS=("js" "py" "java" "go" "cs")
 
 DOCKERFILE_LOCATION="$(dirname "${BASH_SOURCE[0]}")"/"helper_dockerfiles"
 UTILS_LOCATION="$(dirname "${BASH_SOURCE[0]}")"/"utils"
@@ -44,6 +43,7 @@ HIGHEST_RC=0
 # Initialize options
 #
 COLOR_OUTPUT="true"
+FORCED_EXT="false"
 
 while (("$#")); do
   case $1 in
@@ -55,6 +55,10 @@ while (("$#")); do
     shift
     OUTPUT_DIR="$1"
     ;;
+  --ext | -extenstion)
+    shift
+    FORCED_EXT="$1"
+    ;;
   --force)
     DOCKER_EXTRA_ARGS="${DOCKER_EXTRA_ARGS} --no-cache"
     ;;
@@ -68,6 +72,11 @@ while (("$#")); do
   --no-color | -c)
     COLOR_OUTPUT="false"
     ;;
+  --version | -v)
+    echo -n "ASH version $VERSION"
+    EXITCODE=0
+    exit $EXITCODE
+    ;;
   *)
     print_usage
     exit 1
@@ -76,11 +85,6 @@ while (("$#")); do
   shift
 done
 
-print_found_msg() {
-  EXTENSIONS=$1
-  echo -e "${LPURPLE}Found one of ${EXTENSIONS} items in your source dir${NC}"
-}
-
 TIMESTAMP=$(date +%s)
 USERID=$(echo -n "$(whoami)$(hostname)" | openssl dgst -sha512)
 TOOLID=$(basename "$0")
@@ -133,20 +137,16 @@ map_extensions_anf_files() {
 # Try to locate specific extension type (ie yaml, py) from all the extensions found in $SOURCE_DIR
 search_extension() {
   items_to_search=("$@") # passed as parameter to the function
-  item_found=false
+  local item_found=0
   for item in "${items_to_search[@]}"; do
-    if [[ "${extenstions_found[*]}" =~ ${item} || "${files_found[*]}" =~ ${item} ]]; then
-      item_found=true
+    if [[ "${extenstions_found[*]}" =~ ${item} ]]; then
+      local item_found=1
+      echo "$item_found"
+      break
     fi
   done
 }
 
-search_file_content() {
-  items_to_search=("$@") # passed as parameter to the function
-  item_found=false
-  cfn_files=$(grep -lri 'AWSTemplateFormatVersion' ${SOURCE_DIR} --exclude-dir="cdk.out")
-}
-
 # Validate the input and set default values
 # shellcheck disable=SC2120
 validate_input() {
@@ -162,16 +162,14 @@ validate_input() {
 # The first argument passed to this method is the dockerfile that executes the actual scan
 # The remaining arguments (can be treated as *args in python) are the extensions we wish to scan for
 run_security_check() {
-  EXTENSIONS_USED=( "${EXTENSIONS_USED[@]}" "$1" )
-  echo "EXTENSIONS_USED is " $1
-  DOCKERFILE_TO_EXECUTE="$1"
-  ITEMS_TO_SCAN=("${@:2}") # take all the array of commands which are the extensions to scan (slice 2nd to end)
-  RUNTIME_CONTAINER_NAME="scan-$RANDOM"
-
-  search_extension "${ITEMS_TO_SCAN[@]}" # First lets verify this extension even exists in the $SOURCE_DIR directory
-  if [[ $item_found == "true" ]]; then
-    print_found_msg "${ITEMS_TO_SCAN}"
-
+  local EXTENSIONS_USED=( "${EXTENSIONS_USED[@]}" "$1" )
+  local DOCKERFILE_TO_EXECUTE="$1"
+  local ITEMS_TO_SCAN=("${@:2}") # take all the array of commands which are the extensions to scan (slice 2nd to end)
+  local RUNTIME_CONTAINER_NAME="scan-$RANDOM"
+   # First lets verify this extension even exists in the $SOURCE_DIR directory
+  #echo "${EXTENSIONS_USED[@]}" $(search_extension "${ITEMS_TO_SCAN[@]}")
+  if [[ " ${ITEMS_TO_SCAN[*]} " =~ " ${FORCED_EXT} " ]] || [[ $(search_extension "${ITEMS_TO_SCAN[@]}") == "1" ]]; then
+    echo -e "${LPURPLE}Found one of: ${RED}"${ITEMS_TO_SCAN[@]}" ${LPURPLE}items in your source dir,${NC} ${GREEN}running $1 ...${NC}"
     docker build -t "${RUNTIME_CONTAINER_NAME}" -f "${DOCKERFILE_LOCATION}"/"${DOCKERFILE_TO_EXECUTE}" ${DOCKER_EXTRA_ARGS} "${SOURCE_DIR}" > /dev/null
     set +e # the scan will fail the command if it finds any finding. we don't want it to stop our script execution
     docker run --name "${RUNTIME_CONTAINER_NAME}" -v "${CFNRULES_LOCATION}":/cfnrules -v "${UTILS_LOCATION}":/utils -v "${SOURCE_DIR}":/app "${RUNTIME_CONTAINER_NAME}"
@@ -227,13 +225,16 @@ do
 done
 unset IFS
 
-run_security_check "Dockerfile-git" "${GIT_EXTENSIONS[@]}"
-run_security_check "Dockerfile-py" "${PY_EXTENSIONS[@]}"
-run_security_check "Dockerfile-yaml" "${INFRA_EXTENSIONS[@]}"
-run_security_check "Dockerfile-js" "${JS_EXTENSIONS[@]}"
-search_file_content "${CFN_EXTENSIONS}" || echo "No CFN files found"
+echo -e "ASH version $VERSION\n"
+
+run_security_check "Dockerfile-git" "${GIT_EXTENSIONS[@]}" & 
+run_security_check "Dockerfile-py" "${PY_EXTENSIONS[@]}" &
+run_security_check "Dockerfile-yaml" "${INFRA_EXTENSIONS[@]}" &
+run_security_check "Dockerfile-js" "${JS_EXTENSIONS[@]}" &
+run_security_check "Dockerfile-grype" "${GRYPE_EXTENSIONS[@]}" &
 run_security_check "Dockerfile-cdk" "${CFN_EXTENSIONS[@]}"
-run_security_check "Dockerfile-grype" "${GRYPE_EXTENSIONS[@]}"
+wait
+
 
 # Cleanup any previous file
 rm -f "${OUTPUT_DIR}"/"${AGGREGATED_RESULTS_REPORT_FILENAME}"

helper_dockerfiles/Dockerfile-cdk

@@ -1,15 +1,16 @@
 # Get Ubuntu Image
 #FROM public.ecr.aws/ubuntu/ubuntu:latest
-FROM public.ecr.aws/ubuntu/ubuntu:20.04
+#FROM public.ecr.aws/bitnami/python:latest
+FROM public.ecr.aws/docker/library/node:18.0.0
 ENV TZ=Europe/London
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 
 # Instal prerequisites
 RUN apt-get update && \
     apt-get upgrade -y && \
-    apt-get install -y python3 python3-pip curl && \
+    apt-get install -y curl && \
     rm -rf /var/lib/apt/lists/*
-RUN curl -sL https://deb.nodesource.com/setup_14.x | bash - && apt -y install nodejs
+RUN wget https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py
 RUN npm install -g aws-cdk
 RUN python3 -m pip install -U aws-cdk-lib cdk-nag jinja2
 WORKDIR /app

helper_dockerfiles/Dockerfile-git

@@ -1,5 +1,5 @@
 # Get Ubuntu Image
-FROM public.ecr.aws/ubuntu/ubuntu:latest
+FROM public.ecr.aws/bitnami/python:latest
 
 # Instal prerequisites
 RUN apt-get update && \

helper_dockerfiles/Dockerfile-grype

@@ -3,7 +3,8 @@ FROM public.ecr.aws/bitnami/python:latest
 
 # Instal prerequisites
 RUN curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin && \
-    curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin
+    curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin && \
+    python3 -m pip install semgrep
 
 WORKDIR /app
 VOLUME /app

helper_dockerfiles/Dockerfile-js

@@ -1,5 +1,5 @@
 # Get NPM Image
-FROM public.ecr.aws/docker/library/node:latest
+FROM public.ecr.aws/docker/library/node:18.0.0
 
 WORKDIR /app
 VOLUME /app

helper_dockerfiles/Dockerfile-yaml

@@ -1,16 +1,16 @@
 # Get Ubuntu Image
-FROM public.ecr.aws/ubuntu/ubuntu:latest
+FROM public.ecr.aws/bitnami/python:latest
 
 ENV TZ=Europe/London
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 
 # Instal prerequisites
 RUN apt-get update && \
     apt-get upgrade -y && \
-    apt-get install -y git-all python3 python3-pip ruby-full && \
+    apt-get install -y ruby-full && \
     rm -rf /var/lib/apt/lists/*
 
 RUN pip3 install -U checkov && gem install cfn-nag
 WORKDIR /app
 
-CMD bash -C /utils/yaml-docker-execute.sh
+CMD bash -C /utils/yaml-docker-execute.sh