initial commit

README.md

@@ -0,0 +1,94 @@
+# lcvr-to-timesketch
+Pipeline to process LimaCharlie Velociraptor Triages in Timesketch
+
+## Ubuntu Deployment Steps
+* Deploy Docker - [Deployment Directions](https://docs.docker.com/engine/install/ubuntu/)
+    ```bash
+    sudo apt-get update
+    sudo apt-get install ca-certificates curl gnupg -y
+    sudo install -m 0755 -d /etc/apt/keyrings
+    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+    sudo chmod a+r /etc/apt/keyrings/docker.gpg
+    echo \
+    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
+    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+    sudo apt-get update
+    sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
+    sudo apt-get install docker-compose -y
+    ```
+
+* Deploy Timesketch - [Deployment Directions](https://github.com/google/timesketch/blob/master/docs/guides/admin/install.md)
+    ```bash
+    cd /opt
+    curl -s -O https://raw.githubusercontent.com/google/timesketch/master/contrib/deploy_timesketch.sh
+    chmod 755 deploy_timesketch.sh
+    sudo ./deploy_timesketch.sh # At the end, choose to "not start containers"
+    ```
+    ```bash
+    cd timesketch
+    sudo docker compose up -d
+    sudo docker compose exec timesketch-web tsctl create-user admin
+    ```
+    > **Note**  
+    > **I strongly recommend deploying Timesketch with HTTPS**--additional instructions are provided [here](https://github.com/google/timesketch/blob/master/docs/guides/admin/install.md#4-enable-tls-optional). For this proof of concept, we're using HTTP. Modify your configs to reflect HTTPS if you deploy for production use. 
+* Copy files
+    ```bash
+    cd /opt
+    git clone https://github.com/shortstack/lcvr-to-timesketch.git
+    cd lcvr-to-timesketch
+    ```
+* Modify the environment variables in `systemd/webhook.service`
+    * `TIMESKETCH_USER` - Timesketch admin username
+    * `TIMESKETCH_PW` - Timesketch password
+    * `LC_API_KEY` - LimaCharlie API Key
+    * `LC_UID` - LimaCharlie User ID
+    * `SLACK_WEBHOOK_URL` - Slack webhook URL. Leave blank if `SLACK_NOTIFICATIONS` is `no`
+    * `SLACK_NOTIFICATIONS` - Change to `yes` if you wish to recieve progress notifications
+    * `WEBHOOK_IP` - External IP address of the system the webhook is running on (same as Timesketch)
+* Modify the variables in `limacharlie/output.yaml`
+    * `WEBHOOK_IP` - External IP address of the system the webhook is running on  (same as Timesketch)
+    * `WEBHOOK_PORT`- Port of the system the webhook is running on--the default for the webhook service is `9000`
+* Configuration script:
+    ```bash
+    # Install webhook and unzip
+    sudo apt install webhook unzip -y 
+
+    # Install timesketch_importer
+    sudo docker exec timesketch-worker bash -c "pip3 install timesketch-import-client"
+
+    # Fix permissions
+    chmod +x /opt/lcvr-to-timesketch/bash/run.sh
+
+    # Make sure Plaso dir exists
+    mkdir -p /opt/timesketch/upload/plaso
+
+    # Configure webhook as a service
+    sudo cp systemd/webhook.service /etc/systemd/system/webhook.service
+    sudo systemctl enable webhook.service
+    sudo systemctl start webhook.service
+    ```
+    > **Note**  
+    > **I strongly recommend deploying your webhooks with HTTPS.** If you wish to deploy your webhook with HTTPS, additional instructions are provided [here](https://github.com/adnanh/webhook?tab=readme-ov-file#using-https). For this proof of concept, we're using HTTP. Modify your configs to reflect HTTPS if you deploy for production use. 
+* Add the `artifacts-tailored` tailored output in LimaCharlie - `limacharlie/output.yaml` - ensure `WEBHOOK_IP` and `WEBHOOK_PORT` have been updated to reflect your external IP and port
+
+    ![](<./screenshots/Screenshot 2024-01-19 at 3.42.00 PM.png>)
+* Add the `artifacts-to-output` D&R rule in LimaCharlie - `limacharlie/rules.yaml`
+
+    ![](<./screenshots/Screenshot 2024-01-19 at 3.41.26 PM.png>)
+* Kick off `Windows.KapeFiles.Targets` artifact collection in the LimaCharlie Velociraptor extension. 
+  * Argument options:
+    * `EventLogs=Y` - quicker processing time for proof of concept
+    * `KapeTriage=Y` - typically takes much longer 
+
+  ![](<./screenshots/Screenshot 2024-01-22 at 2.57.34 PM.png>)
+* You can watch the `Live Feed` for your `ext-velociraptor` adapter to see incoming activity -- you will see `velociraptor_collection` events come in when triage artifacts have completed and will soon be sent to your webhook output for processing
+
+    ![](<./screenshots/Screenshot 2024-01-19 at 3.59.28 PM.png>)
+* You can see the data being sent through your output by clicking `View Samples` on the outputs screen
+    * This JSON is what is being sent to your webhook, and you can see what parts of it we are using in the `webhook/hooks.json` file
+
+    ![](<./screenshots/Screenshot 2024-01-19 at 4.00.43 PM.png>)
+* If there are any errors sending data to your webhook, you will see them under `Platform Logs` -> `Error`
+* If you have Slack notifications enabled in the webhook service, you will get progress updates in Slack
+* Plaso files tend to take a while to generate--once the plaso file has been generated, it will begin importing into Timesketch. You will be able to see the import progress in the Timesketch GUI.

bash/run.sh

@@ -0,0 +1,106 @@
+#!/bin/bash
+
+if [[ $SLACK_NOTIFICATIONS == "yes" ]]; then
+    curl -X POST -H 'Content-type: application/json' --data '{"text":"Artifact event recieved, downloading artifact"}' $SLACK_WEBHOOK_URL
+fi
+
+ADMIN=$TIMESKETCH_USER
+PW=$TIMESKETCH_PW
+ZIP=$(python3 /opt/lcvr-to-timesketch/python/get_artifact.py $OID $SID $PAYLOAD_ID)
+PARENT_DATA_DIR="/opt/timesketch/upload"
+
+if [[ $SLACK_NOTIFICATIONS == "yes" ]]; then
+    curl -X POST -H 'Content-type: application/json' --data '{"text":"Artifact downloaded..."}' $SLACK_WEBHOOK_URL
+fi
+
+if [[ $ZIP == *".zip.gz"* ]]; then
+    # Get system name and params
+    SYSTEM=${ZIP%.zip.gz}
+    HOSTNAME=$(echo $SYSTEM|cut -d"_" -f 1)
+    OID=$(echo $SYSTEM|cut -d"_" -f 2)
+    FILENAME=$(echo $SYSTEM|cut -d"_" -f 3)
+
+    # Gunzip
+    echo A | gunzip $PARENT_DATA_DIR/$ZIP
+
+    # Unzip
+    echo A | unzip $PARENT_DATA_DIR/$SYSTEM.zip -d $PARENT_DATA_DIR/$SYSTEM
+elif [[ $ZIP == *".zip"* ]]; then
+    # Get system name and params
+    SYSTEM=${ZIP%.*}
+    HOSTNAME=$(echo $SYSTEM|cut -d"_" -f 1)
+    OID=$(echo $SYSTEM|cut -d"_" -f 2)
+    FILENAME=$(echo $SYSTEM|cut -d"_" -f 3)
+
+    # Unzip
+    echo A | unzip $PARENT_DATA_DIR/$ZIP -d $PARENT_DATA_DIR/$SYSTEM
+fi
+
+if [ -d $PARENT_DATA_DIR/$SYSTEM/uploads ]; then
+
+    if [[ $SLACK_NOTIFICATIONS == "yes" ]]; then
+    curl -X POST -H 'Content-type: application/json' --data '{"text":"Triage artifact unzipped, cleaning up files"}' $SLACK_WEBHOOK_URL
+    fi
+
+    # Remove collection data from subdir
+    mv $PARENT_DATA_DIR/$SYSTEM/uploads/* $PARENT_DATA_DIR/$SYSTEM/
+
+    # Delete unnecessary collection data
+    rm -r $PARENT_DATA_DIR/$SYSTEM/results $PARENT_DATA_DIR/$SYSTEM/uploads.json* $PARENT_DATA_DIR/$SYSTEM/uploads $PARENT_DATA_DIR/$SYSTEM/log* $PARENT_DATA_DIR/$SYSTEM/collection* $PARENT_DATA_DIR/$SYSTEM/requests.json
+
+    if [[ $SLACK_NOTIFICATIONS == "yes" ]]; then
+    curl -X POST -H 'Content-type: application/json' --data '{"text":"Triage files organized, generating plaso file"}' $SLACK_WEBHOOK_URL
+    fi
+
+    # Run log2timeline and generate Plaso file
+    docker exec -i timesketch-worker /bin/bash -c "log2timeline.py --status_view window --storage_file /usr/share/timesketch/upload/plaso/$SYSTEM.plaso /usr/share/timesketch/upload/$SYSTEM"
+
+    # Wait for file to become available
+    sleep 40
+
+    # Get ID of sketch if it exists, otherwise create new sketch with OID
+    SKETCHES=`docker exec -i timesketch-web tsctl list-sketches`
+    while IFS= read -r line; do
+        name=`echo $line|cut -f 2 -d " "`
+        id=`echo $line|cut -f 1 -d " "`
+        if [[ "$name" == "$OID" ]]; then
+                SKETCH_ID=$id
+        else
+                SKETCH_ID="none"
+        fi
+    done <<< "$SKETCHES"
+
+    if [[ $SLACK_NOTIFICATIONS == "yes" ]]; then
+    curl -X POST -H 'Content-type: application/json' --data '{"text":"Plaso file generated, importing into Timesketch--progress will be visible in the Timesketch GUI"}' $SLACK_WEBHOOK_URL
+    fi
+
+    # Run timesketch_importer to import Plaso data into Timesketch
+    if [[ "$SKETCH_ID" == "none" ]]; then
+        docker exec -i timesketch-worker /bin/bash -c "timesketch_importer -u $ADMIN  -p $PW --host http://timesketch-web:5000 --timeline_name $HOSTNAME-$FILENAME --sketch_name $OID /usr/share/timesketch/upload/plaso/$SYSTEM.plaso"
+
+        # Get new ID of sketch 
+        SKETCHES=`docker exec -i timesketch-web tsctl list-sketches`
+        while IFS= read -r line; do
+            name=`echo $line|cut -f 2 -d " "`
+            id=`echo $line|cut -f 1 -d " "`
+            if [[ "$name" == "$OID" ]]; then
+                    SKETCH_ID=$id
+            fi
+        done <<< "$SKETCHES"
+    else
+        docker exec timesketch-worker /bin/bash -c "timesketch_importer -u $ADMIN -p $PW --host http://timesketch-web:5000 --timeline_name $HOSTNAME-$FILENAME --sketch_id $SKETCH_ID /usr/share/timesketch/upload/plaso/$SYSTEM.plaso"
+    fi
+
+    if [[ $SLACK_NOTIFICATIONS == "yes" ]]; then
+    curl -X POST -H 'Content-type: application/json' --data '{"text":"Finished importing plaso file into Timesketch - http://'$WEBHOOK_IP'/sketch/'$SKETCH_ID'/explore"}' $SLACK_WEBHOOK_URL
+    fi
+
+else 
+
+    echo "base64 contents of collection downloaded to $PARENT_DATA_DIR/$SYSTEM"
+
+    if [[ $SLACK_NOTIFICATIONS == "yes" ]]; then
+    curl -X POST -H 'Content-type: application/json' --data '{"text":"Not a triage artifact - payload converted and saved to '$PARENT_DATA_DIR/$SYSTEM'"}' $SLACK_WEBHOOK_URL
+    fi
+
+fi

limacharlie/output.yaml

@@ -0,0 +1,7 @@
+version: 3
+outputs:
+    artifacts-tailored:
+        dest_host: http://WEBHOOK_IP:WEBHOOK_PORT/hooks/kick-off-timesketch
+        module: webhook
+        name: artifacts-tailored
+        type: tailored

limacharlie/rules.yaml

@@ -0,0 +1,26 @@
+version: 3
+hives:
+    dr-general:
+        artifacts-to-output:
+            data:
+                detect:
+                    op: is
+                    path: routing/log_type
+                    target: artifact_event
+                    value: velociraptor
+                respond:
+                    - action: output
+                      name: artifacts-tailored
+                      suppression:
+                        is_global: false
+                        keys:
+                            - '{{ .event.original_path }}'
+                            - '{{ .routing.log_id }}'
+                        max_count: 1
+                        period: 1m
+                    - action: report
+                      name: VR artifact ingested
+            usr_mtd:
+                enabled: true
+                expiry: 0
+                tags: []

python/get_artifact.py

@@ -0,0 +1,101 @@
+
+import os 
+import json 
+import requests 
+import base64
+import sys 
+import time 
+
+
+oid = sys.argv[1]
+sid = sys.argv[2]
+payload_id = sys.argv[3]
+api_key = os.getenv("LC_API_KEY")
+uid = os.getenv("LC_UID")
+
+
+def generate_org_jwt(oid):
+    base_url = "https://jwt.limacharlie.io"
+
+    url = "%s?uid=%s&secret=%s&oid=%s" % (base_url, uid, api_key, oid)
+
+    try:
+        r = requests.get(url)
+        jwt = r.json()["jwt"]
+        return jwt
+
+    except:
+        return ""
+
+
+def get_artifact(oid, artifact_id):
+    url = "%s/v1/insight/%s/artifacts/originals/%s" % (
+        "https://api.limacharlie.io",
+        oid,
+        artifact_id,
+    )
+
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": "Bearer %s" % (generate_org_jwt(oid)),
+    }
+
+    response = requests.request("GET", url, headers=headers)
+
+    # if this is a small/non triage artifact, we can just grab the payload base64 contents
+    try:
+        if "/" in json.loads(response.text)["path"]:
+            return "base64", json.loads(response.text)["payload"], json.loads(response.text)["path"].split("/")[-1]
+        else:
+            return "base64", json.loads(response.text)["payload"], json.loads(response.text)["path"].split("\\")[-1]
+    # triage artifacts get exported to google storage, so we will retrieve the url from the export param
+    except:
+        return "export", json.loads(response.text)["export"], json.loads(response.text)["path"].split("\\")[-1]
+
+
+def get_sensor(sid):
+    url = "%s/v1/%s" % (
+        "https://api.limacharlie.io",
+        sid,
+    )
+
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": "Bearer %s" % (generate_org_jwt(oid)),
+    }
+
+    response = requests.request("GET", url, headers=headers)
+
+    return json.loads(response.text)
+
+
+def convert_and_save(b64_string, file_name):
+
+    with open("/opt/timesketch/upload/%s_%s_%s.gz" % (get_sensor(sid)["info"]["hostname"], oid, file_name), "wb") as fh:
+        fh.write(base64.decodebytes(b64_string.encode() + b"=="))
+
+    return "%s_%s_%s.gz" % (get_sensor(sid)["info"]["hostname"], oid, file_name)
+
+
+def download_file(url, file_name):
+
+    filename = "/opt/timesketch/upload/%s_%s_%s" % (get_sensor(sid)["info"]["hostname"], oid, file_name)
+
+    response = requests.request("GET", url=url)
+
+    with open(filename, mode='wb') as localfile:
+        localfile.write(response.content)
+
+    return "%s_%s_%s" % (get_sensor(sid)["info"]["hostname"], oid, file_name)
+
+
+# get artifact from LC
+artifact_type, artifact, file_name = get_artifact(oid, payload_id)
+
+# if base64 artifact, convert and save gz file
+if artifact_type == "base64":
+    print(convert_and_save(artifact, file_name))
+# otherwise, download from google storage and process files in timesketch
+else:
+    time.sleep(10)
+    print(download_file(artifact, file_name))

systemd/webhook.service

@@ -0,0 +1,23 @@
+[Unit]
+Description=LCVR Webhook
+
+[Service]
+Type=simple
+User=root
+Restart=always
+RestartSec=10
+WorkingDirectory=/opt/lcvr-to-timesketch/webhook
+ExecStart=/usr/bin/webhook --hooks /opt/lcvr-to-timesketch/webhook/hooks.json -verbose
+ExecStop=/usr/bin/kill -9 $MAINPID
+StandardOutput=journal
+StandardError=journal
+Environment="TIMESKETCH_PW="
+Environment="TIMESKETCH_USER="
+Environment="LC_API_KEY="
+Environment="LC_UID="
+Environment="SLACK_WEBHOOK_URL="
+Environment="SLACK_NOTIFICATIONS=no"
+Environment="WEBHOOK_IP="
+
+[Install]
+WantedBy=multi-user.target

webhook/hooks.json

@@ -0,0 +1,25 @@
+[
+    {
+      "id": "kick-off-timesketch",
+      "execute-command": "/opt/lcvr-to-timesketch/bash/run.sh",
+      "command-working-directory": "/opt/lcvr-to-timesketch/bash",
+      "pass-environment-to-command": [
+        {
+          "envname": "OID",
+          "source": "payload",
+          "name": "routing.oid"
+        },
+        {
+          "envname": "PAYLOAD_ID",
+          "source": "payload",
+          "name": "routing.log_id"
+        },
+        {
+          "envname": "SID",
+          "source": "payload",
+          "name": "event.source"
+        }
+      ]
+    }
+  ]
+