Fix build and rename app

README.md

@@ -1,5 +1,5 @@
-#  DataSet Add-on for Splunk
-The DataSet Add-on for Splunk provides integration with [DataSet](https://www.dataset.com) and [XDR](https://www.sentinelone.com/platform/xdr-ingestion) by [SentinelOne](https://sentinelone.com). The key functions allow two-way integration:
+#  Security Data Lake Add-On for Splunk
+The Security Data Lake Add-On for Splunk provides integration with [DataSet](https://www.dataset.com) and [XDR](https://www.sentinelone.com/platform/xdr-ingestion) by [SentinelOne](https://sentinelone.com). The key functions allow two-way integration:
 - SPL custom command to query directly from the Splunk UI.
 - Inputs to index alerts as CIM-compliant, or any user-defined query results.
 - Alert action to send events from Splunk.
@@ -60,7 +60,7 @@ The add-on uses Splunk encrypted secrets storage, so admins require `admin_all_o
 
 3. Optionally, configure logging level and proxy information on the associated tabs.
 4. Click Save.
-5. The included DataSet by Example dashboard can be used to confirm connectivity and also shows example searches to get started.
+5. The included Security Data Lake by Example dashboard can be used to confirm connectivity and also shows example searches to get started.
 
 ## SPL Command
 The `| dataset` command allows queries against the [DataSet APIs](https://app.scalyr.com/help/api) directly from Splunk's search bar. 
@@ -174,7 +174,7 @@ Error saving configuration "CSRF validation failed" - This is a Splunk browser i
 
 Search errors `Account token error, review search log for details` or `Splunk configuration error, see search log for details.` - API token was unable to be retrieved. Common issues include user role missing list_storage_passwords permission, API token not set or incorrect account name given that has not been configured. Review job inspector search log for errors returned by Splunk. `Error retrieving account settings, error = UrlEncoded('broken')` indicates a likely misconfigured or incorrect account name; `splunklib.binding.HTTPError: HTTP 403 Forbidden -- You (user=username) do not have permission to perform this operation (requires capability: list_storage_passwords OR admin_all_objects)` indicates missing Splunk user permissions (list_storage_passwords).
 
-To troubleshoot the custom command, check the Job Inspector search log, also available in the internal index: `index=_internal app="TA-dataset" sourcetype=splunk_search_messages`.
+To troubleshoot the custom command, check the Job Inspector search log, also available in the internal index: `index=_internal app="TA_security_data_lake" sourcetype=splunk_search_messages`.
 
 For support, open a ticket with DataSet (or SentinelOne for XDR) support including any logged errors, or open a GitHub issue.
 
@@ -188,3 +188,54 @@ If Splunk events all show the same time, ensure results are returning a `timesta
 ##### Note
 This add-on was built with the [Splunk Add-on UCC framework](https://splunk.github.io/addonfactory-ucc-generator/) and uses the [Splunk Enterprise Python SDK](https://github.com/splunk/splunk-sdk-python).
 Splunk is a trademark or registered trademark of Splunk Inc. in the United States and other countries.
+
+# Development
+
+Since Splunk support Python 3.7 (deprecated as of June 2023), and cant be easily installed we use Python 3.8 to build app.
+In order to use use python 3.8 we use Python Virtual environment.
+
+## Prerequisites
+- Install [Splunk AppInspect CLI](https://dev.splunk.com/enterprise/docs/developapps/testvalidate/appinspect/useappinspectclitool)
+* Install Python 3.8 `brew install python@3.8`
+* Create venv with Python3.8: `python3.8 -m venv venv`
+* Activate it: `source venv/bin/activate`
+* Install dependencies:
+    * `pip install "cython<3.0.0"`
+    * `pip install wheel`
+    * `pip install --no-build-isolation pyyaml==5.4.1`
+    * `pip install --upgrade-strategy only-if-needed splunk-appinspect`
+    * `pip install --upgrade-strategy only-if-needed splunk-add-on-ucc-framework`
+    * `pip install --upgrade-strategy only-if-needed splunk-packaging-toolkit`
+
+## Build App
+- `ucc-gen build --source TA_security_data_lake --ta-version 2.0.1`
+- `slim package output/TA_security_data_lake -o release`
+
+## Run Docker Splunk locally (Mac M1 machines)
+Since Splunk does not have [Docker image for Apple Sillicon](https://github.com/splunk/docker-splunk/issues/493) you may need to
+- Use Docker Desktop 4.16.1 or newer and enable `Rosetta` in `Features in Development`, see [more](https://levelup.gitconnected.com/docker-on-apple-silicon-mac-how-to-run-x86-containers-with-rosetta-2-4a679913a0d5)
+- Run Splunk Docker image with `--platform=linux/amd64` parameter, eg:
+```docker run -it -e SPLUNK_START_ARGS=--accept-license -e SPLUNK_PASSWORD=Test0101 --platform=linux/amd64 --name splunk -p 8000:8000 splunk/splunk:latest start```
+
+To clean up container run `docker container rm splunk` command
+
+## Install Security Data Lake Add-On for Splunk to running Docker container
+Assuming application was previously built
+
+### From existing release
+- `docker cp release/TA_security_data_lake-2.0.0-Rxxx.tar.gz  splunk:/opt/splunk/etc/apps/`
+- `docker exec splunk sudo tar -xvzf /opt/splunk/etc/apps/TA_security_data_lake-2.0.0-Rxxx.tar.gz -C /opt/splunk/etc/apps/`
+- `docker exec splunk sudo chown -R splunk:splunk /opt/splunk/etc/apps/TA_security_data_lake/`
+- `docker exec splunk sudo -u splunk /opt/splunk/bin/splunk restart`
+
+### Using mounted volume from built app
+- Mount folder with built app `docker run -it -v "$(pwd)/output/TA_security_data_lake:/opt/splunk/etc/apps/TA_security_data_lake/" -e SPLUNK_START_ARGS=--accept-license -e SPLUNK_PASSWORD=Test0101 --platform=linux/amd64 --name splunk -p 8000:8000 splunk/splunk:latest start`
+
+To apply changes build app again `ucc-gen build --source TA_security_data_lake`
+- Changes in python scripts take effect immediately without any restart
+- Changes in static files like XML template take effect after restart `docker exec splunk sudo -u splunk /opt/splunk/bin/splunk restart`
+
+Once application is installed create connection to DataSet environment under `Configuration` tab using `Add` button.
+Note that build cleans previously created configuration. To prevent removal of configuration while build 
+- copy configured configuration to home folder `mkdir -p ~/splunk_dataset_app_configuration && cp -R ./output/TA_security_data_lake/local/* ~/splunk_dataset_app_configuration/`
+- copy back to splunk `mkdir -p ./output/TA_security_data_lake/local/ && cp -R ~/splunk_dataset_app_configuration/* ./output/TA_security_data_lake/local/`

Splunk Dashboards/dataset_by_example.xml

@@ -1,5 +1,5 @@
 <form version="1.1">
-  <label>DataSet by Example</label>
+  <label>Security Data Lake by Example</label>
   <init>
     <set token="myMaxCount">maxcount=10</set>
     <set token="baseQuery"></set>
@@ -260,4 +260,4 @@
       </chart>
     </panel>
   </row>
-</form>
+</form>

package/README.md → TA_dataset/README.md

@@ -1,7 +1,7 @@
-#  TA-dataset
+#  TA_dataset
 This add-on integrates with [DataSet](https://www.dataset.com) by [SentinelOne](https://www.sentinelone.com).
 
 For more information, see the [GitHub](https://github.com/scalyr/dataset-addon-for-splunk) repository.
 ##### Note
 This add-on was built with the [Splunk Add-on UCC framework](https://splunk.github.io/addonfactory-ucc-generator/).
-Splunk is a registered trademark of Splunk Inc. in the United States and other countries.
+Splunk is a registered trademark of Splunk Inc. in the United States and other countries.

package/app.manifest → TA_dataset/app.manifest

@@ -1,11 +1,11 @@
 {
   "schemaVersion": "2.0.0",
   "info": {
-    "title": "DataSet Add-on for Splunk",
+    "title": "Security Data Lake Add-On for Splunk",
     "id": {
       "group": null,
-      "name": "TA-dataset",
-      "version": "2.0.0"
+      "name": "TA_dataset",
+      "version": "2.0.1"
     },
     "author": [
       {
@@ -15,7 +15,7 @@
       }
     ],
     "releaseDate": null,
-    "description": "The DataSet Add-on for Splunk provides integration with DataSet by SentinelOne.",
+    "description": "The Security Data Lake Add-On for Splunk provides integration with DataSet by SentinelOne.",
     "classification": {
       "intendedAudience": "IT",
       "categories": ["Security, Fraud & Compliance", "IT Operations", "Business Analytics"],
@@ -55,5 +55,6 @@
     "splunk": {
       "Enterprise": "*"
     }
-  }
-}
+  },
+  "targetWorkloads" : ["*"]
+}

package/bin/dataset_common.py → TA_dataset/bin/dataset_common.py

@@ -196,4 +196,4 @@ def relative_to_epoch(relative):
         num_seconds = num_seconds * 60 * 60 * 24
 
     time_relative = time_current - (relative_num * num_seconds)
-    return time_relative
+    return time_relative

package/bin/dataset_event.py → TA_dataset/bin/dataset_event.py

@@ -51,5 +51,5 @@ def process_event(self, *args, **kwargs):
         return status
 
 if __name__ == "__main__":
-    exitcode = AlertActionWorkerdataset_event("TA-dataset", "dataset_event").run(sys.argv)
-    sys.exit(exitcode)
+    exitcode = AlertActionWorkerdataset_event("TA_dataset", "dataset_event").run(sys.argv)
+    sys.exit(exitcode)

package/bin/import_declare_test.py → TA_dataset/bin/import_declare_test.py

@@ -4,7 +4,7 @@
 import re
 from os.path import dirname
 
-ta_name = 'TA-dataset'
+ta_name = 'TA_dataset'
 pattern = re.compile(r'[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$')
 new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
 new_paths.append(os.path.join(dirname(dirname(__file__)), "lib"))

package/default/app.conf → TA_dataset/default/app.conf

@@ -6,21 +6,21 @@ build = 1
 
 [launcher]
 author = DataSet by SentinelOne
-version = 2.0.0
-description = The DataSet Add-on for Splunk integrates with DataSet by SentinelOne
+version = 2.0.1
+description = The Security Data Lake Add-On for Splunk integrates with DataSet by SentinelOne
 
 [ui]
 is_visible = 1
-label = DataSet Add-on for Splunk
+label = Security Data Lake Add-On for Splunk
 
 [package]
-id = TA-dataset
+id = TA_dataset
 
 [triggers]
 reload.ta_dataset_account = simple
 reload.ta_dataset_settings = simple
 reload.passwords = simple
 
 [id]
-name = TA-dataset
-version = 2.0.0
+name = TA_dataset
+version = 2.0.1

globalConfig.json

@@ -1,8 +1,8 @@
 {
     "meta": {
-        "name": "TA-dataset",
-        "displayName": "DataSet Add-on for Splunk",
-        "version": "2.0.0",
+        "name": "TA_dataset",
+        "displayName": "Security Data Lake Add-On for Splunk",
+        "version": "2.0.1",
         "restRoot": "TA_dataset",
         "schemaVersion": "0.0.3"
     },