DataSet Add-on for Splunk

The DataSet Add-on for Splunk provides integration with DataSet by SentinelOne. The key functions allow two-way integration:

• SPL custom command to query DataSet directly from the Splunk UI without having to reindex data to Splunk.
• Inputs to index alerts as CIM-compliant, or user-defined query results, from DataSet to Splunk.
• Alert action to send events from Splunk to DataSet.

Installation

The included .tgz file is ready for installation in Splunk. The package subdirectory contains all artifacts. To compile, reference Splunk's UCC Framework instructions to use ucc-gen and slim package.

Reference Splunk documentation for installing add-ons. Note admins require admin_all_objects to create secret storage objects and users require list_storage_passwords capability to retrieve secrets.

Splunk Enterprise

Splunk component	Required	Comments
Search heads	Yes	Required to use the custom search command.
Indexers	No	Parsing is performed during data collection.
Forwarders	Yes	For distributed deployments, this add-on requires heavy forwarders for modular inputs.

Splunk Cloud

Splunk component	Required	Comments
Search heads	Yes	Required to use the custom search command. Splunk Cloud Victoria Experience also handles modular inputs on the search heads.
Indexers	No	Parsing is performed during data collection.
Inputs Data Manager	Yes	For Splunk Cloud Classic Experience, this add-on requires an IDM for modular inputs.

Configuration

Dataset

1. Navigate to API Keys.

2. Click Add Key > Add Read Key (required for search command and inputs).
3. Click Add Key > Add Write Key (required for alert action).
4. Optionally, click the pencil icon to rename the keys.

Splunk

1. In Splunk, open the Add-on

2. In configuration on DataSet Account tab:

• Enter the DataSet URL (e.g.: https://app.scalyr.com).
• Enter the DataSet read key from above.
• Enter the DataSet write key from above.

3. Optionally, configure logging level and proxy information on the associated tabs.
4. Click Save.
5. To confirm connectivity, simply search |dataset and validate results.

SPL Command

The | dataset command allows queries against the DataSet API directly from Splunk's search bar.

Optional parameters are supported:

• method - Define query, powerquery, facet or timeseries to call the appropriate REST endpoint. Default is query.
• query - The DataSet query or filter used to select events. Default is no filter (return all events limited by time and maxCount).
• starttime - The Splunk time picker can be used (not "All Time"), but if starttime is defined it will take precedence to define the start time for DataSet events to return. Use epoch time or relative shorthand in the form of a number followed by d, h, m or s (for days, hours, minutes or seconds), e.g.: 24h. Default is 24h.
• endtime - The Splunk time picker can be used (not "All Time"), but if endtime is defined it will take precedence to define the end time for DataSet events to return. Use epoch time or relative shorthand in the form of a number followed by d, h, m or s (for days, hours, minutes or seconds), e.g.: 5m. Default is current time at search.

For query and powerquery:

• maxcount - Number of events to return.
• columns - Specified fields to return from DataSet query (or powerquery, analogous to using | columns in a powerquery). Yields performance gains for high volume queries instead of returning and merging all fields.

For facet:

• field - Define field to get most frequent values of. Default is logfile.

For timeseries:

• function - Define value to compute from matching events. Default is rate.
• buckets - The number of numeric values to return by dividing time range into equal slices. Default is 1.
• createsummaries - Specify whether to create summaries to automatically update on ingestion pipeline. Default is true; recommend setting to false for one-off or while testing new queries.
• useonlysummaries - Specify whether to only use preexisting timeseries for fastest speed.

For all queries, be sure to "wrap the entire query in double quotes, and use 'single quotes' inside" or double quotes \"escaped with a backslash\", as shown in the following examples.

Query Example: | dataset method=query search="serverHost = * AND Action = 'allow'" maxcount=50 starttime=10m endtime=1m

Power Query Example 1: | dataset method=powerquery search="dataset = \"accesslog\" | group requests = count(), errors = count(status == 404) by uriPath | let rate = errors / requests | filter rate > 0.01 | sort -rate"

Power Query Example 2: | dataset method=powerQuery search="$serverHost == 'cloudWatchLogs' | parse 'RequestId: $RID$ Duration: $DUR$ ms Billed Duration: $BDUR$ ms Memory Size: $MEM$ MB Max Memory Used: $UMEM$ MB' | let deltaDUR= BDUR - DUR, deltaMEM = MEM - UMEM | sort -DUR | columns 'Request ID' = RID, 'Duration(ms)' = DUR, 'Charged delta (ms)' = deltaDUR, 'Used Memory (MB)' = UMEM, 'Charged delta Memory (MB)' = deltaMEM" starttime=5m

Facet Query Example: | dataset method=facet search="serverHost = *" field=serverHost maxcount=25 | spath | table value, count

Timeseries Query Example: | dataset method=timeseries search="serverHost='scalyr-metalog'" function="p90(delayMedian)" starttime="24h" buckets=24 createsummaries=false onlyusesummaries=false

Since events are returned in JSON format, the Splunk spath command is useful. Additionally, the Splunk collect command can be used to add the events to a summary index:

| dataset query="serverHost = * AND Action = 'allow'" maxcount=50 starttime=10m endtime=1m | spath | collect index=dataset

Inputs

For use cases requiring data indexed in Splunk, optional inputs are provided utilizing time-based checkpointing to prevent reindexing the same data:

Source Type	Description	CIM Data Model
dataset:alerts	Predefined Power Query API call to index alert state change records	Alerts
dataset:query	User-defined standard query API call to index events	-
dataset:powerquery	User-defined PowerQuery API call to index events	-

1. On the inputs page, click Create New Input and select the desired input

2. For DataSet alerts, enter:

• A name for the input.
• Interval, in seconds. A good starting point is 300 seconds to collect every five mintues.
• Splunk index name
• Start time, in relative shorthand form, e.g.: 24h for 24 hours before input execution time.

3. For DataSet queries, enter:

• A name for the input.
• Interval, in seconds. A good starting point is 300 seconds to collect every five mintues.
• Splunk index name
• Start time, in relative shorthand form, e.g.: 24h for 24 hours before input execution time.
• (optional) End time, in relative shorthand form, e.g.: 5m for 5 minutes before input execution time.
• (optional) Query string used to return matching events.
• (optional) Maximum number of events to return.

4. For DataSet Power Queries, enter:

• A name for the input.
• Interval, in seconds. A good starting point is 300 seconds to collect every five mintues.
• Splunk index name
• Start time, in relative shorthand form, e.g.: 24h for 24 hours before input execution time.
• (optional) End time, in relative shorthand form, e.g.: 5m for 5 minutes before input execution time.
• Query string used to return matching events, including commands such as | columns, | limit, etc.

Alert Action

An alert action allows sending an event to the DataSet addEvents API.

Support and troubleshooting

Error saving configuration "CSRF validation failed" - this is a Splunk browser issue; try using a private window or clearing cache and cookies then retrying.

To troubleshoot the custom command, check the Job Inspector search log, also available in the internal index: index=_internal app="TA-dataset" sourcetype=splunk_search_messages. Common issues include incorrect API key or firewalls blocking outbound traffic on port 443.

For support, open a ticket with support, or open a GitHub issue.

Note

This add-on was built with the Splunk Add-on UCC framework. Splunk is a trademark or registered trademark of Splunk Inc. in the United States and other countries.