Skip to content

[DPDV-3497] Upgrade splunk-appinspect to the latest stable version, fix lint #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Sep 29, 2023
Merged

[DPDV-3497] Upgrade splunk-appinspect to the latest stable version, fix lint #86

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
bec544e
Upgrade splunk-appinspect to the latest stable version.
tomaz-s1 Sep 29, 2023
1b562c4
Try upgrading pyyaml since splunk appinspect depends on a new version.
tomaz-s1 Sep 29, 2023
ecf3a58
Update e2e test which was failing since the element text has been
tomaz-s1 Sep 29, 2023
0884680
Update contribution file with information on installing needed "libma…
tomaz-s1 Sep 29, 2023
3e0f851
Add info on gnu-getopt dependency.
tomaz-s1 Sep 29, 2023
5f9bb8c
Update more out of date e2e selections.
tomaz-s1 Sep 29, 2023
06b0c89
Add a comment on slim validate false positive warning.
tomaz-s1 Sep 29, 2023
ce74a0d
Fix various lint violations.
tomaz-s1 Sep 29, 2023
a40e1cc
Revert "Update more out of date e2e selections."
tomaz-s1 Sep 29, 2023
8606078
Revert "Update e2e test which was failing since the element text has …
tomaz-s1 Sep 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,12 @@ You need Python 3.8 for development and package building.
sudo make dev-install-dependencies-pack-sudo
# install library dependencies
make dev-install-dependencies-lib
# If you get "ImportError: failed to find libmagic. Check your installation" error when
# running inspect target, you also need to install libmagic dependency
brew install libmagic
# If you get "getopt: illegal option -- n" this means you don't have GNU getopt tool installed
# You can install it using command below
brew install gnu-getopt
```
* Test it:
* By packing the application: `make pack`
Expand Down
6 changes: 3 additions & 3 deletions Splunk Dashboards/ingestion_summary.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
<label>Ingestion Summary</label>
<description>This dashboard is provided to estimate daily ingestion for various sources of data in Splunk.</description>
<search id="base_search">
<query>index="_internal" source="*metrics.log" group="per_sourcetype_thruput"
<query>index="_internal" source="*metrics.log" group="per_sourcetype_thruput"
| bucket _time span=1d
| eval GB=kb/1024/1024
| stats sum(GB) as "GB Ingest" avg(GB) as "Average GB" max(GB) as "Max GB" avg(eps) as "Events per Second" by _time, series
| stats sum(GB) as "GB Ingest" avg(GB) as "Average GB" max(GB) as "Max GB" avg(eps) as "Events per Second" by _time, series
| stats sum("GB Ingest") as "Total Ingest(GB)", avg("GB Ingest") as "Daily Avg Ingest(GB)", max("GB Ingest") as "Daily Max Ingest(GB)" by series
| eval "Total Ingest(GB)"=round('Total Ingest(GB)',4), "Daily Avg Ingest(GB)"=round('Daily Avg Ingest(GB)',4), "Daily Max Ingest(GB)"=round('Daily Max Ingest(GB)',4), "Events per Second"=round('Events per Second',4)
| rename series as sourcetype
Expand Down Expand Up @@ -76,4 +76,4 @@
</table>
</panel>
</row>
</form>
</form>
8 changes: 4 additions & 4 deletions Splunk Dashboards/sdl_by_example.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@
<li>
<b>Timeseries Query</b> - This will calculate numeric values over time. For repeated queries, summaries allow precomputed results (fastest).</li>
</ol>
Depending on your use case, you may have a need for any or all of these.
Depending on your use case, you may have a need for any or all of these.
<h1>Let's get started searching! </h1>
</html>
</panel>
Expand Down Expand Up @@ -120,7 +120,7 @@
<fieldForLabel>serverHost</fieldForLabel>
<fieldForValue>serverHost</fieldForValue>
<search>
<query>| dataset method=facet field=serverHost search="serverHost=* "
<query>| dataset method=facet field=serverHost search="serverHost=* "
| spath
| rename value as serverHost
| table serverHost count
Expand Down Expand Up @@ -152,7 +152,7 @@
<table>
<title>SPL: | dataset method=query search="$baseQuery$" $myMaxCount$ | spath</title>
<search>
<query>| dataset method=query search="$baseQuery$" $myMaxCount$
<query>| dataset method=query search="$baseQuery$" $myMaxCount$
| spath</query>
<earliest>$myTime.earliest$</earliest>
<latest>$myTime.latest$</latest>
Expand Down Expand Up @@ -263,4 +263,4 @@
</chart>
</panel>
</row>
</form>
</form>
2 changes: 1 addition & 1 deletion Splunk Dashboards/sentinelone_use_case_query_examples.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,4 +85,4 @@
</table>
</panel>
</row>
</form>
</form>
32 changes: 16 additions & 16 deletions Splunk Dashboards/splunk_app_usage.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@
<search id="base_usage">
<query>
index=_internal sourcetype=splunk_web_access host=* user=*
| rex field=uri_path ".*/(?&lt;title&gt;[^/]*)$"
| rex field=uri_path ".*/(?&lt;title&gt;[^/]*)$"
| join title
[| rest /servicesNS/-/-/data/ui/views splunk_server=*
| search isDashboard=1 isVisible=1
| rename eai:acl.app as app
| search isDashboard=1 isVisible=1
| rename eai:acl.app as app
| fields title app ]
| rename title as dashboard
| search NOT app IN($myExcludedApps$)
Expand All @@ -20,14 +20,14 @@ index=_internal sourcetype=splunk_web_access host=* user=*
<latest>$myTime.latest$</latest>
</search>
<search id="base_panels">
<query>| rest /servicesNS/-/-/data/ui/views
<query>| rest /servicesNS/-/-/data/ui/views

``` get fields we want from all the dashboards for all the apps ```
| fields eai:acl.app label id eai:data
| fields eai:acl.app label id eai:data
| rename eai:acl.app as app_name, eai:data as xml_source, label as dashboard_title

``` now separate all the panels to individual events (rows) ```
| rex field=id "http(s)?://([^/]+/)+(?&lt;file_name&gt;[^/]+)"
| rex field=id "http(s)?://([^/]+/)+(?&lt;file_name&gt;[^/]+)"
| spath input=xml_source output=panel path=form.row.panel
| mvexpand panel
| fields app_name dashboard_title file_name panel xml_source
Expand All @@ -37,14 +37,14 @@ index=_internal sourcetype=splunk_web_access host=* user=*
| spath input=panel output=panel_title path=title
| xpath field=panel outfield=query "//*/search/query"
| xpath field=panel outfield=drilldown "//*/drilldown/*/link"
| rex field=panel "\s+\&lt;(?&lt;panel_type&gt;[^\&lt;]+)\&gt;"
| rex field=panel "\s+\&lt;(?&lt;panel_type&gt;[^\&lt;]+)\&gt;"

``` filter for the types of panels that typically have searches ```
| search panel_type IN ("chart","single","table","event","search","viz","map")
| search NOT app_name IN($myExcludedApps$)

``` total, sort and print the results ```
| eventstats count as total_panels, dc(dashboard_title) as total_dashboards
| eventstats count as total_panels, dc(dashboard_title) as total_dashboards
| eventstats dc(dashboard_title) as dashboards_in_app by app_name
| eventstats count as panels_on_dashboard by app_name, dashboard_title
| sort app_name file_name dashboard_title panel_title
Expand Down Expand Up @@ -218,12 +218,12 @@ index=_internal sourcetype=splunk_web_access host=* user=*
<title>Most Used SPL Commands</title>
<table>
<search>
<query>index=_audit action=search info=completed search=* NOT "search_id='scheduler" NOT "search=|history" NOT "user=splunk-system-user" NOT "search='typeahead" NOT "search='| metadata type=* | search totalCount&gt;0" app="*"
| fields search
| rex field=search "\|\s*(?&lt;command&gt;\w+)\s(?&lt;attributes&gt;[^|]*)" max_match=1000
| mvexpand command
<query>index=_audit action=search info=completed search=* NOT "search_id='scheduler" NOT "search=|history" NOT "user=splunk-system-user" NOT "search='typeahead" NOT "search='| metadata type=* | search totalCount&gt;0" app="*"
| fields search
| rex field=search "\|\s*(?&lt;command&gt;\w+)\s(?&lt;attributes&gt;[^|]*)" max_match=1000
| mvexpand command
| search command!=""
| stats count as "execution", distinct_count(search) as "distinct_searches" by command
| stats count as "execution", distinct_count(search) as "distinct_searches" by command
| sort -distinct_search, -execution
| rename command as "SPL Command"</query>
<earliest>$myTime.earliest$</earliest>
Expand Down Expand Up @@ -332,9 +332,9 @@ index=_internal sourcetype=splunk_web_access host=* user=*
| search NOT app_name IN(Splunk_Security_Essentials,lookup_editor,splunk_monitoring_console,splunk_secure_gateway,splunk_instrumentation,Splunk_SA_CIM)

``` total, sort and print the results ```
| eventstats count as total_searches
| eventstats count as total_searches
| eventstats dc(searches) as searches_in_app by app_name
| sort app_name title
| sort app_name title
| table app_name title search viz cron_schedule alert_type alert_comparator alert_threshold alert_condition
| search app_name = "*" search="*"
| dedup app_name, title, search
Expand All @@ -347,4 +347,4 @@ index=_internal sourcetype=splunk_web_access host=* user=*
</table>
</panel>
</row>
</form>
</form>
2 changes: 2 additions & 0 deletions TA_dataset/default/commands.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ filename = dataset_search_command.py
chunked = true
python.version = python3

# slim validate may complatin with "line 9: Undefined setting in commands.conf, stanza [s1query]: python.version"
# but that is non fatal, see https://community.splunk.com/t5/Building-for-the-Splunk-Platform/Why-does-slim-validate-complain-about-undefined-setting-python/m-p/539911 for details
[s1query]
filename = dataset_search_command.py
chunked = true
Expand Down
6 changes: 3 additions & 3 deletions TA_dataset/default/data/ui/views/ingestion_summary.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
<label>Ingestion Summary</label>
<description>This dashboard is provided to estimate daily ingestion for various sources of data in Splunk.</description>
<search id="base_search">
<query>index="_internal" source="*metrics.log" group="per_sourcetype_thruput"
<query>index="_internal" source="*metrics.log" group="per_sourcetype_thruput"
| bucket _time span=1d
| eval GB=kb/1024/1024
| stats sum(GB) as "GB Ingest" avg(GB) as "Average GB" max(GB) as "Max GB" avg(eps) as "Events per Second" by _time, series
| stats sum(GB) as "GB Ingest" avg(GB) as "Average GB" max(GB) as "Max GB" avg(eps) as "Events per Second" by _time, series
| stats sum("GB Ingest") as "Total Ingest(GB)", avg("GB Ingest") as "Daily Avg Ingest(GB)", max("GB Ingest") as "Daily Max Ingest(GB)" by series
| eval "Total Ingest(GB)"=round('Total Ingest(GB)',4), "Daily Avg Ingest(GB)"=round('Daily Avg Ingest(GB)',4), "Daily Max Ingest(GB)"=round('Daily Max Ingest(GB)',4), "Events per Second"=round('Events per Second',4)
| rename series as sourcetype
Expand Down Expand Up @@ -76,4 +76,4 @@
</table>
</panel>
</row>
</form>
</form>
8 changes: 4 additions & 4 deletions TA_dataset/default/data/ui/views/sdl_by_example.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@
<li>
<b>Timeseries Query</b> - This will calculate numeric values over time. For repeated queries, summaries allow precomputed results (fastest).</li>
</ol>
Depending on your use case, you may have a need for any or all of these.
Depending on your use case, you may have a need for any or all of these.
<h1>Let's get started searching! </h1>
</html>
</panel>
Expand Down Expand Up @@ -120,7 +120,7 @@
<fieldForLabel>serverHost</fieldForLabel>
<fieldForValue>serverHost</fieldForValue>
<search>
<query>| dataset method=facet field=serverHost search="serverHost=* "
<query>| dataset method=facet field=serverHost search="serverHost=* "
| spath
| rename value as serverHost
| table serverHost count
Expand Down Expand Up @@ -152,7 +152,7 @@
<table>
<title>SPL: | dataset method=query search="$baseQuery$" $myMaxCount$ | spath</title>
<search>
<query>| dataset method=query search="$baseQuery$" $myMaxCount$
<query>| dataset method=query search="$baseQuery$" $myMaxCount$
| spath</query>
<earliest>$myTime.earliest$</earliest>
<latest>$myTime.latest$</latest>
Expand Down Expand Up @@ -263,4 +263,4 @@
</chart>
</panel>
</row>
</form>
</form>
2 changes: 1 addition & 1 deletion TA_dataset/default/data/ui/views/sentinelone_use_case_query_examples.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,4 +85,4 @@
</table>
</panel>
</row>
</form>
</form>
2 changes: 1 addition & 1 deletion TA_dataset/default/data/ui/views/soc_search_examples.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,4 +117,4 @@
</chart>
</panel>
</row>
</dashboard>
</dashboard>
32 changes: 16 additions & 16 deletions TA_dataset/default/data/ui/views/splunk_app_usage.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@
<search id="base_usage">
<query>
index=_internal sourcetype=splunk_web_access host=* user=*
| rex field=uri_path ".*/(?&lt;title&gt;[^/]*)$"
| rex field=uri_path ".*/(?&lt;title&gt;[^/]*)$"
| join title
[| rest /servicesNS/-/-/data/ui/views splunk_server=*
| search isDashboard=1 isVisible=1
| rename eai:acl.app as app
| search isDashboard=1 isVisible=1
| rename eai:acl.app as app
| fields title app ]
| rename title as dashboard
| search NOT app IN($myExcludedApps$)
Expand All @@ -20,14 +20,14 @@ index=_internal sourcetype=splunk_web_access host=* user=*
<latest>$myTime.latest$</latest>
</search>
<search id="base_panels">
<query>| rest /servicesNS/-/-/data/ui/views
<query>| rest /servicesNS/-/-/data/ui/views

``` get fields we want from all the dashboards for all the apps ```
| fields eai:acl.app label id eai:data
| fields eai:acl.app label id eai:data
| rename eai:acl.app as app_name, eai:data as xml_source, label as dashboard_title

``` now separate all the panels to individual events (rows) ```
| rex field=id "http(s)?://([^/]+/)+(?&lt;file_name&gt;[^/]+)"
| rex field=id "http(s)?://([^/]+/)+(?&lt;file_name&gt;[^/]+)"
| spath input=xml_source output=panel path=form.row.panel
| mvexpand panel
| fields app_name dashboard_title file_name panel xml_source
Expand All @@ -37,14 +37,14 @@ index=_internal sourcetype=splunk_web_access host=* user=*
| spath input=panel output=panel_title path=title
| xpath field=panel outfield=query "//*/search/query"
| xpath field=panel outfield=drilldown "//*/drilldown/*/link"
| rex field=panel "\s+\&lt;(?&lt;panel_type&gt;[^\&lt;]+)\&gt;"
| rex field=panel "\s+\&lt;(?&lt;panel_type&gt;[^\&lt;]+)\&gt;"

``` filter for the types of panels that typically have searches ```
| search panel_type IN ("chart","single","table","event","search","viz","map")
| search NOT app_name IN($myExcludedApps$)

``` total, sort and print the results ```
| eventstats count as total_panels, dc(dashboard_title) as total_dashboards
| eventstats count as total_panels, dc(dashboard_title) as total_dashboards
| eventstats dc(dashboard_title) as dashboards_in_app by app_name
| eventstats count as panels_on_dashboard by app_name, dashboard_title
| sort app_name file_name dashboard_title panel_title
Expand Down Expand Up @@ -218,12 +218,12 @@ index=_internal sourcetype=splunk_web_access host=* user=*
<title>Most Used SPL Commands</title>
<table>
<search>
<query>index=_audit action=search info=completed search=* NOT "search_id='scheduler" NOT "search=|history" NOT "user=splunk-system-user" NOT "search='typeahead" NOT "search='| metadata type=* | search totalCount&gt;0" app="*"
| fields search
| rex field=search "\|\s*(?&lt;command&gt;\w+)\s(?&lt;attributes&gt;[^|]*)" max_match=1000
| mvexpand command
<query>index=_audit action=search info=completed search=* NOT "search_id='scheduler" NOT "search=|history" NOT "user=splunk-system-user" NOT "search='typeahead" NOT "search='| metadata type=* | search totalCount&gt;0" app="*"
| fields search
| rex field=search "\|\s*(?&lt;command&gt;\w+)\s(?&lt;attributes&gt;[^|]*)" max_match=1000
| mvexpand command
| search command!=""
| stats count as "execution", distinct_count(search) as "distinct_searches" by command
| stats count as "execution", distinct_count(search) as "distinct_searches" by command
| sort -distinct_search, -execution
| rename command as "SPL Command"</query>
<earliest>$myTime.earliest$</earliest>
Expand Down Expand Up @@ -332,9 +332,9 @@ index=_internal sourcetype=splunk_web_access host=* user=*
| search NOT app_name IN(Splunk_Security_Essentials,lookup_editor,splunk_monitoring_console,splunk_secure_gateway,splunk_instrumentation,Splunk_SA_CIM)

``` total, sort and print the results ```
| eventstats count as total_searches
| eventstats count as total_searches
| eventstats dc(searches) as searches_in_app by app_name
| sort app_name title
| sort app_name title
| table app_name title search viz cron_schedule alert_type alert_comparator alert_threshold alert_condition
| search app_name = "*" search="*"
| dedup app_name, title, search
Expand All @@ -347,4 +347,4 @@ index=_internal sourcetype=splunk_web_access host=* user=*
</table>
</panel>
</row>
</form>
</form>
4 changes: 2 additions & 2 deletions requirements-pack.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ cython<3.0.0; python_version <= '3.8'
# splunk is using 3.7, but it's deprecated
# we use 3.8 for development
pip==23.2.1; python_version <= '3.8'
pyyaml==5.4.1; python_version <= '3.8'
pyyaml==6.0.1; python_version <= '3.8'
splunk-add-on-ucc-framework==5.13.2; python_version <= '3.8'
splunk-appinspect==2.35.0; python_version <= '3.8'
splunk-appinspect==2.37.2; python_version <= '3.8'
wheel==0.41.0; python_version <= '3.8'