BREAKING CHANGE: remove btool and replace it with splunk.rest #441

sgoral-splunk · 2025-05-29T20:05:34Z

Issue number: ADDON-79884

PR Type

What kind of change does this PR introduce?

Summary

Changes

solnlib will no longer use Splunk's btool. btool was replaced with configuration endpoint from Splunk splunk.rest module. Since session_key is required for an authorization, solnlib no longer supports get_session_key() function. Also function make_splunkhome_path will now use native Splunk's implementation from splunk.clilib.bundle_paths

User experience

It's a breaking change for the user as functions related to getting stanza from config files will now required app_name and session_key to work properly.

If users uses get_session_key() from solnlib they will need to replace it with their own implementation.

Checklist

introduced new method _get_conf_stanzas_from_splunk_api
Following functions will now require app_name and/or session_key as a arguments to work:

get_conf_stanza
get_conf_key_value
get_splunkd_uri
get_scheme_from_hec_settings
get_splunkd_access_info
get_splunk_host_info

make_splunkhome_path() now uses splunk.clilib.bundle_paths.make_splunkhome_path

Review

self-review - I have performed a self-review of this change according to the development guidelines
Changes are documented. The documentation is understandable, examples work (more info)
PR title and description follows the contributing principles
meeting - I have scheduled a meeting or recorded a demo to explain these changes (if there is a video, put a link below and in the ticket)

Tests

See the testing doc.

Unit - tests have been added/modified to cover the changes
Smoke - tests have been added/modified to cover the changes
UI - tests have been added/modified to cover the changes
coverage - I have checked the code coverage of my changes (see more)

Demo/meeting:

Reviewers are encouraged to request meetings or demos if any part of the change is unclear

hetangmodi-crest · 2025-06-11T06:33:22Z

solnlib/_settings.py

+#
+"""This module provide settings that can be used to disable/switch features."""
+
+use_btool = False


I think we would require this setting to be passed from add-on developers, or provide a way to them to override this setting if they want to use btool.

The idea here was to make some kind of backdoor. It's not a feature that user can choose which apprach use, he will be forced to use api but in case of any issue on production env, admins will be able to temporary switch this flag.

Can we add a test where we switch this flag to True and then ensure the test suite passes, thus ensuring the same behaviour when some developers make this change in prod or pre-prod env?

I've added some test for that e.g. tests.unit.test_splunkenv.test_splunkd_uri_backdoor

solnlib/credentials.py

tests/integration/context.py

tests/unit/test_splunkenv.py

tests/unit/test_modular_input_event_writer.py

tests/unit/test_credentials.py

tests/unit/common.py

solnlib/splunkenv.py

artemrys · 2025-06-12T11:56:18Z

solnlib/splunkenv.py

+    server_response, server_content = simpleRequest(
+        url,
+        sessionKey=session_key,
+        getargs={"output_mode": "json"},
+    )
+
+    result = json.loads(server_content.decode())


this will raise an unexpected exception if simpleRequest is not defined (in case we operate outside of Splunk), do you expect splunkenv to be used outside of Splunk's environment?

hmm, I thought as this is splunkenv module maybe we should support it only in the splunk env? I've added some exception to notify user about this fact. I don't think this is actually used outside but if there will be some requests in future to extend it's functionality we can implement some request-based solution. What do you think?

let's update documentation for this module as well and we can live with it

There is already this information: "Additionally, the splunkenv module can only be used in environments where Splunk is installed, as it relies on Splunk-specific methods for making internal calls." Do you want me to extend it?

artemrys · 2025-06-12T11:56:57Z

solnlib/splunkenv.py

+    conf_name: str,
+    app_name: str,
+    session_key: Optional[str] = None,
+    user: Optional[str] = "nobody",


user: Optional[str] = None, OR user: str = "nobody",

artemrys · 2025-06-12T11:57:37Z

solnlib/splunkenv.py

-        result = op.join(_splunk_home(), ETC_LEAF)
-
-    return os.path.normpath(result)
+use_btool = bool(use_btool)


some historical change, removed

solnlib/modular_input/event_writer.py

solnlib/splunkenv.py

hetangmodi-crest · 2025-06-16T12:59:03Z

tests/unit/data/mock_splunk/etc/apps/splunk_httpinput/local/inputs.json

+    {
+      "content": {
+        "disabled": 0,
+        "enableSSL": 0


Shouldn't these two values be 1 as per the setting in tests/unit/data/mock_splunk/etc/apps/splunk_httpinput/local/inputs.conf?

this is just to test backdoor solution. We are no longer using btool so "valid" config is in the inputs.json, but I set different value here just to verify if use_btool is taking value from .conf.

hetangmodi-crest · 2025-06-16T12:59:22Z

tests/unit/data/mock_splunk/etc/system/default/server.conf

@@ -46,7 +46,7 @@ parallelIngestionPipelines = 1
 instanceType = download

 [sslConfig]
-enableSplunkdSSL = true
+enableSplunkdSSL = false


Any reason to set this to false?

this is just to test backdoor solution. We are no longer using btool so "valid" config is in the server-sslconfig.json, but I set different value here just to verify if use_btool is taking value from .conf.

hetangmodi-crest · 2025-06-16T13:01:56Z

tests/unit/data/mock_splunk/etc/system/default/web.json

+        "maxSockets": "0",
+        "maxThreads": "0",
+        "max_view_cache_size": "1000",
+        "mgmtHostPort": "127.0.0.1:8089",


Shouldn't this value be 127.0.0.2:8079 as per tests/unit/data/mock_splunk/etc/system/default/web.conf?

Same situation as above. I've changed value here to verify "use_btool" flag. Basically .conf files from unit/data/mock_splunkn/etc are no longer in use

artemrys · 2025-06-17T09:22:14Z

solnlib/splunkenv.py

Can't make comment on the line that was not changed...

Please take a look at on_shared_storage, it does not seem to be used in the project and # See validateSearchHeadPooling() in src/libbundle/ConfSettings.cpp is not correct anymore.

Yes, this should be removed as we are using make_splunkhome_path from splunk

artemrys · 2025-06-17T09:24:11Z

solnlib/splunkenv.py

+    if not session_key:
+        session_key = getSessionKey()


Do we then need to provide session_key at all if that is calculated for us in getSessionKey method from Splunk directly? I remember there was a case where it did not work quite well, I think it's worth describing it here.

Basically the problem is that splunk's getSessionKey or main module have access to session key if it's used in the REST context but if used e.g. in modinput script, those function won't provide session key. I can add some comment line here

artemrys · 2025-06-17T09:59:01Z

solnlib/splunkenv.py

@@ -36,6 +63,7 @@
    "get_conf_key_value",
    "get_conf_stanza",
    "get_conf_stanzas",
+    "_get_conf_stanzas_from_splunk_api",


Why expose it here?

right, sorry. It shoudn't be expose here. I will make changes

artemrys · 2025-06-17T09:59:56Z

solnlib/splunkenv.py

+    if not session_key and hasattr(__main__, "___sessionKey"):
+        session_key = getattr(__main__, "___sessionKey")


What is __main__ in this case?

it depends on which module, that is using this splunkenv functions, was started. E.g. for the rest_handlers modules MConfigHandler is setting this ___sessionKey attribute so it's available here. If mod input started it, ___sessionKey will be missing and that's why user will have to provide it

artemrys · 2025-06-17T10:00:11Z

solnlib/splunkenv.py

+        session_key = getattr(__main__, "___sessionKey")
+
+    if not session_key:
+        raise SessionKeyNotFound()


Any recommendations in this case?

right, we can add information "Session key is missing. If you are using 'splunkenv' module in your TA, please ensure you are providing session_key to it's functions. For more information please see: https://splunk.github.io/addonfactory-solutions-library-python/release_7_0_0/"

artemrys

LGTM overall, I did not review every single line but I am fine if we follow up with couple of other PR after it's merged to develop.

srv-rr-github-token · 2025-06-19T10:11:03Z

🎉 This PR is included in version 6.3.0-beta.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

srv-rr-github-token · 2025-06-19T11:38:25Z

🎉 This PR is included in version 7.0.0-beta.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

srv-rr-github-token · 2025-06-19T13:19:46Z

🎉 This PR is included in version 7.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

sgoral-splunk and others added 9 commits May 23, 2025 09:21

feat: init commit, new function to retrieve stanzas based on rest

619a53e

feat: init commit, new function to retrieve stanzas based on rest

ae2b3ab

breaking change: replace btool with splunk rest

b069211

test: added splunk to sys.path

012e6c5

test: update sys.path for splunk libraries

087f2a7

test: add splunk's module patch

7cd7a9c

test: add splunk's module patch

dfb4aed

test: mock simpleRequest

48eeb6e

test: fix make splunk home

e2a5712

hetangmodi-crest reviewed Jun 11, 2025

View reviewed changes

hetangmodi-crest assigned sgoral-splunk Jun 12, 2025

artemrys reviewed Jun 12, 2025

View reviewed changes

chore: add docs, changes after review

519ad53

sgoral-splunk requested review from artemrys and hetangmodi-crest June 16, 2025 08:28

sgoral-splunk added 3 commits June 16, 2025 12:04

test: change procArgs for splunk cli_common

d291cd3

test: fix integration tests, add deprecation for create_from_token

713b3ef

chore: add ImportError to getSessionKey and make_splunkhome_path

eb152ef

sgoral-splunk marked this pull request as ready for review June 16, 2025 12:21

sgoral-splunk requested a review from a team as a code owner June 16, 2025 12:21

hetangmodi-crest reviewed Jun 16, 2025

View reviewed changes

chore: review fidings

ae17291

sgoral-splunk requested a review from hetangmodi-crest June 16, 2025 14:27

hetangmodi-crest previously approved these changes Jun 17, 2025

View reviewed changes

artemrys reviewed Jun 17, 2025

View reviewed changes

chore: changes after review

6214372

sgoral-splunk dismissed hetangmodi-crest’s stale review via 6214372 June 17, 2025 13:50

chore: fix precommit

aeb46d8

artemrys approved these changes Jun 18, 2025

View reviewed changes

hetangmodi-crest merged commit 13dcd5f into develop Jun 19, 2025
19 checks passed

hetangmodi-crest deleted the breaking/change_btool_to_rest_with_backup branch June 19, 2025 10:04

github-actions bot locked and limited conversation to collaborators Jun 19, 2025

srv-rr-github-token added the released on @beta label Jun 19, 2025

hetangmodi-crest restored the breaking/change_btool_to_rest_with_backup branch June 19, 2025 10:13

srv-rr-github-token added the released label Jun 19, 2025

		if not session_key and hasattr(__main__, "___sessionKey"):
		session_key = getattr(__main__, "___sessionKey")

BREAKING CHANGE: remove btool and replace it with splunk.rest #441

BREAKING CHANGE: remove btool and replace it with splunk.rest #441

Uh oh!

Conversation

sgoral-splunk commented May 29, 2025 • edited by artemrys Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Type

Summary

Changes

User experience

Checklist

Review

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sgoral-splunk Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hetangmodi-crest Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sgoral-splunk Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

sgoral-splunk commented May 29, 2025 •

edited by artemrys

Loading

sgoral-splunk Jun 11, 2025 •

edited

Loading

hetangmodi-crest Jun 12, 2025 •

edited

Loading

sgoral-splunk Jun 17, 2025 •

edited

Loading