feat: base code for supporting custom search command #1693

hetangmodi-crest · 2025-04-15T13:20:22Z

Issue number: ADDON-76780, ADDON-79646

PR Type

What kind of change does this PR introduce?

Summary

Changes

This PR contains the centralized code which is required for supporting the custom search command feature.

From this branch we will be having at least 3 separate PR's

PR for the generation of commands.conf -> feat: generation of commands.conf #1694
PR for the generation of searchbnf.conf -> feat: generation of searchbnf.conf #1695
PR for the generation of Python files -> feat: generation of python files for custom search command #1697

User experience

No change in the user experience as of now. As we have not covered generation of files in this PR.

Checklist

If an item doesn't apply to your changes, leave it unchecked.

Review

self-review - I have performed a self-review of this change according to the development guidelines
Changes are documented. The documentation is understandable, examples work (more info)
PR title and description follows the contributing principles
meeting - I have scheduled a meeting or recorded a demo to explain these changes (if there is a video, put a link below and in the ticket)

Tests

See the testing doc.

Unit - tests have been added/modified to cover the changes
Smoke - tests have been added/modified to cover the changes
UI - tests have been added/modified to cover the changes
coverage - I have checked the code coverage of my changes (see more)

Demo/meeting:

Reviewers are encouraged to request meetings or demos if any part of the change is unclear

artemrys · 2025-04-16T12:10:32Z

splunk_add_on_ucc_framework/global_config_validator.py

@@ -58,7 +65,7 @@ def _validate_config_against_schema(self) -> None:
        Validates config against JSON schema.
        Raises jsonschema.ValidationError if config is not valid.
        """
-        schema_path = os.path.join(self._source_dir, "schema", "schema.json")
+        schema_path = os.path.join(self._internal_root_dir, "schema", "schema.json")


Why this change?

The "source_dir" seemed an ambiguous variable name - does source_dir point to source code of add-on that is being built or UCC code source, hence, I have renamed the _source_dir to _internal_root_dir, otherwise there is no functionality change.

artemrys · 2025-04-16T12:11:49Z

tests/unit/test_const.py

+from splunk_add_on_ucc_framework.const import SPLUNK_COMMANDS
+
+
+def test_command_list_up_to_date():


Not sure it belongs to the unit tests. Maybe a script that eventually runs periodically.

@kkedziak-splunk @sgoral-splunk what do you think?

The way that ServiceNow add-on commands are mentioned in the Splunk docs, and if we make this list periodically updated, we won't be able to know if Splunk ever introduced new commands in its docs that actually belong to an add-on. Hence, I think keeping this as a test (as contributed by @kkedziak-splunk) would make more sense, as we would require a human to verify the alteration in command list, if it ever happens.

maybe we will be able to fetch data without snow commands from this url: "https://docs.splunk.com/Documentation/Splunk/9.4.1/SearchReference/ListOfSearchCommands" ?
this way we would be able to periodically update base commands via auto scripts. If no, I'm ok to go with this "test" approach.

@artemrys do you propose to schedule a github workflow with a cron schedule?

Maybe let's proceed with this approach for now and we will see how it works. If needed we can create ticket to change that later. What do you think @hetangmodi-crest @kkedziak-splunk @artemrys ?

I have created the ticket for it. We can look into it later.
ADDON-80607

artemrys · 2025-04-16T12:13:25Z

tests/testdata/test_addons/package_global_config_everything/package/bin/countmatches.py

If this PR does not generate these files, let's leave them out.

Same below.

These files are not generated, they are intended to contain the search command logic and must be provided by the user.
Removing them will cause the build to fail because of this validation. We cannot move the validation elsewhere, as this is a pre-check for generating custom command files.

splunk_add_on_ucc_framework/schema/schema.json

sgoral-splunk · 2025-04-17T08:48:45Z

tests/unit/test_const.py

+from splunk_add_on_ucc_framework.const import SPLUNK_COMMANDS
+
+
+def test_command_list_up_to_date():


maybe we will be able to fetch data without snow commands from this url: "https://docs.splunk.com/Documentation/Splunk/9.4.1/SearchReference/ListOfSearchCommands" ?
this way we would be able to periodically update base commands via auto scripts. If no, I'm ok to go with this "test" approach.

splunk_add_on_ucc_framework/global_config_validator.py

tests/unit/test_global_config_validator.py

splunk_add_on_ucc_framework/global_config_validator.py

sgoral-splunk · 2025-05-07T20:20:05Z

@hetangmodi-crest fossa is failing so let's rebase after ucc-gen validation changes are reverted

**Issue number:** ADDON-76780 ### PR Type **What kind of change does this PR introduce?** * [x] Feature * [ ] Bug Fix * [ ] Refactoring (no functional or API changes) * [x] Documentation Update * [ ] Maintenance (dependency updates, CI, etc.) ## Summary ### Changes This PR includes the generation of the `searchbnf.conf` file for the custom search command. ### User experience After defining a valid configuration for the custom search command in `globalConfig.json`, the `searchbnf.conf` file will be generated. ## Checklist If an item doesn't apply to your changes, leave it unchecked. ### Review * [x] self-review - I have performed a self-review of this change according to the [development guidelines](https://splunk.github.io/addonfactory-ucc-generator/contributing/#development-guidelines) * [x] Changes are documented. The documentation is understandable, examples work [(more info)](https://splunk.github.io/addonfactory-ucc-generator/contributing/#documentation-guidelines) * [x] PR title and description follows the [contributing principles](https://splunk.github.io/addonfactory-ucc-generator/contributing/#pull-requests) * [ ] meeting - I have scheduled a meeting or recorded a demo to explain these changes (if there is a video, put a link below and in the ticket) ### Tests See [the testing doc](https://splunk.github.io/addonfactory-ucc-generator/contributing/#build-and-test). * [x] Unit - tests have been added/modified to cover the changes * [x] Smoke - tests have been added/modified to cover the changes * [ ] UI - tests have been added/modified to cover the changes * [x] coverage - I have checked the code coverage of my changes [(see more)](https://splunk.github.io/addonfactory-ucc-generator/contributing/#checking-the-code-coverage) **Demo/meeting:** *Reviewers are encouraged to request meetings or demos if any part of the change is unclear*

**Issue number:** ADDON-76780, ADDON-79647 ### PR Type **What kind of change does this PR introduce?** * [x] Feature * [ ] Bug Fix * [ ] Refactoring (no functional or API changes) * [x] Documentation Update * [ ] Maintenance (dependency updates, CI, etc.) ## Summary ### Changes This PR includes the generation of the `commands.conf` file for the custom search command. ### User experience After defining a valid configuration for the custom search command in `globalConfig.json`, the `commands.conf` file will be generated. ## Checklist If an item doesn't apply to your changes, leave it unchecked. ### Review * [x] self-review - I have performed a self-review of this change according to the [development guidelines](https://splunk.github.io/addonfactory-ucc-generator/contributing/#development-guidelines) * [x] Changes are documented. The documentation is understandable, examples work [(more info)](https://splunk.github.io/addonfactory-ucc-generator/contributing/#documentation-guidelines) * [x] PR title and description follows the [contributing principles](https://splunk.github.io/addonfactory-ucc-generator/contributing/#pull-requests) * [ ] meeting - I have scheduled a meeting or recorded a demo to explain these changes (if there is a video, put a link below and in the ticket) ### Tests See [the testing doc](https://splunk.github.io/addonfactory-ucc-generator/contributing/#build-and-test). * [x] Unit - tests have been added/modified to cover the changes * [x] Smoke - tests have been added/modified to cover the changes * [ ] UI - tests have been added/modified to cover the changes * [x] coverage - I have checked the code coverage of my changes [(see more)](https://splunk.github.io/addonfactory-ucc-generator/contributing/#checking-the-code-coverage) **Demo/meeting:** *Reviewers are encouraged to request meetings or demos if any part of the change is unclear*

**Issue number:** ADDON-76780 ### PR Type **What kind of change does this PR introduce?** * [x] Feature * [ ] Bug Fix * [ ] Refactoring (no functional or API changes) * [x] Documentation Update * [ ] Maintenance (dependency updates, CI, etc.) ## Summary ### Changes This PR contains generation of template `python files` (which will import logic from the files provided by the users) for the custom search command. For now, we are not supporting generation of python files for `Transforming/Reporting` command. ### User experience After defining required and valid configuration for custom search command in globalconfig.json, required template python files (based on the command type specified by the user) will be generated for the users, which will import logic from the files provided by the users. ## Checklist If an item doesn't apply to your changes, leave it unchecked. ### Review * [x] self-review - I have performed a self-review of this change according to the [development guidelines](https://splunk.github.io/addonfactory-ucc-generator/contributing/#development-guidelines) * [x] Changes are documented. The documentation is understandable, examples work [(more info)](https://splunk.github.io/addonfactory-ucc-generator/contributing/#documentation-guidelines) * [x] PR title and description follows the [contributing principles](https://splunk.github.io/addonfactory-ucc-generator/contributing/#pull-requests) * [ ] meeting - I have scheduled a meeting or recorded a demo to explain these changes (if there is a video, put a link below and in the ticket) ### Tests See [the testing doc](https://splunk.github.io/addonfactory-ucc-generator/contributing/#build-and-test). * [x] Unit - tests have been added/modified to cover the changes * [x] Smoke - tests have been added/modified to cover the changes * [ ] UI - tests have been added/modified to cover the changes * [x] coverage - I have checked the code coverage of my changes [(see more)](https://splunk.github.io/addonfactory-ucc-generator/contributing/#checking-the-code-coverage) **Demo/meeting:** *Reviewers are encouraged to request meetings or demos if any part of the change is unclear*

github-actions · 2025-05-13T12:17:46Z

Code Coverage 🎉

Type	PR	Develop	Change	Status
Line Coverage	88.25%	87.84%	0.41%	🟢 Increased
Branch Coverage	81.94%	81.33%	0.61%	🟢 Increased

hetangmodi-crest added 3 commits April 15, 2025 18:35

feat: centralized code for supporting csc

4b215e3

tests: added respective unit test cases

5a87671

chore: added different csc logic files in _everything addon

65007d4

hetangmodi-crest added the enhancement New feature or request label Apr 15, 2025

hetangmodi-crest self-assigned this Apr 15, 2025

pull-request-size bot added the size/XL label Apr 15, 2025

chore: updated syntax for csc in valid_config.json

0816ddd

hetangmodi-crest marked this pull request as ready for review April 16, 2025 11:07

hetangmodi-crest requested review from a team as code owners April 16, 2025 11:07

hetangmodi-crest requested review from artemrys and kkedziak-splunk April 16, 2025 11:07

This was referenced Apr 16, 2025

feat: add custom search command support #1672

Closed

test: add unit, smoke tests for custom search command #1654

Closed

artemrys reviewed Apr 16, 2025

View reviewed changes

hetangmodi-crest requested a review from sgoral-splunk April 17, 2025 05:41

sgoral-splunk reviewed Apr 17, 2025

View reviewed changes

Merge branch 'develop' into feat/add-validations-for-csc

a373ed0

kkedziak-splunk reviewed Apr 17, 2025

View reviewed changes

splunk_add_on_ucc_framework/global_config_validator.py Outdated Show resolved Hide resolved

tests/unit/test_global_config_validator.py Show resolved Hide resolved

kkedziak-splunk reviewed Apr 17, 2025

View reviewed changes

splunk_add_on_ucc_framework/global_config_validator.py Show resolved Hide resolved

hetangmodi-crest and others added 2 commits April 18, 2025 15:24

chore: removed keyword arguments

2a4c32b

Merge branch 'develop' into feat/add-validations-for-csc

42c8796

Merge branch 'develop' into feat/add-validations-for-csc

a4adaaa

kkedziak-splunk previously approved these changes May 8, 2025

View reviewed changes

hetangmodi-crest dismissed kkedziak-splunk’s stale review via 863d5a0 May 9, 2025 10:04

pull-request-size bot added size/XXL and removed size/XL labels May 9, 2025

hetangmodi-crest added 3 commits May 9, 2025 18:05

Merge branch 'develop' into feat/add-validations-for-csc

40ac685

Merge branch 'develop' into feat/add-validations-for-csc

004561d

sgoral-splunk previously approved these changes May 13, 2025

View reviewed changes

kkedziak-splunk previously approved these changes May 13, 2025

View reviewed changes

Merge branch 'develop' into feat/add-validations-for-csc

b54ca97

hetangmodi-crest dismissed stale reviews from kkedziak-splunk and sgoral-splunk via b54ca97 May 13, 2025 12:14

kkedziak-splunk approved these changes May 13, 2025

View reviewed changes

hetangmodi-crest merged commit 03cb0d2 into develop May 13, 2025
94 checks passed

hetangmodi-crest deleted the feat/add-validations-for-csc branch May 13, 2025 12:58

github-actions bot locked and limited conversation to collaborators May 13, 2025

		from splunk_add_on_ucc_framework.const import SPLUNK_COMMANDS


		def test_command_list_up_to_date():

feat: base code for supporting custom search command #1693

feat: base code for supporting custom search command #1693

Uh oh!

Conversation

hetangmodi-crest commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Type

Summary

Changes

User experience

Checklist

Review

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hetangmodi-crest Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hetangmodi-crest Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sgoral-splunk commented May 7, 2025

Uh oh!

github-actions bot commented May 13, 2025

Code Coverage 🎉

Uh oh!

Uh oh!

Uh oh!

hetangmodi-crest commented Apr 15, 2025 •

edited

Loading

hetangmodi-crest Apr 17, 2025 •

edited

Loading

hetangmodi-crest Apr 17, 2025 •

edited

Loading