Description
Repository Feature
Core Repo - (rule management, validation, testing, lib, cicd, etc.)
Problem Description
We still have logic marked as experimental within our repo that should be deprecated and ultimately removed. This logic was intended to support ml functionality that is no longer supported.
(detection-rules-build) ➜ detection-rules git:(main) ✗ python -m detection_rules es experimental ml -h
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json
█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
Elasticsearch client:
Options:
--ignore-ssl-errors TEXT
-et, --timeout INTEGER Timeout for elasticsearch client
-ep, --es-password TEXT
-eu, --es-user TEXT
--elasticsearch-url TEXT
--api-key TEXT
--cloud-id TEXT
* experimental commands are use at your own risk and may change without warning *
Usage: detection_rules es experimental ml [OPTIONS] COMMAND [ARGS]...
Experimental machine learning commands.
Options:
-h, --help Show this message and exit.
Commands:
check-files Check ML model files on an elasticsearch...
delete-job Remove experimental ML jobs.
remove-model Remove ML model files.
remove-scripts-pipelines Remove ML scripts and pipeline files.
setup Upload ML model and dependencies to enrich data.
upload-job Upload experimental ML jobs.When you try to use this existing code today, it throws an error because it uses a deprecated way to call the ES API.
detection_rules es experimental ml upload-job my-job.json
TypeError: Positional arguments can't be used with Elasticsearch API methods. Instead only use keyword arguments.
Desired Solution
These CLI commands should be officially removed. We should tag the commit for folks to have a reference point in case they still want to use this logic.
Considered Alternatives
We could update the call to supply the kwargs func(job_id=name, body=body) here as a starting point, but with the additional context below, this capability should no longer be supported.
Additional Context
We now have ML jobs supported via integrations.
Additionally, the original issue where the experimental ML features were merged in states that the ML release processes are to be managed via a different internal repo (dremel), which also has been deprecated with the statement:
NOTE: We are no longer using dreml to ship experimental machine learning models. All existing models have been/are being delivered via integration packages or other mechanisms. Please get in touch with the Security ML team for more information on packaging ML models.
