Synthetic source does not still support copy_to
#652
Conversation
|
Since |
When using logsdb index mode copy_to is disabled because it is not supported by synthetic source. So we need to change queries so to reflect the fact that the message field would be empty.
| @@ -582,7 +582,11 @@ | |||
| "include_unmapped": true | |||
| }, | |||
| { | |||
| {% if index_mode != "logsdb" %} | |||
There was a problem hiding this comment.
It looks like our template engine does not process files in the workflows directory. As a result we are not able to conditionally execute the query on message or kubernetes.even.message. I will just unconditionally execute it using kubernetes.even.message that is the original field name. After we introduce support for copy_to we can restore the original behaviour.
This is done since copy_to does not work in synthetic source and the message field would be empty.
We also introduce a track parameter which we can use to select the workflow directory to use to read workflow queries. The default is `workflows` that is the existing one. Using `workflows-logsdb` allows us to use workflow queries which do not realy on the usage of copy_to used by the metricbeat template. This workaround is introduced to deal with synthetic source (and logsdb) not supporting copy_to. Once support for copy_to is available we will revert this change and rely on the existing and default workflows.
|
I tried adding a @gareth-ellis @charlie-pichette does it sound reasonable? We will revert this once synthetic source supports |
|
BTW this would have been much easier if this track used the standard |
| @@ -0,0 +1,13 @@ | |||
| This workflow represents a user using the Hosts dashboard from the Security application in Kibana. | |||
There was a problem hiding this comment.
Maybe just refer to the other README and indicate this is just a temp solutions for logsdb?
No need to repeat READMEs, right?
There was a problem hiding this comment.
And maybe do this for the other READMEs too?
| @@ -31,6 +31,9 @@ | |||
| "operation-type": "composite", | |||
| "param-source": "workflow-selector", | |||
| "workflow": {{workflow | tojson }}, | |||
| {% if p_index_mode == "logsdb" %} | |||
There was a problem hiding this comment.
Shouldn't this be determined by the work_flow track param? Or is the idea that the work_flow param is determined by the index mode track param?
There was a problem hiding this comment.
the workflow parameter just says hosts, network or overview - the workflows-folder is what refers to either currently workflows, but for logsdb we want to use the workflows-logsdb folder
gareth-ellis
force-pushed
the
fix/security-message-copy-to
branch
from
d88ff33
to
ab2e5a4
Compare
|
I executed a test for the stateful case and double checkd all datastream backing indices are actually using LogsDB. |
|
The track completed succesfully |
I have no information about |
|
@salvatore-campagna Regarding the |
|
@elasticmachine update branch |
This reverts commit 2b42a38.
…astic#652)"" This reverts commit b8b2f27.
This causes the
elastic/securitytrack to fail execution whenindex_modeis set tologsdb. This is happening because LogsDB uses synthetic source which, in turn, doesnot support
copy_to. Supportingcopy_tois expected to come in Elasticsearch 8.16.In the meanwhile we just exclude the
copy_tosetting from the mapping so to avoidtriggering the error.
This needs backporting to 8.15.