Custom tags

[pasteque-pal]

Enable the possibility to add custom tags to detections. They should be referenced as a list in the yaml under tags>others. Example:
others:

• tag1:
  • value1
• tag2:
  • value21
  • value22

[pyth0n1c]

Provided some feedback on the PR - I am not sure we should include this in contentctl for the reasons detailed.
Not approving but leaving open for now to facilitate discussion.

Please see the alternative "user-specific" implementation suggestion below.

[pyth0n1c]

This is not a change I believe we can, or should, make in the core contentctl.
Before the release of contentctl 5.x, we did not include extra="forbid" in the definitions of our objects. What that led to was a SIGNIFICANT number of otherwise "optional" keys, such as manual_test, cve, etc being in the wrong place (for example nesting them under the detection object rather than tags or vice versa) or typo-ing a field and not having it caught by validation since it was parsed as an extra field. Being stricter with object definitions helps catch these at the validate phase and generate descriptive warnings.

Also, since these others are not typed in the PR (and do not have a use in ESCU) I would prefer not to add them to contentctl.

My recommendation would be that this field is instead added to your own copy of the detection.py file here:

contentctl/contentctl/objects/detection.py

Line 8 in 449128b

class Detection(Detection_Abstract):

Looking at the comment, you can see this is a file that future contentctl updates should not touch and is meant for user-specific implementations or modifications to the core Detection specification (which is actually Detection_Abstract).

I would recommend, at least, adding
others: dict = {}
to your local copy of the Detection Object or, even better, creating an Others object that looks like this or similar

class Others(BaseModel):
  tag1: list[str] = []
  tag2: list[str] = []

You should then be able to override the annotations method/property by calling the parent's annotations and updating it with the extra info you've included in your other field:

@computed_field
@property
def annotations(self) -> dict[str, Union[List[str], int, str]]:
   base_annotations = super().annotations
   #new_annotations = for loop logic for serializing whatever is in others```
   #Return combined annotations - note this can+will overwrite values in base_annotations with new_annotations if the keys match
   return base_annotations | new_annotations 

I would agree that a strict definition like this does seem overly prescriptive, but with the complexity of our YML objects and possibility for error, I think that the prescriptiveness is actually a positive rather than a negative.

Please let me know your thoughts 😄

[pasteque-pal]

Thanks for your feedback! Maybe I did not code it the right way and did not use adequate variable names, I admit. But for me, the unmanaged annotations are a basic functionality in use in ES, so that would of great help to have this available through contentctl (see below). Let me know how we can move forward on this, really willing to help.

[pyth0n1c]

Hmm, okay this does make a lot of sense! Thank you for the clarification. Perhaps the core contentctl should include the following change, then, in detection_tags.py

unmanaged_annotations: dict[str,list[str]]

and that field, if present, should be serialized as part of the annotations function recommended above?
I think by clearly calling the field unmanaged_annotations we communicate pretty clearly to users:

1. Where exactly these are going to show up in Enterprise Security (in........ Unmanaged Annotations!)
2. That they don't get the same level of in-depth checking/validations as other fields.

This is actually much closer to what you originally contributed.
Let me tag in another contentctl contributor to see if they can provide their thoughts.

What do you think @ljstella ?

[pasteque-pal]

Hi all! Any news regarding this? Let me know if you need anything. Thanks!