Add Facility for deploying ElasticSearch Transform

[nnamdifrankie]

Background

And the ability to deploy a defined ElasticSearch transform to be be deployed when a package is applied or upgraded.

Acceptance Criteria

• As a user I should be able to define an ElasticSearch transform as part of a package.
• The ElasticSearch transform should be started after it is added to the search database.
• As a user I should be able to update the attributes of the transform including the name possibly. This should not result in two Transform running.
• As a user we should be able to delete a transform through an update to the package.
• As a user I should be able to view statistics and information about a transform after deployment using the ElasticSearch API or Kibana if available.

Sample Transform Creation Statements Captured From Kibana Devtools

PUT _transform/endpoint_host_metadata_transform
{
  "source": {
    "index": "metrics-endpoint.metadata-default"
  },
  "dest": {
    "index": "metrics-endpoint.metadata_current-default"
  },
  "pivot": {
    "group_by": {
      "agent.id": {
        "terms": {
          "field": "agent.id"
        }
      }
    },
    "aggregations": {
      "HostDetails": {
        "scripted_metric": {
          "init_script": "state.timestamp_latest = 0L; state.last_doc=''",
          "map_script": "def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli(); if (current_date > state.timestamp_latest) {state.timestamp_latest = current_date;state.last_doc = new HashMap(params['_source']);}",
          "combine_script": "return state",
          "reduce_script": "def last_doc = '';def timestamp_latest = 0L; for (s in states) {if (s.timestamp_latest > (timestamp_latest)) {timestamp_latest = s.timestamp_latest; last_doc = s.last_doc;}} return last_doc"
        }
      }
    }
  },
  "description": "collapse and update the latest document for each host",
  "frequency": "1m",
  "sync": {
    "time": {
      "field": "event.created",
      "delay": "60s"
    }
  }
}

POST _transform/endpoint_host_metadata_transform/_start
DELETE _transform/endpoint_host_metadata_transform

[ycombinator]

@ruflin could use your input here.

As a user I should be able to define an ElasticSearch transform as part of a package.

This item impacts the package spec. We could enhance the spec to allow transforms to be optionally defined as <packageRoot>/dataset/<dataset>/elasticsearch/transform/<transform>.json files. So your sample transform above would be defined in endpoint/dataset/metadata_current/elasticsearch/transform/endpoint_host_metadata_transform.json. The contents of the JSON file would be the same JSON that you'd send to the PUT _transform Elasticsearch API. The only part I'm not 100% sure about is whether the JSON should hardcode the values of the source.index and dest.index properties or if those need to be dynamically generated by Kibana before loading the transform into Elasticsearch?

The ElasticSearch transform should be started after it is added to the search database.
As a user I should be able to update the attributes of the transform including the name possibly. This should not result in two Transform running.
As a user we should be able to delete a transform through an update to the package.
As a user I should be able to view statistics and information about a transform after deployment using the ElasticSearch API or Kibana if available.

These items impact the actual implementation, which would happen in Kibana. Once we agree on the spec changes (discussion in the previous paragraph), I'll make a PR to update the spec in this repo here and then we can file an issue in Kibana for the implementation items.

[ruflin]

It seems the transform depends on one dataset to create a second one. The index metrics-endpoint.metadata-default would be part of the existing metadata dataset. The transform part would be its own dataset and most follow the indexing strategy. With metrics-endpoint.metadata_current-default this seems to be the case, so the dataset would be called metadata_current. This dataset also contains the mapping for the data after the transformation.

So what <packageRoot>/dataset/<dataset>/elasticsearch/transform/<transform>.json makes sense to me. @ycombinator Should we support json and yaml? ;-)

But what about the namespace part? Will the transform be applied to all namespaces? Can the namespace somehow be passed in as a param on start?

@nnamdifrankie Could you open a second issue in Kibana around the part installing / removing / updating the transforms as this not directly applies to the spec.

[nnamdifrankie]

@ruflin What repository for the kibana ticket. And to answer your question about the metadata_current we have already defined it and it is managed by ingest. https://github.com/elastic/package-storage/pull/325/files#diff-fd155145fe68664d8a5cbd8b7727ac5f

[ruflin]

@nnamdifrankie https://github.com/elastic/kibana

Could you also share your thoughts on how you would see it working with the namespaces?

[nnamdifrankie]

Could you also share your thoughts on how you would see it working with the namespaces?
@ruflin

This is a good I will answer and ask some question on how the indices are created.

Answer:

We currently write to the default namespace and read from all namespaces -*, with our index pattern and we do not specify any namespaces. So to begin we will be successful if we target the default namespace.

Questions:

1. Are all indices created by the default in a namespace, and the default namespace is always the fall back. I ask this because of the destination index which is defined by a template. Assuming we specified metrics-endpoint.metadata_current will this create an index metrics-endpoint.metadata_current-default

[ruflin]

Related Kibana issue can be found here: elastic/kibana#75153

[ruflin]

Indices are not created by default, there is only the template. The index (data stream) is created when the first document is ingested.

[polyfractal]

Side note: it looks like the scripted_metric agg is trying to capture the "most recent" document for each host? If that's true, I think it can be done with a TopMetrics agg instead

We're trying to discourage the use of scripted-metric... it can be very dangerous, and the top-metrics agg should be faster too. :)

[nnamdifrankie]

@polyfractal

I know we spoke offline on slack at the time of your comment. But I wanted to comment here for posterity. We are not trying to gather specific metrics but we are trying to retrieve the latest document related to the pivot property, and replace the last document in the destination. Hence we are using this approach, but if there are other better approaches please comment. Thanks.

[polyfractal]

If you need the latest document, perhaps TopHits agg instead?

[mtojek]

It's a bit old issue. Feel free to reopen it if it's still valid.