Enrich processor: allow scheduling of policy executions

[tahaderouiche]

Before an enrich processor can be used, an enrich policy must be executed. When executed, an enrich policy uses enrich data from the policy’s source indices to create a streamlined system index called the enrich index.
The execution is executed manually by running PUT /_enrich/policy/my-policy/_execute, giving the user control on when the new data becomes part of the enriching policy.

In cases where the policy’s source indices are constantly changing, the policy execution can also be scheduled.

Having the ability to schedule (say daily, hourly) the execution natively in elasticsearch would make it more approachable and would benefit this case of a constantly changing source indices.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[philippkahr]

I would like to add my two cents here.
Let's assume the following example with currencies and exchange.rates

my enrich processors matches on pair and sets the exchange.rate
I have the following indices

index: exchange-rate
contains {pair : EURO-DOLLAR and exchange.rate: 1.1}
are updated all 30 minutes using filebeat.

index: latest-exchange-rate
contains: {pair : .... } as above
are updated every hour using transforms.
it only includes the latest values for `pair`

index: currencies
contains { pair: EURO-DOLLAR }
which uses an ingest pipeline with an enrich to populate the `exchange.rate` value.

my enrich policy append-exchange-rate

{
  "match": {
    "indices": "latest-exchange-rate",
    "match_field": "pair",
    "enrich_fields": ["exchange.rate"]
  }
}

Since I am using transforms to dynamically build a small index, I would expect the enrich processor to pick up the change to the original index and do the POST /_enrich/policy/append-exchange-rate/_execute automagically.

Personally this would definitely be needed for indices that have frequent changes and/or transforms that run often. I know that enrich policy building is costly.

Scheduling the policy update alone is definitively a nice feature, but there might be a window where the old data is already refreshed but the policy update has not run yet. Thus the documents are getting populated with old and maybe wrong data.

[ar-mi]

Hey! Have any work already been done on this?

[puppylpg]

It's been year 2022 😸 issue still open

[smnschneider]

Would be great to get this feature!

[leandrojmp]

Hello, is there any update on this feature? Is this still being considered?

[timor-raiman]

+1
Alternatively - #58925

[Rick25-dev]

+1

[Rick25-dev]

+1

[kossde]

I was able to work around this issue by creating a watcher that performs an http call to the cluster on a scheduled interval and re-executes the enrich policy.

Initially when I did this, I was running into trouble with the execution failing periodically but that ended up being related to the auto_expand_replicas settings of the .enrich indices and our high disk utilization on one node. To get around that, I created an index template for .enrich indices and turned off auto_expand_replicas and setting the replica count to 1. The auto-execution now works like a charm!

[leandrojmp]

@kossde hello, can you share the watcher json configuration that you used?