[Transform] Transform might skip buckets with restrictive bucket selector aggregation

[hendrikmuhs]

Affected version: 7.5-
Fixed with: tbd

This affects only users of a top-level bucket_selector aggregation in a pivot transform

The bucket selector deletes buckets that do not meet the criteria. If the criteria is very strict a lot of documents are skipped. If a full page of a composite aggregation result run returns 0 results after filtering Transform seems to stop further processing.

Mitigation

By increasing the page size it gets less likely to get into this. As long as every paged result set has at least 1 result left after the bucket selector transform won't run into the bug. Nevertheless there is no guarantee, it might still happen that 10k buckets in a row get removed by the bucket selector and therefore the issue happens.

A more reliable workaround is to replace the bucket selector and connect an ingest pipeline to the transform. Instead of using a bucket selector aggregation drop documents as part of the ingest pipeline using a drop processor. Because this drops documents later compared to the bucket selector this will be slightly more expensive in terms of performance.

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[hendrikmuhs]

Reproduction:

Using the kibana_sample_data_ecommerce sample data this can be reproduced with:

1. Create the following transform:

PUT _transform/bucket_selector_issue
{
  "source": {
    "index": [
      "kibana_sample_data_ecommerce"
    ]
  },
  "dest": {
    "index": "bucket_selector_issue"
  },
  "frequency": "1m",
  "pivot": {
    "group_by": {
      "customer_full_name": {
        "terms": {
          "field": "customer_full_name.keyword"
        }
      }
    },
    "aggregations": {
      "price_sum": {
        "sum": {
          "field": "products.taxful_price"
        }
      },
      "bs": {
        "bucket_selector": {
          "buckets_path": {
            "ntypes": "price_sum.value"
          },
          "script": "params.ntypes > 600"
        }
      }
    }
  },
  "settings": {
    "max_page_search_size": 500
  }
}

2. Create a 2nd transform bucket_selector_issue-2 with the same configuration but set max_page_search_size to 10000 and change dest to bucket_selector_issue-2.

3. Search both dest indices, due to the restrictive bucket selector they have only a few documents, but bucket_selector_issue-2 has 1 document more than bucket_selector_issue

[hendrikmuhs]

I suggest to start the investigation in TransformIndexer::processBuckets (around line 1040). I guess processSearchResponse returns null due to an empty result set.

[przemekwitek]

I was able to reproduce the issue by following the steps listed in #82546 (comment)

I'll work on the fix now.

[przemekwitek]

The fix is outlined in #82852
Working on the tests now.

[przemekwitek]

As far as 8.1.0 is concerned, this issue is fixed.

[111andre111]

@przemekwitek Is there any chance that we could still get this into 8.0?

[droberts195]

Is there any chance that we could still get this into 8.0?

It can go into 8.0.1. It's too late for 8.0.0. That is blockers only now.

[111andre111]

That sounds good. I don't know if there is a backport PR needed for getting it into 8.0.1? @droberts195