OAK-12353: Group Elasticsearch dynamic boost nested documents by boost score

[fabriziofortino]

Summary

Dynamic boost properties (e.g. predictedTagsDynamicBoost) are indexed in Elasticsearch
as nested documents, with one nested document per value:

"predictedTagsDynamicBoost": [
  { "value": "Replacement Cost", "boost": 1 },
  { "value": "Theft", "boost": 1 },
  { "value": "Alberta", "boost": 1 },
  { "value": "GENERAL INSURANCE COMPANY", "boost": 0.988 }
]

For properties with many values sharing the same boost score, this creates a large
number of nested documents, which is expensive to index and store.

This change groups values that share the same boost score into a single nested
document, with value holding an array instead of a scalar:

"predictedTagsDynamicBoost": [
  { "value": ["Replacement Cost", "Theft", "Alberta"], "boost": 1 },
  { "value": "GENERAL INSURANCE COMPANY", "boost": 0.988 }
]

No mapping or query changes were needed: value is a plain analyzed text field,
which Elasticsearch accepts as an array natively, and the existing nested match /
field_value_factor query in ElasticRequestHandler works unchanged against the
grouped structure.

Feature toggle

Guarded by FT_OAK-12353, enabled by default. Set to false at runtime to revert
to the previous one-nested-document-per-value behavior.

A time-bombed test (ElasticDocumentTest#ft_oak_12353_toggleShouldBeRemoved) will start
failing after 2027-08-12 as a reminder to remove the toggle and its guards once the
grouped format has been running in production long enough.

Changes

• ElasticDocument: introduces the FT_OAK-12353 toggle; groups dynamic boost values
  by boost score in addDynamicBoostField/getProperties when enabled.
• ElasticIndexProviderService: registers the toggle on the OSGi Whiteboard.
• Tests: unit coverage for the grouping logic in ElasticDocumentTest, and end-to-end
  query coverage (grouped and ungrouped) in ElasticDynamicBoostTest.

Test plan

• ElasticDocumentTest — verifies grouping by boost, single-group unwrapping, and
  toggle-disabled fallback to the original per-value nested documents
• ElasticDynamicBoostTest — end-to-end query tests against a real Elasticsearch
  instance, confirming queries still match on values grouped into a shared nested
  document, both with grouping enabled (default) and disabled

[bhabegger]

I don't see how the test below (which asserts on query results) "proves" that the different tags are found within the same nested document. Maybe I missed something ?

[bhabegger]

OK, I got confused by the first phrase of the comment. Thought it was a statement. Maybe rephrase ? "Verify that querying give the same results whether grouping by boost (see ...) is active or not.

[bhabegger]

I would even try to write the test in such a way that the body of the test with and without the toggle is shared making it straight forward.

@Test
void ungroupedDynamicBoostedQueriesWork() {
   // Given
   ElasticDocument.FT_OAK_12353_ENABLE.set(false);

  // Then
  assertSimpleBoostedQueriesWork();
}


@Test
void groupedDynamicBoostedQueriesWork() {
   // Given
   ElasticDocument.FT_OAK_12353_ENABLE.set(true); // <- explicit intent

  // Then
  assertSimpleBoostedQueriesWork();
}

void assertSimpleBoostedQueriesWork() {
        Tree testParent = createNodeWithType(root.getTree("/"), "test", JcrConstants.NT_UNSTRUCTURED, "");

        Tree predicted1 = createAssetNodeWithPredicted(testParent, "asset1", "flower with a lot of red and a bit of blue");
        createPredictedTag(predicted1, "red", 5.0);
        createPredictedTag(predicted1, "blue", 5.0);
        createPredictedTag(predicted1, "green", 5.0);
        createPredictedTag(predicted1, "special", 9.0);

        root.commit();

        assertEventually(() -> {
            assertQuery("//element(*, dam:Asset)[jcr:contains(., 'red')]", XPATH, List.of("/test/asset1"));
            assertQuery("//element(*, dam:Asset)[jcr:contains(., 'blue')]", XPATH, List.of("/test/asset1"));
            assertQuery("//element(*, dam:Asset)[jcr:contains(., 'green')]", XPATH, List.of("/test/asset1"));
            assertQuery("//element(*, dam:Asset)[jcr:contains(., 'special')]", XPATH, List.of("/test/asset1"));
        });
}

[bhabegger]

Interesting. So the field length of multi-valued fields is the sum of the lengths of the values ? It varied as well before no ? (Just a curiosity)

[ChlineSaurus]

Overall looks good. Left one comment

[ChlineSaurus]

One thing an Agent flagged (and then verified via testing):

It seems that the order can change when comparing the old vs new behavior.
Without grouping (as expected), the score is the sum of the matching values boosts divided by the number of matching values. With grouping, the boost still adds up all matching values, but the divider counts each boost-group only once instead of each value. So values sharing a group get summed on top yet divide only once.

Example

  private static final String RED_BLUE_GREEN =
          "select [jcr:path] from [dam:Asset] where contains(*, 'red blue green')";

  private void createRankingAssets() throws Exception {
      createAssetsIndexAndProperties(false, false);
      Tree test = createNodeWithType(root.getTree("/"), "test", JcrConstants.NT_UNSTRUCTURED, "");

      // asset1: three tags sharing one boost group (boost 1)
      Tree many = createAssetNodeWithPredicted(test, "asset1", "titleone");
      createPredictedTag(many, "red", 1.0);
      createPredictedTag(many, "blue", 1.0);
      createPredictedTag(many, "green", 1.0);

      // asset2: one high-boost tag in its own group, the other two effectively zero
      Tree single = createAssetNodeWithPredicted(test, "asset2", "titletwo");
      createPredictedTag(single, "red", 4.0);
      createPredictedTag(single, "blue", 0.01);
      createPredictedTag(single, "green", 0.01);
  
      root.commit();
  }

  @Test
  public void rankingWithGroupingDisabled() throws Exception {
      // one nested doc per value: score_mode=avg divides by 3 children
      // -> asset2's single high boost (4) wins over asset1's three boost-1 values
      ElasticDocument.FT_OAK_12353_ENABLE.set(false); // must be set before indexing (root.commit)
      createRankingAssets();
      assertEventually(() -> assertOrderedQuery(RED_BLUE_GREEN, List.of("/test/asset2", "/test/asset1")));
  }
  
  @Test
  public void rankingWithGroupingEnabled() throws Exception {
      // asset1's three boost-1 values collapse into ONE child; its text score sums the three
      // matched terms and score_mode=avg divides by 1 -> asset1 now outranks asset2. Order flips.
      ElasticDocument.FT_OAK_12353_ENABLE.set(true); // the OAK-12353 default
      createRankingAssets();
      assertEventually(() -> assertOrderedQuery(RED_BLUE_GREEN, List.of("/test/asset1", "/test/asset2")));
  }

I'm not sure if this is a problem, but I thought it was still worth mentioning, especially as in my understanding we will have a mixed behavior for some time.