Support for IndexSort in Nested Fields

[vishdivs]

Description

Since Lucene now supports IndexWriterConfig#setParentField for creating an internal field to enable index sorting, OpenSearch needs to integrate this functionality by setting the Parent Field in appropriate workflows with proper validations.
#17026 (comment)

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#17026

Testing Details:

Testing case 1: Before the Changes

curl -XPUT "localhost:9200/test_nested_sort" -H 'Content-Type: application/json' -d '{      
  "settings": {
    "index": { 
      "sort.field": ["users.age", "users.name.keyword"],
      "sort.order": ["desc", "asc"]
    }                  
  },   
  "mappings": {             
    "properties": {
      "users": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "age": {
            "type": "integer"
          }
        }
      }
    }
  }
}'

Result for this:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"cannot have nested fields when index sort is activated"}],"type":"illegal_argument_exception","reason":"cannot have nested fields when index sort is activated"},"status":400}%

Testing case 2: Testing when Index Sort field is inside Nested Field (With Validation)

curl -XPUT "localhost:9200/test_nested_sort" -H 'Content-Type: application/json' -d '{
  "settings": {
    "index": {
      "sort.field": ["users.age", "users.name.keyword"],
      "sort.order": ["desc", "asc"]
    }
  },
  "mappings": {
    "properties": {
      "users": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "age": {
            "type": "integer"
          }
        }
      }
    }
  }
}'

Result:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"index sorting on nested fields is not supported: found nested sort field [users.age] in [test_nested_sort]"}],"type":"illegal_argument_exception","reason":"index sorting on nested fields is not supported: found nested sort field [users.age] in [test_nested_sort]"},"status":400}%

Test case 3:

curl -XPUT "localhost:9200/test_nested_sort" -H 'Content-Type: application/json' -d '{
  "settings": {
    "index": {
      "sort.field": ["foo"],
      "sort.order": ["desc"]
    }
  },
  "mappings": {
    "properties": {
      "foo": {
        "type": "integer"
      },
      "users": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "age": {
            "type": "integer"
          }
        }
      }
    }
  }
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"test_nested_sort"}

Tested with the search query and result:

curl -XGET "localhost:9200/test_nested_sort/_search" -H 'Content-Type: application/json' -d '{
  "query": {
    "nested": {                      
      "path": "users",
      "query": {                                        
        "bool": {                  
          "should": [  
            {"match": {"users.name": "Alice"}},
            {"range": {"users.age": {"gte": 20}}}
          ]        
        }
      }
    }                         
  },            
  "sort": [       
    {"foo": {"order": "desc"}},
    {
      "users.age": {
        "order": "desc",
        "nested": {
          "path": "users"
        }
      }
    }
  ]
}'
{"took":619,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":3,"relation":"eq"},"max_score":null,"hits":[{"_index":"test_nested_sort","_id":"uArTfJcBNvxL7QBM7V8f","_score":null,"_source":{
  "foo": 40,
  "users": [                         
    {          
      "name": "Bob Johnson",                            
      "age": 35                    
    },                 
    {           
      "name": "Jane Wilson",
      "age": 50    
    },
    {
      "name": "armane Wilson",
      "age": 50 
      }           
  ]                      
},"sort":[40,50]},{"_index":"test_nested_sort","_id":"tgrTfJcBNvxL7QBMcV9g","_score":null,"_source":{
  "foo": 30,
  "users": [
    {
      "name": "Alice Smith",
      "age": 25
    },
    {
      "name": "John Doe",
      "age": 30
    }
  ]
},"sort":[30,30]},{"_index":"test_nested_sort","_id":"ugrYfJcBNvxL7QBMBl_t","_score":null,"_source":{
  "foo": 25,
  "users": [                         
    {          
      "name": "Bob Johnson",                            
      "age": 35                    
    },                 
    {           
      "name": "Jane Wilson",
      "age": 50    
    },
    {
      "name": "armane Wilson",
      "age": 50 
      }           
  ]                      
},"sort":[25,50]}]}}%

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for b6fadef: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[vibrantvarun]

@vishdivs Can you add a bwc test for the same ?

[vishdivs]

@vishdivs Can you add a bwc test for the same ?

@vibrantvarun, I have already added validation check related to Index Version here
Do we need any other BWC tests besides this? Please let me know if there are any specific tests required.

[github-actions]

❕ Gradle check result for 2fcc7ab: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for 9a6bc69: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8290e71: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8290e71: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 30a4c33: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 2417780: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for ec36808: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for ec36808: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for be97692: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 1e1fffe: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 84.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 72.87%. Comparing base (c1dfa6a) to head (1e1fffe).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...va/org/opensearch/index/mapper/DocumentMapper.java	83.33%	0 Missing and 2 partials ⚠️
...java/org/opensearch/index/shard/StoreRecovery.java	60.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18536      +/-   ##
============================================
+ Coverage     72.75%   72.87%   +0.11%     
- Complexity    68520    68641     +121     
============================================
  Files          5570     5572       +2     
  Lines        314998   315196     +198     
  Branches      45697    45750      +53     
============================================
+ Hits         229185   229686     +501     
+ Misses        67260    66865     -395     
- Partials      18553    18645      +92     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.