Skip to content

Support struct field with dynamic disabled #3829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 9, 2025

Conversation

qianheng-aws
Copy link
Collaborator

Description

Support struct field with dynamic disabled

If dynamic mapping disabled, although information of type missed for the new added fields, we can still parse value from the content by itself only.

Related Issues

Resolves #3343

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Heng Qian <qianheng@amazon.com>
@qianheng-aws
Copy link
Collaborator Author

qianheng-aws commented Jun 30, 2025

Another issue is that we don't serialize json whose value is null.

For example:

POST {{baseUrl}}/_bulk/
Content-Type: application/x-ndjson

{"index": {"_index": "test001"}}
{"profile": {"age": 1}}
{"index": {"_index": "test001"}}
{"profile": {"address": null}}

We will currently get result for PPL source=test001:

{
  "schema": [
    {
      "name": "profile",
      "type": "struct"
    }
  ],
  "datarows": [
    [
      {
        "age": 1
      }
    ],
    [
      {}
    ]
  ],
  "total": 2,
  "size": 2
}

But will get this if configuring serializeNulls for GSON:

{
  "schema": [
    {
      "name": "profile",
      "type": "struct"
    }
  ],
  "datarows": [
    [
      {
        "age": 1
      }
    ],
    [
      {
        "address": null
      }
    ]
  ],
  "total": 2,
  "size": 2
}

Which result do we want to generate? I would prefer the latter one. Shall we fix this as well? @penghuo

Signed-off-by: Heng Qian <qianheng@amazon.com>
@LantaoJin
Copy link
Member

+1 for serializeNulls

return ExprNullValue.of();
}

if (fieldType.isEmpty()) {
Copy link
Member

@LantaoJin LantaoJin Jul 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add some comment to explain why we parse context when field type is not present?

Signed-off-by: Heng Qian <qianheng@amazon.com>
Signed-off-by: Heng Qian <qianheng@amazon.com>
Signed-off-by: Heng Qian <qianheng@amazon.com>
@LantaoJin
Copy link
Member

LantaoJin commented Jul 8, 2025

Ping @penghuo @dai-chen @Swiddis

@penghuo penghuo merged commit 76b3ec2 into opensearch-project:main Jul 9, 2025
23 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 9, 2025
* Support struct field with dynamic disabled

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix UT

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Revert change in GSON

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Revert change in GSON

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
(cherry picked from commit 76b3ec2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 9, 2025
* Support struct field with dynamic disabled

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix UT

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Revert change in GSON

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Revert change in GSON

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
(cherry picked from commit 76b3ec2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
qianheng-aws pushed a commit that referenced this pull request Jul 10, 2025
* Support struct field with dynamic disabled



* Fix UT



* Address comments



* Revert change in GSON



* Revert change in GSON



---------


(cherry picked from commit 76b3ec2)

Signed-off-by: Heng Qian <qianheng@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
qianheng-aws added a commit that referenced this pull request Jul 14, 2025
* Support struct field with dynamic disabled (#3829)

* Support struct field with dynamic disabled

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix UT

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Revert change in GSON

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Revert change in GSON

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
(cherry picked from commit 76b3ec2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix compiling error

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add UT

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Heng Qian <qianheng@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] PPL query executed on an object field with dynamic mapping disabled returns an empty result
3 participants