Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding depth check in doc parser for deep nested document #5199

Conversation

nkumar04
Copy link
Contributor

Description

Added a depth level check within the document parser context. The depth level is incremented for each nested object parsing and decremented when parsing is done. If the depth level crosses a threshold, document parser throws a parsing exception and an error is returned in the response, stating the reason of the failure

Issues Resolved

Closes issue (#5195)

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@nkumar04 nkumar04 requested review from a team and reta as code owners November 10, 2022 16:27
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@nkumar04 nkumar04 force-pushed the 5195_fix_deep_nested_doc_parsing branch 2 times, most recently from c2b1c23 to da33086 Compare November 30, 2022 07:23
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@nkumar04 nkumar04 force-pushed the 5195_fix_deep_nested_doc_parsing branch from da33086 to 174883a Compare November 30, 2022 09:30
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Copy link
Member

@dblock dblock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add to CHANGELOG. We'll also need to document this.

How does this alter the default behavior? If I understand correctly now the mapping depth limit is now enforced - what is its default value?

@nkumar04 nkumar04 force-pushed the 5195_fix_deep_nested_doc_parsing branch from 174883a to ae7253f Compare December 1, 2022 15:08
@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2022

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

@nkumar04 nkumar04 force-pushed the 5195_fix_deep_nested_doc_parsing branch from ae7253f to f9b61d7 Compare December 5, 2022 15:19
@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2022

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link

codecov-commenter commented Dec 5, 2022

Codecov Report

Merging #5199 (2c6c8b9) into main (28e9b11) will increase coverage by 0.02%.
The diff coverage is 87.93%.

@@             Coverage Diff              @@
##               main    #5199      +/-   ##
============================================
+ Coverage     70.98%   71.01%   +0.02%     
+ Complexity    58669    58655      -14     
============================================
  Files          4763     4764       +1     
  Lines        279945   280010      +65     
  Branches      40418    40425       +7     
============================================
+ Hits         198731   198843     +112     
+ Misses        64988    64858     -130     
- Partials      16226    16309      +83     
Impacted Files Coverage Δ
...a/org/opensearch/test/OpenSearchIntegTestCase.java 56.35% <0.00%> (-0.32%) ⬇️
...pensearch/common/settings/FeatureFlagSettings.java 50.00% <50.00%> (ø)
.../java/org/opensearch/common/util/FeatureFlags.java 80.00% <88.88%> (+30.00%) ⬆️
...va/org/opensearch/index/mapper/DocumentParser.java 86.45% <89.65%> (+0.21%) ⬆️
...org/opensearch/common/settings/SettingsModule.java 85.81% <100.00%> (+0.29%) ⬆️
...java/org/opensearch/index/mapper/ParseContext.java 85.99% <100.00%> (+2.94%) ⬆️
server/src/main/java/org/opensearch/node/Node.java 87.17% <100.00%> (+0.01%) ⬆️
...regations/metrics/AbstractHyperLogLogPlusPlus.java 51.72% <0.00%> (-44.83%) ⬇️
...opensearch/persistent/PersistentTasksExecutor.java 22.22% <0.00%> (-44.45%) ⬇️
...pensearch/action/ingest/DeletePipelineRequest.java 31.25% <0.00%> (-37.50%) ⬇️
... and 502 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Member

@dblock dblock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will leave this to @reta since he had some concerns.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 4, 2023

Gradle Check (Jenkins) Run Completed with:

@nkumar04 nkumar04 force-pushed the 5195_fix_deep_nested_doc_parsing branch from 997e763 to 2c6c8b9 Compare January 5, 2023 09:12
@github-actions
Copy link
Contributor

github-actions bot commented Jan 5, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.index.termvectors.TermVectorsServiceTests.testWithIndexedPhrases

@reta
Copy link
Collaborator

reta commented Feb 3, 2023

@nkumar04 could you rebase please?

@nkumar04 nkumar04 force-pushed the 5195_fix_deep_nested_doc_parsing branch from 2c6c8b9 to d45b0eb Compare February 8, 2023 03:56
@nkumar04 nkumar04 force-pushed the 5195_fix_deep_nested_doc_parsing branch from d45b0eb to ee3d1ec Compare February 8, 2023 04:13
@nkumar04
Copy link
Contributor Author

nkumar04 commented Feb 8, 2023

Will leave this to @reta since he had some concerns.

Hi @dblock , @reta has approved the PR. Please let me know if you have any comment or suggestion. Thanks

@github-actions
Copy link
Contributor

github-actions bot commented Feb 8, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Feb 8, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      2 org.opensearch.extensions.ExtensionsManagerTests.classMethod
      1 org.opensearch.extensions.ExtensionsManagerTests.testInitialize

@nkumar04
Copy link
Contributor Author

nkumar04 commented Feb 8, 2023

@nkumar04 could you rebase please?

done.

@dblock
Copy link
Member

dblock commented Feb 22, 2023

Looks like this needs another rebase, sorry. I am traveling, will be slow to reply. Hope other maintainers can help merge.

Nikhil Kumar added 4 commits February 23, 2023 19:01
Fixing the issue (opensearch-project#5195)

Signed-off-by: Nikhil Kumar <nikhkumn@amazon.com>
Fixing the issue (opensearch-project#5195)

Signed-off-by: Nikhil Kumar <nikhkumn@amazon.com>
Fixing the issue (opensearch-project#5195)

Signed-off-by: Nikhil Kumar <nikhkumn@amazon.com>
Fixing the issue (opensearch-project#5195)

Signed-off-by: Nikhil Kumar <nikhkumn@amazon.com>
@nkumar04 nkumar04 force-pushed the 5195_fix_deep_nested_doc_parsing branch from ee3d1ec to c7b3897 Compare February 23, 2023 13:32
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@shwetathareja shwetathareja merged commit 950b86a into opensearch-project:main Feb 28, 2023
@andrross
Copy link
Member

I'm assuming this is safe to backport, as it is a fix to a stack overflow label. I'm adding the backport label now but we'll also have to fix up the changelog entry and move it to the Unreleased 2.x section.

@andrross andrross added the backport 2.x Backport to 2.x branch label Feb 28, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5199-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 950b86ae697d95db537e7809f2675ebdf2b0dc10
# Push it to GitHub
git push --set-upstream origin backport/backport-5199-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5199-to-2.x.

nkumar04 added a commit to nkumar04/OpenSearch that referenced this pull request Mar 3, 2023
…-project#5199)

* Adding depth check in doc parser for deep nested document

Fixing the issue (opensearch-project#5195)

Signed-off-by: Nikhil Kumar <nikhkumn@amazon.com>
(cherry picked from commit 950b86a)
nkumar04 added a commit to nkumar04/OpenSearch that referenced this pull request Mar 3, 2023
…-project#5199)

* Adding depth check in doc parser for deep nested document

Fixing the issue (opensearch-project#5195)

Signed-off-by: Nikhil Kumar <nikhkumn@amazon.com>
(cherry picked from commit 950b86a)
nkumar04 added a commit to nkumar04/OpenSearch that referenced this pull request Mar 3, 2023
…-project#5199)

* Adding depth check in doc parser for deep nested document

Fixing the issue (opensearch-project#5195)

Signed-off-by: Nikhil Kumar <nikhkumn@amazon.com>
(cherry picked from commit 950b86a)
reta pushed a commit that referenced this pull request Mar 3, 2023
)

* Adding depth check in doc parser for deep nested document

Fixing the issue (#5195)

Signed-off-by: Nikhil Kumar <nikhkumn@amazon.com>
(cherry picked from commit 950b86a)
mingshl pushed a commit to mingshl/OpenSearch-Mingshl that referenced this pull request Mar 24, 2023
…-project#5199)

* Adding depth check in doc parser for deep nested document

Fixing the issue (opensearch-project#5195)

Signed-off-by: Nikhil Kumar <nikhkumn@amazon.com>
Signed-off-by: Mingshi Liu <mingshl@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants