[fixes #16599] Field-level `ignore_malformed` should override index-level setting #18706

mishail · 2025-07-08T19:44:21Z

Description

There are 9 types of fields that support setting of ignore_malformed property, see the table below.

The current logic checks for the index level FieldMapper.IGNORE_MALFORMED_SETTING in catch section in parse(ParseContext context) in the FieldMapper class. The current implementation has no visibility into field level settings.

The proposed changes adding an ability for FieldMapper to get access field-level setting via ignoreMalformed() method.

Type	Controlling class
`ip`	`IpFieldMapper`
`ip_range`	`RangeFieldMapper`
`geo_point`	`AbstractGeometryFieldMapper`
`geo_shape`	`AbstractGeometryFieldMapper`
`xy_point`	`AbstractGeometryFieldMapper`
`xy_shape`	`AbstractGeometryFieldMapper`
numerics	`NumberFieldMapper`, `ScaledFloatFieldMapper`
`derived`	`DerivedFieldMapper`
`date`	`DateFieldMapper`

Related Issues

Resolves #16599

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…d override index-level setting Signed-off-by: Mikhail Stepura <mstepura@apple.com>

server/src/test/java/org/opensearch/index/mapper/NumberFieldMapperTests.java

github-actions · 2025-07-08T21:05:41Z

✅ Gradle check result for be93286: SUCCESS

codecov · 2025-07-08T21:06:15Z

Codecov Report

Attention: Patch coverage is 82.75862% with 5 lines in your changes missing coverage. Please review.

Project coverage is 72.77%. Comparing base (548e467) to head (21d0cd5).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
.../java/org/opensearch/index/mapper/FieldMapper.java	55.55%	2 Missing and 2 partials ⚠️
...rg/opensearch/index/mapper/DerivedFieldMapper.java	80.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18706      +/-   ##
============================================
- Coverage     72.90%   72.77%   -0.14%     
+ Complexity    68587    68512      -75     
============================================
  Files          5566     5566              
  Lines        314701   314712      +11     
  Branches      45653    45653              
============================================
- Hits         229434   229032     -402     
- Misses        66655    67057     +402     
- Partials      18612    18623      +11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Mikhail Stepura <mstepura@apple.com>

github-actions · 2025-07-16T12:50:33Z

❌ Gradle check result for 5f37584: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mishail · 2025-07-16T19:25:26Z

Do we have a test anywhere that verifies the field level setting overrides an index-level setting? Perhaps a new case can be added here:

https://github.com/opensearch-project/OpenSearch/blob/3e9a420533f67157ea521fe23212a3a07754c2bf/rest-api-spec/src/main/resources/rest-api-spec/test/search/200_ignore_malformed.yml

Now I've added a new test there. Let me know if that's sufficient

github-actions · 2025-07-16T19:47:17Z

❌ Gradle check result for 76990b8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-16T21:22:43Z

❌ Gradle check result for f44fbf4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-17T03:49:10Z

❕ Gradle check result for 4336b3a: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Andrew Ross <andrross@amazon.com>

Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Mikhail Stepura <mikhail.stepura@outlook.com>

Signed-off-by: Mikhail Stepura <mstepura@apple.com>

github-actions · 2025-07-17T05:32:26Z

✅ Gradle check result for 5a095f2: SUCCESS

mishail · 2025-07-17T06:02:14Z

@andrross this is ready for review. Thanks for the feedback!

Signed-off-by: Mikhail Stepura <mstepura@apple.com>

github-actions · 2025-07-17T23:15:46Z

✅ Gradle check result for 21d0cd5: SUCCESS

…d override index-level setting (opensearch-project#18706) * [fixes opensearch-project#16599] Field-level `ignore_malformed` should override index-level setting Signed-off-by: Mikhail Stepura <mstepura@apple.com> * CHANGELOG.md Signed-off-by: Mikhail Stepura <mstepura@apple.com> * Update CHANGELOG.md Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Mikhail Stepura <mikhail.stepura@outlook.com> * API level test Signed-off-by: Mikhail Stepura <mstepura@apple.com> * Ignore for the versions without the fix Signed-off-by: Mikhail Stepura <mstepura@apple.com> * Cleanup Signed-off-by: Mikhail Stepura <mstepura@apple.com> --------- Signed-off-by: Mikhail Stepura <mstepura@apple.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Mikhail Stepura <mikhail.stepura@outlook.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

[fixes opensearch-project#16599] Field-level ignore_malformed shoul…

be93286

…d override index-level setting Signed-off-by: Mikhail Stepura <mstepura@apple.com>

mishail requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners July 8, 2025 19:44

github-actions bot added bug Something isn't working good first issue Good for newcomers Indexing Indexing, Bulk Indexing and anything related to indexing labels Jul 8, 2025

mishail commented Jul 8, 2025

View reviewed changes

server/src/test/java/org/opensearch/index/mapper/NumberFieldMapperTests.java Show resolved Hide resolved

CHANGELOG.md

1454d32

Signed-off-by: Mikhail Stepura <mstepura@apple.com>

mishail force-pushed the ms/16599-ignore-malformed branch from 2eb4878 to 1454d32 Compare July 9, 2025 00:47

This was referenced Jul 12, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexStatsIT #15836

Open

[AUTOCUT] Gradle Check Flaky Test Report for FullRollingRestartIT #18490

Closed

opensearch-ci-bot mentioned this pull request Jul 16, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteCloseIndexIT #18695

Open

mishail requested a review from andrross July 16, 2025 19:21

mishail force-pushed the ms/16599-ignore-malformed branch from 76990b8 to f44fbf4 Compare July 16, 2025 20:14

andrross and others added 4 commits July 16, 2025 21:03

Merge branch 'main' into ms/16599-ignore-malformed

1fac875

Signed-off-by: Andrew Ross <andrross@amazon.com>

Update CHANGELOG.md

de83af0

Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Mikhail Stepura <mikhail.stepura@outlook.com>

API level test

c4f00d9

Signed-off-by: Mikhail Stepura <mstepura@apple.com>

Ignore for the versions without the fix

5a095f2

Signed-off-by: Mikhail Stepura <mstepura@apple.com>

mishail force-pushed the ms/16599-ignore-malformed branch from 4336b3a to 5a095f2 Compare July 17, 2025 04:04

Cleanup

21d0cd5

Signed-off-by: Mikhail Stepura <mstepura@apple.com>

mishail force-pushed the ms/16599-ignore-malformed branch from 83b0b4f to 21d0cd5 Compare July 17, 2025 21:45

andrross approved these changes Jul 17, 2025

View reviewed changes

andrross merged commit 6e2ff5c into opensearch-project:main Jul 18, 2025
31 checks passed

opensearch-ci-bot mentioned this pull request Jul 16, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ShardIndexingPressureSettingsIT #14331

Open

cwperks mentioned this pull request Jul 24, 2025

Replace usages of ThreadContext.stashContext with pluginSubject.runAs opensearch-project/geospatial#715

Merged

5 tasks

opensearch-ci-bot mentioned this pull request Jun 6, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreRefreshListenerTests #17174

Open

BrewTestBot mentioned this pull request Aug 20, 2025

opensearch 3.2.0 Homebrew/homebrew-core#234146

Merged

opensearch-ci-bot mentioned this pull request Oct 22, 2025

[AUTOCUT] Gradle Check Flaky Test Report for FlightClientManagerTests #19728

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fixes #16599] Field-level `ignore_malformed` should override index-level setting #18706

[fixes #16599] Field-level `ignore_malformed` should override index-level setting #18706

Uh oh!

mishail commented Jul 8, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

codecov bot commented Jul 8, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 16, 2025

Uh oh!

mishail commented Jul 16, 2025

Uh oh!

github-actions bot commented Jul 16, 2025

Uh oh!

github-actions bot commented Jul 16, 2025

Uh oh!

github-actions bot commented Jul 17, 2025

Uh oh!

github-actions bot commented Jul 17, 2025

Uh oh!

mishail commented Jul 17, 2025

Uh oh!

github-actions bot commented Jul 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[fixes #16599] Field-level ignore_malformed should override index-level setting #18706

[fixes #16599] Field-level ignore_malformed should override index-level setting #18706

Uh oh!

Conversation

mishail commented Jul 8, 2025

Description

Related Issues

Check List

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

codecov bot commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jul 16, 2025

Uh oh!

mishail commented Jul 16, 2025

Uh oh!

github-actions bot commented Jul 16, 2025

Uh oh!

github-actions bot commented Jul 16, 2025

Uh oh!

github-actions bot commented Jul 17, 2025

Uh oh!

github-actions bot commented Jul 17, 2025

Uh oh!

mishail commented Jul 17, 2025

Uh oh!

github-actions bot commented Jul 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[fixes #16599] Field-level `ignore_malformed` should override index-level setting #18706

[fixes #16599] Field-level `ignore_malformed` should override index-level setting #18706

codecov bot commented Jul 8, 2025 •

edited

Loading