Skip to content

Revert "Enable skip_list for @timestamp field or index sort field #19661

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Revert "Enable skip_list for @timestamp field or index sort field #19661

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f7b1ee0
Revert "Enable skip_list for @timestamp field or index sort field (#1…
asimmahmood1 Oct 16, 2025
14e006f
Merge branch 'main' into revert_timestamp_skip_list
jainankitk Oct 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Avoid primary shard failure caused by merged segment warmer exceptions ([#19436](https://github.com/opensearch-project/OpenSearch/pull/19436))
- Fix pull-based ingestion out-of-bounds offset scenarios and remove persisted offsets ([#19607](https://github.com/opensearch-project/OpenSearch/pull/19607))
- [Star Tree] Fix sub-aggregator casting for search with profile=true ([19652](https://github.com/opensearch-project/OpenSearch/pull/19652))
- Remove default enable skip_list for @timestamp field or index sort field to fix upgrades ([19611](https://github.com/opensearch-project/OpenSearch/pull/19661))
- Fix issue with updating core with a patch number other than 0 ([#19377](https://github.com/opensearch-project/OpenSearch/pull/19377))

### Dependencies
Expand Down
18 changes: 1 addition & 17 deletions server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@
import org.opensearch.common.util.FeatureFlags;
import org.opensearch.common.util.LocaleUtils;
import org.opensearch.common.xcontent.support.XContentMapValues;
import org.opensearch.index.IndexSortConfig;
import org.opensearch.index.compositeindex.datacube.DimensionType;
import org.opensearch.index.fielddata.IndexFieldData;
import org.opensearch.index.fielddata.IndexNumericFieldData.NumericType;
Expand Down Expand Up @@ -754,7 +753,6 @@ public DocValueFormat docValueFormat(@Nullable String format, ZoneId timeZone) {
private final boolean indexed;
private final boolean hasDocValues;
private final boolean skiplist;
private final boolean isSkiplistConfigured;
private final Locale locale;
private final String format;
private final String printFormat;
Expand All @@ -780,7 +778,6 @@ private DateFieldMapper(
this.indexed = builder.index.getValue();
this.hasDocValues = builder.docValues.getValue();
this.skiplist = builder.skiplist.getValue();
this.isSkiplistConfigured = builder.skiplist.isConfigured();
this.locale = builder.locale.getValue();
this.format = builder.format.getValue();
this.printFormat = builder.printFormat.getValue();
Expand Down Expand Up @@ -838,7 +835,7 @@ protected void parseCreateField(ParseContext context) throws IOException {
context.doc().add(new LongPoint(fieldType().name(), timestamp));
}
if (hasDocValues) {
if (skiplist || isSkiplistDefaultEnabled(context.indexSettings().getIndexSortConfig(), fieldType().name())) {
if (skiplist) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this result in same error if someone enables skiplist setting on a cluster having indices created with <= 2.19 version?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if we can leverage the dateFieldMapper.indexCreatedVersion property to validate the index version?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, skip_list mapping is on creation only, so OS will reject the request to update the mapping.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I was repling to your 1st question.

I am wondering if we can leverage the dateFieldMapper.indexCreatedVersion property to validate the index version?

That should work for upgrade case. Let me confirm it manually, that'll be easy, then I can come up with bwc test case.

context.doc().add(SortedNumericDocValuesField.indexedField(fieldType().name(), timestamp));
} else {
context.doc().add(new SortedNumericDocValuesField(fieldType().name(), timestamp));
Expand All @@ -851,19 +848,6 @@ protected void parseCreateField(ParseContext context) throws IOException {
}
}

boolean isSkiplistDefaultEnabled(IndexSortConfig indexSortConfig, String fieldName) {
if (!isSkiplistConfigured) {
if (indexSortConfig.hasPrimarySortOnField(fieldName)) {
return true;
}
if (DataStreamFieldMapper.Defaults.TIMESTAMP_FIELD.getName().equals(fieldName)) {
return true;
}

}
return false;
}

@Override
protected String getFieldValue(ParseContext context) throws IOException {
if (context.externalValueSet()) {
Expand Down
34 changes: 0 additions & 34 deletions server/src/test/java/org/opensearch/index/mapper/DateFieldMapperTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,10 @@
import org.apache.lucene.index.StoredFields;
import org.apache.lucene.index.VectorEncoding;
import org.apache.lucene.index.VectorSimilarityFunction;
import org.opensearch.Version;
import org.opensearch.cluster.metadata.IndexMetadata;
import org.opensearch.common.settings.Settings;
import org.opensearch.common.time.DateFormatter;
import org.opensearch.common.util.FeatureFlags;
import org.opensearch.common.xcontent.XContentFactory;
import org.opensearch.core.xcontent.XContentBuilder;
import org.opensearch.index.IndexSettings;
import org.opensearch.index.IndexSortConfig;
import org.opensearch.index.fieldvisitor.SingleFieldsVisitor;
import org.opensearch.index.termvectors.TermVectorsService;
import org.opensearch.search.DocValueFormat;
Expand Down Expand Up @@ -867,35 +862,6 @@ public void testSkipListIntegrationMappingDefinitionSerialization() throws IOExc
assertTrue("skip_list should be true in date_nanos mapper", dateFieldMapperNanos.skiplist());
}

public void testIsSkiplistDefaultEnabled() throws IOException {
DocumentMapper mapper = createDocumentMapper(fieldMapping(b -> b.field("type", "date")));
DateFieldMapper dateFieldMapper = (DateFieldMapper) mapper.mappers().getMapper("field");

// Test with no index sort and non-timestamp field
IndexMetadata noSortindexMetadata = new IndexMetadata.Builder("index").settings(getIndexSettings()).build();
IndexSettings noSolrIndexSettings = new IndexSettings(noSortindexMetadata, getIndexSettings());
IndexSortConfig noSortConfig = new IndexSortConfig(new IndexSettings(noSortindexMetadata, getIndexSettings()));
assertFalse(dateFieldMapper.isSkiplistDefaultEnabled(noSortConfig, "field"));

// timestamp field
assertTrue(dateFieldMapper.isSkiplistDefaultEnabled(noSortConfig, "@timestamp"));

// Create index settings with an index sort.
Settings settings = Settings.builder()
.put(IndexMetadata.SETTING_VERSION_CREATED, Version.CURRENT)
.put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, 1)
.put(IndexMetadata.SETTING_NUMBER_OF_REPLICAS, 1)
.putList("index.sort.field", "field")
.build();

// Test with timestamp field
IndexMetadata indexMetadata = new IndexMetadata.Builder("index").settings(settings).build();
IndexSettings indexSettings = new IndexSettings(indexMetadata, settings);
IndexSortConfig sortConfig = new IndexSortConfig(indexSettings);
assertTrue(dateFieldMapper.isSkiplistDefaultEnabled(sortConfig, "field"));
assertTrue(dateFieldMapper.isSkiplistDefaultEnabled(sortConfig, "@timestamp"));
}

public void testSkipListIntegrationFieldBehaviorConsistency() throws IOException {
// Test that field behavior is consistent between skip_list enabled and disabled

Expand Down
6 changes: 1 addition & 5 deletions test/framework/src/main/java/org/opensearch/index/mapper/MapperServiceTestCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,7 @@

public abstract class MapperServiceTestCase extends OpenSearchTestCase {

protected static final Settings SETTINGS = Settings.builder()
.put("index.version.created", Version.CURRENT)
.put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, 1)
.put(IndexMetadata.SETTING_NUMBER_OF_REPLICAS, 1)
.build();
protected static final Settings SETTINGS = Settings.builder().put("index.version.created", Version.CURRENT).build();

protected static final ToXContent.Params INCLUDE_DEFAULTS = new ToXContent.MapParams(
Collections.singletonMap("include_defaults", "true")
Expand Down
Loading