Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure index templates are not applied to system indices #16418

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Fix typo super->sb in method toString() of RemoteStoreNodeAttribute ([#15362](https://github.com/opensearch-project/OpenSearch/pull/15362))
- [Workload Management] Fixing Create/Update QueryGroup TransportActions to execute from non-cluster manager nodes ([16422](https://github.com/opensearch-project/OpenSearch/pull/16422))
- Fix flaky test in `testApproximateRangeWithSizeOverDefault` by adjusting totalHits assertion logic ([#16434](https://github.com/opensearch-project/OpenSearch/pull/16434#pullrequestreview-2386999409))
- Ensure index templates are not applied to system indices ([#16418](https://github.com/opensearch-project/OpenSearch/pull/16418))

### Security

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -434,15 +434,21 @@ public ClusterState applyCreateIndexRequest(
// in which case templates don't apply, so create the index from the source metadata
return applyCreateIndexRequestWithExistingMetadata(currentState, request, silent, sourceMetadata, metadataTransformer);
} else {
// The backing index may have a different name or prefix than the data stream name.
final String name = request.dataStreamName() != null ? request.dataStreamName() : request.index();

// Do not apply any templates to system indices
if (systemIndices.isSystemIndex(name)) {
reta marked this conversation as resolved.
Show resolved Hide resolved
return applyCreateIndexRequestWithNoTemplates(currentState, request, silent, metadataTransformer);
}

// Hidden indices apply templates slightly differently (ignoring wildcard '*'
// templates), so we need to check to see if the request is creating a hidden index
// prior to resolving which templates it matches
final Boolean isHiddenFromRequest = IndexMetadata.INDEX_HIDDEN_SETTING.exists(request.settings())
? IndexMetadata.INDEX_HIDDEN_SETTING.get(request.settings())
: null;

// The backing index may have a different name or prefix than the data stream name.
final String name = request.dataStreamName() != null ? request.dataStreamName() : request.index();
// Check to see if a v2 template matched
final String v2Template = MetadataIndexTemplateService.findV2Template(
currentState.metadata(),
Expand Down Expand Up @@ -676,7 +682,18 @@ public void addRemoteStoreCustomMetadata(IndexMetadata.Builder tmpImdBuilder, bo
tmpImdBuilder.putCustom(IndexMetadata.REMOTE_STORE_CUSTOM_KEY, remoteCustomData);
}

private ClusterState applyCreateIndexRequestWithV1Templates(
ClusterState applyCreateIndexRequestWithNoTemplates(
final ClusterState currentState,
final CreateIndexClusterStateUpdateRequest request,
final boolean silent,
final BiConsumer<Metadata.Builder, IndexMetadata> metadataTransformer
) throws Exception {
// Using applyCreateIndexRequestWithV1Templates with empty list instead of applyCreateIndexRequestWithV2Template
// with null template as applyCreateIndexRequestWithV2Template has assertions when template is null
return applyCreateIndexRequestWithV1Templates(currentState, request, silent, Collections.emptyList(), metadataTransformer);
cwperks marked this conversation as resolved.
Show resolved Hide resolved
}

ClusterState applyCreateIndexRequestWithV1Templates(
final ClusterState currentState,
final CreateIndexClusterStateUpdateRequest request,
final boolean silent,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,11 @@
import static org.hamcrest.Matchers.notNullValue;
import static org.hamcrest.Matchers.nullValue;
import static org.hamcrest.Matchers.startsWith;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;

public class MetadataCreateIndexServiceTests extends OpenSearchTestCase {
Expand Down Expand Up @@ -2559,6 +2563,62 @@ public void testApplyContextWithSettingsOverlap() throws IOException {
}
}

public void testApplyCreateIndexRequestWithNoTemplates() throws Exception {
BiConsumer<Metadata.Builder, IndexMetadata> mockMetadataTransformer = mock(BiConsumer.class);
SystemIndices mockSystemIndices = mock(SystemIndices.class);
when(mockSystemIndices.isSystemIndex(anyString())).thenReturn(true);

ClusterService clusterService = mock(ClusterService.class);
ClusterState clusterState = ClusterState.builder(org.opensearch.cluster.ClusterName.CLUSTER_NAME_SETTING.getDefault(Settings.EMPTY))
.metadata(Metadata.EMPTY_METADATA)
.build();

ThreadPool threadPool = new TestThreadPool(getTestName());

ClusterSettings clusterSettings = new ClusterSettings(Settings.EMPTY, ClusterSettings.BUILT_IN_CLUSTER_SETTINGS);
when(clusterService.getSettings()).thenReturn(Settings.EMPTY);
when(clusterService.getClusterSettings()).thenReturn(clusterSettings);
when(clusterService.state()).thenReturn(clusterState);

MetadataCreateIndexService metadataCreateIndexService = spy(
new MetadataCreateIndexService(
Settings.EMPTY,
clusterService,
indicesServices,
null,
null,
createTestShardLimitService(randomIntBetween(1, 1000), false, clusterService),
new Environment(Settings.builder().put("path.home", "dummy").build(), null),
IndexScopedSettings.DEFAULT_SCOPED_SETTINGS,
threadPool,
null,
mockSystemIndices,
true,
new AwarenessReplicaBalance(Settings.EMPTY, clusterService.getClusterSettings()),
DefaultRemoteStoreSettings.INSTANCE,
repositoriesServiceSupplier
)
);

metadataCreateIndexService.applyCreateIndexRequest(clusterState, request, false, mockMetadataTransformer);
verify(metadataCreateIndexService).applyCreateIndexRequestWithNoTemplates(
eq(clusterState),
eq(request),
eq(false),
eq(mockMetadataTransformer)
);

verify(metadataCreateIndexService).applyCreateIndexRequestWithV1Templates(
eq(clusterState),
eq(request),
eq(false),
eq(Collections.emptyList()),
eq(mockMetadataTransformer)
);
Comment on lines +2603 to +2617
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally don't think tests like these add a lot of value. You're really just testing that you're calling an internal method that you think you're calling. This isn't verifying that the code is actually doing the thing you expect it to be doing (i.e. that index templates are not actually applied to a system index). Furthermore, this becomes brittle to internal refactorings of the class under test even if the refactoring does not change the behavior of the class at all.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any way to create an integration test of the scenario you observed that would validate this fix?

Good call-out. This seems the better approach here. The repro in the original issue should be easy to set up in an integ test, the only hitch here is making sure systemIndices is populated with whatever test index we're choosing, unless the test is isolated enough that we can use one of the actual indices specified.

Copy link
Collaborator

@reta reta Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having an YAML spec or IT test case would definitely be helpful

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, IT test will actually validate this behavior rather than simply check for a method call. Is there a guide to how such a test can be added? I could take a look.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pyek-bot Here is an example test that asserts that deprecation warnings are emitted when a user writes to a system index:

public void testSystemIndexAccessBlockedByDefault() throws Exception {


threadPool.shutdown();
}

private IndexTemplateMetadata addMatchingTemplate(Consumer<IndexTemplateMetadata.Builder> configurator) {
IndexTemplateMetadata.Builder builder = templateMetadataBuilder("template1", "te*");
configurator.accept(builder);
Expand Down
Loading