Skip to content

feat: support label selectors #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 35 additions & 2 deletions docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,41 @@ and `["list", "watch"]` for related resources.

The project is mainly tested with cluster-scoped deployment, however, QOSDK namespace-scoped deployments are also supported.

See also the upcoming deployment modes/options: [sharding with label selectors](https://github.com/csviri/kubernetes-glue-operator/issues/50),
[watching only one custom resources type](https://github.com/csviri/kubernetes-glue-operator/issues/54)
### Sharding with Label Selectors

The operator can be deployed to only target certain `Glue` or `GlueOperator` resources based on [label selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/).
You can use simply the [configuration](https://docs.quarkiverse.io/quarkus-operator-sdk/dev/includes/quarkus-operator-sdk.html#quarkus-operator-sdk_quarkus-operator-sdk-controllers-controllers-selector)
from Quarkus Operator SDK to set the label selector for the reconciler.

The configuration for `Glue` looks like:

`quarkus.operator-sdk.controllers.glue.selector=mylabel=myvalue`

for `GlueOperator`:

`quarkus.operator-sdk.controllers.glue-operator.selector=mylabel=myvalue`

This will work with any label selector for `GlueOperator` and with simple label selectors for `Glue`,
thus in `key=value` or just `key` form.


With `Glue` there is a caveat. `GlueOperator` works in a way that it creates a `Glue` resource for every
custom resource tracked, so if there is a label selector defined for `Glue` it needs to add this label
to the `Glue` resource when it is created. Since it is not trivial to parse label selectors, in more
complex forms of label selectors (other the ones mentioned above), the labels to add to the `Glue` resources
by a `GlueOperator` needs to be specified explicitly using
[`glue.operator.glue-operator-managed-glue-labels`](https://github.com/csviri/kubernetes-glue-operator/blob/main/src/main/java/io/csviri/operator/glue/ControllerConfig.java#L10-L10)
config key (which is a type of map). Therefore, for a label selector that specified two values for a glue:

`quarkus.operator-sdk.controllers.glue.selector=mylabel1=value1,mylabel2=value2`

the following two configuration params needs to be added:

`glue.operator.glue-operator-managed-glue-labels.mylabel1=value1`
`glue.operator.glue-operator-managed-glue-labels.mylabel2=value2`

This will ensure that the labels are added correctly to the `Glue`. See the related
[integration test](https://github.com/csviri/kubernetes-glue-operator/blob/main/src/test/java/io/csviri/operator/glue/GlueOperatorComplexLabelSelectorTest.java#L23-L23).

## Implementation details and performance

Expand Down
12 changes: 12 additions & 0 deletions src/main/java/io/csviri/operator/glue/ControllerConfig.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
package io.csviri.operator.glue;

import java.util.Map;

import io.smallrye.config.ConfigMapping;

@ConfigMapping(prefix = "glue.operator")
public interface ControllerConfig {

Map<String, String> glueOperatorManagedGlueLabels();

}
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,13 @@
import static io.csviri.operator.glue.Utils.getResourceForSSAFrom;
import static io.csviri.operator.glue.reconciler.operator.GlueOperatorReconciler.PARENT_RELATED_RESOURCE_NAME;

@ControllerConfiguration
@ControllerConfiguration(name = GlueReconciler.GLUE_RECONCILER_NAME)
public class GlueReconciler implements Reconciler<Glue>, Cleaner<Glue>, ErrorStatusHandler<Glue> {

private static final Logger log = LoggerFactory.getLogger(GlueReconciler.class);
public static final String DEPENDENT_NAME_ANNOTATION_KEY = "io.csviri.operator.resourceflow/name";
public static final String PARENT_GLUE_FINALIZER_PREFIX = "io.csviri.operator.resourceflow.glue/";
public static final String GLUE_RECONCILER_NAME = "glue";

@Inject
ValidationAndErrorHandler validationAndErrorHandler;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
package io.csviri.operator.glue.reconciler.operator;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.*;

import org.eclipse.microprofile.config.inject.ConfigProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import io.csviri.operator.glue.ControllerConfig;
import io.csviri.operator.glue.GlueException;
import io.csviri.operator.glue.customresource.glue.Glue;
import io.csviri.operator.glue.customresource.glue.GlueSpec;
import io.csviri.operator.glue.customresource.glue.RelatedResourceSpec;
Expand All @@ -25,9 +25,12 @@
import io.javaoperatorsdk.operator.processing.event.source.EventSource;
import io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource;

import jakarta.annotation.PostConstruct;
import jakarta.inject.Inject;

@ControllerConfiguration
import static io.csviri.operator.glue.reconciler.glue.GlueReconciler.GLUE_RECONCILER_NAME;

@ControllerConfiguration(name = GlueOperatorReconciler.GLUE_OPERATOR_RECONCILER_NAME)
public class GlueOperatorReconciler
implements Reconciler<GlueOperator>, EventSourceInitializer<GlueOperator>,
Cleaner<GlueOperator>, ErrorStatusHandler<GlueOperator> {
Expand All @@ -37,11 +40,25 @@ public class GlueOperatorReconciler
public static final String GLUE_LABEL_KEY = "foroperator";
public static final String GLUE_LABEL_VALUE = "true";
public static final String PARENT_RELATED_RESOURCE_NAME = "parent";
public static final String GLUE_OPERATOR_RECONCILER_NAME = "glue-operator";

@Inject
ValidationAndErrorHandler validationAndErrorHandler;

private InformerEventSource<Glue, GlueOperator> resourceFlowEventSource;
@ConfigProperty(name = "quarkus.operator-sdk.controllers." + GLUE_RECONCILER_NAME + ".selector")
Optional<String> glueLabelSelector;

@Inject
ControllerConfig controllerConfig;

private Map<String, String> defaultGlueLabels;

private InformerEventSource<Glue, GlueOperator> glueEventSource;

@PostConstruct
void init() {
defaultGlueLabels = initDefaultLabelsToAddToGlue();
}

@Override
public UpdateControl<GlueOperator> reconcile(GlueOperator glueOperator,
Expand All @@ -54,9 +71,10 @@ public UpdateControl<GlueOperator> reconcile(GlueOperator glueOperator,

var targetCREventSource = getOrRegisterCustomResourceEventSource(glueOperator, context);
targetCREventSource.list().forEach(cr -> {
var actualResourceFlow = resourceFlowEventSource
.get(new ResourceID(glueName(cr), cr.getMetadata().getNamespace()));
var desiredResourceFlow = createResourceFlow(cr, glueOperator);
var actualResourceFlow = glueEventSource
.get(new ResourceID(glueName(cr.getMetadata().getName(), cr.getKind()),
cr.getMetadata().getNamespace()));
var desiredResourceFlow = createGlue(cr, glueOperator);
if (actualResourceFlow.isEmpty()) {
context.getClient().resource(desiredResourceFlow).serverSideApply();
} else if (!actualResourceFlow.orElseThrow().getSpec()
Expand All @@ -72,17 +90,22 @@ public UpdateControl<GlueOperator> reconcile(GlueOperator glueOperator,
return UpdateControl.noUpdate();
}

private Glue createResourceFlow(GenericKubernetesResource targetParentResource,
private Glue createGlue(GenericKubernetesResource targetParentResource,
GlueOperator glueOperator) {
var glue = new Glue();

glue.setMetadata(new ObjectMetaBuilder()
.withName(glueName(targetParentResource))
.withName(
glueName(targetParentResource.getMetadata().getName(), targetParentResource.getKind()))
.withNamespace(targetParentResource.getMetadata().getNamespace())
.withLabels(Map.of(GLUE_LABEL_KEY, GLUE_LABEL_VALUE))
.build());
glue.setSpec(toWorkflowSpec(glueOperator.getSpec()));

if (!defaultGlueLabels.isEmpty()) {
glue.getMetadata().getLabels().putAll(defaultGlueLabels);
}

var parent = glueOperator.getSpec().getParent();
RelatedResourceSpec parentRelatedSpec = new RelatedResourceSpec();
parentRelatedSpec.setName(PARENT_RELATED_RESOURCE_NAME);
Expand Down Expand Up @@ -129,12 +152,12 @@ private InformerEventSource<GenericKubernetesResource, GlueOperator> getOrRegist
@Override
public Map<String, EventSource> prepareEventSources(
EventSourceContext<GlueOperator> eventSourceContext) {
resourceFlowEventSource = new InformerEventSource<>(
glueEventSource = new InformerEventSource<>(
InformerConfiguration.from(Glue.class, eventSourceContext)
.withLabelSelector(GLUE_LABEL_KEY + "=" + GLUE_LABEL_VALUE)
.build(),
eventSourceContext);
return EventSourceInitializer.nameEventSources(resourceFlowEventSource);
return EventSourceInitializer.nameEventSources(glueEventSource);
}

@Override
Expand All @@ -155,9 +178,34 @@ public DeleteControl cleanup(GlueOperator glueOperator,
return DeleteControl.defaultDelete();
}

private static String glueName(GenericKubernetesResource cr) {
return KubernetesResourceUtil.sanitizeName(cr.getMetadata().getName() + "-" + cr.getKind());
public static String glueName(String name, String kind) {
return KubernetesResourceUtil.sanitizeName(name + "-" + kind);
}

private Map<String, String> initDefaultLabelsToAddToGlue() {
Map<String, String> res = new HashMap<>();
if (!controllerConfig.glueOperatorManagedGlueLabels().isEmpty()) {
res.putAll(controllerConfig.glueOperatorManagedGlueLabels());
} else {
glueLabelSelector.ifPresent(ls -> {
if (ls.contains(",") || ls.contains("(")) {
throw new GlueException(
"Glue reconciler label selector contains non-simple label selector: " + ls +
". Specify Glue label selector in simple form ('key=value' or 'key') " +
"or configure 'glue.operator.glue-operator-managed-glue-labels'");
}
String[] labelSelectorParts = ls.split("=");
if (labelSelectorParts.length > 2) {
throw new GlueException("Invalid label selector: " + ls);
}
if (labelSelectorParts.length == 1) {
res.put(labelSelectorParts[0], "");
} else {
res.put(labelSelectorParts[0], labelSelectorParts[1]);
}
});
}
return res;
}

}
54 changes: 54 additions & 0 deletions src/test/java/io/csviri/operator/glue/GlueLabelSelectorTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
package io.csviri.operator.glue;


import java.util.Map;

import org.junit.jupiter.api.Test;

import io.csviri.operator.glue.customresource.glue.Glue;
import io.fabric8.kubernetes.api.model.ConfigMap;
import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.test.junit.QuarkusTestProfile;
import io.quarkus.test.junit.TestProfile;

import static io.csviri.operator.glue.TestUtils.INITIAL_RECONCILE_WAIT_TIMEOUT;
import static io.csviri.operator.glue.reconciler.glue.GlueReconciler.GLUE_RECONCILER_NAME;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;

@QuarkusTest
@TestProfile(GlueLabelSelectorTest.LabelSelectorTestProfile.class)
public class GlueLabelSelectorTest extends TestBase {


public static final String LABEL_KEY = "test-glue";
public static final String LABEL_VALUE = "true";

@Test
void testLabelSelectorHandling() {
Glue glue =
TestUtils.loadResoureFlow("/glue/SimpleGlue.yaml");
glue = create(glue);

await().pollDelay(INITIAL_RECONCILE_WAIT_TIMEOUT).untilAsserted(() -> {
assertThat(get(ConfigMap.class, "simple-glue-configmap")).isNull();
});

glue.getMetadata().getLabels().put(LABEL_KEY, LABEL_VALUE);
update(glue);

await().untilAsserted(() -> {
assertThat(get(ConfigMap.class, "simple-glue-configmap")).isNotNull();
});
}

public static class LabelSelectorTestProfile implements QuarkusTestProfile {

@Override
public Map<String, String> getConfigOverrides() {
return Map.of("quarkus.operator-sdk.controllers." + GLUE_RECONCILER_NAME + ".selector",
LABEL_KEY + "=" + LABEL_VALUE);
}
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
package io.csviri.operator.glue;

import java.util.Map;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import io.csviri.operator.glue.customresource.TestCustomResource;
import io.csviri.operator.glue.customresource.TestCustomResource2;
import io.csviri.operator.glue.customresource.glue.Glue;
import io.csviri.operator.glue.reconciler.operator.GlueOperatorReconciler;
import io.fabric8.kubernetes.api.model.ConfigMap;
import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.test.junit.QuarkusTestProfile;
import io.quarkus.test.junit.TestProfile;

import static io.csviri.operator.glue.reconciler.glue.GlueReconciler.GLUE_RECONCILER_NAME;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;

@QuarkusTest
@TestProfile(GlueOperatorComplexLabelSelectorTest.GlueOperatorComplexLabelSelectorTestProfile.class)
public class GlueOperatorComplexLabelSelectorTest extends TestBase {

public static final String GLUE_LABEL_KEY1 = "test-glue1";
public static final String GLUE_LABEL_KEY2 = "test-glue2";
public static final String LABEL_VALUE = "true";

@BeforeEach
void applyCRD() {
TestUtils.applyTestCrd(client, TestCustomResource.class, TestCustomResource2.class);
}

@Test
void testGlueOperatorLabelSelector() {
var go = create(TestUtils
.loadResourceFlowOperator("/glueoperator/SimpleGlueOperator.yaml"));

var testCR = create(TestData.testCustomResource());

await().untilAsserted(() -> {
assertThat(get(ConfigMap.class, testCR.getMetadata().getName())).isNotNull();
var glue = get(Glue.class, GlueOperatorReconciler.glueName(testCR.getMetadata().getName(),
testCR.getKind()));
assertThat(glue).isNotNull();
assertThat(glue.getMetadata().getLabels())
.containsEntry(GLUE_LABEL_KEY1, LABEL_VALUE)
.containsEntry(GLUE_LABEL_KEY2, LABEL_VALUE);
});

delete(testCR);
await().untilAsserted(() -> {
var glue = get(Glue.class, GlueOperatorReconciler.glueName(testCR.getMetadata().getName(),
testCR.getKind()));
assertThat(glue).isNull();
});
delete(go);
}

public static class GlueOperatorComplexLabelSelectorTestProfile implements QuarkusTestProfile {

@Override
public Map<String, String> getConfigOverrides() {
return Map.of("quarkus.operator-sdk.controllers." + GLUE_RECONCILER_NAME + ".selector",
// complex label selector with 2 values checked
GLUE_LABEL_KEY1 + "=" + LABEL_VALUE + "," + GLUE_LABEL_KEY2 + "=" + LABEL_VALUE,
// explicit labels added to glue
"glue.operator.glue-operator-managed-glue-labels." + GLUE_LABEL_KEY1, LABEL_VALUE,
"glue.operator.glue-operator-managed-glue-labels." + GLUE_LABEL_KEY2, LABEL_VALUE);
}
}

}
Loading