This repository demonstrates some of the basic features of Red Hat’s latest Data Grid release and how to deploy a Data Grid cluster locally and on OCP.
- 1. Introduction
- 2. Deploying standalone
- 3. Deploying RHDG on OCP using the Operator
- 4. Creating a Cross-site replication cluster
- 5. Deploying RHDG on OCP using Helm Charts
- 6. Monitoring RHDG with Grafana
- 7. Java Flight Recorder
- Annex A: Adding Datasource configuration
- Annex B: Interact with caches using REST
- Annex C: Configure the scope of your operator
- Annex D: Stern - Tail logs from multiple pods
- Annex E: Advanced stats, and reporting for RHDG
- Annex F: Convert cache configurations
- Annex G: Getting full CR example
- Useful links
Red Hat Data Grid is an in-memory, distributed, NoSQL datastore solution. Your applications can access, process, and analyze data at in-memory speed to deliver a superior user experience.
Red Hat Data Grid provides value as a standard architectural component in application infrastructures for a variety of real-world scenarios and usecases:
-
Data caching and transient data storage.
-
Primary data store.
-
Low latency compute grid.
Currently, Data Grid is supported both using the binary on VMs and using the container images on Openshift. Due to the huge benefits of deploying our applications containerized in Openshift, we will use the same image to launch the Data Grid cluster locally using podman-compose:
podman-compose -f rhdg-local/docker-compose.yaml up eu-01podman-compose -f rhdg-local/docker-compose.yaml upIf the cluster formation goes well, you will see that the JGroups Cluster View is updated to the two servers:
[eu-01] | 2023-06-27 07:48:06,213 INFO (non-blocking-thread-infinispan-europe-01-p2-t1) [org.infinispan.LIFECYCLE] [Context=___hotRodTopologyCache_hotrod-default][Scope=infinispan-europe-01]ISPN100010: Finished rebalance with members [infinispan-europe-02, infinispan-europe-01], topology id 7
[eu-02] | 2023-06-27 07:48:06,224 INFO (jgroups-5,infinispan-europe-02) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 8
[eu-02] | 2023-06-27 07:48:06,227 INFO (jgroups-6,infinispan-europe-02) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 9
[eu-02] | 2023-06-27 07:48:06,230 INFO (jgroups-5,infinispan-europe-02) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100010: Finished rebalance with members [infinispan-europe-02, infinispan-europe-01], topology id 10
//...
[eu-01] | 2023-06-27 07:48:06,401 INFO (main) [org.infinispan.SERVER] ISPN080001: Infinispan Server 14.0.1.Final started in 2777ms|
ℹ️
|
For ways of deploying the server using the .zip, check the following version of this repository. |
Install Data Grid Operator into an OpenShift namespace to create and manage Data Grid clusters.
To deploy the RHDG operator, you will need to create four objects: The namespace of the Operator, the namespace where the Data Grid cluster will live, the Subscription and the Operator Group.
|
âť—
|
Bear in mind that you will need cluster-admin permissions to deploy an operator, as it is necessary to create cluster-wide CRDs (Custom Resource Definitions).
|
Execute the following command to deploy the operator:
oc process -f rhdg-operator/rhdg-01-operator.yaml -p OPERATOR_NAMESPACE="rhdg8-operator" -p CLUSTER_NAMESPACE="rhdg8" | oc apply -f -It is also possible to install the operator from the web console. For more information, please check the official documentation.
The Data Grid operator defines the Infinispan CRD to simplify the process of creation, configuration, and management of DG clusters.
I have created several OCP templates to quickly deploy different set-ups of Data Grid:
| Template | Authentication | Authorization | SSL Encryption |
|---|---|---|---|
❌ |
❌ |
❌ |
|
✔️ |
✔️ |
❌ |
|
✔️ |
✔️ |
✔️ |
Execute the following commands depending on the cluster that you want to deploy:
oc process -f rhdg-operator/rhdg-02-cluster-basic.yaml -p CLUSTER_NAME="cluster-a" | oc apply -f -
oc process -f rhdg-operator/rhdg-02-cluster-auth.yaml -p CLUSTER_NAME="cluster-b" | oc apply -f -
oc process -f rhdg-operator/rhdg-02-cluster-auth-ssl.yaml -p CLUSTER_NAME="cluster-c" | oc apply -f -|
ℹ️
|
In this section, we will explore how to create caches using the Operator. For other ways of creating caches, please check this other Git repository with information about the Data Grid client. |
I have created an OCP template to quickly set up several caches on the RHDG cluster:
-
operator-cache-01: Based on an XML configuration. -
operator-cache-02: Based on a YAML configuration. -
operator-cache-03: Based on an already defined templated.
In order to apply this template, just execute the following command:
oc process -f rhdg-operator/rhdg-03-caches.yaml -p CLUSTER_NAMESPACE="rhdg8" -p CLUSTER_NAME="rhdg" | oc apply -f -For more information about how to create caches using the CRD, please check the official documentation.
Data Grid exposes a metrics endpoint that provides statistics and events in Prometheus format.
|
|
Enabling monitoring for user-defined projects
Do not execute it before checking if this was done before, you can override work from your colleagues: oc apply -f ocp/ocp-01-user-workload-monitoring.yamlAfter executing the command above, you will see several pods in the following namespace: oc get pods -n openshift-user-workload-monitoring |
I have created an OCP template to quickly configure metrics monitoring of an RHDG cluster. Execute the following command:
oc process -f rhdg-operator/rhdg-04-monitoring.yaml -p CLUSTER_NAMESPACE="rhdg8" -p CLUSTER_NAME="rhdg" | oc apply -f -For more information, access the Openshift documentation for the monitoring stack and the RHDG documentation to configure monitoring for RHDG 8 on OCP.
oc process -f rhdg-operator/rhdg-02-cluster-cross-site.yaml \
-p CLUSTER_NAMESPACE="rhdg8-a" \
-p CLUSTER_NAME="cluster-a" \
-p CROSS_SITE_NAMESPACE="rhdg8-b" \
-p CROSS_SITE_CLUSTER="cluster-b" | oc apply -f -oc process -f rhdg-operator/rhdg-02-cluster-cross-site.yaml \
-p CLUSTER_NAMESPACE="rhdg8-b" \
-p CLUSTER_NAME="cluster-b" \
-p CROSS_SITE_NAMESPACE="rhdg8-a" \
-p CROSS_SITE_CLUSTER="cluster-a" | oc apply -f -CLUSTER_A_URL=$(oc get route cluster-a-external -n rhdg8-a -o template='http://{{.spec.host}}')
CLUSTER_B_URL=$(oc get route cluster-b-external -n rhdg8-b -o template='http://{{.spec.host}}')
# Check the configuration of the cache in each environment (It points to the other environment)
curl -H "Content-Type: application/json" ${CLUSTER_A_URL}/rest/v2/caches/cross-site-cache?action=config | jq .
# Add an entry in clusterA
curl -X POST ${CLUSTER_A_URL}/rest/v2/caches/cross-site-cache/hello \
-H "Key-Content-Type: text/plain" \
-H "Content-Type: text/plain" \
-d 'world'
# Retrieve keys from clusterB
curl ${CLUSTER_B_URL}/rest/v2/caches/cross-site-cache?action=keysIn order to create your first deployment easily, first add the OpenShift Helm Charts repository:
helm repo add openshift-helm-charts https://charts.openshift.io/Create a new OCP project:
oc new-project rhdg8-helm --display-name="RHDG 8 - Helm" --description="This namespace contains a deployment of RHDG using the official Helm Chart"Then, modify the rhdg-chart/default-values.yaml to configure your deployment:
helm install rhdg openshift-helm-charts/redhat-data-grid -f rhdg-chart/minimal-values.yamlYou will be able to authenticate to the cluster using the credentials obtained from the following command:
oc get secret rhdg-generated-secret \
-o jsonpath="{.data.identities-batch}" | base64 --decodeIf you want to make changes, you need to update the values file and use the helm upgrade command:
helm upgrade rhdg openshift-helm-charts/redhat-data-grid -f rhdg-chart/default-values.yamlIf you want to customize the server deployment - the infinispan.yaml file -, you will need to provide server configuration in YAML format. You can use the following examples:
-
rhdg-chart/default-values.yaml: Example provided in the Helm Charts GitHub repository. -
extras/config/server-example-infinispan.yaml: Example provided as a testing file for the server in the Infinispan GitHub repository. -
extras/config/default-operator-infinispan.yaml: Example obtained from the RHDG image for Infinispan 13.0.2.Final. -
Upstream documentation with examples.
To customize the Helm Chart, you will need to fork the official upstream chart and modify the configuration needed:
-
Clone your git repo in the parent folder:
git clone https://github.com/alvarolop/infinispan-helm-charts.git cd infinispan-helm-charts -
Create a new OCP project:
oc new-project rhdg8-helm-customized --display-name="RHDG 8 - Helm Customized" --description="This namespace contains a deployment of RHDG using a customized Helm Chart"
-
In order to deploy this unpackaged version of the Helm Chart, you just have to use Helm to render the OCP objects using the default values file and apply the result in your OCP cluster:
helm template --validate --set deploy.nameOverride="infinispan" . | oc apply -f -
Alternatively, you can use the values.yaml files defined in this repository:
helm template --validate --set deploy.nameOverride="infinispan" -f ../rhdg8-server/rhdg-chart/default-values.yaml . | oc apply -f -|
ℹ️
|
In the previous commands, you need the following parameters:
|
For more information, check the following links:
A typical OpenShift monitoring stack includes Prometheus - for monitoring both systems and services-, and Grafana - for analyzing and visualizing metrics-.
To deploy the community-powered Grafana operator on OCP 4.13 just follow these steps:
oc process -f https://raw.githubusercontent.com/alvarolop/quarkus-observability-app/main/openshift/grafana/10-operator.yaml | oc apply -f -Now, we will create a Grafana instance using the operator:
oc process -f https://raw.githubusercontent.com/alvarolop/quarkus-observability-app/main/openshift/grafana/20-instance.yaml | oc apply -f -Now, we will create a Grafana data source:
BEARER_TOKEN=$(oc get secret $(oc describe sa grafana-sa -n grafana | awk '/Tokens/{ print $2 }') -n $GRAFANA_NAMESPACE --template='{{ .data.token | base64decode }}')
oc process -f https://raw.githubusercontent.com/alvarolop/quarkus-observability-app/main/openshift/grafana/30-datasource.yaml -p BEARER_TOKEN=$BEARER_TOKEN | oc apply -f -Now, we will create a Grafana dashboard:
oc process -f https://raw.githubusercontent.com/alvarolop/quarkus-observability-app/main/openshift/grafana/40-dashboard.yaml \
-p DASHBOARD_GZIP="$(cat grafana/grafana-default-operator-dashboard.json | gzip | base64 -w0)" \
-p DASHBOARD_NAME=dashboard-rhdg8-default | oc apply -f -|
ℹ️
|
Here you can find information on other ways of creating dashboards. |
|
|
This is currently not possible in Operator v5. Still investigating. |
After accessing Grafana using the OCP SSO, you may log in as admin. Retrieve the credentials from the secret using the following commands:
oc get secret grafana-admin-credentials -n grafana -o jsonpath='{.data.GF_SECURITY_ADMIN_USER}' | base64 --decode
oc get secret grafana-admin-credentials -n grafana -o jsonpath='{.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 --decodeFor more information, access the Grafana main documentation or the Grafana operator documentation.
Java Flight Recorder (JFR) is a tool for collecting diagnostic and profiling data about a running Java application. It is integrated into the Java Virtual Machine (JVM) and causes almost no performance overhead. The Red Hat build of Cryostat is a container-native implementation of JDK Flight Recorder (JFR).
The Cryostat Operator requires cert-manager to be preinstalled on the cluster for traffic encryption. If the cert-manager is enabled but not installed, the deployment fails.
Here you can find the Infinispan documentation on how to use Cryostat for JFR on Red Hat Data Grid.
The cert-manager Operator for Red Hat OpenShift is a cluster-wide service that provides application certificate lifecycle management. You can install it simply by creating the Operator subscription:
oc process -f openshift/cert-manager/10-operator.yaml | oc apply -f -For more configuration options, check the official documentation.
JFR provides insights into various aspects of JVM performance to ease cluster inspection and debugging. Depending on your requirements, you can store and analyze your recordings using the integrated tools provided by Cryostat or export the recordings to an external monitoring application.
Cryostat is a container-native Java application based on JDK Flight Recorder (JFR) that you can use to monitor Java Virtual Machine (JVM) performance for containerized workloads that run on a Red Hat OpenShift cluster.
Install the operator in the cryostat namespace using the following OCP template:
oc process -f openshift/cryostat/10-operator.yaml | oc apply -f -Now, create a ClusterCryostat object in order to be able to monitor JVM containers from different namespaces:
oc process -f openshift/cryostat/20-cryostat.yaml | oc apply -f -For more configuration options, check the following documentation:
Once Cryostat is installed in the cluster, you can access it using the autogenerated route. Check it out using the following command:
oc get routes clustercryostat -n cryostat --template='https://{{ .spec.host }}'First of all, on the top left corner, you have to click on Targets > Create Target. We will use this section to add coordinates to our RH Data Grid cluster:
Second, you can access the Dashboard section to analyze the current status of the JVM and see metrics about:
-
CPU Load.
-
Heap Memory Usage.
-
Threads.
Third, you can click on the Recordings section to start a Java Flight recording:
podman-compose -f rhdg-local/docker-compose-datasource.yaml up# Check current Databases (Should show "datagrid")
psql postgres://rhdguser:password@localhost:5432/datagrid -c '\l'
# Check cache statistics
curl -s -k -u admin:password $RHDG_URL/rest/v2/caches/datasource-cache | yq -P .
# Check DB table contents
psql postgres://rhdguser:password@localhost:5432/datagrid -c 'SELECT * FROM "TBL_datasource_cache";'Interact with the newly created caches with the following commands:
# Set your variables. These are default:
CLUSTER_NAMESPACE="rhdg8"
CLUSTER_NAME="rhdg"
RHDG_URL=$(oc get route ${CLUSTER_NAME}-external -n ${CLUSTER_NAMESPACE} -o template='https://{{.spec.host}}')
# Check all the caches on your cluster
curl -X GET -k -H "Content-Type: application/json" ${RHDG_URL}/rest/v2/caches | jq
# Check information about an specific cache
curl -X GET -k -H "Content-Type: application/json" ${RHDG_URL}/rest/v2/caches/${CACHE_NAME} | jq
# Delete a cache
curl -X DELETE -k ${RHDG_URL}/rest/v2/caches/${CACHE_NAME}An Operator group, defined by the OperatorGroup resource, provides multitenant configuration to OLM-installed Operators. An Operator group selects target namespaces in which to generate required RBAC access for its member Operators.
If you want to modify the default behavior of the template provided in this repository, modify lines 26 to 33 of this template.
1) AllNamespaces: The Operator can be a member of an Operator group that selects all namespaces (target namespace set is the empty string ""). This configuration allows us to create DG clusters in every namespace of the cluster:
- apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: datagrid
namespace: ${OPERATOR_NAMESPACE}
spec: {}2) MultiNamespace: The Operator can be a member of an Operator group that selects more than one namespace. Choose this option if you want to have several operators that manage RHDG clusters. For example, if you want to have a different operator per Business Unit managing several Openshift projects:
- apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: datagrid
namespace: ${OPERATOR_NAMESPACE}
spec:
targetNamespaces:
- ${CLUSTER_NAMESPACE-1}
- ${CLUSTER_NAMESPACE-2}3) SingleNamespace: The Operator can be a member of an Operator group that selects one namespace. This is useful if we want every application (Each OCP namespace) to be able to configure and deploy its own DG clusters:
- apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: datagrid
namespace: ${OPERATOR_NAMESPACE}
spec:
targetNamespaces:
- ${CLUSTER_NAMESPACE}For more information, check the Openshift documentation about Operator Groups and the official documentation to install DG on Openshift.
In some situations, you will need to monitor logs from several pods of the same application and maybe you want to check to which pod the request arrived. Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging.
First, you will need to install it on your machine. After that, log in to your cluster and monitoring the previous deployment is as simple as executing the following command:
stern --namespace=$CLUSTER_NAMESPACE -l clusterName=$CLUSTER_NAMEThe previous command will show all the logs from all the pods from a namespace that contains a given label. There are many filters and configuration options. Check the documentation for a full list of them
Since Infinispan 12.0, Data Grid includes metrics specifically related to Queries on the server side. Retrieve them using the following script:
CACHE_NAME="operator-cache-01"
oc project $RHDG_NAMESPACE
for pod in $(oc get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}')
do
echo "$pod: Get stats"
oc exec $pod -- bash -c 'curl $HOSTNAME:$RHDG_SERVICE_PORT_INFINISPAN/rest/v2/caches/$CACHE_NAME/search/stats' | jq
doneSince Infinispan 12.0, Data Grid includes an option to download a server report from each pod. Retrieve it using the following script:
oc project $RHDG_NAMESPACE
for pod in $(oc get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}')
do
echo "$pod: Generate report"
oc exec $pod -- bash -c 'echo "server report" | ./bin/cli.sh -c $HOSTNAME:$RHDG_SERVICE_PORT_INFINISPAN -f -'
echo "$pod: Download report"
oc exec $pod -- bash -c 'files=( *tar.gz* ); cat "${files[0]}"' > $(date +"%Y-%m-%d-%H-%M")-$pod-report.tar.gz
echo "$pod: Remove report"
oc exec rhdg-0 -- bash -c 'rm -rf *tar.gz*'
doneIn Data Grid 7, caches were defined in XML format. Since RHDG 8, it is possible to use XML, JSON or YAML. The server includes some tools to automatically convert from one to the other.
CACHE_NAME="___protobuf_metadata"
# Get in XML
curl --digest -u developer:$DEV_PASS -H "Accept: application/xml" $INFINISPAN_SERVICE_HOST:11222/rest/v2/caches/$CACHE_NAME?action=config
# Get in JSON
curl --digest -u developer:$DEV_PASS -H "Accept: application/json" $INFINISPAN_SERVICE_HOST:11222/rest/v2/caches/$CACHE_NAME?action=config
# Get in YAML
curl --digest -u developer:$DEV_PASS -H "Accept: application/yaml" $INFINISPAN_SERVICE_HOST:11222/rest/v2/caches/$CACHE_NAME?action=configThe following example converts an XML definition to YAML:
curl localhost:11222/rest/v2/caches?action=convert\
--digest -u developer:developer \
-X POST \
-H "Accept: application/yaml" \
-H "Content-Type: application/xml" \
-d '<?xml version="1.0" encoding="UTF-8"?><replicated-cache mode="SYNC" statistics="false"><encoding media-type="application/x-protostream"/><expiration lifespan="300000" /><memory max-size="400MB" when-full="REMOVE"/><state-transfer enabled="true" await-initial-transfer="false"/></replicated-cache>'The result is the following YAML:
replicatedCache:
mode: "SYNC"
statistics: "false"
encoding:
key:
mediaType: "application/x-protostream"
value:
mediaType: "application/x-protostream"
expiration:
lifespan: "300000"
memory:
maxSize: "400MB"
whenFull: "REMOVE"
stateTransfer:
enabled: "true"
awaitInitialTransfer: "false"-
Download the Infinispan CRD:
# Infinispan Operator 2.1.X URL="https://raw.githubusercontent.com/infinispan/infinispan-operator/2.1.x/deploy/crds/infinispan.org_infinispans_crd.yaml" # Infinispan Operator 2.2.X URL="https://raw.githubusercontent.com/infinispan/infinispan-operator/2.2.x/config/crd/bases/infinispan.org_infinispans.yaml" curl -o rhdg-crds/infinispan-2.2.x.yaml $URL
-
Edit the file in order to create a new CRD instead of modifying the previous one.
-
Create the object in the cluster:
oc apply -f rhdg-crds/infinispan-2.2.x.yaml
-
Get the full list of options:
oc explain custominfinispan --recursive