Skip to content

Oracle XStream Out CDC Source Connector #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Oracle XStream Out CDC Source Connector #383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3192f82
Oracle XStream Out CDC Source Connector
karthik-appu Aug 4, 2025
8a66d83
oracle database 21c ee
karthik-appu Aug 6, 2025
977bccf
oracle database 21c ee
karthik-appu Aug 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 194 additions & 0 deletions connector/oracle-xstream-cdc-source/21c-ee/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
# Oracle XStream CDC Source Connector with Oracle 21c Enterprise Edition

In this workflow, we will set up, configure, and validate the Oracle XStream CDC Source Connector with Oracle Database 21c Enterprise Edition and the Confluent Platform.

- Provision Oracle Database 21c Enterprise Edition using Oracle Operator for Kubernetes (OraOperator)

- Prepare and configure Oracle XStream on Oracle Database 21c Enterprise Edition

- Deploy Confluent Platform using CFK operator and deploy Oracle XStream CDC Source Connector

- Validate and test the Oracle XStream CDC Source Connector with enterprise-grade Oracle database

## 🖼️ Architecture Overview

<p align="left" width="60%">
<img width="660%" src="./images/confluent-oraclexstream.png" alt="Confluent Platform with Oracle XStream CDC Source Connector" >
</p>

---

## 📋 Prerequisites

- Requires Kubernetes cluster

- Access and set KubeContext to the target Kubernetes cluster

- Default Storage Class configured

- Worker nodes with sufficient resources

- `git, kubectl, helm, jq` commands exist

```sh
command -v git kubectl helm jq
```

- Git clone the repository

```sh
git clone https://github.com/confluentinc/confluent-kubernetes-examples.git

# Change to the 21c-ee Oracle XStream CDC Source Connector example directory
cd connector/oracle-xstream-cdc-source/21c-ee
```

- Set environment variables

```sh
{
export CP_HOME=$(pwd)
export CP_NS='confluent'
export ORACLE_HOME=$(pwd)/oracle
export ORACLE_NS='oracle'
echo "CP_HOME=${CP_HOME} \nCP_NS=${CP_NS} \nORACLE_HOME=${ORACLE_HOME} \nORACLE_NS=${ORACLE_NS}"
}
```

---

## 🗄️ Provision and Configure Oracle Database with XStream Out Interface

Follow the steps in [Provision and Configure Oracle Database with XStream Out Interface](./oracle/database.md) to provision the Oracle Database and complete the database pre-requisites required for Oracle XStream CDC Source Connector.

> **Note:**
> Oracle Database prerequisites must be **completed and validated** before the XStream CDC Source Connector will function as expected in Confluent Platform.

---

## 🚀 Deploy Confluent Platform using CFK Operator

### Deploy Confluent for Kubernetes Operator

```sh
{
helm repo add confluentinc https://packages.confluent.io/helm
helm repo update confluentinc
# 3.0.0 - 0.1263.8
helm upgrade --install confluent-operator \
confluentinc/confluent-for-kubernetes \
--set kRaftEnabled=true \
--version 0.1263.8 \
--namespace ${CP_NS} \
--create-namespace
kubectl config set-context --current --namespace=${CP_NS}
helm ls
kubectl wait pod -l app.kubernetes.io/name=confluent-operator --for=condition=ready --timeout=180s
}
```

### Deploy Confluent Platform

```sh
{
kubectl apply -f ${CP_HOME}/confluent-platform.yaml \
-f ${CP_HOME}/configmap-oracle-xstream-pod-overlay.yaml
echo "\n *** Waiting for all pods to be ready ***"
kubectl wait pod --all --for=condition=ready --timeout=360s
kubectl get pods
}
```

> **Note:**
> While the recommended approach is to build a custom Connect image with the Oracle XStream CDC Source Connector plugin and dependencies, this example uses an pod overlay/Init Container feature to install the plugin, Oracle Instant Client and required libraries. Following tasks are performed by the Init Container, configured in the [configmap-oracle-xstream-pod-overlay.yaml](./configmap-oracle-xstream-pod-overlay.yaml):
>
> - Download and extract Oracle Instant Client
> - Download and extract Oracle XStream CDC Source Connector
> - Setup libaio
> - Copy Oracle JARs to plugin
> - Configure LD_LIBRARY_PATH
>
> To check the logs of this Init Container, open a new terminal and run the following command:
> ```sh
> kubectl logs -f connect-0 -c oracle-xstream-cdc-installer
> ```

## 🔗 Deploy Oracle XStream CDC Source Connector

Once all prerequisites the Oracle Database and Confluent Platform are deployed and ready, we can deploy the Oracle XStream CDC Source Connector by using a Connector custom resource [connector-oracle-xstream-cdc.yaml](./connector-oracle-xstream-cdc.yaml).

### Deploy Oracle XStream CDC Source Connector

```sh
{
kubectl apply -f connector-oracle-xstream-cdc.yaml
kubectl get connector oracle-xstream-cdc-source
echo "Waiting for the Oracle XStream CDC Source Connector to be ready..."
kubectl wait connector oracle-xstream-cdc-source --for=jsonpath='{.status.connectorState}'=RUNNING --timeout=180s
kubectl get connector oracle-xstream-cdc-source
}
```
Connector should has status `CREATED`, connector status `RUNNING` and tasks ready `1/1`.

### Validate Oracle XStream CDC Source Connector

Open a new terminal and port-forward the Control Center service.

```sh
{
export CP_NS='confluent'
kubectl port-forward svc/controlcenter 9021:9021 -n ${CP_NS}
}
```

Open the browser and navigate to http://localhost:9021.

Click on **controlcenter.cluster** cluster and click on **Topics** tab on the left sidebar, we should see the following topics related to the Oracle XStream CDC Source Connector:

- `ORCLPDB1.SAMPLE.EMPLOYEES` - CDC data topic
- `__orclpdb1-schema-changes.cflt` - Schema change events topic

Select the `ORCLPDB1.SAMPLE.EMPLOYEES` topic and click on **Messages** tab to see the initial records.

### Test CDC Functionality

**Insert Test Data:**

```sh
{
kubectl exec -it oracle-db-0 -n oracle -- sqlplus sample/password@ORCLPDB1 <<EOF
ALTER SESSION SET CURRENT_SCHEMA = SAMPLE;
INSERT INTO employees VALUES (2001, 'Neil', 'Armstrong', 'neil.armstrong@spacecorp.com', '+1-555-MOON', SYSDATE, 'IT_PROG', 78000, NULL, 1003, 10, SYSDATE, SYSDATE);
INSERT INTO employees VALUES (2002, 'Sally', 'Ride', 'sally.ride@spacecorp.com', '+1-555-STAR', SYSDATE, 'SA_REP', 62000, 0.10, 1004, 20, SYSDATE, SYSDATE);
INSERT INTO employees VALUES (2003, 'Buzz', 'Aldrin', 'buzz.aldrin@spacecorp.com', '+1-555-BUZZ', SYSDATE, 'FI_ACCOUNT', 65000, NULL, 1006, 30, SYSDATE, SYSDATE);
UPDATE employees SET salary = 85000 WHERE employee_id = 2001;
DELETE FROM employees WHERE employee_id = 2003;
COMMIT;
EXIT;
EOF
}
```

**Verify in Control Center:**

We should see: 6 initial records + 3 INSERTs + 1 UPDATE + 1 DELETE = 11 CDC events total.

Select the `__orclpdb1-schema-changes.cflt` topic and click on **Messages** tab to see the schema change events.

That's it! We have successfully deployed and validated the Oracle XStream CDC Source Connector with end-to-end CDC data flow to Kafka topics.

---

## 📝 Cleanup

```sh
{
kubectl delete -f connector-oracle-xstream-cdc.yaml \
-f confluent-platform.yaml \
-f configmap-oracle-xstream-pod-overlay.yaml
helm uninstall confluent-operator
kubectl delete namespace ${CP_NS}
kubectl delete -f ${ORACLE_HOME}/
}
```
---
196 changes: 196 additions & 0 deletions connector/oracle-xstream-cdc-source/21c-ee/configmap-oracle-xstream-pod-overlay.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap-oracle-xstream-pod-overlay
namespace: confluent
data:
pod-template.yaml: |
spec:
template:
spec:
# Init container to install Oracle Instant Client, Oracle XStream CDC plugin, and dependencies
initContainers:
- name: oracle-xstream-cdc-installer
image: container-registry.oracle.com/database/free:latest-lite
env:
- name: ORACLE_INSTANT_CLIENT_VERSION
value: "21.8.0.0.0"
- name: ORACLE_XSTREAM_CDC_PLUGIN_VERSION
value: "1.0.2"
command:
- /bin/sh
- -c
- |
set -e
echo "=== Oracle XStream CDC Complete Setup ==="

# Set the version of Oracle Instant Client (configurable)
ORACLE_VERSION="${ORACLE_INSTANT_CLIENT_VERSION:-21.8.0.0.0}"

# Derive URL path from version for download
# Extract major and minor versions (e.g., "21.8.0.0.0" -> "21" and "8")
ORACLE_MAJOR=$(echo "$ORACLE_VERSION" | cut -d. -f1)
ORACLE_MINOR=$(echo "$ORACLE_VERSION" | cut -d. -f2)

# Generate URL path (e.g., "21" + "8" + "000" = "218000")
ORACLE_URL_PATH="${ORACLE_MAJOR}${ORACLE_MINOR}000"

# Set the version of Oracle XStream CDC Plugin (configurable)
ORACLE_XSTREAM_CDC_VERSION="${ORACLE_XSTREAM_CDC_PLUGIN_VERSION:-1.0.2}"

echo "📋 Using Oracle Instant Client version: $ORACLE_VERSION"
echo "📋 Using Oracle XStream CDC plugin version: $ORACLE_XSTREAM_CDC_VERSION"

# ============================================================================
# STEP 0: Verification - Check required tools are available
# ============================================================================
echo "🔍 [Step 0] Verifying required tools..."

# Check for required tools
command -v unzip >/dev/null 2>&1 || { echo "❌ unzip not found - cannot extract files"; exit 1; }
command -v curl >/dev/null 2>&1 || { echo "❌ curl not found - cannot download files"; exit 1; }

echo "✅ Required tools verified: unzip, curl available"

# ============================================================================
# STEP 1: Oracle Instant Client - Download and extract
# ============================================================================
echo "📦 [Step 1] Downloading Oracle Instant Client..."
cd /tmp

# Download Oracle Instant Client packages
curl -o "instantclient-basic-linux.x64-${ORACLE_VERSION}dbru.zip" \
"https://download.oracle.com/otn_software/linux/instantclient/${ORACLE_URL_PATH}/instantclient-basic-linux.x64-${ORACLE_VERSION}dbru.zip"
curl -o "instantclient-jdbc-linux.x64-${ORACLE_VERSION}dbru.zip" \
"https://download.oracle.com/otn_software/linux/instantclient/${ORACLE_URL_PATH}/instantclient-jdbc-linux.x64-${ORACLE_VERSION}dbru.zip"
curl -o "instantclient-sqlplus-linux.x64-${ORACLE_VERSION}dbru.zip" \
"https://download.oracle.com/otn_software/linux/instantclient/${ORACLE_URL_PATH}/instantclient-sqlplus-linux.x64-${ORACLE_VERSION}dbru.zip"

# Extract all packages to temporary directory
echo "📦 Extracting Oracle Instant Client packages..."
unzip -o -q "instantclient-basic-linux.x64-${ORACLE_VERSION}dbru.zip" -d /oracle-libs/
unzip -o -q "instantclient-jdbc-linux.x64-${ORACLE_VERSION}dbru.zip" -d /oracle-libs/
unzip -o -q "instantclient-sqlplus-linux.x64-${ORACLE_VERSION}dbru.zip" -d /oracle-libs/

# Rename downloaded directory to fixed name for easier path management
mv /oracle-libs/instantclient_* /oracle-libs/instantclient

# Verify Oracle Instant Client extraction
echo "🔍 Verifying Oracle Instant Client extraction..."
ls -la /oracle-libs/instantclient/

echo "✅ Oracle Instant Client downloaded and extracted"

# ============================================================================
# STEP 2: Oracle XStream CDC Plugin - Download from Confluent Hub
# ============================================================================
echo "🔌 [Step 2] Downloading Oracle XStream CDC Plugin..."

PLUGIN_ZIP="/tmp/oracle-xstream-cdc-plugin.zip"

# Download from official Confluent Hub API
if curl -o "$PLUGIN_ZIP" "https://api.hub.confluent.io/api/plugins/confluentinc/kafka-connect-oracle-xstream-cdc-source/versions/${ORACLE_XSTREAM_CDC_VERSION}/archive" 2>/dev/null; then
Copy link

Copilot AI Aug 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Downloading plugins from external URLs without verification poses a security risk. Consider implementing checksum validation or using a trusted registry to ensure the integrity of downloaded components.

Copilot uses AI. Check for mistakes.
echo "✅ Downloaded Oracle XStream CDC plugin from Confluent Hub API"
else
echo "❌ Plugin download failed - cannot proceed"
exit 1
fi

# Extract plugin to components directory
mkdir -p /usr/share/confluent-hub-components
echo "📦 Extracting Oracle XStream CDC plugin..."
unzip -o -q "$PLUGIN_ZIP" -d /usr/share/confluent-hub-components/

PLUGIN_DIR="/usr/share/confluent-hub-components/confluentinc-kafka-connect-oracle-xstream-cdc-source-${ORACLE_XSTREAM_CDC_VERSION}"
echo "✅ Plugin extracted successfully to $PLUGIN_DIR"
ls -la "$PLUGIN_DIR"

# ============================================================================
# STEP 3: Setup - libaio, copy Oracle JARs, configure LD_LIBRARY_PATH
# ============================================================================
echo "🔧 [Step 3] Setting up libaio and Oracle dependencies..."

# Setup libaio
echo "📦 Installing libaio..."
mkdir -p /oracle-libs/system-libs

# Oracle Free image has libaio in /usr/lib64/ - copy it to shared volume
echo "📋 Copying libaio from Oracle Free image..."
cp /usr/lib64/libaio.so* /oracle-libs/system-libs/

# Create standard symlinks for libaio
cd /oracle-libs/system-libs/
ln -sf libaio.so.1.0.1 libaio.so.1
ln -sf libaio.so.1.0.1 libaio.so
echo "✅ libaio ready: $(ls libaio.so.1.*)"

# Copy Oracle JARs to plugin lib directory
echo "📋 Copying Oracle JARs to plugin..."
cp /oracle-libs/instantclient/ojdbc8.jar "$PLUGIN_DIR/lib/"
cp /oracle-libs/instantclient/xstreams.jar "$PLUGIN_DIR/lib/"
cp /oracle-libs/instantclient/orai18n.jar "$PLUGIN_DIR/lib/"

# Configure LD_LIBRARY_PATH
export LD_LIBRARY_PATH="/oracle-libs/instantclient:/oracle-libs/system-libs:${LD_LIBRARY_PATH}"
export ORACLE_HOME="/oracle-libs/instantclient"

echo "🔧 LD_LIBRARY_PATH configured: $LD_LIBRARY_PATH"
echo "🔧 ORACLE_HOME configured: $ORACLE_HOME"

# Verify Oracle JARs in plugin
echo "📦 Oracle JARs in plugin lib:"
ls -la "$PLUGIN_DIR/lib/" | grep -E "(ojdbc8|xstreams|orai18n)"

echo "✅ Setup complete - libaio, JARs, and library paths configured"

# ============================================================================
# STEP 4: Verification - Test Oracle client and SQL*Plus
# ============================================================================
echo "🧪 [Step 4] Testing Oracle client and SQL*Plus..."

echo "🔍 Testing Oracle Instant Client"
echo "📋 dependencies:"
ldd /oracle-libs/instantclient/libclntsh.so.21.1 2>/dev/null | grep -E "(libaio|not found)" || echo "Dependencies OK"

# Test SQL*Plus functionality
echo "🧪 Testing SQL*Plus..."
if /oracle-libs/instantclient/sqlplus -? >/dev/null 2>&1; then
echo "✅ SQL*Plus test passed - Oracle client ready for XStream CDC"
else
echo "⚠️ SQL*Plus test failed - checking dependencies"
echo "📋 SQL*Plus dependencies:"
ldd /oracle-libs/instantclient/sqlplus 2>/dev/null | head -5 || echo "Dependency check complete"
fi

# Final verification
echo "🎯 Final verification:"
echo " 📁 Oracle Home: $ORACLE_HOME"
echo " 📚 Library Path: $LD_LIBRARY_PATH"
echo " 🔌 Plugin: $PLUGIN_DIR"
echo " 🗄️ libaio: $([ -f "/oracle-libs/system-libs/libaio.so.1" ] && echo "Available" || echo "Not found")"

echo "🎉 Oracle XStream CDC setup complete!"

volumeMounts:
- name: oracle-libs
mountPath: /oracle-libs
- name: confluent-hub-components
mountPath: /usr/share/confluent-hub-components
containers:
- name: connect
env:
# Oracle paths (fixed directory name, version only used for download)
- name: LD_LIBRARY_PATH
value: "/oracle-libs/instantclient:/oracle-libs/system-libs:${LD_LIBRARY_PATH}"
- name: ORACLE_HOME
value: "/oracle-libs/instantclient"
volumeMounts:
- name: oracle-libs
mountPath: /oracle-libs
- name: confluent-hub-components
mountPath: /usr/share/confluent-hub-components
volumes:
- name: oracle-libs
emptyDir: {}
- name: confluent-hub-components
emptyDir: {}
Loading