Altinity · hodgesrm · Nov 29, 2023 · Nov 29, 2023
diff --git a/README.md b/README.md
@@ -326,7 +326,7 @@ mvn install -DskipTests=true
 - [Architecture](doc/architecture.md)
 - [Local Setup - Docker Compose](doc/setup.md)
 - [Debezium Setup](doc/debezium_setup.md)
-- [Kubernetes Setup](doc/k8s_pipeline_setup.md)
+- [Kubernetes Setup](doc/k8s/k8s_pipeline_setup.md)
 - [Sink Configuration](doc/sink_configuration.md)
 - [Testing](doc/TESTING.md)
 - [Performance Benchmarking](doc/Performance.md)

diff --git a/doc/Kafka.md b/doc/Kafka.md
diff --git a/doc/Kafka_metadata.md b/doc/Kafka_metadata.md
@@ -1,3 +1,7 @@
+## Kafka MetaData
+
+These are the columns that are added when the `store.kafka.metadata` configuration is enabled.
+
 | Column           | Data Type                | Description                                                                                                                                                           |
 |------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | `_offset`        | Nullable(UInt64),        | Kafka offset                                                                                                                                                          |

diff --git a/doc/Milestones.md b/doc/Milestones.md
diff --git a/doc/Monitoring.md b/doc/Monitoring.md
@@ -1,3 +1,5 @@
+## Sink Connector (Kafka) monitoring
+
 Sink Connector Config
 OpenJDK 11.0.14.1 
 

diff --git a/doc/Performance.md b/doc/Performance.md
@@ -1,4 +1,6 @@
-# Setup
+# Load Testing using SysBench.
+
+## Setup
 We use `SysBench` to perform load testing.
 https://github.com/akopytov/sysbench
 

diff --git a/doc/TESTING.md b/doc/TESTING.md
@@ -1,5 +1,5 @@
 # Unit tests
-The unit tests are added in the following directory. This is a work in progress.
+The unit tests are added in the following directory.
 [test](../src/test/com/altinity/clickhouse/sink/connector)
 
 # Integration tests.

diff --git a/doc/Troubleshooting.md b/doc/Troubleshooting.md
@@ -1,3 +1,4 @@
+# Troubleshooting.
 
 ### Caused by: io.debezium.DebeziumException: java.sql.SQLSyntaxErrorException: Access denied; you need (at least one of) 
 ### the SUPER, REPLICATION CLIENT privilege(s) for this operation

diff --git a/doc/architecture.md b/doc/architecture.md
@@ -1,25 +1,4 @@
-# Architecture
-
-![arch](img/arch.png)
-
-## DeDuplicator
-
-**Assumption**: each table has to have a **Deduplication key** specified. 
-Deduplication key is a set of fields, explicitly specified to be used as a basis for deduplication.
-In case PK is defined it is natural to use PK as Deduplication Key as well, It is not mandatory, however, and for tables
-without PK, Deduplication key is still required.
-
-Connector maintains a map (Deduplication Map), where the key is a Deduplication key and the value is the record itself. 
-As soon as a new record with the same Deduplication key arrives, it either replaces the existing record 
-in the Deduplication Map or is dropped, based on the **Deduplication Policy**. Records from the Deduplication Map are 
-formed into a Batch and are flushed into the ClickHouse on either time or size-based **Dump Policy**.
-
-> **NB** it should be noted, that time-based batch flush can not form the same batches upon replay.
-
-Flushed rows are removed from the Deduplication Map based on either time or size-based **Clear Policy**
-
-As a result of the Deduplication Map application, connector has a set of records, which are de-duplicated within 
-a certain time or size - limited window of records.
+# Kafka Sink Connector Architecture
 
 ## Exactly Once Semantics
 

diff --git a/doc/k8s_build_connect_images.md → doc/k8s/k8s_build_connect_images.md b/doc/k8s_build_connect_images.md → doc/k8s/k8s_build_connect_images.md
diff --git a/doc/k8s_minikube_load.md → doc/k8s/k8s_minikube_load.md b/doc/k8s_minikube_load.md → doc/k8s/k8s_minikube_load.md
diff --git a/doc/k8s_pipeline_setup.md → doc/k8s/k8s_pipeline_setup.md b/doc/k8s_pipeline_setup.md → doc/k8s/k8s_pipeline_setup.md
diff --git a/doc/k8s_pipeline_tools.md → doc/k8s/k8s_pipeline_tools.md b/doc/k8s_pipeline_tools.md → doc/k8s/k8s_pipeline_tools.md
diff --git a/doc/mutable_data.md b/doc/mutable_data.md
@@ -1,4 +1,4 @@
-## ReplacingMergeTree
+## How Updates/Deletes are Handled - ReplacingMergeTree
 Sink Connector will attempt to read the `engine_full` column from system.tables for the corresponding table and will 
 identify the `engine` and the `ver` column.
 

diff --git a/doc/postgres_wal.md b/doc/postgres_wal.md
@@ -1,3 +1,5 @@
+## PostgreSQL WAL Dump utility.
+
 Using the `pg_waldump` utility to dump the WAL log information. `pg_waldump` utility needs to be provided the postgresql data directory path.
 ```
 pg_waldump pg_wal/000000010000000000000001

diff --git a/doc/schema_registry.md b/doc/schema_registry.md
@@ -1,4 +1,5 @@
-#### Confluent Schema Registry
+# Confluent Schema Registry
+### Schema Registry API calls.
 
 REST API to get subjects
 ```

diff --git a/doc/setup.md b/doc/setup.md
@@ -1,6 +1,6 @@
 This doc describes how to set up CDC pipeline
 
-# Pipeline
+# Setup Kafka Sink Connector Pipeline
 
 ![pipeline](img/pipeline.png)
 

diff --git a/doc/sink_configuration.md b/doc/sink_configuration.md
@@ -1,3 +1,6 @@
+# Kafka Sink connector configuration.
+
+
 All the Sink Connector configuration variables are documented here.
 
 `topics`: Kafka topics delimited by comma.\

diff --git a/doc/transactions.md b/doc/transactions.md
@@ -1,3 +1,4 @@
+# MySQL Transactions(currently not supported)
 Transactions have to be enabled in the source connector, For MySQL
 the following flag has to be set
 `    "provide.transaction.metadata": "true"`