Add cassandra to docs

A30002503 · Jul 17, 2019 · 0bc1f52 · 0bc1f52
1 parent 012b0cc
commit 0bc1f52
Show file tree

Hide file tree

Showing 17 changed files with 5,399 additions and 5,380 deletions.
diff --git a/docs-src/content/execute/index.md b/docs-src/content/execute/index.md
@@ -6,6 +6,35 @@ type: blog
 
 `*Execute` stages are used to execute arbitrary commands against external systems such as Databases and APIs.
 
+## CassandraExecute
+##### Since: 2.0.0 - Supports Streaming: False
+{{< note title="Plugin" >}}
+The `CassandraExecute` is provided by the https://github.com/tripl-ai/arc-cassandra-pipeline-plugin package.
+{{</note>}}
+
+The `CassandraExecute` executes a CQL statement against an external [Cassandra](https://cassandra.apache.org/) cluster.
+
+### Parameters
+
+| Attribute | Type | Required | Description |
+|-----------|------|----------|-------------|
+|name|String|true|{{< readfile file="/content/partials/fields/stageName.md" markdown="true" >}}|
+|environments|Array[String]|true|{{< readfile file="/content/partials/fields/environments.md" markdown="true" >}}|
+|inputURI|URI|true|URI of the input file containing the CQL statement.|
+|authentication|Map[String, String]|false|{{< readfile file="/content/partials/fields/authentication.md" markdown="true" >}}|
+|description|String|false|{{< readfile file="/content/partials/fields/description.md" markdown="true" >}}|
+|params|Map[String, String]|false|{{< readfile file="/content/partials/fields/params.md" markdown="true" >}}. Any parameters provided will be added to the Cassandra connection object.|
+|sqlParams|Map[String, String]|false|{{< readfile file="/content/partials/fields/sqlParams.md" markdown="true" >}}|
+
+### Examples
+
+#### Minimal
+{{< readfile file="/resources/docs_resources_plugins/CassandraExecuteMin" highlight="json" >}} 
+
+#### Complete
+{{< readfile file="/resources/docs_resources_plugins/CassandraExecuteComplete" highlight="json" >}} 
+
+
 ## HTTPExecute
 ##### Since: 1.0.0 - Supports Streaming: False
 

diff --git a/docs-src/content/extract/index.md b/docs-src/content/extract/index.md
@@ -92,6 +92,38 @@ The `BytesExtract` stage reads one or more binary files and returns a `DataFrame
 {{< readfile file="/resources/docs_resources/BytesExtractComplete" highlight="json" >}} 
 
 
+## CassandraExtract
+##### Since: 2.0.0 - Supports Streaming: False
+{{< note title="Plugin" >}}
+The `CassandraExtract` is provided by the https://github.com/tripl-ai/arc-cassandra-pipeline-plugin package.
+{{</note>}}
+
+The `CassandraExtract` reads directly from a [Cassandra](https://cassandra.apache.org/) cluster and returns a `DataFrame`. 
+
+### Parameters
+
+| Attribute | Type | Required | Description |
+|-----------|------|----------|-------------|
+|name|String|true|{{< readfile file="/content/partials/fields/stageName.md" markdown="true" >}}|
+|environments|Array[String]|true|{{< readfile file="/content/partials/fields/environments.md" markdown="true" >}}|
+|keyspace|String|true|The name of the Cassandra keyspace to extract from.|
+|table|String|true|The name of the Cassandra table to extract from.|
+|outputView|String|true|{{< readfile file="/content/partials/fields/outputView.md" markdown="true" >}}|
+|description|String|false|{{< readfile file="/content/partials/fields/description.md" markdown="true" >}}|
+|numPartitions|Integer|false|{{< readfile file="/content/partials/fields/numPartitions.md" markdown="true" >}} This also determines the maximum number of concurrent JDBC connections.|
+|params|Map[String, String]|false|{{< readfile file="/content/partials/fields/params.md" markdown="true" >}}. Any parameters provided will be added to the Cassandra connection object.|
+|partitionBy|Array[String]|false|{{< readfile file="/content/partials/fields/partitionBy.md" markdown="true" >}}|
+|persist|Boolean|false|{{< readfile file="/content/partials/fields/persist.md" markdown="true" >}}|
+
+### Examples
+
+#### Minimal
+{{< readfile file="/resources/docs_resources_plugins/CassandraExtractMin" highlight="json" >}} 
+
+#### Complete
+{{< readfile file="/resources/docs_resources_plugins/CassandraExtractComplete" highlight="json" >}} 
+
+
 ## DeltaLakeExtract
 ##### Since: 2.0.0 - Supports Streaming: True
 {{< note title="Plugin" >}}

diff --git a/docs-src/content/load/index.md b/docs-src/content/load/index.md
@@ -39,6 +39,38 @@ The `AvroLoad` writes an input `DataFrame` to a target [Apache Avro](https://avr
 {{< readfile file="/resources/docs_resources/AvroLoadComplete" highlight="json" >}} 
 
 
+## CassandraLoad
+##### Since: 2.0.0 - Supports Streaming: False
+{{< note title="Plugin" >}}
+The `CassandraLoad` is provided by the https://github.com/tripl-ai/arc-cassandra-pipeline-plugin package.
+{{</note>}}
+
+The `CassandraLoad` writes an input `DataFrame` to a target [Cassandra](https://cassandra.apache.org/) cluster. 
+
+### Parameters
+
+| Attribute | Type | Required | Description |
+|-----------|------|----------|-------------|
+|name|String|true|{{< readfile file="/content/partials/fields/stageName.md" markdown="true" >}}|
+|environments|Array[String]|true|{{< readfile file="/content/partials/fields/environments.md" markdown="true" >}}|
+|inputView|String|true|{{< readfile file="/content/partials/fields/inputView.md" markdown="true" >}}|
+|keyspace|String|true|The name of the Cassandra keyspace to write to.|
+|table|String|true|The name of the Cassandra table to write to.|
+|description|String|false|{{< readfile file="/content/partials/fields/description.md" markdown="true" >}}|
+|numPartitions|Integer|false|{{< readfile file="/content/partials/fields/numPartitions.md" markdown="true" >}} This also determines the maximum number of concurrent JDBC connections.|
+|params|Map[String, String]|false|{{< readfile file="/content/partials/fields/params.md" markdown="true" >}}. Any parameters provided will be added to the Cassandra connection object.|
+|partitionBy|Array[String]|false|{{< readfile file="/content/partials/fields/partitionBy.md" markdown="true" >}}|
+|saveMode|String|false|{{< readfile file="/content/partials/fields/saveMode.md" markdown="true" >}}|
+
+### Examples
+
+#### Minimal
+{{< readfile file="/resources/docs_resources_plugins/CassandraLoadMin" highlight="json" >}} 
+
+#### Complete
+{{< readfile file="/resources/docs_resources_plugins/CassandraLoadComplete" highlight="json" >}} 
+
+
 ## ConsoleLoad
 ##### Since: 1.2.0 - Supports Streaming: True
 

diff --git a/docs-src/resources/docs_resources_plugins/CassandraExecuteComplete b/docs-src/resources/docs_resources_plugins/CassandraExecuteComplete
@@ -0,0 +1,15 @@
+{
+  "type": "CassandraExecute",
+  "name": "create table",
+  "environments": [
+    "production",
+    "test"
+  ],  
+  "inputURI": "hdfs://datalake/sql/create_table.cql",
+  "params": {
+    "spark.cassandra.connection.host": "cassandra"
+  },
+  "sqlParams": {
+    "keyspace": ${ETL_CONF_ENVIRONMENT}
+  }     
+}
diff --git a/docs-src/resources/docs_resources_plugins/CassandraExecuteMin b/docs-src/resources/docs_resources_plugins/CassandraExecuteMin
@@ -0,0 +1,15 @@
+{
+  "type": ".=CassandraExecute",
+  "name": "create table",
+  "environments": [
+    "production",
+    "test"
+  ],  
+  "inputURI": "hdfs://datalake/sql/create_table.cql",
+  "params": {
+    "spark.cassandra.connection.host": "cassandra"
+  },
+  "sqlParams": {
+    "keyspace": ${ETL_CONF_ENVIRONMENT}
+  }                 
+}
diff --git a/docs-src/resources/docs_resources_plugins/CassandraExtractComplete b/docs-src/resources/docs_resources_plugins/CassandraExtractComplete
@@ -0,0 +1,19 @@
+{
+  "type": "CassandraExtract",
+  "name": "read",
+  "environments": [
+    "production",
+    "test"
+  ],  
+  "keyspace": "default",
+  "table": "customer",
+  "outputView": "customer",
+  "params": {
+    "spark.cassandra.connection.host": "cassandra"
+  },
+  "numPartitions": 10,
+  "partitionBy": [
+    "country"
+  ],
+  "persist": true         
+}
diff --git a/docs-src/resources/docs_resources_plugins/CassandraExtractMin b/docs-src/resources/docs_resources_plugins/CassandraExtractMin
@@ -0,0 +1,11 @@
+{
+  "type": "CassandraExtract",
+  "name": "read",
+  "environments": [
+    "production",
+    "test"
+  ],  
+  "keyspace": "default",
+  "table": "customer",
+  "outputView": "customer"      
+}
diff --git a/docs-src/resources/docs_resources_plugins/CassandraLoadComplete b/docs-src/resources/docs_resources_plugins/CassandraLoadComplete
@@ -0,0 +1,21 @@
+{
+  "type": "CassandraLoad",
+  "name": "write",
+  "environments": [
+    "production",
+    "test"
+  ],  
+  "inputView": "customer",
+  "keyspace": "default",
+  "table": "customer"
+  "saveMode": "Overwrite",
+  "params": {
+    "confirm.truncate": "true",
+    "spark.cassandra.connection.host": "cassandra"
+  },
+  "numPartitions": 10,
+  "partitionBy": [
+    "country"
+  ],
+  "saveMode": "Overwrite"
+}
diff --git a/docs-src/resources/docs_resources_plugins/CassandraLoadMin b/docs-src/resources/docs_resources_plugins/CassandraLoadMin
@@ -0,0 +1,11 @@
+{
+  "type": "CassandraLoad",
+  "name": "write",
+  "environments": [
+    "production",
+    "test"
+  ],  
+  "inputView": "customer",
+  "keyspace": "default",
+  "table": "customer"
+}