Skip to content

Commit aff41ae

Browse files
HashanCalceyHashanCalcey
committed
init
1 parent 7c78bc0 commit aff41ae

File tree

224 files changed

+22851
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

224 files changed

+22851
-0
lines changed

‎READ_ME.txt‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#Connect Sql Server using kafka connect and jdbc connector
2+
3+
to run this example you need to have SQL server installed.
4+
5+
6+
and then follow the steps to start
7+
8+
1. start zookeeper server
9+
10+
kafka_2.12-2.2.1/bin/zookeeper-server-start.sh zookeeper.properties
11+
12+
2. then start kafka server
13+
14+
kafka_2.12-2.2.1/bin/kafka-server-start.sh server.properties
15+
16+
3. register the connector
17+
18+
kafka_2.12-2.2.1/bin/connect-standalone.sh connect-standalone.properties sql-server-jdbc-connector.properties
19+
20+
table.whitelist - specify tables you want to monitor
21+
mode - specifiy the mode incrementing, timestamp, both
22+
for more - https://docs.confluent.io/current/connect/kafka-connect-jdbc/source-connector/source_config_options.html#mode
23+
24+
4. check topic list
25+
26+
kafka_2.12-2.2.1/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
27+
28+
and check wheather there are any topics starting from "incrementing-"
29+
* you can change the prefix by changing topic.prefix in connector properties
30+
31+
5. start console consumer and subscribe to a topic
32+
33+
kafka_2.12-2.2.1/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic incrementing-students
34+
35+
6. make changes (insert, update) and see those are reflected on consumer.

‎connect-standalone.properties‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
# These are defaults. This file just demonstrates how to override some settings.
17+
bootstrap.servers=localhost:9092
18+
19+
# The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will
20+
# need to configure these based on the format they want their data in when loaded from or stored into Kafka
21+
key.converter=org.apache.kafka.connect.json.JsonConverter
22+
value.converter=org.apache.kafka.connect.json.JsonConverter
23+
# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter we want to apply
24+
# it to
25+
key.converter.schemas.enable=true
26+
value.converter.schemas.enable=true
27+
28+
offset.storage.file.filename=/tmp/connect.offsets
29+
# Flush much faster than normal, which is useful for testing/debugging
30+
offset.flush.interval.ms=10000
31+
32+
# Set to a list of filesystem paths separated by commas (,) to enable class loading isolation for plugins
33+
# (connectors, converters, transformations). The list should consist of top level directories that include
34+
# any combination of:
35+
# a) directories immediately containing jars with plugins and their dependencies
36+
# b) uber-jars with plugins and their dependencies
37+
# c) directories immediately containing the package directory structure of classes of plugins and their dependencies
38+
# Note: symlinks will be followed to discover dependencies or plugins.
39+
# Examples:
40+
# plugin.path=
41+
plugin.path=plugins

‎kafka_2.12-2.2.1/LICENSE‎

Lines changed: 396 additions & 0 deletions
Large diffs are not rendered by default.

‎kafka_2.12-2.2.1/NOTICE‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
Apache Kafka
2+
Copyright 2019 The Apache Software Foundation.
3+
4+
This product includes software developed at
5+
The Apache Software Foundation (http://www.apache.org/).
6+
7+
This distribution has a binary dependency on jersey, which is available under the CDDL
8+
License. The source code of jersey can be found at https://github.com/jersey/jersey/.

‎kafka_2.12-2.2.1/bin/connect-distributed.sh‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/bin/sh
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
if [ $# -lt 1 ];
18+
then
19+
echo "USAGE: $0 [-daemon] connect-distributed.properties"
20+
exit 1
21+
fi
22+
23+
base_dir=$(dirname $0)
24+
25+
if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
26+
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/connect-log4j.properties"
27+
fi
28+
29+
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
30+
export KAFKA_HEAP_OPTS="-Xms256M -Xmx2G"
31+
fi
32+
33+
EXTRA_ARGS=${EXTRA_ARGS-'-name connectDistributed'}
34+
35+
COMMAND=$1
36+
case $COMMAND in
37+
-daemon)
38+
EXTRA_ARGS="-daemon "$EXTRA_ARGS
39+
shift
40+
;;
41+
*)
42+
;;
43+
esac
44+
45+
exec $(dirname $0)/kafka-run-class.sh $EXTRA_ARGS org.apache.kafka.connect.cli.ConnectDistributed "$@"

‎kafka_2.12-2.2.1/bin/connect-standalone.sh‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/bin/sh
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
if [ $# -lt 1 ];
18+
then
19+
echo "USAGE: $0 [-daemon] connect-standalone.properties"
20+
exit 1
21+
fi
22+
23+
base_dir=$(dirname $0)
24+
25+
if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
26+
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/connect-log4j.properties"
27+
fi
28+
29+
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
30+
export KAFKA_HEAP_OPTS="-Xms256M -Xmx2G"
31+
fi
32+
33+
EXTRA_ARGS=${EXTRA_ARGS-'-name connectStandalone'}
34+
35+
COMMAND=$1
36+
case $COMMAND in
37+
-daemon)
38+
EXTRA_ARGS="-daemon "$EXTRA_ARGS
39+
shift
40+
;;
41+
*)
42+
;;
43+
esac
44+
45+
exec $(dirname $0)/kafka-run-class.sh $EXTRA_ARGS org.apache.kafka.connect.cli.ConnectStandalone "$@"

‎kafka_2.12-2.2.1/bin/kafka-acls.sh‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/bin/bash
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
exec $(dirname $0)/kafka-run-class.sh kafka.admin.AclCommand "$@"

‎kafka_2.12-2.2.1/bin/kafka-broker-api-versions.sh‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/bin/bash
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
exec $(dirname $0)/kafka-run-class.sh kafka.admin.BrokerApiVersionsCommand "$@"

‎kafka_2.12-2.2.1/bin/kafka-configs.sh‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/bin/bash
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
exec $(dirname $0)/kafka-run-class.sh kafka.admin.ConfigCommand "$@"

‎kafka_2.12-2.2.1/bin/kafka-console-consumer.sh‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/bin/bash
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
18+
export KAFKA_HEAP_OPTS="-Xmx512M"
19+
fi
20+
21+
exec $(dirname $0)/kafka-run-class.sh kafka.tools.ConsoleConsumer "$@"

0 commit comments

Comments
 (0)