Skip to content

Commit ffdbfd3

Browse files
committed
docs: add batching example
1 parent e303bf8 commit ffdbfd3

File tree

5 files changed

+31
-18
lines changed

5 files changed

+31
-18
lines changed

README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,37 @@ with Google Cloud Platform as a cloud provider.
3131
10. Run one of the provided examples.
3232
3333
#### Batching example
34+
An example that contains batch writing and reading from Snowflake. Inspired by [Apache Beam/WordCount-example](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java)
3435
36+
1. Run batching example by executing following command:
37+
```
38+
./gradlew run --args=" /
39+
--inputFile=gs://apache-beam-samples/shakespeare/ /
40+
--output=gs://<GCS BUCKET NAME>/counts /
41+
--serverName=<SNOWFLAKE SERVER NAME> /
42+
--username=<SNOWFLAKE USERNAME> /
43+
--password=<SNOWFLAKE PASSWORD> /
44+
--database=<SNOWFLAKE DATABASE> /
45+
--schema=<SNOWFLAKE SCHEMA> /
46+
--storageIntegration=<SNOWFLAKE STORAGE INTEGRATION NAME> /
47+
--stagingBucketName=<GCS BUCKET NAME> /
48+
--runner=<DirectRunner/DataflowRunner> /
49+
--project=<FOR DATAFLOW RUNNER: GCP PROJECT NAME> /
50+
--gcpTempLocation=<FOR DATAFLOW RUNNER: GCS TEMP LOCATION STARTING> /
51+
--region=<FOR DATAFLOW RUNNER: GCP REGION> /
52+
--appName=<OPTIONAL: DATAFLOW JOB NAME PREFIX>"
53+
```
54+
2. Go to Snowflake console to check saved counts
55+
```
56+
select * from <DATABASE NAME>.<SCHEMA NAME>.WORD_COUNT;
57+
```
58+
![Batching snowflake result](./images/batching_snowflake_result.png)
59+
3. Go to GCS bucket to check saved files
60+
![Batching gcs result](./images/batching_gcs_result.png)
61+
4. Go to DataFlow to check submitted jobs
62+
![Batching DataFlow result](./images/batching_dataflow_result.png)
63+
64+
3565
#### Streaming example
3666
3767
#### Cross-language example
144 KB
Loading

images/batching_gcs_result.png

186 KB
Loading
251 KB
Loading

src/main/java/batching/SnowflakeWordCount.java

Lines changed: 1 addition & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -44,24 +44,7 @@
4444
* 1. Reading counts from Snowflake
4545
* 2. Writing counts into output
4646
*
47-
* TODO
48-
*
49-
* ./gradlew run --args=" /
50-
* --inputFile=gs://apache-beam-samples/shakespeare/* /
51-
* --output=gs://beam-snowflake-test/counts /
52-
* --serverName=<SNOWFLAKE SERVER NAME> /
53-
* --username=<SNOWFLAKE USERNAME> /
54-
* --password=<SNOWFLAKE PASSWORD> /
55-
* --database=<SNOWFLAKE DATABASE> /
56-
* --schema=<SNOWFLAKE SCHEMA> /
57-
* --storageIntegration=<SNOWFLAKE STORAGE INTEGRATION NAME> /
58-
* --stagingBucketName=<GCS BUCKET NAME> /
59-
* --runner=<DirectRunner/DataflowRunner> /
60-
* --project=<FOR DATAFLOW RUNNER: GCP PROJECT NAME> /
61-
* --gcpTempLocation=<FOR DATAFLOW RUNNER: GCS TEMP LOCATION STARTING> /
62-
* --region=<FOR DATAFLOW RUNNER: GCP REGION> /
63-
* --appName=<OPTIONAL: DATAFLOW JOB NAME PREFIX>"
64-
*
47+
* Check main README for executing
6548
*/
6649
public class SnowflakeWordCount {
6750

0 commit comments

Comments
 (0)