Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Master #1

Merged
merged 161 commits into from
Oct 18, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
161 commits
Select commit Hold shift + click to select a range
bdffcf4
KSQL screencast image
Aug 30, 2017
5ea83ba
remove here hyperlink
Aug 30, 2017
e81f61f
Merge pull request #268 from confluentinc/joel-hamill/demo-image
joel-hamill Aug 30, 2017
c0487e1
Added LIKE examples to syntax guide and examples.
hjafarpour Aug 31, 2017
5589b2e
Fixed typo.
hjafarpour Aug 31, 2017
639e164
Minor doc changes.
hjafarpour Aug 31, 2017
621ccc3
Merge pull request #271 from hjafarpour/KSQL-324-Example-Queries
hjafarpour Aug 31, 2017
70b980f
Improvements to syntax guide
Sep 1, 2017
cc86321
Merge pull request #277 from confluentinc/0.1.x-syntax-guide
miguno Sep 4, 2017
a75dff3
Restructure TOC
Sep 4, 2017
6b896d7
Fixed misspelled words
Sep 5, 2017
056fa14
Fixed a minor error in the syntax guide.
hjafarpour Sep 5, 2017
e1522cb
Merge pull request #281 from hjafarpour/KSQL-324-Example-Queries
hjafarpour Sep 5, 2017
1e5496b
Merge pull request #280 from juneng603/feature-fix-misspelled-words
hjafarpour Sep 5, 2017
15180dd
Clarify PARTITION BY in CSAS
Sep 6, 2017
39c04f8
remove build-tools copy, use pluginRepository in pom (#286)
aayars Sep 6, 2017
032d1b5
Add section Use Cases and Examples
Sep 6, 2017
885cf10
Fix typo
Sep 7, 2017
4637617
Add missing word to roadmap
elliotcm Sep 10, 2017
7f0141b
Merge pull request #288 from elliotcm/patch-1
miguno Sep 11, 2017
c52137d
Clarify FAQ
Sep 12, 2017
3af28e7
Remove trailing whitespace
Sep 12, 2017
b2df850
Add Configure KSQL to Syntax Guide, fix/clariy config examples
Sep 12, 2017
1bf980c
Merge pull request #289 from confluentinc/0.1.x-configuration
miguno Sep 12, 2017
7a9acb9
Emit succinct error for missing query file
Sep 12, 2017
ce7c880
Add upstream project so build are triggered automatically
ewencp Sep 14, 2017
869f598
Clarify client-server mode
Sep 15, 2017
76b0bcc
Clarifications
Sep 15, 2017
01d488c
Add `$` to CLI commands
Sep 15, 2017
ed4f0e6
Clarifications
Sep 15, 2017
b5030ba
Clarifications
Sep 15, 2017
4211fe7
Clarifications
Sep 15, 2017
0f6150c
Fix typo
Sep 15, 2017
7db5983
Remove bullet formatting
Sep 15, 2017
1cb5307
fix checkstyle errors
dguy Sep 15, 2017
e614bd7
Merge pull request #298 from confluentinc/fix-checkstyle
logscape Sep 15, 2017
50ffcac
Document how to configure the Kafka bootstrap servers
Sep 15, 2017
bf8a719
Merge pull request #296 from ewencp/add-upstream-project
ewencp Sep 15, 2017
34c5f9f
Merge remote-tracking branch 'origin/0.1.x'
ewencp Sep 15, 2017
ac19358
Merge pull request #297 from confluentinc/0.1.x-bootstrap-servers
miguno Sep 18, 2017
bbe07e5
Update non-docker-clickstream.md
Sep 18, 2017
a8d1e5d
Merge pull request #301 from himani1/patch-2
logscape Sep 18, 2017
6e2cc63
Merge pull request #291 from kirktrue/omit-stacktrace-for-missing-file
logscape Sep 20, 2017
961eafa
Clarify KSQL works with AK and CP
Sep 20, 2017
9d678d9
Merge pull request #302 from confluentinc/0.1.x-faq-works-with-apache…
logscape Sep 20, 2017
c6d7773
small-code-quality-improvements Added code quality improvements.
TheRealHaui Sep 24, 2017
22b898e
small-code-quality-improvements Added code quality improvements.
TheRealHaui Sep 24, 2017
dfea526
added persistent information for streams and table
Sep 26, 2017
88fda38
Merge pull request #319 from bluemonk3y/0.1.x-DOC-added-persistent-in…
logscape Sep 26, 2017
bf90854
Merge pull request #313 from TheRealHaui/small-code-quality-improvements
logscape Sep 27, 2017
34273f7
fixing commandstore hanging
Sep 27, 2017
a2f8680
reduced poll on priorCommands processing
Sep 27, 2017
c2f12f2
removed header p html
Sep 27, 2017
268ba18
added comment on poll instead of subscribe
Sep 27, 2017
e71f7ca
Merge pull request #321 from bluemonk3y/0.1.x-BUG-fix-command-queue-r…
logscape Sep 27, 2017
36f7624
Clarify how streams and tables are stored in topics
Sep 28, 2017
b8cf069
Merge pull request #323 from confluentinc/0.1.x-concepts-clarification
logscape Sep 28, 2017
a94639e
Including the streams config properties in the session with CTAS and …
hjafarpour Sep 29, 2017
fb5800e
refactoring parser, core and common
Sep 29, 2017
ade44ce
Merge pull request #324 from hjafarpour/KSQL-375-include-query-config…
hjafarpour Sep 29, 2017
ddfec38
extracted serde module
Sep 29, 2017
35e6a40
Merge pull request #325 from bluemonk3y/0.1.x-CODE-restructure
hjafarpour Sep 29, 2017
a33045c
Refactored the package for GenericRow.
hjafarpour Sep 29, 2017
f2a1a9f
Merge pull request #327 from hjafarpour/refactor-common-GenericRow
hjafarpour Sep 29, 2017
bbf636d
Fixed the drop table issue.
hjafarpour Sep 30, 2017
dc1ad22
Merge pull request #328 from hjafarpour/KSQL-376-droptable-issue
hjafarpour Sep 30, 2017
09a7755
extracted pipeline
Oct 2, 2017
115b4b9
Merge pull request #330 from bluemonk3y/0.1.x-CODE-restructure-Queued…
logscape Oct 2, 2017
295fc8b
rename ksql-core to ksql-engine
Oct 2, 2017
850bbbd
Add to Examples: WHERE clause and content-based routing (#333)
ybyzek Oct 3, 2017
ddc9396
Merge pull request #332 from bluemonk3y/0.1.x-CODE-331-rename-ksql-co…
logscape Oct 3, 2017
1584815
First step in better testing tools. Added an integration test tool to…
hjafarpour Oct 3, 2017
58f9a25
Merge pull request #335 from hjafarpour/KSQL-379-Better-Test-Tools
hjafarpour Oct 3, 2017
973c7dc
No need to set the streams application.id for the rest server.
hjafarpour Oct 4, 2017
17fb73f
Merge pull request #336 from hjafarpour/NoNeedForStreamAppIdForRESTSe…
hjafarpour Oct 4, 2017
04094a9
First pass on refactoring KSQL confing.
hjafarpour Oct 4, 2017
fe2296f
extracting new integration test using KSQLContext
Oct 5, 2017
839cb84
Refined admin client config.
hjafarpour Oct 5, 2017
794da0f
remove dead code. make field and method access more restricted
dguy Oct 5, 2017
117d760
Merge branch '0.1.x'
hjafarpour Oct 5, 2017
46f349e
Merge branch 'master' of https://github.com/confluentinc/ksql
hjafarpour Oct 5, 2017
c63aee2
bump common dependency to 3.3.1-SNAPSHOT
Oct 5, 2017
243c00f
Merge remote-tracking branch 'origin/0.1.x'
Oct 5, 2017
392c4d1
using 0.11.0.1 of client to use fixed AdminClient
Oct 5, 2017
443fc5e
kafka version setting to 11.0.1
Oct 5, 2017
c12ca7a
Merge pull request #337 from confluentinc/remove-some-dead-code
hjafarpour Oct 5, 2017
f527704
Merge remote-tracking branch 'upstream/0.1.x' into 0.1.x
hjafarpour Oct 5, 2017
e166ede
Merge branch '0.1.x-CODE-379-started-int-test-refactor' of https://gi…
hjafarpour Oct 6, 2017
520d779
remove unused imports
dguy Oct 6, 2017
5ba12a0
Merge pull request #342 from dguy/unused-imports
dguy Oct 6, 2017
9a17925
remove dead code.
dguy Oct 6, 2017
63228fd
Minor change to use the same Application Id for embedded mode.
hjafarpour Oct 6, 2017
c0836ae
Merge pull request #344 from hjafarpour/bluemonk3y-0.1.x-CODE-379-sta…
hjafarpour Oct 6, 2017
813c850
merge 0.1.x
dguy Oct 6, 2017
2196ea6
Merge pull request #343 from dguy/ksql-engine-tidy
hjafarpour Oct 6, 2017
7c0a11d
Merge remote-tracking branch 'origin/0.1.x'
Oct 7, 2017
e6aaf4d
Merge remote-tracking branch 'upstream/0.1.x' into KSQLConfigRefactor
hjafarpour Oct 9, 2017
71f2d78
Some changes based on the new KSQLConfig.
hjafarpour Oct 9, 2017
8ab2828
Merge pull request #338 from hjafarpour/KSQLConfigRefactor
hjafarpour Oct 9, 2017
6f27e7e
replace window-unit with TimeUnit (#346)
dguy Oct 10, 2017
ff36f7f
skip bad data in json and delimited deserializer (#347)
dguy Oct 10, 2017
f97e11b
Clean up the poms after the refactor that created new packages.
hjafarpour Oct 10, 2017
74beb47
More pom clean up.
hjafarpour Oct 10, 2017
b38cc4b
Merge pull request #350 from hjafarpour/pom-cleanup-after-refactor
hjafarpour Oct 10, 2017
901b3b6
Reverting adminclient singleton temporarily.
hjafarpour Oct 10, 2017
8a6b6a7
Merge pull request #351 from hjafarpour/Revert-AdminClient-Singleton
hjafarpour Oct 10, 2017
5d0978a
Merge branch '0.1.x'
hjafarpour Oct 10, 2017
b9a99fe
Merge pull request #352 from confluentinc/master
hjafarpour Oct 10, 2017
0af3fb8
Bump to common 4.0.0-SNAPSHOT
Oct 11, 2017
0aca9a7
Bump to common 4.1.0-SNAPSHOT
Oct 11, 2017
2e7eb68
update slack channel for build notifications
Oct 11, 2017
c086ca5
Merge remote-tracking branch 'origin/0.1.x' into 4.0.x
Oct 11, 2017
2c82044
Merge remote-tracking branch 'origin/4.0.x'
Oct 11, 2017
aba0f7c
fix artifact version in compose file
Oct 11, 2017
32e41e8
Merge remote-tracking branch 'origin/0.1.x' into 4.0.x
Oct 11, 2017
ed1ee1e
Merge remote-tracking branch 'origin/4.0.x'
Oct 11, 2017
ce9e2da
After bumping the dependency to 4.0.0 we encountered some issues that…
hjafarpour Oct 11, 2017
22e481d
Merge pull request #363 from hjafarpour/KSQL-388-fix-400-build-issues
hjafarpour Oct 11, 2017
27a0e1f
Merge remote-tracking branch 'upstream/master'
hjafarpour Oct 11, 2017
06b59a9
Better clean up for queries. Internal topics will be deletted after q…
hjafarpour Oct 12, 2017
181fa50
Added singleton AdminClient back.
hjafarpour Oct 12, 2017
cc57a3e
extract methods, remove duplication, etc from PhysicalPlanBuilder (#356)
dguy Oct 12, 2017
c480c42
Use streams 1.0 api methods (#365)
dguy Oct 12, 2017
f9e4416
new PR with migrated tests
Oct 12, 2017
dbffadd
tidying stuff up
Oct 12, 2017
cc2177c
trigger build
Oct 12, 2017
2c8ac7b
remove more crud
Oct 12, 2017
32c20ae
Merge pull request #367 from bluemonk3y/4.0.x-TEST-migrating-integrat…
logscape Oct 12, 2017
8b299f3
Merge remote-tracking branch 'upstream/4.0.x' into KSQL-383-clean-up-…
hjafarpour Oct 12, 2017
bcec9eb
Fixed the pom version.
hjafarpour Oct 12, 2017
93634bf
Pom version fix.
hjafarpour Oct 12, 2017
a4c863c
Refactored Function registry.
hjafarpour Oct 13, 2017
8b5b883
adding windowing integration tests, updated streams int test util to …
Oct 13, 2017
5a165c6
trigger
Oct 13, 2017
9c1cc3c
moving packages
Oct 13, 2017
ac3b527
Applied review feedback.
hjafarpour Oct 13, 2017
d18a471
Remove SourceNode as it is redundant (#368)
dguy Oct 16, 2017
858a71b
Merge pull request #371 from bluemonk3y/4.0.x-TEST-windowing-integrat…
logscape Oct 16, 2017
196d4db
adding stream-table join
Oct 16, 2017
a148075
exposing time controls when publishing test data
Oct 16, 2017
73ec9f2
exposing time controls for int tests
Oct 16, 2017
91dd2d5
removed old comments
Oct 16, 2017
e03d4d1
Merge remote-tracking branch 'upstream/4.0.x' into KSQL-386-KsqlFunct…
hjafarpour Oct 16, 2017
87671d0
trigger build
Oct 17, 2017
7055aca
remove jackson dependency in ksql-rest-app. (#369)
dguy Oct 17, 2017
36829c3
Merge pull request #375 from bluemonk3y/4.0.x-TEST-stream-table-join-…
logscape Oct 17, 2017
eb12963
move the createsink method out of SchemaKStream (#376)
dguy Oct 17, 2017
1aecb91
make join test more reliable (#381)
dguy Oct 17, 2017
587b233
rework threading on QueryStreamWriter
Oct 17, 2017
ccd9e13
import
Oct 17, 2017
1fa24e3
import
Oct 17, 2017
1676efc
import
Oct 17, 2017
66c7f13
Use streams deserializtion exception handler (#385)
dguy Oct 17, 2017
28a832d
extract SelectValueMapper from SchemaKXxxx (#379)
dguy Oct 17, 2017
0af6c5b
minor tidyups
Oct 17, 2017
c84ef9d
Merge branch '4.0.x' of https://github.com/confluentinc/ksql into 4.0…
Oct 17, 2017
52ef4ef
Put window aggregate logic into the various KsqlWindowExpression clas…
dguy Oct 17, 2017
863fa31
Merge pull request #383 from bluemonk3y/4.0.x-CODE-QueryStreamWriterT…
logscape Oct 17, 2017
984e51a
Minor modification in KSQLContext.
hjafarpour Oct 17, 2017
c9b3f75
Merge remote-tracking branch 'upstream/4.0.x' into KSQL-386-KsqlFunct…
hjafarpour Oct 17, 2017
de6772e
Merge pull request #370 from hjafarpour/KSQL-386-KsqlFunctions-non-st…
hjafarpour Oct 17, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add Configure KSQL to Syntax Guide, fix/clariy config examples
  • Loading branch information
Michael G. Noll committed Sep 12, 2017
commit b2df850d42dfbdcbf02374dcf69ccbc2b5eeeacb
26 changes: 9 additions & 17 deletions docs/examples.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Examples

| [Overview](/docs#ksql-documentation) |[Quick Start](/docs/quickstart#quick-start) | [Concepts](/docs/concepts.md#concepts) | [Syntax Reference](/docs/syntax-reference.md#syntax-reference) |[Demo](/ksql-clickstream-demo#clickstream-analysis) | Examples | [FAQ](/docs/faq.md#frequently-asked-questions) | [Roadmap](/docs/roadmap.md#roadmap) |
| [Overview](/docs#ksql-documentation) |[Quick Start](/docs/quickstart#quick-start) | [Concepts](/docs/concepts.md#concepts) | [Syntax Reference](/docs/syntax-reference.md#syntax-reference) |[Demo](/ksql-clickstream-demo#clickstream-analysis) | Examples | [FAQ](/docs/faq.md#frequently-asked-questions) | [Roadmap](/docs/roadmap.md#roadmap) |
|---|----|-----|----|----|----|----|----|

**Table of Contents**
Expand Down Expand Up @@ -252,30 +252,22 @@ CREATE STREAM pageviews_interest_contact AS \

## Configuring KSQL

You can set config properties for KSQL and your queries with the SET statement. This includes configuring settings
relating to Kafka's Streams API as well as settings for Kafka's producer and consumer clients.
Common configuration properties that you might want to change from their default values include:

```sql
SET '<property-name>'='<property-value>';
```

Both property name and property value should be enclosed in single quotes.
A property that is set using the SET statement will remain in effect for the remainder of the KSQL CLI session
until you issue another SET statement to change it.

Here are some of the common config properties that you might want to change from their default values:

- `auto.offset.reset`: The default value in KSQL is `latest` meaning all the Kafka topics will be read
from the current offset (aka latest available data). You can change it using the following statement:
- [auto.offset.reset](https://kafka.apache.org/documentation/#newconsumerconfigs):
The default value in KSQL is `latest` meaning all the Kafka topics will be read from the current offset (aka latest
available data). You can change it using the following statement:

```sql
SET 'auto.offset.reset'='earliest';
```

- `commit.interval.ms`: The default value is `2000`. Here is an example to change the value to `5000`:
- [commit.interval.ms](https://kafka.apache.org/documentation/#streamsconfigs):
The default value in KSQL is `2000`. Here is an example to change the value to `5000`:

```sql
SET 'commit.interval.ms'='5000';
```

- `cache.max.bytes.buffering`: The default value is `10485760` (10 MB);
- [cache.max.bytes.buffering](https://kafka.apache.org/documentation/#streamsconfigs):
The default value in KSQL is `10000000` (~ 10 MB);
60 changes: 59 additions & 1 deletion docs/syntax-reference.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Syntax Reference

| [Overview](/docs#ksql-documentation) |[Quick Start](/docs/quickstart#quick-start) | [Concepts](/docs/concepts.md#concepts) | Syntax Reference | [Demo](/ksql-clickstream-demo#clickstream-analysis) | [Examples](/docs/examples.md#examples) | [FAQ](/docs/faq.md#frequently-asked-questions) | [Roadmap](/docs/roadmap.md#roadmap) |
| [Overview](/docs#ksql-documentation) | [Quick Start](/docs/quickstart#quick-start) | [Concepts](/docs/concepts.md#concepts) | Syntax Reference | [Demo](/ksql-clickstream-demo#clickstream-analysis) | [Examples](/docs/examples.md#examples) | [FAQ](/docs/faq.md#frequently-asked-questions) | [Roadmap](/docs/roadmap.md#roadmap) |
|---|----|-----|----|----|----|----|----|


Expand All @@ -12,6 +12,7 @@ The KSQL CLI provides a terminal-based interactive shell for running queries.
- [KSQL statements](#ksql-statements)
- [Scalar functions](#scalar-functions)
- [Aggregate functions](#aggregate-functions)
- [Configuring KSQL](#configuring-ksql)


# CLI-specific commands
Expand Down Expand Up @@ -428,6 +429,19 @@ SHOW QUERIES;
List the running persistent queries.


### SHOW PROPERTIES

**Synopsis**

```sql
SHOW PROPERTIES;
```

**Description**

List the [configuration settings](#configuring-ksql) that are currently in effect.


### TERMINATE

**Synopsis**
Expand Down Expand Up @@ -477,3 +491,47 @@ Terminate a persistent query. Persistent queries run continuously until they are
| MAX | `MAX(col1)` | Return the maximum value for a given column and window |
| MIN | `MIN(col1)` | Return the minimum value for a given column and window |
| SUM | `SUM(col1)` | Sums the column values |


# Configuring KSQL

You can set configuration properties for KSQL and your queries with the SET statement. This includes
[settings for Kafka's Streams API](https://kafka.apache.org/documentation/#streamsconfigs)
(e.g., `cache.max.bytes.buffering`) as well as
settings for Kafka's [producer client](https://kafka.apache.org/documentation/#producerconfigs) and
[consumer client](https://kafka.apache.org/documentation/#newconsumerconfigs) (e.g., `auto.offset.reset`).

```sql
SET '<property-name>'='<property-value>';
```

Examples:

```
ksql> SET 'auto.offset.reset'='earliest';
ksql> SET 'commit.interval.ms'='5000';
```

Both property name and property value should be enclosed in single quotes.

Once a property has been set, it will remain in effect for the remainder of the KSQL CLI session until you issue another
SET statement to change it.

You can also use a *properties file* instead of the SET statement. The syntax of properties files follow Java
conventions, which are slightly different to the syntax of the SET statement above.

```bash
# Show the example contents of a properties file
$ cat ksql.properties
auto.offset.reset=earliest

# Start KSQL in standalone mode with the custom properties above
$ ksql-cli local --properties-file ./ksql.properties

# Start a KSQL server node (for client-server mode) with the custom properties above
$ ksql-server-start --properties-file ./ksql.properties
```

Note: Be careful when you are using KSQL in Docker because the properties file must be available inside the Docker
container. If you don't want to customize your Docker setup so that it contains an appropriate properties file, you
should not use a properties file but the SET statement.