-
Notifications
You must be signed in to change notification settings - Fork 73
Update of the documentation for the 0.12.1 release #556
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
c1a009b
Add SQL schema generation guide to gradleReference.md
zaleslaw 8927800
Finished gradleReference.md changes
zaleslaw 8a48733
Removed OpenAPI schemas section from schemasGradle.md file because it…
zaleslaw 969c68e
Add SQL schema import tutorial and update documentation
zaleslaw ceabcab
Update SQL schema import guide and enhance documentation.
zaleslaw 655d562
The modification of the guide in the SQL schema import document provi…
zaleslaw File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,7 +10,7 @@ dataframes { | |
} | ||
} | ||
``` | ||
Note than name of the file and the interface are normalized: split by '_' and ' ' and joined to camel case. | ||
Note that the name of the file and the interface are normalized: split by '_' and ' ' and joined to CamelCase. | ||
You can set parsing options for CSV: | ||
```kotlin | ||
dataframes { | ||
|
@@ -23,24 +23,36 @@ dataframes { | |
} | ||
} | ||
``` | ||
In this case output path will depend on your directory structure. For project with package `org.example` path will be `build/generated/dataframe/main/kotlin/org/example/dataframe/JetbrainsRepositories.Generated.kt | ||
`. Note that name of the Kotlin file is derived from the name of the data file with the suffix `.Generated` and the package | ||
is derived from the directory structure with child directory `dataframe`. The name of the **data schema** itself is `JetbrainsRepositories`. You could specify it explicitly: | ||
In this case, the output path will depend on your directory structure. | ||
For project with package `org.example` path will be `build/generated/dataframe/main/kotlin/org/example/dataframe/JetbrainsRepositories.Generated.kt | ||
`. | ||
|
||
Note that the name of the Kotlin file is derived from the name of the data file with the suffix | ||
`.Generated` and the package | ||
is derived from the directory structure with child directory `dataframe`. | ||
|
||
The name of the **data schema** itself is `JetbrainsRepositories`. | ||
You could specify it explicitly: | ||
|
||
```kotlin | ||
schema { | ||
// output: build/generated/dataframe/main/kotlin/org/example/dataframe/MyName.Generated.kt | ||
data = "https://raw.githubusercontent.com/Kotlin/dataframe/master/data/jetbrains_repositories.csv" | ||
name = "MyName" | ||
} | ||
``` | ||
If you want to change default package for all schemas: | ||
|
||
If you want to change the default package for all schemas: | ||
|
||
```kotlin | ||
dataframes { | ||
packageName = "org.example" | ||
// Schemas... | ||
} | ||
``` | ||
|
||
Then you can set packageName for specific schema exclusively: | ||
|
||
```kotlin | ||
dataframes { | ||
// output: build/generated/dataframe/main/kotlin/org/example/data/OtherName.Generated.kt | ||
|
@@ -50,7 +62,9 @@ dataframes { | |
} | ||
} | ||
``` | ||
If you want non-default name and package, consider using fully-qualified name: | ||
|
||
If you want non-default name and package, consider using fully qualified name: | ||
|
||
```kotlin | ||
dataframes { | ||
// output: build/generated/dataframe/main/kotlin/org/example/data/OtherName.Generated.kt | ||
|
@@ -60,7 +74,10 @@ dataframes { | |
} | ||
} | ||
``` | ||
By default, plugin will generate output in specified source set. Source set could be specified for all schemas or for specific schema: | ||
|
||
By default, the plugin will generate output in a specified source set. | ||
Source set could be specified for all schemas or for specific schema: | ||
|
||
```kotlin | ||
dataframes { | ||
packageName = "org.example" | ||
|
@@ -76,7 +93,9 @@ dataframes { | |
} | ||
} | ||
``` | ||
But if you need generated files in other directory, set `src`: | ||
|
||
If you need the generated files to be put in another directory, set `src`: | ||
|
||
```kotlin | ||
dataframes { | ||
// output: schemas/org/example/test/OtherName.Generated.kt | ||
|
@@ -87,10 +106,63 @@ dataframes { | |
} | ||
} | ||
``` | ||
## Schema Definitions from SQL Databases | ||
|
||
To generate a schema for an existing SQL table, | ||
you need to define a few parameters to establish a JDBC connection: | ||
URL (passing to `data` field), username, and password. | ||
|
||
Also, the `tableName` parameter should be specified to convert the data from the table with that name to the dataframe. | ||
|
||
```kotlin | ||
dataframes { | ||
schema { | ||
data = "jdbc:mariadb://localhost:3306/imdb" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we add a link to this database? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I want to release example with this database and will update the links |
||
name = "org.example.imdb.Actors" | ||
jdbcOptions { | ||
user = "root" | ||
password = "pass" | ||
tableName = "actors" | ||
} | ||
} | ||
} | ||
``` | ||
|
||
To generate a schema for the result of an SQL query, | ||
you need to define the same parameters as before together with the SQL query to establish connection. | ||
|
||
```kotlin | ||
dataframes { | ||
schema { | ||
data = "jdbc:mariadb://localhost:3306/imdb" | ||
name = "org.example.imdb.TarantinoFilms" | ||
jdbcOptions { | ||
user = "root" | ||
password = "pass" | ||
sqlQuery = """ | ||
SELECT name, year, rank, | ||
GROUP_CONCAT (genre) as "genres" | ||
FROM movies JOIN movies_directors ON movie_id = movies.id | ||
JOIN directors ON directors.id=director_id LEFT JOIN movies_genres ON movies.id = movies_genres.movie_id | ||
WHERE directors.first_name = "Quentin" AND directors.last_name = "Tarantino" | ||
GROUP BY name, year, rank | ||
ORDER BY year | ||
""" | ||
} | ||
} | ||
} | ||
``` | ||
|
||
**NOTE:** This is an experimental functionality and, for now, | ||
we only support four databases: MariaDB, MySQL, PostgreSQL, and SQLite. | ||
zaleslaw marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Additionally, support for JSON and date-time types is limited. | ||
Please take this into consideration when using these functions. | ||
|
||
## DSL reference | ||
Inside `dataframes` you can configure parameters that will apply to all schemas. Configuration inside `schema` will override these defaults for specific schema. | ||
Here is full DSL for declaring data schemas: | ||
Inside `dataframes` you can configure parameters that will apply to all schemas. | ||
Configuration inside `schema` will override these defaults for a specific schema. | ||
Here is the full DSL for declaring data schemas: | ||
|
||
```kotlin | ||
dataframes { | ||
|
@@ -101,8 +173,8 @@ dataframes { | |
// KOTLIN SCRIPT: DataSchemaVisibility.INTERNAL DataSchemaVisibility.IMPLICIT_PUBLIC, DataSchemaVisibility.EXPLICIT_PUBLIC | ||
// GROOVY SCRIPT: 'internal', 'implicit_public', 'explicit_public' | ||
|
||
withoutDefaultPath() // disable default path for all schemas | ||
// i.e. plugin won't copy "data" property of the schemas to generated companion objects | ||
withoutDefaultPath() // disable a default path for all schemas | ||
zaleslaw marked this conversation as resolved.
Show resolved
Hide resolved
|
||
// i.e., plugin won't copy "data" property of the schemas to generated companion objects | ||
|
||
// split property names by delimiters (arguments of this method), lowercase parts and join to camel case | ||
// enabled by default | ||
|
@@ -125,8 +197,8 @@ dataframes { | |
withNormalizationBy('_') // enable property names normalization for this schema and use these delimiters | ||
withoutNormalization() // disable property names normalization for this schema | ||
|
||
withoutDefaultPath() // disable default path for this schema | ||
withDefaultPath() // enable default path for this schema | ||
withoutDefaultPath() // disable the default path for this schema | ||
withDefaultPath() // enable the default path for this schema | ||
} | ||
} | ||
``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.