Skip to content

Commit

Permalink
GITBOOK-1667: Update supported function types for ingestion transform…
Browse files Browse the repository at this point in the history
…ation.
  • Loading branch information
klsince authored and gitbook-bot committed May 3, 2023
1 parent f12b9df commit 711b187
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions developers/advanced/ingestion-level-transformations.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@ If not set, Groovy for ingestion transformation is disabled by default.

### Inbuilt Pinot functions

There are also several inbuilt functions that can be used directly as ingestion transform functions
All the functions defined in [this directory](https://github.com/apache/pinot/tree/02cb2d4970c71a2ea5b4c140a860fbf220e11bd3/pinot-common/src/main/java/org/apache/pinot/common/function/scalar) annotated with `@ScalarFunction` (e.g. [toEpochSeconds](https://github.com/apache/pinot/blob/02cb2d4970c71a2ea5b4c140a860fbf220e11bd3/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/DateTimeFunctions.java#L78)) are supported ingestion transformation functions.

Below are some commonly used inbuilt Pinot functions for ingestion transformations.

#### DateTime functions

Expand Down Expand Up @@ -101,7 +103,7 @@ Letters that are not part of Simple Date Time legend ([https://docs.oracle.com/j

| Function name | Description |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| json_format | <p>Converts a JSON/AVRO complex object to a string. This json map can then be queried using <a href="https://docs.pinot.apache.org/users/user-guide-query/pinot-query-language#transform-function-in-aggregation-and-grouping">jsonExtractScalar</a> function.</p><p><code>"json_format(jsonMapField)"</code></p> |
| json\_format | <p>Converts a JSON/AVRO complex object to a string. This json map can then be queried using <a href="https://docs.pinot.apache.org/users/user-guide-query/pinot-query-language#transform-function-in-aggregation-and-grouping">jsonExtractScalar</a> function.</p><p><code>"json_format(jsonMapField)"</code></p> |

## Types of transformation

Expand Down Expand Up @@ -143,8 +145,8 @@ Consider a table that has a string column `campaign` and a multi-value column do
}
```

Filter config also supports SQL-like expression of inbuilt [scalar functions](../../users/user-guide-query/scalar-functions.md#scalar-functions) for filtering records (starting v 0.11.0+).
Example:
Filter config also supports SQL-like expression of inbuilt [scalar functions](../../users/user-guide-query/scalar-functions.md#scalar-functions) for filtering records (starting v 0.11.0+). Example:

```javascript
"ingestionConfig": {
"filterConfig": {
Expand Down Expand Up @@ -174,7 +176,6 @@ Transform functions can be defined on columns in the ingestion config of the tab
For example, imagine that our source data contains the `prices` and `timestamp` fields. We want to extract the maximum price and store that in the `maxPrices` field and convert the timestamp into the number of hours since the epoch and store it in the `hoursSinceEpoch` field. You can do this by applying the following transformation:
{% code title="pinot-table-offline.json" %}
```javascript
{
"tableName": "myTable",
Expand All @@ -191,7 +192,6 @@ For example, imagine that our source data contains the `prices` and `timestamp`
}
}
```
{% endcode %}
Below are some examples of commonly used functions.
Expand Down

0 comments on commit 711b187

Please sign in to comment.