Skip to content

Commit

Permalink
added docs for utf8 encoding
Browse files Browse the repository at this point in the history
  • Loading branch information
SabrinaZhaozyf authored and xiangfu0 committed Aug 30, 2022
1 parent 86b7676 commit c62d69e
Show file tree
Hide file tree
Showing 5 changed files with 43 additions and 6 deletions.
1 change: 1 addition & 0 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,7 @@
* [trim](configuration-reference/functions/trim.md)
* [upper](configuration-reference/functions/upper.md)
* [Url](configuration-reference/functions/url.md)
* [UTF8](configuration-reference/functions/utf8.md)
* [VALUEIN](configuration-reference/functions/valuein.md)
* [week](configuration-reference/functions/week.md)
* [year](configuration-reference/functions/year.md)
Expand Down
1 change: 1 addition & 0 deletions configuration-reference/functions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ This page contains reference documentation for functions in Apache Pinot.
{% page-ref page="trim.md" %}
{% page-ref page="upper.md" %}
{% page-ref page="url.md" %}
{% page-ref page="utf8.md" %}
{% page-ref page="valuein.md" %}
{% page-ref page="week.md" %}
{% page-ref page="year.md" %}
Expand Down
8 changes: 4 additions & 4 deletions configuration-reference/functions/base64.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ description: This section contains reference documentation for base64 encode and

Encoding scheme follows [java.util.Base64.Encoder](https://docs.oracle.com/javase/8/docs/api/java/util/Base64.Encoder.html)

`toBase64` returns Base64-encoded string of input binary data (`bytes` type).
`fromBase64` returns binary data (represented as a Hex string) from Base64-encoded string.
* `toBase64` returns Base64 encoded string of input binary data (`bytes` type).
* `fromBase64` returns binary data (represented as a Hex string) from Base64-encoded string.

## Signature

Expand All @@ -18,8 +18,8 @@ Encoding scheme follows [java.util.Base64.Encoder](https://docs.oracle.com/javas
## Usage Examples

{% hint style="info" %}
For better readability, the following examples converts string `hello!` into BYTES using [toUtf8]() function
and converts the decoded BYTES into string using [fromUtf8]().
For better readability, the following examples converts string `hello!` into BYTES using [toUtf8](https://docs.pinot.apache.org/configuration-reference/functions/utf8) function
and converts the decoded BYTES into string using [fromUtf8](https://docs.pinot.apache.org/configuration-reference/functions/utf8).
{% endhint %}

```sql
Expand Down
34 changes: 34 additions & 0 deletions configuration-reference/functions/utf8.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
description: This section contains reference documentation for UTF8 encode/decode functions.
---

# UTF8

* `fromUtf8` returns UTF8 encoded string of input binary data (`bytes` type).
* `toUtf8` returns binary data (represented as a Hex string) from a UTF8 encoded string.

## Signature

> fromUtf8(bytesCol)
>
> toUtf8(string)
## Usage Examples

```sql
SELECT bytesCol1, fromUtf8(bytesCol1) AS utf8Str
FROM testTable
LIMIT 1
```
| bytesCol1 | utf8Str|
|-------------|--------|
| 68656c6c6f21| hello! |

```sql
SELECT toUtf8('hello!') AS binaryOutput
FROM ignoreMe
```

| binaryOutput |
|--------------|
| 68656c6c6f21 |
5 changes: 3 additions & 2 deletions users/user-guide-query/supported-transformations.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ Multiple string functions are supported out of the box from release-0.5.0 .
| <p><a href="../../configuration-reference/functions/remove.md"><strong>remove(input, search)</strong></a><br>removes all instances of search from string</p> |
| <p><a href="../../configuration-reference/functions/url.md"><strong>urlEncoding(string)</strong></a><br>url-encode a string with UTF-8 format</p> |
| <p><a href="../../configuration-reference/functions/url.md"><strong>urlDecoding(string)</strong></a><br>decode a url to plaintext string</p> |
| <p><a href="../../configuration-reference/functions/base64.md"><strong>fromBase64(string)</strong></a><br>decode a Base64-encoded to bytes represented as a hex string</p> |

| <p><a href="../../configuration-reference/functions/base64.md"><strong>fromBase64(string)</strong></a><br>decode a Base64-encoded string to bytes represented as a hex string</p> |
| <p><a href="../../configuration-reference/functions/utf8.md"><strong>toUtf8(string)</strong></a><br>decode a UTF8-encoded string to bytes represented as a hex string</p> |
## DateTime Functions

Date time functions allow you to perform transformations on columns that contain timestamps or dates.
Expand Down Expand Up @@ -131,6 +131,7 @@ These functions can be used for column transformation in table ingestion configs
| <p><a href="../../configuration-reference/functions/sha512.md"><strong>SHA512(bytesCol)</strong></a><br>Return SHA-512 digest of binary column(<code>bytes</code> type) as hex string</p> |
| <p><a href="../../configuration-reference/functions/md5.md"><strong>MD5(bytesCol)</strong></a><br>Return MD5 digest of binary column(<code>bytes</code> type) as hex string</p> |
| <p><a href="../../configuration-reference/functions/base64.md"><strong>toBase64(bytesCol)</strong></a><br>Return the Base64-encoded string of binary column(<code>bytes</code> type)</p> |
| <p><a href="../../configuration-reference/functions/utf8.md"><strong>fromUtf8(bytesCol)</strong></a><br>Return the UTF8-encoded string of binary column(<code>bytes</code> type)</p> |

## Multi-value Column Functions

Expand Down

0 comments on commit c62d69e

Please sign in to comment.