feat(docs): add kv store concepts page (#1566)

content/docs/05.concepts/05.kv-store.md

@@ -0,0 +1,277 @@
+---
+title: KV Store
+icon: /docs/icons/concepts.svg
+version: ">= 0.18.0"
+---
+
+KV Store allows you to have key-value pairs at the namespace level.
+
+The Key Value Store, generally called as KV Store, is an implementation on top of Kestra's internal storage. As the name suggests, it is a store for key value pairs.
+
+The motivation behind introducing KV store is:
+
+**Privacy**: We never want Kestra to store user private data. This means that all values will be stored in the user’s private cloud storage bucket, and the Kestra's database only contains metadata about it, such as the key, file URI, any attached metadata about the object like TTL, creation date, last updated timestamp, etc.
+
+**Ease of implementation/migration**: Users can easily switch from open-source to cloud/EE because the implementation and data storage will be the same regardless of whether Kestra runs on top of Kafka or JDBC.
+
+## Keys and Values
+
+`Keys` are arbitrary strings. Keys can contain:
+
+* characters in uppercase, lowercase, or mixed
+* standard ASCII characters
+
+`Values` are stored as ION files in Kestra's internal storage. Values are strongly typed, and can be of one of the following types:
+
+- string
+- number
+- boolean
+- datetime
+- date
+- duration
+- JSON
+
+You can associate TTL with the KV pair. The KV pair can only be used until it is active as per the TTL.
+
+## Namespace binding
+
+Key value pairs are tied to a namespace.
+
+Users should be able to create and read KV pairs across namespaces as long as those namespaces are allowed.
+
+### Namespace Management in OSS
+
+Starting in 0.18.0, Kestra has opened up a lightweight version of Namespace Management in OSS that will help you manage the KV store. You will now have access to the `Namespaces` tab from the left navigation menu on the Kestra's UI. You will be able to see all the available namespaces on this tab, as well as create one.
+
+For any namespace, you will be able to see the namespace's overview, its corresponding editor, flows, dependencies and the KV store.
+
+## Managing KV Store
+
+You can manage the KV store via:
+
+- the UI
+- the tasks
+- the API
+- the terraform resource
+
+### Managing KV Store via UI
+
+In the Kestra UI,
+
+1. Navigate to the `Namespaces` tab from the left navigation menu.
+2. Select the namespace where you want to create the KV pair.
+3. Navigate to the `KV Store` tab. This is where you can see all the KV pairs associated with this namespace.
+4. Click on `New Key-Value` button on the top right to create a new KV pair.
+5. Put an appropriate name for the `Key`.
+6. Choose an appropriate `Type` for the value.
+7. Put an appropriate value in the `Value` textbox.
+8. You can choose an expiration time (TTL) for the KV pair from the dropdown. Note that the dropdown contains some standard durations. In case you want some custom duration, you can enter it in the `Custom duration` textbox.
+9. Click on `Save` button at the bottom to save this KV pair.
+
+With this, a new KV pair is created.
+
+![navigate_to_namespace](/docs/concepts/kv-store/navigate_to_namespace.png)
+
+![navigate_to_keystore](/docs/concepts//kv-store/navigate_to_keystore.png)
+
+![create_kv_pair](/docs/concepts//kv-store/create_kv_pair.png)
+
+You can edit / delete the KV pair by clicking on the appropriate button towards the right of the corresponding KV pair.
+
+![edit_delete_kv_pair](/docs/concepts//kv-store/edit_delete_kv_pair.png)
+
+### Managing KV Store via tasks
+
+You can manage the KV store via tasks in the following fashion:
+
+#### Set the KV pair
+
+```yaml
+id: set_kv
+type: io.kestra.plugin.kv.Set
+key: my_key
+value: "{{ outputs.query.uri }}"
+namespace: dev # the current namespace of the flow can be used by default
+overwrite: true # whether to overwrite or fail if a value for that key already exists; default true
+ttl: P30D # optional TTL
+```
+
+You can use set to create or edit the KV pair. Ensure `overwrite` is set to `true` while editing the KV pair.
+
+#### Get the KV pair
+
+The easiest way to retrieve a value by key is to use the Pebble function following this syntax:
+
+```
+{{ kv('KEY_NAME', namespace_name, errorOnMissing_boolean) }}
+```
+
+For example,
+
+```
+{{ kv('my_key') }} # assuming you retrieve it in a flow in the same namespace as the one for which key was created
+```
+
+If you prefer, you can also retrieve the value using a task:
+
+```yaml
+id: get_value_by_key
+type: io.kestra.core.tasks.kv.Get
+key: my_key
+namespace: dev # the current namespace of the flow can be used by default
+errorOnMissing: false # bool
+```
+
+And if you want to check if some values already exist for a given key, you can search keys by prefix:
+
+```yaml
+id: get_keys_by_prefix
+type: io.kestra.core.tasks.kv.GetKeys
+prefix: "test_"
+namespace: dev # the current namespace of the flow can be used by default
+errorOnMissing: false # bool
+```
+
+The output will be a list of keys - if no keys were found, an empty list will be returned.
+
+#### Delete a KV pair
+
+```yaml
+id: delete_kv_pair
+type: io.kestra.core.tasks.kv.Delete
+key: my_key
+namespace: dev # the current namespace of the flow can be used by default
+errorOnMissing: false 
+```
+
+### Managing KV Store via API
+
+You can manage the KV store via APIs with the following URLs:
+
+#### Set the KV pair
+
+The API call to set the KV pair follows the structure:
+
+```bash
+% curl -X PUT -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/{namespace}/kv/{key} -d '<value>'
+```
+
+For example:
+
+```bash
+% curl -X PUT -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/my_key -d '"Hello World"'
+```
+
+The above `curl` call will create the KV pair with key `my_key` with value `Hello World` string in the `company.team` namespace. The API does not return any response as such.
+
+#### Get the KV pair
+
+You can get any particular KV pair using:
+
+```bash
+% curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/{namespace}/kv/{key}
+```
+
+For example:
+
+```bash
+curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/my_key
+```
+
+This would retrieve a KV pair with the key `my_key` in the `company.team` namespace. The output of the API will contain the data type of the value and the retrived value of the KV pair. The output will look like:
+
+```json
+{"type": "STRING", "value": "Hello World"}
+```
+
+You can also get all the keys in the namespace using the curl call:
+
+```bash
+curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/{namespace}/kv
+```
+
+For example, the following curl call will return all the keys in the `company.team` namespace:
+
+```bash
+curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv
+```
+
+The output will be as follows:
+```json
+{"key":"my_key","creationDate":"2024-07-15T06:10:33.422Z","updateDate":"2024-07-15T06:11:08.911Z"},{"key":"test_key","creationDate":"2024-07-15T04:37:18.196Z","updateDate":"2024-07-15T04:37:18.196Z"}]
+```
+
+#### Delete the KV pair
+
+You can delete the KV paior using the following API:
+
+```bash
+curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/{namespace}/kv/{key}
+```
+
+This call returns the boolean indicating whether the key existed for deletion or not.
+
+For example, the following curl call will delete the key `my_key` in the `company.team` namespace:
+
+```bash
+curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/my_key
+```
+
+### Managing KV Store via Terraform
+
+You can manage the KV store via Terraform in the following fashion:
+
+#### Create the KV pair
+
+```
+resource "kestra_kv" "new" {
+  namespace = "company.team"
+  key       = "my_key"
+  value     = "Hello Woprld"
+  type      = "STRING"
+}
+```
+
+#### Get the KV pair
+
+```
+data "kestra_kv" "new" {
+  namespace = "company.team"
+  key       = "my_key"
+}
+```
+
+## Accessing JSON value
+
+For accessing JSON value, you need to use the `json()` function on the retrieved value. This is demonstrated in the following flow:
+
+```yaml
+id: kv_json_flow
+namespace: company.team
+tasks:
+  - id: set_json_kv
+    type: io.kestra.plugin.core.kv.Set
+    key: "my_json_key"
+    value: | 
+      {
+        "name": "John Doe",
+        "age": 32,
+        "address": {
+          "city": "Paris",
+          "country": "France"
+        }
+      }
+    overwrite: true
+  - id: get_json_kv_pebble
+    type: io.kestra.plugin.core.log.Log
+    message:
+      - "Name: {{ json(kv('my_json_key')).name }}"
+      - "Age: {{ json(kv('my_json_key')).age }}"
+      - "City: {{ json(kv('my_json_key')).address.city }}"
+  - id: get_json_kv_task
+    type: io.kestra.plugin.core.kv.Get
+    key: my_json_key
+  - id: get_json_kv_nested_value
+    type: io.kestra.plugin.core.log.Log
+    message: "Country: {{ json(outputs.get_json_kv_task.value).address.country }}"
+```