Skip to content

Commit

Permalink
doc: Kestra 0.19
Browse files Browse the repository at this point in the history
  • Loading branch information
loicmathieu authored and brian-mulier-p committed Oct 1, 2024
1 parent ebe5488 commit 579e4a7
Show file tree
Hide file tree
Showing 3 changed files with 65 additions and 1 deletion.
21 changes: 21 additions & 0 deletions content/docs/10.configuration-guide/01.database.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,3 +225,24 @@ kestra:
fixed-delay: 1h
retention: 7d
```

## Protecting against too big messages

::alert{type="info"}
Note: this is an experimental feature available starting with Kestra 0.19.
::

The database backend has no limit on the size of messages it can handle. However, as messages are loaded into memory, this can endanger Kestra itself and push pressure on memory usage.

To prevent that, you can configure a functionality that will refuse to store too big messages and fail the execution instead.

The following configuration will refuse messages that exceed 1MiB by failing the execution.

```yaml
kestra:
jdbc:
queues:
message-protection:
enabled: true
limit: 1048576
```
25 changes: 24 additions & 1 deletion content/docs/10.configuration-guide/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,4 +179,27 @@ For example, for the topic storing executions, you can configure the retention v
execution:
properties:
retention.ms: "86400000"
```
```

## Protecting against too big messages

::alert{type="info"}
Note: this is an experimental feature.
::

Kafka topic has a limit of the size of messages it can handle. By default, we set this limit to 10MiB.
If a message exceeds this limit, it may crash the executor and Kestra will stop.

To prevent that, you can configure a functionality that will automatically store too big messages to the internal storage.

The following configuration will store all messages that exceed 1MiB to the internal storage, and refuse messages that exceed 10MiB by failing the execution, the executor will not crash, and Kestra will keep functioning.

```yaml
kestra:
kafka:
message-protection:
soft-limit: 1048576
hard-limit: 10485760
```

The `hard-limit` is not mandatory, in this case messages of any size will be accepted.
20 changes: 20 additions & 0 deletions content/docs/10.configuration-guide/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,3 +75,23 @@ triggers:
```

In this example, the `recoverMissedSchedules` is set to `NONE`, which means that Kestra will not recover any missed schedules for this specific flow regardless of the global configuration.

## Restrict which plugins can be used

::alert{type="info"}
This is an [Enterprise Edition](/docs/enterprise) feature available starting with Kestra 0.19.
::

You can restrict which plugins can be used in a Kestra instance by configuring an allowlist / exclude list using regexes.

The following configuration only allow plugin from the `io.kestra` package and disallow the `io.kestra.plugin.core.debug.Echo` plugin.

```yaml
kestra:
plugins:
security:
includes:
- io.kestra.*
excludes:
- io.kestra.plugin.core.debug.Echo
```

0 comments on commit 579e4a7

Please sign in to comment.