Improve append-only inference for input connectors #149
Description
Is your feature request related to a problem? Please describe.
Input connectors often infer append-only properties of the collections they read from, but these properties are not always inferred correctly, even though this could (and should) be improved.
In particular:
- Collections read from Kafka are almost always append-only (except possible UpsertSession exception, not even exposed to the users).
- The Python connector should also be append-only by default.
Marking these collections as append-only would allow the runtime engine to apply more aggressive memory optimizations. When a collection is append-only, significantly more efficient memory management strategies are available.
Describe the solution you'd like
- The Python connector append-only by default.
- The Kafka connector append-only by default.
Describe alternatives you've considered
N/A.
Additional context
This change should be fully backward-compatible.
In practice, the collections produced by these connectors are already append-only; the issue is that this property is currently not explicitly set.