Is your feature request related to a problem? Please describe.
There's currently no built-in way to translate a field's raw value into a more human-readable one using a static lookup table. For example, when processing Kafka logs, the kafka_request_api_key field contains numeric API key codes ("0", "1", "2", ...) instead of their human-readable names (produce, fetch, list_offsets, ...). Today this requires an if/else chain in the transform program source, which doesn't scale to the ~70 entries of the Kafka API key table.
Describe the solution you'd like
Add a lookup(value, table, default:) function to the transform stdlib. The table is an ordinary object value, so no new config parameter is needed:
actions:
- type: transform
source: |
api_key = {"0": "produce", "1": "fetch", "2": "list_offsets", "3": "metadata"}
.kafka_request_api_key = lookup(.kafka_request_api_key, api_key)
Behavior notes:
value — the value to translate; reading a missing field yields null and passes through unchanged.
table — an object mapping raw value → replacement value. Objects only.
- Keys are matched by their canonical string form, so the number
0 and the string "0" address the same entry — JSON writes enum codes both ways.
- A value that isn't a key of the table is returned unchanged, rather than erroring or emptying the field.
default: overrides that: lookup(.status, {"500": "crit"}, default: "ok").
- Composite values (arrays, objects) are rejected as keys.
For this to be usable with large tables, a table literal must not be rebuilt on every event — ObjectExpr.Eval currently allocates a fresh map per evaluation. Constant sub-expressions should be folded into a new ConstExpr AST node inside the existing validation walk (ValidateCalls), which already prepares nodes in place: it compiles regex literals and parses timestamp literals. No separate compiler pass.
- Fold in two positions: function call arguments and assignment right-hand sides, so both
lookup(.x, {...}) and the table = {...} idiom are covered.
- An expression that fails to evaluate must never be folded — it stays in the AST and keeps failing per event, so folding can't turn a runtime error into a startup failure.
- Folded values are shared across events and processor goroutines. That's safe only because values are immutable (index and member assignment copy the container), and it must be covered by a test.
Goal: the per-event cost of a lookup doesn't grow with the size of the table.
Describe alternatives you've considered
- A
mapping: section in the plugin config plus map(value, "name") (the shape originally proposed here) — rejected: it adds a config parameter serving exactly one function, and every future function needing startup-prepared data would want its own. Resolving a table by name at runtime also needs per-instance state, but the stdlib registry is a process-global of stateless functions, so it would mean threading a per-instance registry through Start, ValidateCalls and NewContext. And since validating table names at startup requires a literal argument anyway, the runtime name lookup buys nothing. Finally, a per-action mapping: block is no more reusable across actions than an object literal is.
- Inline if/else chains in the program source — doesn't scale and is hard to maintain for enums with many values.
- An object literal without constant folding — correct, but rebuilds the table on every event.
- Loading the table from an external file — out of scope for now; see below.
- Doing the mapping upstream — not always possible when the source of the field (e.g. a Kafka producer) can't be changed.
Additional context
Example use case: Kafka kafka_request_api_key numeric codes → human-readable API names (produce, fetch, offsets, etc.), to make dashboards and log search more readable without needing a lookup table on the query side.
Name the function lookup, not map — in an expression language map reads as map/filter/reduce.
Out of scope:
- loading tables from an external file — additive later as a foldable
from_file(...) in the same argument position, if large tables in YAML become a real complaint;
- a
Const flag on Parameter to reject a non-constant table at startup; a non-constant table should simply work and not be folded.
Definition of done:
- unit tests for
lookup (hits, miss behavior, default:, numeric/string key equivalence, argument validation);
- unit tests for folding, including the don't-fold-on-error rule and the shared-value immutability guarantee;
- an end-to-end test in
transform_test.go;
lookup documented in the plugin's functions section, with README.md regenerated via make gen-doc.
Is your feature request related to a problem? Please describe.
There's currently no built-in way to translate a field's raw value into a more human-readable one using a static lookup table. For example, when processing Kafka logs, the
kafka_request_api_keyfield contains numeric API key codes ("0","1","2", ...) instead of their human-readable names (produce,fetch,list_offsets, ...). Today this requires an if/else chain in thetransformprogram source, which doesn't scale to the ~70 entries of the Kafka API key table.Describe the solution you'd like
Add a
lookup(value, table, default:)function to thetransformstdlib. The table is an ordinary object value, so no new config parameter is needed:Behavior notes:
value— the value to translate; reading a missing field yieldsnulland passes through unchanged.table— an object mapping raw value → replacement value. Objects only.0and the string"0"address the same entry — JSON writes enum codes both ways.default:overrides that:lookup(.status, {"500": "crit"}, default: "ok").For this to be usable with large tables, a table literal must not be rebuilt on every event —
ObjectExpr.Evalcurrently allocates a fresh map per evaluation. Constant sub-expressions should be folded into a newConstExprAST node inside the existing validation walk (ValidateCalls), which already prepares nodes in place: it compiles regex literals and parses timestamp literals. No separate compiler pass.lookup(.x, {...})and thetable = {...}idiom are covered.Goal: the per-event cost of a lookup doesn't grow with the size of the table.
Describe alternatives you've considered
mapping:section in the plugin config plusmap(value, "name")(the shape originally proposed here) — rejected: it adds a config parameter serving exactly one function, and every future function needing startup-prepared data would want its own. Resolving a table by name at runtime also needs per-instance state, but the stdlib registry is a process-global of stateless functions, so it would mean threading a per-instance registry throughStart,ValidateCallsandNewContext. And since validating table names at startup requires a literal argument anyway, the runtime name lookup buys nothing. Finally, a per-actionmapping:block is no more reusable across actions than an object literal is.Additional context
Example use case: Kafka
kafka_request_api_keynumeric codes → human-readable API names (produce,fetch,offsets, etc.), to make dashboards and log search more readable without needing a lookup table on the query side.Name the function
lookup, notmap— in an expression languagemapreads as map/filter/reduce.Out of scope:
from_file(...)in the same argument position, if large tables in YAML become a real complaint;Constflag onParameterto reject a non-constant table at startup; a non-constant table should simply work and not be folded.Definition of done:
lookup(hits, miss behavior,default:, numeric/string key equivalence, argument validation);transform_test.go;lookupdocumented in the plugin's functions section, withREADME.mdregenerated viamake gen-doc.