Skip to content

Feature: transform plugin — translate field values via a configured mapping #999

Description

@timggggggg

Is your feature request related to a problem? Please describe.
There's currently no built-in way to translate a field's raw value into a more human-readable one using a static lookup table. For example, when processing Kafka logs, the kafka_request_api_key field contains numeric API key codes ("0", "1", "2", ...) instead of their human-readable names (produce, fetch, list_offsets, ...). Today this requires an if/else chain in the transform program source, which doesn't scale to the ~70 entries of the Kafka API key table.

Describe the solution you'd like
Add a lookup(value, table, default:) function to the transform stdlib. The table is an ordinary object value, so no new config parameter is needed:

actions:
  - type: transform
    source: |
      api_key = {"0": "produce", "1": "fetch", "2": "list_offsets", "3": "metadata"}
      .kafka_request_api_key = lookup(.kafka_request_api_key, api_key)

Behavior notes:

  • value — the value to translate; reading a missing field yields null and passes through unchanged.
  • table — an object mapping raw value → replacement value. Objects only.
  • Keys are matched by their canonical string form, so the number 0 and the string "0" address the same entry — JSON writes enum codes both ways.
  • A value that isn't a key of the table is returned unchanged, rather than erroring or emptying the field. default: overrides that: lookup(.status, {"500": "crit"}, default: "ok").
  • Composite values (arrays, objects) are rejected as keys.

For this to be usable with large tables, a table literal must not be rebuilt on every event — ObjectExpr.Eval currently allocates a fresh map per evaluation. Constant sub-expressions should be folded into a new ConstExpr AST node inside the existing validation walk (ValidateCalls), which already prepares nodes in place: it compiles regex literals and parses timestamp literals. No separate compiler pass.

  • Fold in two positions: function call arguments and assignment right-hand sides, so both lookup(.x, {...}) and the table = {...} idiom are covered.
  • An expression that fails to evaluate must never be folded — it stays in the AST and keeps failing per event, so folding can't turn a runtime error into a startup failure.
  • Folded values are shared across events and processor goroutines. That's safe only because values are immutable (index and member assignment copy the container), and it must be covered by a test.

Goal: the per-event cost of a lookup doesn't grow with the size of the table.

Describe alternatives you've considered

  • A mapping: section in the plugin config plus map(value, "name") (the shape originally proposed here) — rejected: it adds a config parameter serving exactly one function, and every future function needing startup-prepared data would want its own. Resolving a table by name at runtime also needs per-instance state, but the stdlib registry is a process-global of stateless functions, so it would mean threading a per-instance registry through Start, ValidateCalls and NewContext. And since validating table names at startup requires a literal argument anyway, the runtime name lookup buys nothing. Finally, a per-action mapping: block is no more reusable across actions than an object literal is.
  • Inline if/else chains in the program source — doesn't scale and is hard to maintain for enums with many values.
  • An object literal without constant folding — correct, but rebuilds the table on every event.
  • Loading the table from an external file — out of scope for now; see below.
  • Doing the mapping upstream — not always possible when the source of the field (e.g. a Kafka producer) can't be changed.

Additional context
Example use case: Kafka kafka_request_api_key numeric codes → human-readable API names (produce, fetch, offsets, etc.), to make dashboards and log search more readable without needing a lookup table on the query side.

Name the function lookup, not map — in an expression language map reads as map/filter/reduce.

Out of scope:

  • loading tables from an external file — additive later as a foldable from_file(...) in the same argument position, if large tables in YAML become a real complaint;
  • a Const flag on Parameter to reject a non-constant table at startup; a non-constant table should simply work and not be folded.

Definition of done:

  • unit tests for lookup (hits, miss behavior, default:, numeric/string key equivalence, argument validation);
  • unit tests for folding, including the don't-fold-on-error rule and the shared-value immutability guarantee;
  • an end-to-end test in transform_test.go;
  • lookup documented in the plugin's functions section, with README.md regenerated via make gen-doc.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions