[pkg/ottl] Allow passing Converters as literal parameters

[TylerHelmuth]

Component(s)

pkg/ottl

Is your feature request related to a problem? Please describe.

Currently OTTL provides no way to use any defined Converter within another Editor/Converter. Although Converters can be passed as a parameter, they are always executed and the result is what is actually passed as the parameter. This limits the capabilities of existing functions (#22787).

Describe the solution you'd like

W should allow OTTL to pass Converters themselves as a parameter so they can be executed within the function.

Describe alternatives you've considered

Adding more, unique functions with similar signatures. This is less desirable.

Additional context

No response

[rnishtala-sumo]

@TylerHelmuth I wanted to confirm that this we need to support passing an optional parameter, as mentioned here: #20879 before working on this one?

We probably don't want to add another mandatory parameter to the replace_ functions that takes in a converter as a literal?

[TylerHelmuth]

This issue can be worked independently of optional parameters

[rnishtala-sumo]

ok thanks!, this is what I have in mind for this task

• Pass the function name as a literal string
• Maintain a function that returns a function with the given name

func funcWithName(name string) func() {
    switch name {
    case "sha256":
        return sha256
    case "sha1":
        return sha1
    default:
        return nil
    }
} 

• Pass the substring that matches the regex to the function

I think the replace_ function would then look like below

replace_pattern(attributes["log"], "^device=([0-9A-Za-z]+)", "sha256($$1)")

[TylerHelmuth]

@rnishtala-sumo if we wanted to use a string identifier for the function then replace_pattern could do a switch statement inside its own function and reference the hash functions directly.

I am suggesting we go a step further and allow ottl function to accept a ConverterGetter type struct. It would look something like:

type FunctionGetter[K any] interface {
	Get(ctx context.Context, tCtx K) (ExprFunc[K], error)
}

type StandardFunctionGetter[K any] struct {
        Function ExprFunc[K]
}

func (g StandardFunctionGetter[K]) Get(ctx context.Context, tCtx K) (string, error) {
	return g.Function
}

replace_pattern could then invoke FunctionGetter.Get and be returned an ExprFunc[K] that it could then invoke.

I haven't dug deeper into this yet, and I'm sure there are some edges I haven't thought of, but I think it would be good to try to follow the existing Getter pattern and let the ottl function implement call a Getter to retrieve the passed in function reference and then invoke it.

[TylerHelmuth]

The grammar will need changed as well to allow

replace_pattern(attributes["log"], "^device=([0-9A-Za-z]+)", "$$1", SHA256)

Today the grammar would interpret SHA256 as an enum I think - that will be a hurdle that requires some creativity. We can try to handle it via the grammar by differentiating between Converters and Enums or handle it in the parser by trying to handle the token as an Enum and then, if Enum parsing fails, handling as Function.

[TylerHelmuth]

Invoking the function may also require some reflection - if that is the case hopefully it can be setup during parsing so that runtime execution doesn't take a performance hit. We may also have to "type" functions to limit their parameters like a SingleParamFunctionGetter or something.

You've stumbled into a very complex use case for the OTTL but I still think it is better than passing in string identifiers and doing a switch statement - if we get this right it will significantly increase the power and flexibility of the package.

[rnishtala-sumo]

ok I think I understand what you're suggesting, I'll spend some time on this and circle back

One question though, wouldn't this be a breaking change unless the last param is optional?

replace_pattern(attributes["log"], "^device=([0-9A-Za-z]+)", "$$1", SHA256)

The replace_pattern func currently takes 3 params

replace_pattern(target, regex, replacement)

[TylerHelmuth]

Correct, we can't implement it into the replace functions yet. This issue is only to add the capability to OTTL

[rnishtala-sumo]

I finally got a chance to look at this again. Question - why can't we use the generic Getter type for this. I tested the following on my branch by adding a fourth argument of Getter type.

Arguments

Target       ottl.GetSetter[K]    `ottlarg:"0"`
RegexPattern string               `ottlarg:"1"`
Replacement  ottl.StringGetter[K] `ottlarg:"2"`
Function     ottl.Getter[K]       `ottlarg:"3"`

replace_pattern(attributes["message"], "device=*", "test", SHA256(attributes["device_name"]))

The Getter type allows us to pass in a Converter as a parameter.

[rnishtala-sumo]

oh nvm, lost track of the original premise, which was - they are always executed and the result is what is actually passed as the parameter. So we want to pass the converter as a literal and evaluate inside the function.

[rnishtala-sumo]

Please let me know your thoughts on this draft PR
#24356

cc: @TylerHelmuth @astencel-sumo

[rnishtala-sumo]

@TylerHelmuth could this issue be assigned to me?

[TylerHelmuth]

Closed via #24356