Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support conditional expression to evaluate based on the data type for a given field #4478

Closed
Tracked by #4460
dlvenable opened this issue Apr 30, 2024 · 2 comments
Closed
Tracked by #4460
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@dlvenable
Copy link
Member

dlvenable commented Apr 30, 2024

Background

Some processors we only want to run when the field has a certain type.

Proposal

I propose creating a new operator within the Data Prepper expression language: typeof.

This could be used to determine if a given field has a certain type.

For example:

/myfield typeof string

Additionally, this approach could support inheritance.

For example, the following could be true:

25 typeof integer
25 typeof number
25.5 typeof double
25.5 typeof number

Types

Data Prepper doesn't have have a consistent concept of types. However, we should have some core types. We can base these on what we already have for the convert entries processor.

INTEGER("integer", new IntegerConverter()),
STRING("string", new StringConverter()),
DOUBLE("double", new DoubleConverter()),
BOOLEAN("boolean", new BooleanConverter()),
LONG("long", new LongConverter());

Alternative

One alternative is to add a new function.

$type(/myfield) == 'string'

However, this would not work so well with inheritance.

@dlvenable dlvenable added enhancement New feature or request and removed untriaged labels Apr 30, 2024
dlvenable added a commit to dlvenable/data-prepper that referenced this issue May 2, 2024
…itially created to help connect the convert_entry_type processor with upcoming work for evaluating type information as part of opensearch-project#4478.

Signed-off-by: David Venable <dlv@amazon.com>
@kkondaka
Copy link
Collaborator

kkondaka commented May 2, 2024

Another option is to provide a function in expressions like isType(/field, "string") or isTypeOf(/field, "string").

dlvenable added a commit that referenced this issue May 3, 2024
Adds an enum to represent core data types in Data Prepper. This is initially created to help connect the convert_entry_type processor with upcoming work for evaluating type information as part of #4478.

Signed-off-by: David Venable <dlv@amazon.com>
@dlvenable dlvenable added this to the v2.8 milestone May 14, 2024
@dlvenable
Copy link
Member Author

Completed by: #4500

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

2 participants