Skip to content

get_array_type_columns doesn't support columns with type object #101

Open
@dvanbrug

Description

@dvanbrug

When interacting with a table with columns that contain objects, the get_array_type_columns method below does not return all the array type columns.

for col_name, value in source.items():
# If it's a list (ES Array add to cursor)
if isinstance(value, list):
if len(value) > 0:
# If it's an array of objects add all keys
if isinstance(value[0], dict):
for in_col_name in value[0]:
array_columns.append((f"{col_name}.{in_col_name}",))
array_columns.append((f"{col_name}.{in_col_name}.keyword",))
continue
array_columns.append((col_name,))
array_columns.append((f"{col_name}.keyword",))

For example, when using ECS, the event.category column is an array, but does not get identified by this method as it only looks at the event column and not the sub-columns.

Instead, this block should also analyze the object components and check those for being arrays as well.

Tested on ElasticSearch 8.6.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions