Skip to content

Allow using 1GB+ data buffers in variable width vectors #721

@gszadovszky

Description

@gszadovszky

Describe the enhancement requested

In BaseVariableWidthVector the reallocation logic of the data buffer is to always use the "next power of 2" size instead of the actual desired size (see here). The reallocation fails if the new size exceeds the MAX_BUFFER_SIZE which is Integer.MAX_VALUE by default.
The issue is Integer.MAX_VALUE is just 1 byte less than a power of 2 (231 - 1) which means that exceeding the desired capacity of 1073741824 (1GB), the reallocation would fail. It means, we are unable to get a data buffer larger than 1GB while the default maximum size is almost 2GB.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions