[Java] java.lang.IndexOutOfBoundsException when using VectorSchemaRootAppender.append

[fuhaiq]

Describe the bug, including details regarding any error messages, version, and platform.

I'm using VectorSchemaRootAppender.append append batch data into a final VectorSchemaRoot

VectorSchemaRoot result = null;
ScanOptions options = new ScanOptions(/*batchSize*/ 32768, projection);
try (
        var allocator = new RootAllocator();
        var datasetFactory = new FileSystemDatasetFactory(allocator,
                NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
        Dataset dataset = datasetFactory.finish();
        Scanner scanner = dataset.newScan(options);
        ArrowReader reader = scanner.scanBatches()
) {
    while (reader.loadNextBatch()) {
        try (VectorSchemaRoot root = reader.getVectorSchemaRoot()) {
            if(result == null) {
                result = VectorSchemaRoot.create(root.getSchema(), allocator);
            }
            VectorSchemaRootAppender.append(result, root);
        }
    }
    return result;
} catch (Exception e) {
    throw new RuntimeException(e);
}

But exception as bellowed

Caused by: java.lang.IndexOutOfBoundsException: index: 0, length: 4 (expected: range(0, 0))
	at org.apache.arrow.memory.ArrowBuf.checkIndexD(ArrowBuf.java:319)
	at org.apache.arrow.memory.ArrowBuf.chk(ArrowBuf.java:306)
	at org.apache.arrow.memory.ArrowBuf.getInt(ArrowBuf.java:425)
	at org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:111)
	at org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:46)
	at org.apache.arrow.vector.BaseVariableWidthVector.accept(BaseVariableWidthVector.java:1424)
	at org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:67)
	at org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:81)

Component(s)

Java

[lidavidm]

CC @davisusanibar

[fuhaiq]

allocateNew before append data

if(result == null) {
    result = VectorSchemaRoot.create(root.getSchema(), ExecutionContext.instance().allocator());
    result.allocateNew();
}
VectorSchemaRootAppender.append(result, root); 

[davisusanibar]

Thank you @fuhaiq to share that. Just added your solution as part of the Arrow Java Cookbooks - apache/arrow-cookbook#318

[muyihao]

when I allocateNew before append data, then the data of result before append was cleaned, what I want to do is to append two or even more batch.

public static void newMerge() { VectorSchemaRoot root = null; for (int i = 0; i < 100; i++) { VectorSchemaRoot vectorSchemaRoot = createVectorSchemaRoot(ArrowUtil.rootAllocator); if (root == null) { root = vectorSchemaRoot; } else { if (root.getRowCount() <= 10) { VectorSchemaRootAppender.append(root, vectorSchemaRoot); } else { System.out.println("merged:" + root.contentToTSVString()); root = vectorSchemaRoot; root.allocateNew(); } } vectorSchemaRoot.close(); } }
Writing this way will report an error: java.lang.IndexOutOfBoundsException

while write as bellow will result null data