Blob Encoding not working with Lance Java SDK

[rahil-c]

Issue Summary

Hi all, recently I have been working on adding lance file format support within Apache Hudi: apache/hudi#14127 and wanted to raise a potential issue(hopefully my understanding is correct of this feature)

During the integration, I wanted to test out storing binary content and leverage the blob encoding feature that Lance mentions here: https://lancedb.github.io/lance/guide/blob/?h=blob, to avoid materializing these blobs on each read and just get back a struct of the position and size.

However during my testing with the LanceFileReader via the Java sdk I would find that the blob contents are always materialized despite passing the property "lance-encoding:blob": "true during the intial write using the LanceFileWriter.

To isolate the issue to only lance, I have the following unit test which is just using the LanceFileWriter and LanceFileReader to show the reproduction. Should be easy to copy into an IDE and run and i have attached results as well.

Reproduction

package org.apache.hudi.io.storage;

import com.lancedb.lance.file.LanceFileReader;
import com.lancedb.lance.file.LanceFileWriter;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.LargeVarBinaryVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;
import org.apache.arrow.vector.types.pojo.Schema;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;

import java.nio.file.Path;
import java.util.*;

import static org.junit.jupiter.api.Assertions.*;

public class BlobEncodingTest {

    @Test
    public void testBlobEncodingReturnsDescriptors(@TempDir Path tempDir) throws Exception {
        String filePath = tempDir.resolve("test_blob.lance").toString();
        BufferAllocator allocator = new RootAllocator();

        // Step 1: Write blob-encoded data
        Map<String, String> blobMetadata = new HashMap<>();
        blobMetadata.put("lance-encoding:blob", "true");

        Field blobField = new Field(
                "blob_data",
                new FieldType(true, ArrowType.LargeBinary.INSTANCE, null, blobMetadata),
                Collections.emptyList()
        );

        Schema schema = new Schema(Collections.singletonList(blobField), null);

        try (LanceFileWriter writer = LanceFileWriter.open(
                filePath,
                allocator,
                null,
                Collections.emptyMap()
        )) {
            try (VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) {
                root.allocateNew();

                LargeVarBinaryVector blobVector =
                        (LargeVarBinaryVector) root.getVector("blob_data");

                // Write 5 blobs
                for (int i = 0; i < 5; i++) {
                    byte[] data = new byte[100 * (i + 1)]; // Different sizes
                    Arrays.fill(data, (byte) i);
                    blobVector.setSafe(i, data);
                }

                root.setRowCount(5);
                writer.write(root);
            }
        }

        // Step 2: Read back and verify
        try (LanceFileReader reader = LanceFileReader.open(filePath, allocator)) {

            // Check schema
            Schema readSchema = reader.schema();
            Field readField = readSchema.getFields().get(0);

            System.out.println("=== SCHEMA VERIFICATION ===");
            System.out.println("Field name: " + readField.getName());
            System.out.println("Field type: " + readField.getType());
            System.out.println("Field metadata: " + readField.getMetadata());

            // Check if blob metadata is preserved
            assertTrue(readField.getMetadata().containsKey("lance-encoding:blob"),
                    "Blob metadata should be preserved in schema");

            // Read batch
            try (ArrowReader batch = reader.readAll(null, null, 10)) {
                batch.loadNextBatch();  // Actually load the data
                VectorSchemaRoot root = batch.getVectorSchemaRoot();

                System.out.println("\n=== READ BATCH VERIFICATION ===");
                System.out.println("Batch schema: " + root.getSchema());
                System.out.println("Row count: " + root.getRowCount());

                // Get the blob column
                org.apache.arrow.vector.FieldVector column = root.getVector("blob_data");
                System.out.println("Column type: " + column.getField().getType());

                // Check if it's a struct with position and size (means the blob encoding happened)
                if (column.getField().getType() instanceof ArrowType.Struct) {
                    System.out.println("SUCCESS: Returned DESCRIPTORS (struct type)");
                    System.out.println("Struct has fields: " + column.getField().getChildren());

                    // The struct should have 'position' and 'size' fields
                    assertEquals(2, column.getField().getChildren().size(),
                            "Struct should have 2 fields (position and size)");

                } else if (column.getField().getType() instanceof ArrowType.LargeBinary) {
                    System.out.println("ISSUE: Returned MATERIALIZED BYTES (binary type)");
                    System.out.println("This means blobs were fetched from external buffers");

                    // This is what currently happens - Java materializes
                    LargeVarBinaryVector binaryVector = (LargeVarBinaryVector) column;

                    System.out.println("\n=== ACTUAL DATA (Materialized) ===");
                    for (int i = 0; i < Math.min(5, root.getRowCount()); i++) {
                        byte[] data = binaryVector.get(i);
                        System.out.println("Row " + i + ": " + data.length + " bytes");
                    }

                    // Fail the test to demonstrate the issue
                    fail("Java LanceFileReader materializes blobs instead of returning descriptors. " +
                            "Expected struct<position: uint64, size: uint64> but got " +
                            column.getField().getType());

                } else {
                    fail("Unexpected type: " + column.getField().getType());
                }
            }
        }
        allocator.close();
    }
}

Results of Repro

Questions

• Is blob encoding a feature that is only supported at the lance table format level, or should ideally it just work with lance file format?
• If the file format supports this blob encoding feature then does this issue happen with the rust or python sdk?

[jackye1995]

The LanceFileReader and LanceFileWriter is pretty barebone at this moment, it was only developed for the Iceberg integration POC.

For blob encoding specifically, I agree it is valuable to do it in file level, although blob v2 will be moving more features to the table level.

[rahil-c]

Thanks @jackye1995 for the response. Do you happen to know the plan for LanceFileReader and LanceFileWriter java related apis to catch up to parity with the Rust implementation?

It would be ideal for other table format java clients such as Hudi, Iceberg, Paimon to also be able to leverage this blob encoding or other core features within the lance project.

[jackye1995]

Do you happen to know the plan for LanceFileReader and LanceFileWriter java related apis to catch up to parity with the Rust implementation?

We don't have any active plans from LanceDB side. In Trino, there is a PR open for lance-trino connector that implements a java-native reader against the file format spec and is much faster, but it is implemented against Trino data models. In general most of these efforts are community driven for using the file reader and writer directly. So far even in python, file-only use case is only in benchmarking, so integrating with other table format will probably be the main driving force for improving this area.

[rahil-c]

@jackye1995 @westonpace @eddyxu I have made the following patch here, which allowed me to see the blob encoding returned in the repro. I am not super familiar with lance file format and rust yet so feel free to correct it if this is not the ideal fix.
#5193

[the-other-tim-brown]

Is there any updates on adding support for reading the position and length through the java file reader? I took a look at the Trino-Lance plugin and it looks like it is leveraging the FragmentScanner. I was hoping there would be a way to get this to work with just a file reader instead of requiring the full Lance table format.

[westonpace]

Apologies if I dropped the ball here. I remember discussing this PR with @rahil-c in person last year and see it didn't get merged. Sorry, if that was on my end.

The challenge here is that some columns can be returned in multiple formats. The file reader needs the caller to tell it not just which columns (or field ids) to read but also tell it what the desired output encoding is for that field. Blob columns are the only column we really use this mechanism for at the moment (I have other ideas in mind in the future) and so the capability is generic.

This is why the ReaderProjection is both a set of column indices (which columns to read) and a schema (what output types and names to generate).

The Java API only takes a list of names (projected_names). So what the method does today is use ReaderProjection::from_column_names. This method assumes you want the default output type for each column and for blobs that is binary.

The PR from Rahil switched to using ReaderProjection::from_field_ids which allows the caller to specify both field ids and a schema. The way the PR creates the schema is a little roundabout but the resulting schema will always have blob columns set to the descriptor output type. As a result, it reads the descriptors and not the blobs. These descriptors are the position/size/etc. fields which are used to create a file reader.

My initial concern remains. This is just changing the default (from binary to descriptors). Not only is it a breaking change but it doesn't allow users to get the old behavior if they want it.

I think a more complete fix would be to give the Java file reader a new method (or modify the existing) which allows for the output type to be specified in addition to names or field ids.

[westonpace]

All this being said, if there is consensus to change the default output type in Java from binary format to descriptor format then I have no complaints moving forward with the proposed PR (looks like it needs an update).