feat: merge-insert jni support by kaori-seasons · Pull Request #4172 · lance-format/lance

kaori-seasons · 2025-07-08T04:08:38Z

In order to support the Merge insert API of flink cdc streaming writing, it is necessary to support the relevant Java JNI.

Related to issue-4050

MergeInsertBuilder

The MergeInsertBuilder class provides a fluent API for building merge insert operations, which allow you to merge new data with existing data in a Lance dataset.

Usage Example

import com.lancedb.lance.MergeInsertBuilder;
import com.lancedb.lance.MergeInsertResult;
import org.apache.arrow.vector.VectorSchemaRoot;

// Create a merge insert builder
MergeInsertBuilder builder = MergeInsertBuilder.create(dataset, "id");

// Configure the operation
MergeInsertBuilder configured = builder
    .whenMatchedUpdateAll("source.value > target.value")
    .whenNotMatchedInsertAll()
    .whenNotMatchedBySourceDelete("target.id > 100")
    .conflictRetries(5)
    .retryTimeout(1000L);

// Execute the merge insert
try (VectorSchemaRoot newData = createNewData()) {
    MergeInsertResult result = configured.execute(newData);

    System.out.println("Inserted: " + result.getNumInsertedRows());
    System.out.println("Updated: " + result.getNumUpdatedRows());
    System.out.println("Deleted: " + result.getNumDeletedRows());
}

Performance Considerations

Native memory is managed automatically
Arrow streams are handled efficiently through FFI
Async operations are executed on a dedicated runtime
Resource cleanup is guaranteed through AutoCloseable

github-actions · 2025-07-08T04:08:54Z

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

README.md

java/core/lance-jni/src/lib.rs

java/core/lance-jni/src/tests.rs

majin1102 · 2025-07-08T07:04:57Z

Thanks for this contribution

The code has imported many unnecessary changes(maybe some AI framework?) and the comments are Chinese.
Please check it.

majin1102 · 2025-07-08T06:32:01Z

java/core/src/main/java/com/lancedb/lance/Dataset.java

   * @param schema dataset schema
   * @param params write params
-   * @return Dataset
+   * @return Datase


How is this change imported?

majin1102 · 2025-07-08T06:32:45Z

java/core/src/main/java/com/lancedb/lance/Dataset.java

+  }
+
+  /**
+   * 创建 merge insert 操作构建器（单列版本）


Use English for comments?

majin1102 · 2025-07-08T06:33:48Z

java/core/src/main/java/com/lancedb/lance/Fragment.java

   * @param datasetUri the dataset uri
   * @param allocator the buffer allocator
-   * @param root the vector schema root
+   * @param root the vector schema roo


How did this change happen?

kaori-seasons · 2025-07-08T08:09:24Z

@majin1102 Thanks for your review. This is because I wrote an MCP server myself. It connects DeepWiki and Claude. I will try to fix it this weekend.

jackye1995 · 2025-07-08T16:27:09Z

java/core/lance-jni/README.md

+
+The `MergeInsertBuilder` class provides a fluent API for building merge insert operations, which allow you to merge new data with existing data in a Lance dataset. This is similar to SQL's MERGE statement.
+
+#### Basic Usage


documentations should in general go to the website not in a separated README

jackye1995 · 2025-07-08T16:28:08Z

java/core/lance-jni/README.md

+}
+```
+
+#### API Reference


this looks very AI lol, I think we should just keep the basic usage.

jackye1995 · 2025-07-08T16:29:41Z

java/core/lance-jni/src/lib.rs

 }
+
+#[no_mangle]
+pub extern "system" fn Java_com_lancedb_lance_MergeInsertBuilder_createNativeBuilder(


I think it is better to have these builder and boilerplate classes in Java, and when building the actual job we can pass it to rust, so we don't need to do this back and forth conversion.

jackye1995 · 2025-07-08T16:30:06Z

java/core/lance-jni/src/lib.rs

+) -> jobject {
+    let builder = unsafe { Box::from_raw(builder_handle as *mut LanceMergeInsertBuilder) };
+
+    // 从内存地址创建 ArrowArrayStream


nit: please remove Chinese comments

jackye1995 · 2025-07-08T16:32:44Z

java/core/lance-jni/src/tests.rs

+// limitations under the License.
+
+#[cfg(test)]
+mod tests {


tests should go under the same file of relevant modules, see other rust tests for reference.

jackye1995 · 2025-07-08T16:33:01Z

java/core/lance-jni/src/tests.rs

@@ -0,0 +1,791 @@
+// Copyright 2024 Lance Developers.


nit: wrong license header

jackye1995 · 2025-07-08T17:30:45Z

Thank you for working on this! Added some basic comments, and looks like there are some unintentional changes introduced in each file that need to be removed. Let me know when this is ready for another review!

jackye1995 · 2025-07-08T17:34:56Z

java/core/lance-jni/src/lib.rs

+}
+
+#[no_mangle]
+pub extern "system" fn Java_com_lancedb_lance_MergeInsertBuilder_executeNative(


we should try to consolidate this with what is already there in BlockingDataset and make merge_insert another method of it.

# Conflicts: # java/core/lance-jni/src/blocking_dataset.rs # java/core/src/main/java/com/lancedb/lance/Dataset.java # java/core/src/main/java/com/lancedb/lance/ReadOptions.java

majin1102

Thanks for following up. Give some suggestions.

PTAL

majin1102 · 2025-07-31T16:48:55Z

java/core/src/main/java/com/lancedb/lance/Dataset.java

  }

+  /**
+   * 执行 merge insert 操作（使用 BlockingDataset）


English please

majin1102 · 2025-07-31T16:55:31Z

java/core/src/main/java/com/lancedb/lance/MergeInsertBuilder.java

+    private final BufferAllocator allocator;
+
+    // Configuration options
+    private String whenMatchedConfig = "do_nothing";


Is just null a little better?

majin1102 · 2025-07-31T16:57:34Z

java/core/src/main/java/com/lancedb/lance/MergeInsertBuilder.java

+    private long timeoutMillis = 0;
+
+    // Native method declarations
+    private static native Object executeWithConfigNative(


This datasetHandle a little weird outside of Dataset. I think just passing the dataset works as well. What do you think

Why return Object? I think just return the target class works

majin1102 · 2025-07-31T17:03:27Z

java/core/src/main/java/com/lancedb/lance/MergeInsertBuilder.java

+    );
+
+    static {
+        System.loadLibrary("lance_jni");


I suggest put the static block to the top

majin1102 · 2025-07-31T17:04:39Z

java/core/src/main/java/com/lancedb/lance/MergeInsertBuilder.java

+
+    @Override
+    public void close() {
+        // No cleanup needed for the new design


If not necessary, why not delete it and remove the interface

jackye1995 · 2025-09-12T16:25:10Z

Another contributor has implemented this feature in #4685

kaori-seasons added 11 commits July 8, 2025 10:50

commit

2470a0d

add merge insert java iml && tests

0b03bc6

add merge insert

7368b8f

add cdc arrow converter

850800c

add licenese

2063e75

fix checkstyle && fix license

fb31ff6

fix checkstyle

027a8f9

add merge insert

fe8994e

fix compiler error && fix converttoschema function

373d68b

fix checkstyle

561acfd

fix arrow convert error

b1d4124

github-actions bot added the java label Jul 8, 2025

jackye1995 self-requested a review July 8, 2025 04:20

Xuanwo reviewed Jul 8, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

java/core/lance-jni/src/lib.rs Outdated Show resolved Hide resolved

java/core/lance-jni/src/tests.rs Outdated Show resolved Hide resolved

kaori-seasons added 4 commits July 8, 2025 13:38

remove useless test

372e01a

remove chinese comment

b2e0063

update readme.md

2ba6bf8

format file

647a6ef

kaori-seasons changed the title ~~【ISSUE#4050】Merge-Insert JNI Support~~ feat: Merge-Insert JNI Support Jul 8, 2025

github-actions bot added the enhancement New feature or request label Jul 8, 2025

remove uesless content

d3be04f

kaori-seasons changed the title ~~feat: Merge-Insert JNI Support~~ feat: merge-insert jni support Jul 8, 2025

majin1102 requested changes Jul 8, 2025

View reviewed changes

jackye1995 reviewed Jul 8, 2025

View reviewed changes

java/core/lance-jni/README.md

}

```

#### API Reference

Copy link

Contributor

jackye1995 Jul 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks very AI lol, I think we should just keep the basic usage.

jackye1995 reviewed Jul 8, 2025

View reviewed changes

java/core/lance-jni/src/tests.rs

@@ -0,0 +1,791 @@

// Copyright 2024 Lance Developers.

Copy link

Contributor

jackye1995 Jul 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: wrong license header

jackye1995 reviewed Jul 8, 2025

View reviewed changes

majin1102 mentioned this pull request Jul 2, 2025

Epic: Improving integration with Java ecosystem #3950

Closed

undertaker86001 added 3 commits July 31, 2025 19:44

refactor code

4bb24a2

fix conflict

0b4f406

Merge remote-tracking branch 'upstream/main' into issue-4050

1009d3c

# Conflicts: # java/core/lance-jni/src/blocking_dataset.rs # java/core/src/main/java/com/lancedb/lance/Dataset.java # java/core/src/main/java/com/lancedb/lance/ReadOptions.java

majin1102 requested changes Jul 31, 2025

View reviewed changes

jackye1995 closed this Sep 12, 2025


		The `MergeInsertBuilder` class provides a fluent API for building merge insert operations, which allow you to merge new data with existing data in a Lance dataset. This is similar to SQL's MERGE statement.

		#### Basic Usage

Conversation

kaori-seasons commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MergeInsertBuilder

Usage Example

Performance Considerations

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

majin1102 commented Jul 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaori-seasons commented Jul 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995 commented Jul 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

majin1102 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

majin1102 Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995 commented Sep 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kaori-seasons commented Jul 8, 2025 •

edited

Loading

majin1102 Jul 31, 2025 •

edited

Loading