Skip to content

[Java][Gandiva] Synchronize some methods on Projector. #601

@lriggs

Description

@lriggs

Describe the enhancement requested

Our product, which uses Gandiva, was encountering errors from LLVM during intense query processing stress tests. I traced the issue back to the JIT upgrade (apache/arrow#39098), but it seems it only started occurring then because the new JIT engine requires unique symbol names unlike the old engine.

The error from llvm is: Duplicate definition of symbol 'expr_0_0'.

After updating the llvm version to 18 (apache/arrow#45114) you can see this error occasionally when running the Gandiva ProjectorTest.

Failed to make LLVM module due to Failed to add IR module to LLJIT: Duplicate definition of symbol 'expr_0_0'

	at org.apache.arrow.gandiva.evaluator.JniWrapper.buildProjector(Native Method)
	at org.apache.arrow.gandiva.evaluator.Projector.make(Projector.java:206)
	at org.apache.arrow.gandiva.evaluator.Projector.make(Projector.java:70)
	at org.apache.arrow.gandiva.evaluator.ProjectorTest.lambda$testMakeProjectorParallel$1(ProjectorTest.java:138)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
[WARNING] Tests run: 44, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.656 s -- in org.apache.arrow.gandiva.evaluator.ProjectorTest

Adding synchronization the the make and evaluate methods in gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java will fix the issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions