Python type inference does not work for pybind11 wrapped C++

[chipkent]

I generated a set of examples that integrate C++ into Deephaven. pybind11 was used to integrate C++ directly into python.

The code can be found at https://github.com/chipkent/deephaven-example-use-cases/tree/main/examples/cpp/deephaven-example.

There seem to be problems in both of the mechanisms I tried.

1. In the first case below, a cast is needed to get the right column type. If the cast is not present, the UI says the column is a String.
2. In the second case below, there is an error. Value: no signature found for builtin <built-in method price of PyCapsule object at 0xfffec56ae760>

from deephaven import empty_table
import blackscholes

v = blackscholes.price(100, 95, 0.05, 1.2, 0.4, True, False) 
print(v)
print(type(v))

t = empty_table(10).update([
    "UnderlyingPrice = 100+i",
    "Strke = 95",
    "RiskFree = 0.05",
    "YearsToExpiry = 0.6",
    "Vol = 0.4",
    "IsCall = true",
    "IsStock = false",
    "BlackScholesPythonCpp = (double) blackscholes.price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
    "BlackScholesJavaCpp = io.deephaven.BlackScholes.price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
])


price = blackscholes.price

tx = empty_table(10).update([
    "UnderlyingPrice = 100+i",
    "Strke = 95",
    "RiskFree = 0.05",
    "YearsToExpiry = 0.6",
    "Vol = 0.4",
    "IsCall = true",
    "IsStock = false",
    "BlackScholesPythonCpp = price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
    "BlackScholesJavaCpp = io.deephaven.BlackScholes.price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
])

Error:

r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'deephaven.dherror.DHError'>
Value: table update operation failed. : Error in Python interpreter
Traceback (most recent call last):
  File "/build/venv/lib/python3.12/site-packages/deephaven/table.py", line 994, in update
    return Table(j_table=self.j_table.update(*formulas))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)
	at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:201)
	at io.deephaven.engine.table.impl.select.SwitchColumn.initDef(SwitchColumn.java:64)
	at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.createContext(SelectAndViewAnalyzer.java:128)
	at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$34(QueryTable.java:1527)
	at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
	at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$35(QueryTable.java:1509)
	at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3646)
	at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1508)
	at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1487)
	at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:98)
	at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
	at org.jpy.PyLib.executeCode(Native Method)
	at org.jpy.PyObject.executeCode(PyObject.java:138)
	at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
	at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
	at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
	at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
	at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:165)
	at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
	at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:165)
	at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
	at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
	at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:193)
	at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
	at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
	at java.base/java.lang.Thread.run(Thread.java:829)
caused by io.deephaven.engine.table.impl.lang.QueryLanguageParser$QueryLanguageParseException: 

Having trouble with the following expression:
Full expression           : price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)
Expression having trouble : 
Exception type            : java.lang.RuntimeException
Exception message         : Error in Python interpreter:
Type: <class 'ValueError'>
Value: no signature found for builtin <built-in method price of PyCapsule object at 0xfffec56ae760>
Line: 2361
Namespace: _signature_from_builtin
File: /usr/lib/python3.12/inspect.py
Traceback (most recent call last):
  File "/build/venv/lib/python3.12/site-packages/deephaven/_udf.py", line 529, in _udf_parser
  File "/build/venv/lib/python3.12/site-packages/deephaven/_udf.py", line 496, in _parse_signature
  File "/usr/lib/python3.12/inspect.py", line 3310, in signature
  File "/usr/lib/python3.12/inspect.py", line 3054, in from_callable
  File "/usr/lib/python3.12/inspect.py", line 2571, in _signature_from_callable
  File "/usr/lib/python3.12/inspect.py", line 2361, in _signature_from_builtin


	at org.jpy.PyLib.callAndReturnObject(Native Method)
	at org.jpy.PyObject.call(PyObject.java:449)
	at io.deephaven.engine.util.PyCallableWrapperJpyImpl.prepareSignature(PyCallableWrapperJpyImpl.java:195)
	at io.deephaven.engine.util.PyCallableWrapperJpyImpl.parseSignature(PyCallableWrapperJpyImpl.java:206)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.pyCallableReturnType(QueryLanguageParser.java:2580)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:2389)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:133)
	at com.github.javaparser.ast.expr.MethodCallExpr.accept(MethodCallExpr.java:116)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:319)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:212)
	at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:240)
	at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:122)
	at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:181)
	... 32 more


Line: 996
Namespace: update
File: /build/venv/lib/python3.12/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "<string>", line 24, in <module>
  File "/build/venv/lib/python3.12/site-packages/deephaven/table.py", line 996, in update

	at org.jpy.PyLib.executeCode(Native Method)
	at org.jpy.PyObject.executeCode(PyObject.java:138)
	at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
	at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
	at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
	at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
	at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:165)
	at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
	at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:165)
	at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
	at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
	at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:193)
	at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
	at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
	at java.base/java.lang.Thread.run(Thread.java:829)

[jmao-denver]

1. I have done some investigation and it turns out that to add signatures to functions in C extension modules isn't as one might expect. See the following.

python/cpython#123473
https://stackoverflow.com/questions/1104823/python-c-extension-method-signatures-for-documentation
https://stackoverflow.com/questions/25847035/what-are-signature-and-text-signature-used-for-in-python-3-4/25847066#25847066

2. I have also tried creating .pyi files but they don't help runtime inspection via the inspect module.
3. besides the use of __text_signature__, another alternative way is to wrap C extension modules (making them private) in pure Python code.
4. numba, numpy choose to store the proprietary encoded signature in the types attribute of the function object
5. DH UDF signature parser needs to properly handle the case where signature isn't available