Deadlock results from Batch.execute()

[Michael-A-McMahon]

Executing Oracle R2DBC's implementation of io.r2dbc.spi.Batch can result in a deadlock. The cause is rooted in Oracle JDBC's use of locks to ensure thread safe APIs.

A stack trace similar to the following is a symptom:

"ForkJoinPool.commonPool-worker-3" #16 daemon prio=5 os_prio=0 cpu=844.91ms elapsed=2204.18s tid=0x00007fb818004800 nid=0xbb1 waiting on condition  [0x00007fb80dfb8000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.10/Native Method)
	- parking to wait for  <0x00000000e13d1a38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.10/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.10/AbstractQueuedSynchronizer.java:2081)
	at oracle.jdbc.driver.RestrictedLock.awaitRestrictedModeExit(ojdbc11@21.1.0.0/RestrictedLock.java:208)
	at oracle.jdbc.driver.RestrictedLock.lock(ojdbc11@21.1.0.0/RestrictedLock.java:140)
	at oracle.jdbc.internal.Monitor.acquireLock(ojdbc11@21.1.0.0/Monitor.java:131)
	at oracle.jdbc.internal.Monitor.acquireCloseableLock(ojdbc11@21.1.0.0/Monitor.java:109)
	at oracle.jdbc.driver.InsensitiveScrollableResultSet$RowPublisher$ExpiringRow.getObject(ojdbc11@21.1.0.0/InsensitiveScrollableResultSet.java:1388)
	at oracle.r2dbc.impl.OracleReactiveJdbcAdapter$OracleJdbcRow.getObject(oracle.r2dbc@0.1.0/OracleReactiveJdbcAdapter.java:1185)
	at oracle.r2dbc.impl.OracleRowImpl.convertColumnValue(oracle.r2dbc@0.1.0/OracleRowImpl.java:251)
	at oracle.r2dbc.impl.OracleRowImpl.get(oracle.r2dbc@0.1.0/OracleRowImpl.java:137) 

The deadlock comes about as follows, where a Batch of two SQL commands is executed and Oracle JDBC's thread pool consists of 1 thread:

1. An invocation of OraclePreparedStatement.executeAsyncOracle() executes the 2nd SQL command in the batch. (The first command has already executed).
2. The executeAsyncOracle method acquires the Oracle JDBC Connection's lock and initiates a database call.
3. A callback is registered that will release the lock after the database response is received.
4. On the executor thread, an oracle.jdbc.OracleRow resulting from the 1st SQL command of the batch is processed by a mapping function.
5. The function invokes OracleRow.getObject(int, Class).
6. The implementation of getObject attempts to acquire the Oracle JDBC Connection's lock, but the lock-releasing callback described in step 3 has not executed yet.
7. The executor thread is blocked until the lock is released.
8. The database response is received and the lock-releasing callback described in step 1 is submitted to the executor.
9. The lock-releasing callback can not progress until the executor thread becomes available.
10. The executor thread can not become available until the getObject-calling task can acquire the lock.
11. DEADLOCK: The locker-releaser needs the getObject-caller's thread to progress. The getObject-caller needs the locker-releaser's lock to progress.

Note that this issue exists regardless of the thread pool size. If enough Batch results are being processed concurrently, there is potential to exhaust the thread pool as more threads become blocked waiting for the connection lock.

Note that the general limitation of blocked threads arising from concurrent calls on one connection is known. The Oracle R2DBC documentation advises programmers against doing this. Resolving this limitation is outside the scope of this issue.

This issue is filed because Oracle R2DBC's Batch implementation is not respecting it's own limitations. The implementation is executing a SQL command while application code is processing the result of another command concurrently.