[yugabyte#11266][YSQL] Use session flag to set non transactional writ…

…es for index inserts

Summary:
Today, only index backfill uses the non transactional write for index inserts.
In order to allow non transactional writes for regular writes to index tables, we will use the `yb_disable_transactional_writes` session flag which was initially created to disable transactional writes on regular tables for inserts to the main table: https://phabricator.dev.yugabyte.com/D13703

Note using this session flag could lead to state inconsistency as follows:
Prior to using this session variable, a distributed transaction write lock was used to write to both main table and index table.
But by disabling this distributed transaction write, the writes to the main table and index become independent writes as they are written directly to RocksDB.
So when there’s a connection abort or crash in the middle of the write, there could be a state inconsistency between the main table and index table, like some rows may appear in one but not the other. User would have to truncate and restart in such scenarios.
Despite such pitfall, we are extending this flag onto index tables to speed up the bulk loading of data such as when using the COPY command.

Also, note that this change explicitly covers the write use case for the insert writes like COPY command and no other write use cases like delete or update.

Test Plan:
On a test of 1 GB file, when the `yb_disable_transactional_writes` flag was set to false, the COPY command on indexed table took around 8 hours.
On the same condition, when the `yb_disable_transactional_writes` flag was set to true, the COPY took around 4 hours.
When the difference was calculated in seconds, there was 91% growth in speed when the session flag was set to true.
This performance improvement aligns with the latency improvement seen on regular tables tested in the `D13703` diff.

Modified existing unit test.

Reviewers: jason, smishra

Reviewed By: jason, smishra

Subscribers: yql

Differential Revision: https://phabricator.dev.yugabyte.com/D15159

java/yb-pgsql/src/test/java/org/yb/pgsql/TestBatchCopyFrom.java

@@ -500,6 +500,7 @@ public void testNonTxnWriteSessionVariable() throws Exception {
       statement.execute("SET " + DISABLE_TXN_WRITES_SESSION_VARIABLE_NAME + "=true");
       statement.execute(String.format(
           "CREATE TABLE %s (a Integer, b serial, c varchar, d int)", tableName));
+      statement.execute(String.format("CREATE INDEX ON %s (d)", tableName));
       statement.execute(String.format(
           "COPY %s FROM \'%s\' WITH (FORMAT CSV, HEADER)", tableName, absFilePath));
 

src/postgres/src/backend/executor/ybcModifyTable.c

@@ -477,9 +477,11 @@ void YBCExecuteInsertIndexForDb(Oid dboid,
 	 * `non_distributed_txn` when closing issue #4906.
 	 */
 	const bool is_backfill = (backfill_write_time != NULL);
+	const bool is_non_distributed_txn_write =
+		is_backfill || (!IsSystemRelation(index) && yb_disable_transactional_writes);
 	HandleYBStatus(YBCPgNewInsert(dboid,
 								  relid,
-								  is_backfill /* is_single_row_txn */,
+								  is_non_distributed_txn_write,
 								  &insert_stmt));
 
 	callback(insert_stmt, indexstate, index, values, isnull,