[DocDB][YCQL][Indexes] Drop YCQL Index times out when create and drop index are being executed continuously while loading the data

[shishir2001-yb]

Jira Link: DB-9003

Description

Tried on version: 2.21.0.0-b227

Note: This issue started occurring from a build between 2.21.0.0-b216 and 2.21.0.0-b227.

While executing a heavy Workload on a table if we create and drop indexes frequently then it may cause timeout issue for drop index.

Also, the following was seen on Master error log:
Note: Added the whole error in the comment

E1117 04:24:12.662847 80635 backfill_index.cc:986] Backfill Index Table(s) { test_indexes_034c74byvaluev4_idx4 } failed to backfill the index: [e718ede41a574e94bb855c1e0a321325] due to Invalid argument (yb/tserver/tablet_service.cc:735): Tablet has a different schema 595 vs 591. Requested index is not ready to backfill. IndexMap: 0x000035913cdb1e58 -> [{955035f2fa10421c9d9d379303bd95c7, table_id: "955035f2fa10421c9d9d379303bd95c7" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 2 column_name: "C$_v2" colexpr { column_id: 2 } } . . . . columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1461853}]

Test Details:

1. Start a workload which will load data into the YCQL table
2. Execute the following in a loop
    - Drop a random index
    - Create an index on a random column  of the table       

Find all the logs(Attachments -> http://stress.dev.yugabyte.com/stress_test/a6b1a8b9-0e5d-4e3f-b49d-bcada682ca87)

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[shishir2001-yb]

E1117 04:24:12.662847 80635 backfill_index.cc:986] Backfill Index Table(s) { test_indexes_034c74byvaluev4_idx4 } failed to backfill the index: [e718ede41a574e94bb855c1e0a321325] due to Invalid argument (yb/tserver/tablet_service.cc:735): Tablet has a different schema 595 vs 591. Requested index is not ready to backfill. IndexMap: 0x000035913cdb1e58 -> [{955035f2fa10421c9d9d379303bd95c7, table_id: "955035f2fa10421c9d9d379303bd95c7" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 2 column_name: "C$_v2" colexpr { column_id: 2 } } columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1261725}, {62afea80ed9b4683b58047535ee0464d, table_id: "62afea80ed9b4683b58047535ee0464d" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 8 column_name: "C$_v8" colexpr { column_id: 8 } } columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1323606}, {650791d2d2a841d7a7f1a4cadbdfb200, table_id: "650791d2d2a841d7a7f1a4cadbdfb200" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 3 column_name: "C$_v3" colexpr { column_id: 3 } } columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1481129}, {be6c2ec24dc8471d9dbd094c0cb87107, table_id: "be6c2ec24dc8471d9dbd094c0cb87107" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 10 column_name: "C$_v10" colexpr { column_id: 10 } } columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1500823}, {a62152a92eb743a7bd9b289edff08055, table_id: "a62152a92eb743a7bd9b289edff08055" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 6 column_name: "C$_v6" colexpr { column_id: 6 } } columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1303793}, {ee34f2fb89a040ac914b589a6725e31c, table_id: "ee34f2fb89a040ac914b589a6725e31c" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 9 column_name: "C$_v9" colexpr { column_id: 9 } } columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1421628}, {b282ced7089d4ca791be2ba3f71cee2d, table_id: "b282ced7089d4ca791be2ba3f71cee2d" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 5 column_name: "C$_v5" colexpr { column_id: 5 } } columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1402230}, {76e315e7631f4c2487641ef72895a999, table_id: "76e315e7631f4c2487641ef72895a999" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 7 column_name: "C$_v7" colexpr { column_id: 7 } } columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1441544}, {1bb5010160b64a4e9a3e82fb071b0eba, table_id: "1bb5010160b64a4e9a3e82fb071b0eba" version: 0 is_local: false columns { column_id: 0 indexed_column_id: 1 column_name: "C$_v1" colexpr { column_id: 1 } } columns { column_id: 1 indexed_column_id: 0 column_name: "C$_k" colexpr { column_id: 0 } } hash_column_count: 1 range_column_count: 1 is_unique: false indexed_table_id: "68214fdb2bd2484095f782959aca7482" indexed_hash_column_ids: 0 use_mangled_column_name: true index_permissions: INDEX_PERM_READ_WRITE_AND_DELETE backfill_error_message: "" num_rows_processed_by_backfill_job: 1461853}]