[DocDB] splitting: Disable tablet splitting for tablets with incorrectly persisted data

[arybochkin]

Jira Link: DB-1991
The following cases should be handled at tablet server side:

• Reject tablet splitting and do not try again if the partition bound is misaligned (applicable for the case when split at statement was used to create an index table)
• Reject tablet splitting and do not try again if there are doc keys inside the tablet that are outside of tablet partition boundaries; refer to GetSplitKey() / GetEncodedMiddleKey() a new error code must be returned to identify such tablets for indexes recreation after YBOperation.partition_key encoding issue is fixed
• Add protection into read and write code path for debug builds (for performance reasons and to not crash existing clusters) that we are always put into RocksDB keys that match tablet partition boundaries

Step to reproduce the issue:

0. Create a cluster with enable_automatic_tablet_splitting=true:

./bin/yb-ctl create --master_flags="enable_automatic_tablet_splitting=true,tablet_split_low_phase_shard_count_per_node=16,tablet_split_low_phase_size_threshold_bytes=13421" --tserver_flags="yb_num_shards_per_tserver=1"

1. Index tablet splitting via split at statement
   1.1. Create a table and a unique index

# create table test1(id int primary key, name text);
# create unique index idx1 on test1(name ASC) split at values(('p'));

Tablets info after statements have been executed:

yugabyte        idx1       000033e1000030008000000000004005        d75c1be78e94456d9803758d6650a897        range: [<start>, DocKey([], ["p"]))
yugabyte        idx1       000033e1000030008000000000004005        b5531d9aa54a45fb821fcf2baca3f268        range: [DocKey([], ["p"]), <end>)

1.2. Insert a row with a value that matches partition bounds

# insert into test1 values(3, 'p');

YBOperation is forwarded to the first tablet d75c1be78e94456d9803758d6650a897 as YBOperation.partition_key is less than the second partition lower bound: 537000000021 < 5370000021. But the operation should have been forwarded to the second tablet in accordance with range bounds. Tablets dump confirms the inserted row is persisted in the first tablet (with respect to DocDB format):

tablet-d75c1be78e94456d9803758d6650a897 in readable format:
SubDocKey(DocKey([], ["p", null]), [SystemColumnId(0); HT{ physical: 1649185585666746 }]) -> null; intent doc ht: HT{ physical: 1649185585665110 }
SubDocKey(DocKey([], ["p", null]), [ColumnId(12); HT{ physical: 1649185585666746 w: 1 }]) -> "G\xfc\xa0H\x80\x00\x00\x03!!"; intent doc ht: HT{ physical: 1649185585665110 w: 1 }

Data Block # 1 @ 0051
--------------------------------------
  HEX    5370000024214A80238001785106292545804A: 238001785106292BA9804A24
  ASCII  S p : # Ä  x Q  ) + © Ä J $ 
  ------
  HEX    5370000024214B8C238001785106292545803FAB: 238001785106292BA9803FAB5347FCA048800000032121
  ASCII  S p : # Ä  x Q  ) + © Ä ? ´ S G ¸ † H Ä 
  ------

tablet-b5531d9aa54a45fb821fcf2baca3f268
<empty>

2. Index tablet splitting via split at statement
   2.1. ./bin/ysqlsh and create a table and a unique index:

create table employees(id int primary key, name text, description text);
create unique index idx1 on employees(description ASC);

2.2. mv collect_insert_data.txt to collect_insert_data.py and run python ./collect_insert_data.py
2.3. ./bin/ysqlsh -f sql_data.sql
2.4. Wait for the data to be loaded

Expected result: only 10 records are loaded
Actual result: the record that matches partition bound is duplicated, the following can be observed:

# select count(*) from employees group by description;
 count
-------
     1
     1
     1
     1
     1
     1
  9780
     1
     1
     1
(10 rows)

[arybochkin]

Taking into account changes in #12211, #12190 and #12191, the issues mentioned above might not happen in a real cluster, some king of synthetic simulation with changing test flags may be required.

[rthallamko3]

@arybochkin , Can we close this now?

[arybochkin]

We cannot close. Sanity checks are required for issue, all possible unit test have already been added in the context of #12190.

[rthallamko3]

Reopening for adding sanity checks.