Level by level traversal with pauses to wait for confirmations

[anthony-swirldslabs]

[anthony-swirldslabs]

Using a new ReconnectBench introduced at #11347 , I've tested the fix with level-by-level traversal and the results are as follows (XXX stands for no fix, aka a control run):

XXX
Benchmark                 (keySize)  (maxKey)  (numFiles)  (numRecords)  (numThreads)  (randomSeed)  (recordSize)  (teacherAddProbability)  (teacherModifyProbability)  (teacherRemoveProbability)  Mode  Cnt  Score   Error  Units
ReconnectBench.reconnect          8   1000000         100       5000000            32    9823452658           128                     0.05                        0.05                        0.05  avgt    7  5.612 ± 0.371   s/op

LEVEL-BY-LEVEL
Benchmark                 (keySize)  (maxKey)  (numFiles)  (numRecords)  (numThreads)  (randomSeed)  (recordSize)  (teacherAddProbability)  (teacherModifyProbability)  (teacherRemoveProbability)  Mode  Cnt  Score   Error  Units
ReconnectBench.reconnect          8   1000000         100       5000000            32    9823452658           128                     0.05                        0.05                        0.05  avgt    7  5.389 ± 0.256   s/op

In other words:

• no fix: score = 5.612 ± 0.371 s
• with fix: score = 5.389 ± 0.256 s

which is an improvement of about 4%.

[anthony-swirldslabs]

I've retested this fix with an updated benchmark, and the results are:

XXX
Benchmark                 (keySize)  (maxKey)  (numFiles)  (numRecords)  (numThreads)  (randomSeed)  (recordSize)  (teacherAddProbability)  (teacherModifyProbability)  (teacherRemoveProbability)  Mode  Cnt  Score   Error  Units
ReconnectBench.reconnect          8   1000000         100       5000000            32    9823452658           128                     0.05                        0.05                        0.05  avgt    7  5.389 ± 0.295   s/op

LevelByLevel
Benchmark                 (keySize)  (maxKey)  (numFiles)  (numRecords)  (numThreads)  (randomSeed)  (recordSize)  (teacherAddProbability)  (teacherModifyProbability)  (teacherRemoveProbability)  Mode  Cnt  Score   Error  Units
ReconnectBench.reconnect          8   1000000         100       5000000            32    9823452658           128                     0.05                        0.05                        0.05  avgt    7  5.350 ± 0.390   s/op

So the improvement seem to look like 0.7% at this point.

[anthony-swirldslabs]

I've re-tested on a bare-metal Ubuntu server, and the results are as follows:

XXX:
Benchmark                 (keySize)  (maxKey)  (numFiles)  (numRecords)  (numThreads)  (randomSeed)  (recordSize)  (teacherAddProbability)  (teacherModifyProbability)  (teacherRemoveProbability)  Mode  Cnt  Score   Error  Units
ReconnectBench.reconnect          8   1000000         100       5000000            32    9823452658           128                     0.05                        0.05                        0.05  avgt    7  6.888 ± 0.418   s/op


LevelByLevel
Benchmark                 (keySize)  (maxKey)  (numFiles)  (numRecords)  (numThreads)  (randomSeed)  (recordSize)  (teacherAddProbability)  (teacherModifyProbability)  (teacherRemoveProbability)  Mode  Cnt  Score   Error  Units
ReconnectBench.reconnect          8   1000000         100       5000000            32    9823452658           128                     0.05                        0.05                        0.05  avgt    7  6.566 ± 0.397   s/op

So the improvement is (6.888-6.566)/6.888 = ~4.6% which is consistent with an original prior measurement above.

[anthony-swirldslabs]

JRS test seems to yield random numbers with two runs for each the baseline and the branch with the fix:

Baseline:

Test: VMerkleDual-NDReconnect-500-60m
Duration: 00:54:33.986
Configured... Show more
---- ReconnectPerformanceValidator SUCCEEDED validation ----
Synchronization 370.33s, initialization 0.00s, data 5218.16mb
Synchronization 9.28s, initialization 0.00s, data 71.98mb

Test: VMerkleDual-NDReconnect-500-60m
Duration: 00:57:27.338
Configured... Show more
---- ReconnectPerformanceValidator SUCCEEDED validation ----
Synchronization 538.03s, initialization 0.00s, data 5223.66mb
Synchronization 11.89s, initialization 0.00s, data 65.99mb

Level by Level Traversal:

Test: VMerkleDual-NDReconnect-500-60m
Duration: 00:55:28.275
Configured... Show more
---- ReconnectPerformanceValidator SUCCEEDED validation ----
Synchronization 501.11s, initialization 0.00s, data 5220.92mb
Synchronization 5.60s, initialization 0.00s, data 65.95mb

Test: VMerkleDual-NDReconnect-500-60m
Duration: 00:54:34.406
Configured... Show more
---- ReconnectPerformanceValidator SUCCEEDED validation ----
Synchronization 417.89s, initialization 0.00s, data 5219.59mb
Synchronization 5.26s, initialization 0.00s, data 58.35mb

The first synchronization attempts took anywhere from 370 to 538 seconds, with baseline averaging at 454s and the fix averaging 459s. However, the second reconnect attempts average at 10.6s for the baseline, and only 5.4s for the fix. Note, however, that the numbers differ wildly from run to run.

In the light of the recent discovery by @artemananiev about a less than ideal way in which the JRS test modifies the tree, it's unclear whether the above numbers are meaningful at all.

However, per the discovery, the first attempt is supposed to be a full reconnect and the level by level traversal fix is not supposed to help (or hurt) for full reconnects, and the similar average around 450seconds for both the runs seems to sort of confirm that.

Also, per the same discovery the second reconnect ressembles a partial ("normal") reconnect, and this is exactly where the level by level traversal fix is supposed to help. And again, while varying a lot from run to run, the numbers for the second attempt seem to demonstrate the performance improvement: 5.4s with the fix vs 10.6s for the baseline.

So if the JRS numbers are to be trusted, the Level By Level Traversal fix shows a 50% performance improvement.

[anthony-swirldslabs]

Based on latest findings at https://github.com/hashgraph/hedera-testing/issues/442#issuecomment-2043901561 , I pushed a new commit with reduced timeout. We should get perfnet testing results next week, but for now I re-ran the tests on a bare metal server and here's the best results (best - means that they have the lowest standard deviation):

Baseline:

Benchmark                 (delayNetworkFuzzRangePercent)  (delayNetworkMicroseconds)  (delayStorageFuzzRangePercent)  (delayStorageMicroseconds)  (keySize)  (mapCount)  (maxKey)  (numFiles)  (numRecords)  (numThreads)  (randomSeed)  (recordSize)  (teacherAddProbability)  (teacherModifyProbability)  (teacherRemoveProbability)  Mode  Cnt  Score   Error  Units
ReconnectBench.reconnect                            0.15                          50                            0.15                         100          8           1   1000000         100          1000            32    9823452658           128                     0.01                        0.01                        0.01  avgt    7  5.551 ± 0.892   s/op

LevelByLevel:

Benchmark                 (delayNetworkFuzzRangePercent)  (delayNetworkMicroseconds)  (delayStorageFuzzRangePercent)  (delayStorageMicroseconds)  (keySize)  (mapCount)  (maxKey)  (numFiles)  (numRecords)  (numThreads)  (randomSeed)  (recordSize)  (teacherAddProbability)  (teacherModifyProbability)  (teacherRemoveProbability)  Mode  Cnt  Score   Error  Units
ReconnectBench.reconnect                            0.15                          50                            0.15                         100          8           1   1000000         100          1000            32    9823452658           128                     0.01                        0.01                        0.01  avgt    7  3.943 ± 0.168   s/op

So the fix shows an improvement of (5.551-3.943)/5.551 ~ 28%.

I keep noticing that the fix also reduces the standard deviation of results consistently. With the fix, the avg stddev is ~0.188 while w/o the fix it's ~1.5+. This seems like an expected outcome of the fix because it tries to minimize randomness by waiting until responses are received. W/o the fix, depending on whether they are sometimes received or not, the teacher code may or may not perform optimally.

[anthony-swirldslabs]

I finally got results for 6K from Tom. Here's the deltas:

LbL, 6min	58:03.8	05:20.4	0:07:17						
	24:29.9	33:08.0	0:08:38						
	56:19.8	04:41.6	0:08:22						
	27:43.4	36:00.0	0:08:17						
	58:50.1	06:34.8	0:07:45						
	29:57.7	40:04.9	0:10:07						
	55:30.5	02:45.7	0:07:15	0:08:14	0:00:59	0:08:24	0:00:59		4%
Base, 6min	38:57.3	44:52.6	0:05:55						
	10:45.7	19:58.4	0:09:13						
	43:05.9	50:51.9	0:07:46						
	13:22.1	21:16.3	0:07:54						
	46:46.4	56:03.2	0:09:17						
	10:53.2	19:21.3	0:08:28						
	38:00.8	47:49.9	0:09:49	0:08:20	0:01:18	0:08:44	0:00:49		
LbL, 20min	35:37.2	47:32.1	0:11:55						
	19:03.8	33:22.7	0:14:19						
	02:52.8	17:16.9	0:14:24						
	46:44.7	01:24.9	0:14:40						
	30:22.8	43:39.7	0:13:17						
	14:11.4	29:52.2	0:15:41						
	57:50.3	10:37.7	0:12:47	0:13:52	0:01:16	0:14:11	0:01:02		2%
Base, 20min	23:04.4	35:53.2	0:12:49						
	06:41.5	20:24.0	0:13:43						
	08:54.0	23:37.6	0:14:44						
	10:29.3	24:50.4	0:14:21						
	13:05.2	29:01.1	0:15:56						
	07:01.3	19:22.3	0:12:21						
	59:21.6	14:58.4	0:15:37	0:14:13	0:01:21	0:14:27	0:01:19		
LbL,63min	26:39.9	57:15.1	0:30:35						
	12:16.9	42:54.8	0:30:38						
	57:15.8	23:34.0	0:26:18						
	42:56.2	05:34.1	0:22:38						
	29:31.4	54:19.5	0:24:48						
	14:24.4	35:33.3	0:21:09						
	00:31.8	18:56.9	0:18:25	0:24:56	0:04:38	0:23:59	0:04:16		24%
Base, 63min	38:00.7	10:28.6	0:32:28						
	24:00.4	54:42.8	0:30:42						
	10:05.4	40:25.2	0:30:20						
	56:37.4	27:07.5	0:30:30						
	42:40.9	20:05.8	0:37:25						
	07:01.8	36:49.1	0:29:47						
	30:02.2	59:45.7	0:29:44	0:31:34	0:02:44	0:31:25	0:02:58		

To summarize: at 6K TPS, the current LevelByLevel traversal fix shows a reconnect performance improvement of 4% for 6 minutes downtime, 2% for 20 min, and 24% for 63 min.

[anthony-swirldslabs]

The results for 10K are in:

Test	BEHIND	ACTIVE	Diff	AVG, all	STDEV, all	AVG, meas	STDEV, meas		LbL vs Base
LbL, 6min	51:28.6	07:16.2	0:15:48						
	42:27.1	59:09.3	0:16:42						
	33:38.5	57:50.3	0:24:12						
	13:09.9	34:14.2	0:21:04						
	04:51.1	27:27.0	0:22:36						
	55:39.1	17:24.3	0:21:45						
	46:39.5	10:50.6	0:24:11	0:20:54	0:03:23	0:21:45	0:02:47		-10%
Base, 6min	19:58.0	37:10.9	0:17:13						
	10:30.0	24:19.1	0:13:49						
	02:04.3	25:46.4	0:23:42						
	41:10.5	59:32.8	0:18:22						
	33:03.8	55:06.0	0:22:02						
	24:17.6	44:01.5	0:19:44						
	16:14.3	37:19.1	0:21:05	0:19:25	0:03:18	0:19:47	0:03:27		
LbL, 20min	38:50.7	13:01.8	0:34:11						
	41:23.5	14:00.2	0:32:37						
	43:58.4	15:41.9	0:31:43						
	46:04.1	16:13.4	0:30:09						
	48:19.3	17:26.9	0:29:08						
	51:08.1	21:08.8	0:30:01						
	30:15.1	48:24.7	0:18:10	0:29:25	0:05:16	0:28:38	0:05:17		0%
Base, 20min	08:30.2	38:48.8	0:30:19						
	11:39.5	40:55.4	0:29:16						
	13:52.0	42:37.8	0:28:46						
	17:06.0	48:32.9	0:31:27						
	20:03.0	49:09.6	0:29:07						
	22:30.2	49:34.9	0:27:05						
	24:56.4	51:13.0	0:26:17	0:28:54	0:01:46	0:28:39	0:01:49		
LbL,63min	15:46.8	40:57.5	0:25:11						
	51:29.3	13:23.6	0:21:54						
	37:36.3	55:38.5	0:18:02						
	12:51.7	34:26.2	0:21:35						
	59:35.6	16:31.6	0:16:56						
	31:04.8	47:03.3	0:15:59						
	18:48.4	41:13.0	0:22:25	0:20:17	0:03:21	0:19:28	0:02:49		39%
Base, 63min	12:40.2	52:13.4	0:39:33						
	24:20.3	06:22.6	0:42:02						
	48:54.3	18:28.2	0:29:34						
	34:11.4	03:21.6	0:29:10						
	20:07.6	54:04.8	0:33:57						
	43:34.2	09:48.5	0:26:14						
	56:05.7	25:44.6	0:29:39	0:32:53	0:05:54	0:31:46	0:05:36		

To summarize: at 10K TPS, the the current LevelByLevel traversal fix shows a reconnect performance improvement of -10% for 6 minutes downtime, 0% for 20 min, and 39% for 63 min.