ShardedRocks determinism phase 2 (PSM, Checkpoint) #12203

spraza · 2025-06-16T22:48:20Z

Description

This is part 2 of #11841. It took a few weeks of iteration of identifying different sources of non-determinism and then figuring out the simplest solution.

In terms of root cause behind the non-determinism, all offending instances were in the checkpoint metadata category. When we send this checkpoint over the network, flow transport relies on the byte size of the payload and the run would diverge if the payload size is even slightly different.

Generally, once the root cause is identified, the hard part is done, and the solution can be relatively straightforward using the principles described in #11790. As a reminder the goal is to ensure simulation behavior is as close to production as possible, at the same time, ensure the runs are deterministic in simulation. What made solving sharded rocks with PSM and checkpoint hard is two fold:

There are just lots of metadata fields that are fetched via rocksdb api. It's not scalable to go one field at a time and pick a sane deterministic value.

foundationdb/fdbserver/RocksDBCheckpointUtils.actor.cpp

Lines 77 to 103 in 046d510

    
           liveFileMetaData.relative_filename = fileMetaData.relative_filename; 
        
           liveFileMetaData.directory = fileMetaData.directory; 
        
           liveFileMetaData.file_number = fileMetaData.file_number; 
        
           liveFileMetaData.file_type = static_cast<rocksdb::FileType>(fileMetaData.file_type); 
        
           liveFileMetaData.size = fileMetaData.size; 
        
           liveFileMetaData.temperature = static_cast<rocksdb::Temperature>(fileMetaData.temperature); 
        
           liveFileMetaData.file_checksum = fileMetaData.file_checksum; 
        
           liveFileMetaData.file_checksum_func_name = fileMetaData.file_checksum_func_name; 
        
           liveFileMetaData.smallest_seqno = fileMetaData.smallest_seqno; 
        
           liveFileMetaData.largest_seqno = fileMetaData.largest_seqno; 
        
           liveFileMetaData.smallestkey = fileMetaData.smallestkey; 
        
           liveFileMetaData.largestkey = fileMetaData.largestkey; 
        
           liveFileMetaData.num_reads_sampled = fileMetaData.num_reads_sampled; 
        
           liveFileMetaData.being_compacted = fileMetaData.being_compacted; 
        
           liveFileMetaData.num_entries = fileMetaData.num_entries; 
        
           liveFileMetaData.num_deletions = fileMetaData.num_deletions; 
        
           liveFileMetaData.oldest_blob_file_number = fileMetaData.oldest_blob_file_number; 
        
           liveFileMetaData.oldest_ancester_time = fileMetaData.oldest_ancester_time; 
        
           liveFileMetaData.file_creation_time = fileMetaData.file_creation_time; 
        
           liveFileMetaData.epoch_number = fileMetaData.epoch_number; 
        
           liveFileMetaData.name = fileMetaData.name; 
        
           liveFileMetaData.db_path = fileMetaData.db_path; 
        
           liveFileMetaData.column_family_name = fileMetaData.column_family_name; 
        
           liveFileMetaData.level = fileMetaData.level; 
        
           liveFileMetaData.smallest = fileMetaData.smallest; 
        
           liveFileMetaData.largest = fileMetaData.largest; 
        
           liveFileMetaData.file_type = rocksdb::kTableFile;

The checkpoint goes through network transport layer, which is too low level to change the behavior of in order to get determinism.

So instead, this was the solution I converged towards:

First, limit the entry points of all sets/gets to checkpoint. This is simply done by having set/get APIs. What then this gives is that I can change the internal representation of the checkpoint at one place without letting any of the clients know. This is only done in simulation. Even in simulation, from a client's pov, you write X, you get X.
Once I had control over the checkpoint internal representation without affecting clients, then I can change the representation by adding dynamic padding to it. The dynamic here means we round the payload size to the next Nth byte size. For example, payload of 100 bytes -> padded to 5000 bytes, payload of 5001 bytes -> padded to 10000 bytes, etc. The N is configurable but what this does is makes payload size deterministic -- there's a probability here but more discussion on that in the FAQ section. Finally, there was a footer that encodes the exact payload size picked, so when someone requests it, we can extract the original content back.

FAQs

These are questions I was asking myself as I worked towards solving this issue.

Q1: Why not use the approach of picking sane deterministically random values for the metadata fields?
A1: As described above, there were too many fields, and then we go through the network layer which is too low level and does not have field-by-field granularity, only "payload" or packet level granularity.

Q2: Is there any functional change in production?
A2: No. In production, we exit early at the set/get level. Even in simulation, there should be no functional behavior change since client is not aware of the internal details of dynamic padding and flow transport layer.

Q3: Ok, in simulation, surely there has to be a cost. What is it?
A3: Yes, there is a cost. I would put all of that under "performance" category, which is not the goal of simulation. The fact that we store additional padding and that too inefficiently (it's optimized for readability and debugging) means we're using more memory and in fact more simulated network i/o. All of this is ok as long as performance degradations don't cause significant slow downs, OOM or crashes.

Q4: How do you pick PAYLOAD_ROUND_TO_NEXT, what are the tradeoffs, and how do you know it does not cause issues like OOM?
A4: Empirically via experimentation. The tradeoff is that lower the value, lower memory but higher probability of non-determinism. From my experiments, rocksdb metadata variation, atleast over the wire, is a few KBs, not 10s of KBs. I picked 5K as a result and so far have not seen any OOMs/timeouts/crashes. This can always be tuned as needed.

Testing

Added new unit tests.

Ran PhysicalShardMove.toml 500K times, saw 0 instances of UnseedMismatch: 20250620-211429-praza-shardedrocks-determinism-phase2-e54019 compressed=True data_size=37114014 duration=19311177 ended=500000 fail=8 fail_fast=25 max_runs=500000 pass=499992 priority=100 remaining=0 runtime=20:11:28 sanity=False started=500000 stopped=20250621-172557 submitted=20250620-211429 timeout=5400 username=praza-shardedrocks-determinism-phase2-e54019c6b7b1452581c954dc4c8cb57faa2aedd9

Ran general 500K, saw 0 instances of UnseedMismatch: 20250620-211406-praza-shardedrocks-determinism-phase2-e54019 compressed=True data_size=37084541 duration=24685358 ended=500000 fail=6 fail_fast=25 max_runs=500000 pass=499994 priority=100 remaining=0 runtime=23:02:25 sanity=False started=500000 stopped=20250621-201631 submitted=20250620-211406 timeout=5400 username=praza-shardedrocks-determinism-phase2-e54019c6b7b1452581c954dc4c8cb57faa2aedd9

Ran general 500K while forcing sharded rocks, saw 0 instances of UnseedMismatch: 20250620-212116-praza-shardedrocks-determinism-phase2-forceS compressed=True data_size=37084540 duration=26423186 ended=500000 fail=8 fail_fast=25 max_runs=500000 pass=499992 priority=100 remaining=0 runtime=23:10:01 sanity=False started=500000 stopped=20250621-203117 submitted=20250620-212116 timeout=5400 username=praza-shardedrocks-determinism-phase2-forceSR-8745a0d9340d3a2c6f2c0c7beea251dd33e81ce8

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2025-06-16T22:53:11Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 046d510
Duration 0:04:40
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-16T22:53:29Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 046d510
Duration 0:04:58
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-16T22:53:30Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 046d510
Duration 0:04:58
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-16T22:53:32Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 046d510
Duration 0:04:59
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-06-16T22:53:37Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 046d510
Duration 0:05:05
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-16T23:26:49Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 046d510
Duration 0:38:19
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-16T23:56:43Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 25efe40
Duration 0:25:54
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-17T00:09:22Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 25efe40
Duration 0:38:35
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-17T00:14:07Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 25efe40
Duration 0:42:31
Result: ❌ FAILED
Error: Error while executing command: mvn --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -pl site.ycsb:foundationdb-binding -am clean package. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-06-17T00:17:55Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 25efe40
Duration 0:47:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-17T00:23:44Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 25efe40
Duration 0:52:51
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-17T00:31:43Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 25efe40
Duration 1:00:55
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-17T00:37:30Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 25efe40
Duration 1:06:40
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

…f CheckpointMetaData object

foundationdb-ci · 2025-06-20T21:52:12Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 0d53092
Duration 0:29:03
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-20T22:02:14Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 0d53092
Duration 0:39:07
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-20T22:06:29Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 0d53092
Duration 0:43:14
Result: ❌ FAILED
Error: Error while executing command: mvn --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -pl site.ycsb:foundationdb-binding -am clean package. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-06-20T22:12:10Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 0d53092
Duration 0:49:02
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-20T22:25:18Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 0d53092
Duration 1:02:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-20T22:26:25Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 0d53092
Duration 1:03:18
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-20T22:31:27Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 0d53092
Duration 1:08:19
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77

Looks great!

jzhou77 · 2025-06-21T05:10:49Z

fdbclient/StorageCheckpoint.cpp

@@ -0,0 +1,145 @@
+#include "fdbclient/StorageCheckpoint.h"


Can you add the copyright header?

fdbclient/StorageCheckpoint.cpp

foundationdb-ci · 2025-06-23T19:10:22Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: dda7c99
Duration 0:04:40
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T19:10:31Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: dda7c99
Duration 0:04:54
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T19:10:37Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: dda7c99
Duration 0:04:57
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T19:10:43Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: dda7c99
Duration 0:05:04
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-06-23T19:10:45Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: dda7c99
Duration 0:05:07
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T19:14:37Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: dda7c99
Duration 0:08:56
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /opt/homebrew/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T20:06:37Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: dda7c99
Duration 1:00:57
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T20:29:47Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 6034f49
Duration 0:27:02
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T20:41:34Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 6034f49
Duration 0:38:53
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T20:46:14Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 6034f49
Duration 0:42:49
Result: ❌ FAILED
Error: Error while executing command: mvn --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -pl site.ycsb:foundationdb-binding -am clean package. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-06-23T20:53:05Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 6034f49
Duration 0:50:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T20:58:40Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 6034f49
Duration 0:55:55
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T21:04:24Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 6034f49
Duration 1:01:42
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

fdbserver/KeyValueStoreShardedRocksDB.actor.cpp

jzhou77

LGTM!

foundationdb-ci · 2025-06-23T22:34:06Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: c044011
Duration 0:24:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T22:47:52Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: c044011
Duration 0:38:38
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T22:51:34Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: c044011
Duration 0:42:17
Result: ❌ FAILED
Error: Error while executing command: mvn --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -pl site.ycsb:foundationdb-binding -am clean package. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-06-23T22:58:23Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: c044011
Duration 0:49:07
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T23:10:07Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: c044011
Duration 1:00:54
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T23:17:01Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: c044011
Duration 1:07:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-23T23:18:05Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: c044011
Duration 1:08:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

spraza marked this pull request as draft June 16, 2025 22:48

ShardedRocks determinism phase 2 (PSM, Checkpoint)

25efe40

spraza force-pushed the shardedrocks-determinism-phase2 branch from 046d510 to 25efe40 Compare June 16, 2025 23:30

Tie lifetime of getSerializedCheckpoint's return value same as that o…

0d53092

…f CheckpointMetaData object

spraza marked this pull request as ready for review June 20, 2025 22:22

spraza requested review from neethuhaneesha, jzhou77, kakaiu and yao-xiao-github June 20, 2025 22:22

spraza changed the title ~~[DO_NOT_REVIEW_YET][DRAFT] ShardedRocks determinism phase 2 (PSM, Checkpoint)~~ ShardedRocks determinism phase 2 (PSM, Checkpoint) Jun 20, 2025

jzhou77 reviewed Jun 21, 2025

View reviewed changes

neethuhaneesha previously approved these changes Jun 23, 2025

View reviewed changes

Address feedback

dda7c99

spraza dismissed neethuhaneesha’s stale review via dda7c99 June 23, 2025 19:05

Fix formatting

6034f49

jzhou77 reviewed Jun 23, 2025

View reviewed changes

fdbserver/KeyValueStoreShardedRocksDB.actor.cpp Outdated Show resolved Hide resolved

fdbserver/KeyValueStoreShardedRocksDB.actor.cpp Outdated Show resolved Hide resolved

Address feedback

c044011

jzhou77 approved these changes Jun 23, 2025

View reviewed changes

yao-xiao-github approved these changes Jun 23, 2025

View reviewed changes

spraza merged commit 8c6bc89 into apple:main Jun 24, 2025
6 of 7 checks passed

	liveFileMetaData.relative_filename = fileMetaData.relative_filename;
	liveFileMetaData.directory = fileMetaData.directory;
	liveFileMetaData.file_number = fileMetaData.file_number;
	liveFileMetaData.file_type = static_cast<rocksdb::FileType>(fileMetaData.file_type);
	liveFileMetaData.size = fileMetaData.size;
	liveFileMetaData.temperature = static_cast<rocksdb::Temperature>(fileMetaData.temperature);
	liveFileMetaData.file_checksum = fileMetaData.file_checksum;
	liveFileMetaData.file_checksum_func_name = fileMetaData.file_checksum_func_name;
	liveFileMetaData.smallest_seqno = fileMetaData.smallest_seqno;
	liveFileMetaData.largest_seqno = fileMetaData.largest_seqno;
	liveFileMetaData.smallestkey = fileMetaData.smallestkey;
	liveFileMetaData.largestkey = fileMetaData.largestkey;
	liveFileMetaData.num_reads_sampled = fileMetaData.num_reads_sampled;
	liveFileMetaData.being_compacted = fileMetaData.being_compacted;
	liveFileMetaData.num_entries = fileMetaData.num_entries;
	liveFileMetaData.num_deletions = fileMetaData.num_deletions;
	liveFileMetaData.oldest_blob_file_number = fileMetaData.oldest_blob_file_number;
	liveFileMetaData.oldest_ancester_time = fileMetaData.oldest_ancester_time;
	liveFileMetaData.file_creation_time = fileMetaData.file_creation_time;
	liveFileMetaData.epoch_number = fileMetaData.epoch_number;
	liveFileMetaData.name = fileMetaData.name;
	liveFileMetaData.db_path = fileMetaData.db_path;
	liveFileMetaData.column_family_name = fileMetaData.column_family_name;
	liveFileMetaData.level = fileMetaData.level;
	liveFileMetaData.smallest = fileMetaData.smallest;
	liveFileMetaData.largest = fileMetaData.largest;
	liveFileMetaData.file_type = rocksdb::kTableFile;

ShardedRocks determinism phase 2 (PSM, Checkpoint) #12203

ShardedRocks determinism phase 2 (PSM, Checkpoint) #12203

Uh oh!

Conversation

spraza commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

FAQs

Testing

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Jun 16, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jun 16, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 16, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 16, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 16, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 16, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 16, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 17, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 17, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 17, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jun 17, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 17, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 17, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 20, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 20, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 20, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 20, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jun 20, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 20, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 20, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

jzhou77 left a comment

Choose a reason for hiding this comment

Uh oh!

jzhou77 Jun 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

foundationdb-ci commented Jun 23, 2025

spraza commented Jun 16, 2025 •

edited

Loading