You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With #147, the PartitionProcessor reads for every incoming shuffle message the current sequence number for the producing partition id. This will entail for every such message a RocksDB read. In order to avoid too many RocksDB reads, we could cache a fixed number of sequence numbers via a LRU cache. That way we could potentially save a lot of RocksDB reads.
The text was updated successfully, but these errors were encountered:
tillrohrmann
changed the title
Cache a fixed amount of sequence numbers in the PartitionStorage for deduplication
Cache a fixed amount of sequence numbers in the PartitionStore for deduplication
Aug 21, 2024
With #147, the
PartitionProcessor
reads for every incoming shuffle message the current sequence number for the producing partition id. This will entail for every such message a RocksDB read. In order to avoid too many RocksDB reads, we could cache a fixed number of sequence numbers via a LRU cache. That way we could potentially save a lot of RocksDB reads.The text was updated successfully, but these errors were encountered: