Skip to content

Rebase develop #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 54 commits into from
Closed

Rebase develop #4

wants to merge 54 commits into from

Conversation

guptask
Copy link

@guptask guptask commented Aug 11, 2022 •
edited by rdower
Loading

This change is Reviewable

igchor and others added 30 commits August 10, 2022 17:41
@igchor @guptask
Run centos and debian workflows on push and PR
480e5a7
@igchor @guptask
Introduce FileShmSegment for file-backed shared memory
c4c7f0b 
It's implementation is mostly based on PosixShmSegment.

Also, extend ShmManager and ShmSegmentOpts to support this new
segment type.
@igchor @guptask
Adjust and enable tests for ShmFileSegment
bea2653
@guptask
Add support for shm opts serialization
c3c55ce 
After introducing file segment type, nameToKey_ does not provide
enough information to recover/remove segments on restart.

This commit fixes that by replacing nameToKey_ with nameToOpts_.

Previously, the Key from nameToKey_ map was only used in a single
DCHECK().
@victoria-mcgrath @guptask
Initial version of config API extension to support multiple memory tiers
751ad14 
* New class MemoryTierCacheConfig allows to configure a memory tier.
  Setting tier size and location of a file for file-backed memory are
  supported in this initial implementation;
* New member, vector of memory tiers, is added to class CacheAllocatorConfig.
* New test suite, chelib/allocator/tests/MemoryTiersTest.cpp,
  demonstrates the usage of and tests extended config API.
@igchor @guptask
Integrate Memory Tier config API with CacheAllocator.
ece528c
@igchor @guptask
Add MemoryTierCacheConfig::fromShm()
808677a 
to allow using new configureMemoryTiers() API with legacy behavior.

Move validation code for memory tiers to validate() method and convert
ratios to sizes lazily (on get)..
@igchor @guptask
Fix test_shm_manager.cpp test
d0aa09b 
It wrongly assumed that the only possible segment type is
PosixSysV segment.
@igchor @guptask
Run tests on CI
5ff6576
@igchor @guptask
Run long tests (navy/bench) every day on CI
29760ed
@guptask
Moved common segment code for posix and file shm segments into ShmCommon
75858d3
@victoria-mcgrath @guptask
Enabled memory tier config API for cachebench.
eea5738
@victoria-mcgrath @guptask
Enabled shared memory tier in cachebench.
85cd702
@victoria-mcgrath @guptask
Converted nvmCacheState_ to std::optional to simplify NVM cache state…
f367a77 
… handling when NVM cache state is not enabled
@igchor @guptask
Run CI on prebuild docker image
a825a6d
@igchor @guptask
Run only centos build on CI
2cfd281
@igchor @guptask
Initial multi-tier support implementation
cbbc763
@igchor @guptask
Extend CompressedPtr to work with multiple tiers
1d258ba 
Now it's size is 8 bytes intead of 4.

Original CompressedPtr stored only some offset with a memory Allocator.
For multi-tier implementation, this is not enough. We must also store
tierId and when uncompressing, select a proper allocator.

An alternative could be to just resign from CompressedPtr but they
are leveraged to allow the cache to be mapped to different addresses on shared memory.

Changing CompressedPtr impacted CacheItem size - it increased from 32 to 44 bytes.
@vinser52 @guptask
Implemented async Item movement between tiers
3258786
@vinser52 @guptask
Adding example for multitiered cache
deaa2b2
@igchor @guptask
Enable workarounds in tests
dc67a35
@igchor @guptask
Add basic multi-tier test
c138cbc
@igchor @guptask
Set correct size for each memory tier
feea63f
@igchor @guptask
Extend cachbench with value validation
873a474
@vinser52 @guptask
Aadding new configs to hit_ratio/graph_cache_leader_fobj
f11c60c
@igchor @guptask
Move validateValue call to make sure it is measured by latency tracker
08d7659
@vinser52 @guptask
Fix eviction flow and removeCb calls
ef50769 
Without this fix removeCb called even in case when Item is moved between
tiers.
@igchor @guptask
Remove failing build-cachelib workflow (pmem#42)
2f10f68 
It fails because CentOS is EOL. We might want to consider
using CentOS Streams but for now, just remove it.

Right now, we rely on build-cachelib-centos workflow anyway.
@victoria-mcgrath @guptask
Disabled test suite allocator-test-AllocatorTypeTest (pmem#41)
83a5de7 
Disabled test suite allocator-test-AllocatorTypeTest to skip sporadically failing tests.
@igchor @guptask
Do not compensate for rounding error when calculating tier sizes (pme…
414ab78 
…m#43)

Compensation results in ratios being different than originially specified.
@guptask guptask self-assigned this Aug 11, 2022
@guptask guptask force-pushed the rebase_develop branch 6 times, most recently from 8398e8b to 6896451 Compare August 15, 2022 08:04
@guptask guptask force-pushed the rebase_develop branch 7 times, most recently from bdc6ca0 to aede229 Compare August 25, 2022 00:38
@guptask
Copy link
Author

guptask commented Aug 25, 2022 •
edited
Loading

Proposal to squash the extra 3 rebase commits into existing commits which are relevant:

Squash 1

aede229 can be merged with d852b2f

Squash 2

9770e45 and c9fd02b can be merged with e50a45c

vinser52 pushed a commit that referenced this pull request Sep 12, 2022
@facebook-github-bot
Prevent divide by zero errors in ObjSize controller.
fa5fa64 
Summary:
AdRanker ASAN canary flagged a possible UBSan violation.

## Error
Failed Run: https://fburl.com/servicelab/apytosry
```
    #0 0x562e3adb59bc in facebook::cachelib::objcache2::ObjectCacheSizeController<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::work() buck-out/v2/gen/fbcode/47d914adeee3d982/cachelib/experimental/objcache2/__object-cache__/headers/cachelib/experimental/objcache2/ObjectCacheSizeController-inl.h
    #1 0x562de7610f78 in facebook::cachelib::PeriodicWorker::loop() fbcode/cachelib/common/PeriodicWorker.cpp:55
    #2 0x7f7632c524e4 in execute_native_thread_routine /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:82:18
    #3 0x7f7632f6ec0e in start_thread /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/nptl/pthread_create.c:434:8
    #4 0x7f76330011db in clone3 /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

UndefinedBehaviorSanitizer: integer-divide-by-zero buck-out/v2/gen/fbcode/47d914adeee3d982/cachelib/experimental/objcache2/__object-cache__/headers/cachelib/experimental/objcache2/ObjectCacheSizeController-inl.h:33:40 in
```

Reviewed By: jiayuebao

Differential Revision: D39024188

fbshipit-source-id: 64ad644c360565e638fa3ca74616a315038382ab
@vinser52
Copy link

Closing this PR as we already rebased develop branch

@vinser52 vinser52 closed this Sep 13, 2022
byrnedj pushed a commit that referenced this pull request Jul 23, 2023
@igchor
Merge pull request #4 from igchor/workflows_pr
17eb460 
Run centos and debian workflows on push and PR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@guptask guptask

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@guptask @vinser52 @igchor @victoria-mcgrath @mcengija