Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/tiered caching #11

Draft
wants to merge 134 commits into
base: dummy-tiered-caching-base
Choose a base branch
from
Draft

Feature/tiered caching #11

wants to merge 134 commits into from

Conversation

peteralfonsi
Copy link
Owner

@peteralfonsi peteralfonsi commented Nov 14, 2023 •
edited
Loading

Description

Merges in keystore, caching policy, node stats, cache value rework, and serializers.

Related Issues

Part of tiered caching

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

(All tests DO NOT pass - I cannot get the strikethrough thing to work and i need github actions to assemble this for use in CDK! We still have two failing tests that Sagar is looking into!)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Peter Alfonsi and others added 30 commits September 28, 2023 16:04
Implemented and tested time took for QuerySearchResult objects for us…
fcd7476 
…e in tiered caching
Moved setting to QueryPhase.execute()
87cbe2a
Moved setting to QueryPhase.execute()
cf18eb9
spotlessCheck
c876de8 
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Added first draft of caching policy setup
93a16d4
@sgup432
Tiered caching framework draft changes
8d6c8c4
initial commit
83590e1
fixed permissions
aa0a5d7
Set up event listener to create RemovalNotification objects, also mis…
41dc035 
…c functions
Added misc getter methods
4ba49af
thread pool stuff
dcb9fad
cleanup
b912527
Removed partial cache files from disk
44d5e61
fixed double counting of on heap hit count
43e2d1d
first attempt at fixing cache closing issue in tests
aa2d28d
misc stuff for testing
354c311
Moved more stuff around for debugging issues
66915b6
fixed various serialization issues relating to silent ehcache put() a…
f985e63 
…nd get() failures
Cleanup
ae300e9
More cleanup
7a7b41f
Merged in Sagar's serialization changes
ec729d0
Part 1 of modifying ehcache to take serializable Key
990ba9c
final attempts at doing it generally before learning that wasnt even …
d21af15 
…needed
Changed Ehcache disk tier implementation to not be generic, fixed per…
d152478 
…sistent cache locking issue
Fixed multiple disk tier instance issue, made spillover test pass
2d200fc
Removed persistence, cleaned up
8f65db4
cleanup
7155afd
added test entries metric for on heap
c760594
Fixed issue where multiple mock nodes on one JVM (in IT tests) would …
8510637 
…cause initialization errors in the cache manager
Merge branch 'framework-serialized' into update_node_stats
0b2e506 
Fixing initialization issue that broke IT tests
kiranprakash154 and others added 30 commits December 5, 2023 09:44
@kiranprakash154
Merge branch 'feature/tiered-caching' into kp/delete-stale-keys
27f122b 
Signed-off-by: Kiran Prakash <awskiran@amazon.com>
Added RBM size setting + UT
295ff1a
added UT for setting switch
163d7fe
added skeleton for stale key threshold setting
29d31b4
Added commented-out lines to opensearch.yml
856160e
added feature flag
4bdf459
added ehcache cluster settings
347a14c
added feature flag setting to disk tier IT
64c339a
added null checks to key and dummy string serializers
90c0f9a
Integrated misc changes from other branch
47453e1
changed dummy disk cache size from 1 MB -> 10 GB
216d99d
Merge branch 'misc-settings' into enable-disk-setting
6483c69
added first draft of dynamic disk tier enable/disable logic
8db79e0
added IT testing toggling disk tier dynamic setting
153c575 
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
cleanup
52dad7a 
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
removed disk tier checks from IT where disk tier is not turned on
1671e2f
added checks to disk enable/disable fns, condensed IT
24b5f14
set default disk tier size to 100 mb
1c1a2b5 
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
invalidateAll() also invalidates keys from inactive disk tier
2758ee3 
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
set disk threshold policy to 0
f8b0425
@kiranprakash154
refactor cleanCache & cleanDiskCache to share methods
8f0b5bc
@kiranprakash154
null checks for indexShard
8c996b3
@kiranprakash154
some bugs i found
4bd1503
@kiranprakash154
use assertNumCacheEntries
1911fa6
@kiranprakash154
update print statement
ae0b471
@kiranprakash154
edit sleep time
fce110d
changed setting paths from index to indices for non-index level settings
7d2ea2e
Addressed Kiran's comments
49c9bbd 
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Merge branch 'misc-settings' into feature/tiered-caching
dc3fd1e
Merge branch 'kp/delete-stale-keys' into feature/tiered-caching
f375874
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgup432 sgup432 sgup432 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@peteralfonsi @kiranprakash154 @sgup432