Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Key-Value Map Filter #1

Merged
merged 33 commits into from
Jan 13, 2025
Merged

Implement Key-Value Map Filter #1

merged 33 commits into from
Jan 13, 2025

Conversation

itzmeanjan
Copy link
Owner

  • Implement Key-Value Map filter using Binary Fuse Filters
  • Extend KV Map filter to matrices, so that longer values can be encoded, not just u32
  • Add tests ensuring recoverability of values, when queried using keys

@itzmeanjan
Update gitignore file to conform to Rust project
b3157d4 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add cargo.toml file
0cb2d66 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add Matrix struct
55bb1f1 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Generate matrix of dimension m x n, given a 32 -bytes seed
7bf2539 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add binary fuse filter utility functions
50af1b3 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Implement function for computing Sha3_256 hash of arbitrary length key
00db8d4 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add function to compute batch hash
f52c8c3 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add dependency for generating cryptographically secure PRNG
2facb5c 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Make small functions inline always
410800a 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add function for filling a u64 result, in little-endian order, from…
341a290 
… arbitrary long byte array

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add function for converting a (key, value) pair to a row of the database
cd96a01 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add function for decoding a row as (hashed-key, value) pair
29858e8 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Correctly compute matrix element mask
f01af1c 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Handle encoding value boundary 0x81
3c159b3 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Decode row such that it checks if key-value pair encoding is valid
0d3ecc9 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Check if decoded row is zeroed post boundary mark
a8753a7 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Minor change of variable name
acd9705 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add test to ensure that encoding (key,value) pair as row and recoveri…
d8d0cbb 
…ng works as expected

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add test configuration macro
40ed3c6 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Extend encoding/decoding of (key,value) pair test, covering more scen…
de6bc4c 
…arios

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add function to encode a key-value map into a database matrix
c5a2682 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Extract Binary-Fuse-Filter implementation from function parsing KV da…
efb3356 
…tabase

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Update function encoding KV database as matrix to use explicit functi…
0ddeae5 
…on for constructing Binary Fuse Filter

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Bumped column-limit (max-width in rustfmt lingo) to 160, reformatted …
26df383 
…source

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Extract function for serializing a (key, value) pair to a row of matr…
d77fd34 
…ix and vice versa into a new module

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Extract binary fuse filter associated functions as module functions
eb8e602 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Keep matrix element bit length in BFF struct
091b94f 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add function for recovering value from KV database encoded as matrix
97fe629 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Make slice of appropriate length before copying into it
202cb87 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add test ensuring functional correctness of encoding key-value map as…
d5bccdf 
… database matrix, checking if we can recover values correctly

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Change what parameters test is run with
669600d 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Add Github Actions CI script to run tests
5edd21c 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan
Reduce input param range, so that test run finishes quick enough
e797e85 
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan itzmeanjan merged commit 47cf385 into main Jan 13, 2025
2 checks passed
@itzmeanjan itzmeanjan deleted the implement-kv-map-filter branch January 13, 2025 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@itzmeanjan