Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory store repartitioning behavior #199

Merged
merged 6 commits into from
Oct 26, 2021

Conversation

Roman1us
Copy link
Contributor

Note: Before submitting this pull request, please review our contributing
guidelines
.

Description

I use memory store on production for aggregation tables. So, if we are use 2 partitions on table's changelog topic, we want table keys to be partitioned accross workers. But, in memory store we will have old keys, that, after repartition (rebalance), belongs to new worker. In some cases, when we modify table from stream, it won't be a problem, but i use use_partitioner and modify table by task, and for me this is a big problem.

So i add extra logic for memory store, that stores partitions for keys. Also added clearing no longer used keys on the worker after recovery from table.

@codecov-commenter
Copy link

codecov-commenter commented Oct 15, 2021

Codecov Report

❗ No coverage uploaded for pull request base (master@510051b). Click here to learn what that means.
The diff coverage is 100.00%.

❗ Current head 2bdf935 differs from pull request most recent head b8a2388. Consider uploading reports for the commit b8a2388 to get more accurate results
Impacted file tree graph

@@            Coverage Diff            @@
##             master     #199   +/-   ##
=========================================
  Coverage          ?   94.46%           
=========================================
  Files             ?      100           
  Lines             ?    10665           
  Branches          ?     1206           
=========================================
  Hits              ?    10075           
  Misses            ?      525           
  Partials          ?       65           
Impacted Files Coverage Δ
faust/stores/memory.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 510051b...b8a2388. Read the comment docs.

@patkivikram patkivikram merged commit 654b648 into faust-streaming:master Oct 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants