Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incremental Rollup and Tablet Size Calculation #4972

Merged
merged 25 commits into from
Mar 19, 2020
Merged

Incremental Rollup and Tablet Size Calculation #4972

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
5f07dc9
Bring incremental rollup commit
parasssh Jan 9, 2020
75233b5
Incremental Rollups -- let's track versions of keys.
manishrjain Mar 16, 2020
76d2838
Merge master
manishrjain Mar 16, 2020
fe10d20
Build works with master merge
manishrjain Mar 16, 2020
527a082
Fix map ini
manishrjain Mar 17, 2020
50e7544
Some evidence to show it's not just writes, but write in the range of…
manishrjain Mar 17, 2020
700b54b
Some more tries to understand the behavior
manishrjain Mar 17, 2020
53b1ae5
Merge branch 'master' into mrjn/incremental
Mar 18, 2020
55de90c
Using Martin's fix.
manishrjain Mar 18, 2020
eabfe9a
Make rollup goroutine work with the task tracking
manishrjain Mar 18, 2020
3bbde8c
Cleaned up a bit of the log messages by using strings and converting …
manishrjain Mar 18, 2020
401d6a7
Merge branch 'master' into mrjn/incremental
Mar 18, 2020
7f2a9d7
Merge branch 'mrjn/incremental' of https://github.com/dgraph-io/dgrap…
Mar 18, 2020
1656b38
Fix an issue where we were only sending the keys for rollup which did…
manishrjain Mar 18, 2020
ace03ac
Switch tablet size calculation to use DB.Tables method.
manishrjain Mar 19, 2020
000727f
Ensure that when closing, all registered tasks are properly stopped.
manishrjain Mar 19, 2020
2743576
Revert unnecessary change in list.go
manishrjain Mar 19, 2020
ea1081b
Better handle if another task is going on.
manishrjain Mar 19, 2020
280c634
Call rollup even if we find one delta posting
manishrjain Mar 19, 2020
42fb5e8
Add a note about iterating over tables
manishrjain Mar 19, 2020
71d89dd
fix GoLang CI comments
Mar 19, 2020
a4a7c59
Merge branch 'mrjn/incremental' of https://github.com/dgraph-io/dgrap…
Mar 19, 2020
d4cea37
Revert CI changes done by Paras. Add comment about why.
manishrjain Mar 19, 2020
b4a07c0
Remove question, add comment
manishrjain Mar 19, 2020
50e5b72
Merge master
manishrjain Mar 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Build works with master merge
  • Loading branch information
@manishrjain
manishrjain committed Mar 16, 2020
commit fe10d2072b60381bec1cc15d8ba0436d9ee8926c
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ require (
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd
github.com/dgraph-io/badger/v2 v2.0.1-rc1.0.20200316175624-91c31ebe8c22
github.com/dgraph-io/dgo/v2 v2.2.0
github.com/dgraph-io/ristretto v0.0.2-0.20200115201040-8f368f2f2ab3
github.com/dgrijalva/jwt-go v3.2.0+incompatible
github.com/dgryski/go-farm v0.0.0-20191112170834-c2139c5d712b
github.com/dgryski/go-groupvarint v0.0.0-20190318181831-5ce5df8ca4e1
Expand Down
8 changes: 0 additions & 8 deletions worker/draft.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1172,7 +1172,6 @@ func (n *node) calcTabletSizes(readTs uint64) error {
atomic.AddInt64(size, delta)
}

writer := posting.NewTxnWriter(pstore)
stream := pstore.NewStreamAt(readTs)
stream.LogPrefix = "Tablet Size Calculation"
stream.Prefix = []byte{x.DefaultPrefix}
Expand All @@ -1197,13 +1196,6 @@ func (n *node) calcTabletSizes(readTs uint64) error {
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go func() {
select {
case <-ctx.Done():
case <-op.closer.HasBeenClosed():
cancel()
}
}()
if err := stream.Orchestrate(ctx); err != nil {
return err
}
Expand Down