Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(dgraph): ludicrous mode mutation error #5701

Merged
merged 15 commits into from
Jul 9, 2020

Conversation

harshil-goel
Copy link
Contributor

@harshil-goel harshil-goel commented Jun 22, 2020

https://discuss.dgraph.io/t/ludicrous-mode-fails-queries-on-21-million-dataset/7320/2

Time taken to live load 21 million dataset in ludicrous mode went down from 5:11 min on average to 3:01 min on average.


This change is Reviewable

Docs Preview: Dgraph Preview

@harshil-goel harshil-goel marked this pull request as ready for review June 25, 2020 11:53
Copy link
Contributor

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand everything but based on https://discuss.dgraph.io/t/ludicrous-mode-fails-queries-on-21-million-dataset/7320 and the explanation @harshil-goel gave me, this change looks good to me.

Reviewed 1 of 5 files at r2.
Reviewable status: 1 of 5 files reviewed, 4 unresolved discussions (waiting on @harshil-goel, @manishrjain, and @vvbalaji-dgraph)


systest/21million/test-21million.sh, line 61 at r2 (raw file):

    --mode          normal = run dgraph in normal mode
                    none = run dgraph in normal mode
		    ludicrous = run dgraph in ludicrous mode

Indentation.


worker/draft.go, line 646 at r2 (raw file):

			totalSize += int64(psz)

			if x.WorkerConfig.LudicrousMode && proposal.Mutations != nil && proposal.Mutations.StartTs == 0 {

If we're not assigning startTs to mutations, can they still have a non-zero startTs? If not, I'd convert this proposal.Mutations.StartTs == 0 into an assert inside the if


worker/draft.go, line 1164 at r2 (raw file):

							span.Annotate(nil, "Proposal found in CommittedEntries")
						}
						if x.WorkerConfig.LudicrousMode && len(proposal.Mutations.GetEdges()) > 0 {

I believe this change is not related to this PR.


worker/task.go, line 2185 at r2 (raw file):

	for itr.Seek(countKey); itr.Valid(); itr.Next() {
		item := itr.Item()
		key := append(make([]byte, 0), item.Key()...)

I believe this change is not related to this PR.

Copy link
Contributor Author

@harshil-goel harshil-goel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 1 of 5 files reviewed, 4 unresolved discussions (waiting on @jarifibrahim, @manishrjain, and @vvbalaji-dgraph)


systest/21million/test-21million.sh, line 61 at r2 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

Indentation.

Done.


worker/draft.go, line 646 at r2 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

If we're not assigning startTs to mutations, can they still have a non-zero startTs? If not, I'd convert this proposal.Mutations.StartTs == 0 into an assert inside the if

ALTER Operations already have a timestamp. do end up coming with startTs. The same check == 0 is applied when we assign a startTs to a mutation.


worker/draft.go, line 1164 at r2 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

I believe this change is not related to this PR.

I needed the change to run the tests. I will get merge those other PRs first.


worker/task.go, line 2185 at r2 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

I believe this change is not related to this PR.

I needed the change to run the tests. I will get merge those other PRs first.

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 4 of 5 files at r2, 1 of 1 files at r3.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @jarifibrahim and @vvbalaji-dgraph)


worker/draft.go, line 646 at r2 (raw file):

Previously, harshil-goel (Harshil Goel) wrote…

ALTER Operations already have a timestamp. do end up coming with startTs. The same check == 0 is applied when we assign a startTs to a mutation.

100 chars. Seems like a fine thing to do.

@harshil-goel harshil-goel merged commit 2a8c17f into master Jul 9, 2020
@harshil-goel harshil-goel deleted the harshil-goel/lud-mode-tests branch July 9, 2020 09:39
harshil-goel added a commit that referenced this pull request Jul 9, 2020
Fixes out of order commit in ludicrous mode.

(cherry picked from commit 2a8c17f)
harshil-goel added a commit that referenced this pull request Jul 9, 2020
Fixes out of order commit in ludicrous mode.

(cherry picked from commit 2a8c17f)
parasssh pushed a commit that referenced this pull request Jul 13, 2020
Fixes out of order commit in ludicrous mode.

(cherry picked from commit 2a8c17f)
parasssh pushed a commit that referenced this pull request Jul 13, 2020
Fixes out of order commit in ludicrous mode.

(cherry picked from commit 2a8c17f)

Co-authored-by: parasssh <paras@dgraph.io>
arijitAD pushed a commit that referenced this pull request Jul 14, 2020
Fixes out of order commit in ludicrous mode.
dna2github pushed a commit to dna2fork/dgraph that referenced this pull request Jul 18, 2020
Fixes out of order commit in ludicrous mode.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants