Skip to content

GML-1432: Refactor dataloaders #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 36 commits into
base: master
Choose a base branch
from
Open

GML-1432: Refactor dataloaders #194

wants to merge 36 commits into from

Conversation

@billshitg
Copy link
Contributor

@billshitg billshitg commented Oct 10, 2023

  • Refactor BaseLoader

parkererickson-tg
parkererickson-tg approved these changes Oct 10, 2023
View reviewed changes
Copy link
Contributor

@parkererickson-tg parkererickson-tg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any benchmarks surrounding this or will we get that insight once the neighbor loader is implemented?

@billshitg
Copy link
Contributor Author

billshitg commented Oct 10, 2023

good point. We don't have benchmarks. We should probably do it once we have the actual dataloaders.

qe-tigergraph
qe-tigergraph requested changes Nov 10, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1013/

qe-tigergraph
qe-tigergraph requested changes Nov 15, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1015/

qe-tigergraph
qe-tigergraph requested changes Nov 18, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1016/

qe-tigergraph
qe-tigergraph requested changes Nov 18, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1017/

qe-tigergraph
qe-tigergraph requested changes Nov 22, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1017/

qe-tigergraph
qe-tigergraph requested changes Nov 22, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1018/

qe-tigergraph
qe-tigergraph requested changes Nov 29, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1019/

qe-tigergraph
qe-tigergraph requested changes Dec 6, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1020/

qe-tigergraph
qe-tigergraph requested changes Dec 8, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1032/

@billshitg
refactor(BaseLoader): split read_data and parse_data
55e7049
@billshitg
refactor(BaseLoader): consolidate vertex and edge parse
324fea0
@billshitg
refactor(BaseLoader): consolidate attr_to_tensor and reindex
441a0b3
@billshitg
feat(BaseLoader): handle one data point per msg
65cca46
@billshitg
fix(BaseLoader): upate _start_request for the refactor
9eaef0e
@billshitg
refactor(VertexLoader): update gsql template
e4e407e
@billshitg
refactor(VertexLoader): update reader thread
4c42d4a
@billshitg
fix(VertexLoader): fix wrong comma in gsql
41c45f9
@billshitg
fix(BaseLoader): fix missing function decorator
ce95a21
@billshitg
fix(BaseLoader,VertexLoader): change how we shuffle
a1ff95b
@billshitg
feat(BaseLoader,VertexLoader): fix when to stop kafka
e7f3e94
@billshitg
feat(VertexLoader): update shuffle
1dfa8bd
@billshitg
test(VertexLoader): update tests
d5a97df
@billshitg
feat(VertexLoader): move kafka commands into template
0298bc0
@billshitg
feat: move shuffle to gsql for speed and add EdgeLoader
7315071
@billshitg
feat(GraphLoader): update gsql and add subquery
d8b816b
@billshitg
feat(dataloaders): update for GraphLoader
7b2970c
@billshitg
feat(utilities): add function to install multiple files
7f5f584
@billshitg billshitg force-pushed the GML-1432-refactor-baseloader branch from cf68fe8 to 22a02b6 Compare December 8, 2023 18:36
qe-tigergraph
qe-tigergraph requested changes Dec 8, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1033/

qe-tigergraph
qe-tigergraph requested changes Dec 11, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1034/

qe-tigergraph
qe-tigergraph requested changes Dec 12, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1037/

qe-tigergraph
qe-tigergraph requested changes Dec 12, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1038/

qe-tigergraph
qe-tigergraph requested changes Dec 13, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: SUCCESS, e2e Test: FAILURE, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1039/

qe-tigergraph
qe-tigergraph reviewed Dec 15, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit Test: SUCCESS, e2e Test: SUCCESS, Jenkins_job:http://192.168.99.101:30080/job/mlwb_build/1041/

qe-tigergraph
qe-tigergraph approved these changes Dec 15, 2023
View reviewed changes
Copy link
Collaborator

@qe-tigergraph qe-tigergraph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QE Approved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@qe-tigergraph qe-tigergraph qe-tigergraph approved these changes

+1 more reviewer

@parkererickson-tg parkererickson-tg parkererickson-tg approved these changes

Reviewers whose approvals may not affect merge requirements

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@billshitg @qe-tigergraph @parkererickson-tg