Skip to content

v2.0 with deletes #169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
szarnyasg merged 77 commits into from
Jul 12, 2022
Merged

v2.0 with deletes #169

szarnyasg merged 77 commits into from
Jul 12, 2022

Conversation

@szarnyasg
Copy link
Member

@szarnyasg szarnyasg commented Jun 27, 2022 •
edited
Loading

This PR adds:

  • Delete operations
  • Moving the schema from the Hadoop Datagen to the newer Spark Datagen
  • Loading of updatestreams is done using DuckDB from Parquet files

The move to the Spark Datagen has the following effects:

  • substitution parameters should be created using the new parameter generator, which outputs Parquet files (opposed to the pipe-separated txt files of the old parameter generator) with different filenames

Fixes #160

GLaDAP and others added 30 commits June 22, 2022 21:52
@GLaDAP
Add script for update stream plots
999490e
@GLaDAP
Add delete queries
f76753f
@GLaDAP
Add delete queries to serializer
e54f503
@GLaDAP
Merge branch 'main' into v2.0-dev
f95239d
@GLaDAP
Move filtering of update streams
2b2eb11
@GLaDAP
Change update filenames and add deletions in config
e46e5de
@GLaDAP
Add updateStream decoders using Resultset
1eb5378
@GLaDAP
Update test resources
faff734
@GLaDAP
Fix workload configuration
3de38f3
@GLaDAP
Add new loader update streams in LdbcSnbInteractiveWorkload
d8c023f
@GLaDAP
Remove CharSeeker
a9447d3
@GLaDAP
Add operation type mapping for deletes
abeb719
@GLaDAP
Rename fields to support deduction serialization
1cd9adb
@GLaDAP
Add delete queries to dummy ldbcSnbInteractive test
e67f77d
@GLaDAP
Fix tagIds
85d1f44
@GLaDAP
Add script to convert spark delete data to interactive
cd00071
@szarnyasg
Add conversion script for inserts
cf5929a
@GLaDAP
Change substitution param files from txt to csv
a8ce122
@GLaDAP
Change decoder Q3 to comply with specification
42dbe0d
@GLaDAP
Fix parsing of dates in query event stream reader
12e0293
@GLaDAP
Update test set
8830cbd
@szarnyasg
Move package-mvn-artifacts.sh script to the scripts/ directory
95843b1
@szarnyasg
Add build and install scripts
15ae303
@GLaDAP
Merge operation stream readers
b97ba7f
@GLaDAP
Merge branch 'v2.0-dev' of github.com:ldbc/ldbc_snb_interactive_drive…
977a539 
…r into v2.0-dev
@GLaDAP
Remove obsolete comment
6895f54
@szarnyasg
Adjust README instructions for insert/delete conversion scripts
7631a52
@szarnyasg
Adjust parameter names for the Interactive delete queries
85cc475
@szarnyasg
Update 5 operation: Replace joinDate with creationDate
fb9ad28
@szarnyasg
Use vanilla HashMap instead of Guava's ImmutableMap to allow null values
3b91aad 
The nulls are passed on to JDBC in the implementations repository.
@szarnyasg szarnyasg mentioned this pull request Jul 9, 2022
Merged
szarnyasg and others added 10 commits July 9, 2022 22:16
@szarnyasg szarnyasg marked this pull request as ready for review July 11, 2022 15:13
@szarnyasg szarnyasg merged commit d132fdf into main Jul 12, 2022
@szarnyasg szarnyasg deleted the v2.0-dev branch July 12, 2022 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Introduce delete operations

3 participants

@szarnyasg @GLaDAP