Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KV BulkWrite #223

Merged
merged 45 commits into from
Dec 9, 2019
Merged

KV BulkWrite #223

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
6b5e2af
Changing KV index code so that it doesn't do a 'set(get+1)' increment…
kellrott Jul 13, 2019
0b934bf
Adding bulk write interface to KV interface. It only does something d…
kellrott Jul 16, 2019
db0f614
Fixing bug in the aggrigation processor.
kellrott Jul 16, 2019
639abcf
Removing RocksDB from driver list and testing
kellrott Jul 16, 2019
920e365
Adding rate counter to KVLoading logging
kellrott Jul 19, 2019
55b75f6
Adding a BulkAdd method to all of the drivers. Reworking the server A…
kellrott Jul 19, 2019
3079c11
Merge remote-tracking branch 'origin/master' into kvindex-bulkwrite
kellrott Jul 19, 2019
7b062a6
Making kvgraph bulk add use the bulk transaction
kellrott Jul 19, 2019
b6e00c2
Removing redundant code
kellrott Jul 19, 2019
8fa93a8
Fixing lint issue
kellrott Jul 19, 2019
b7a8598
Removing redundant code
kellrott Jul 22, 2019
cd0ed62
Having internal bulkAdd call errors get copied if if they occur
kellrott Jul 23, 2019
c9b3e09
Addressing issues raised in PR
kellrott Sep 11, 2019
e5241d0
Merge remote-tracking branch 'origin/master' into kvindex-bulkwrite
kellrott Sep 11, 2019
c49018b
Adding some tests related to StreamBatch method
kellrott Sep 11, 2019
02cb246
Adding graph name validation in SteamBatch
kellrott Sep 11, 2019
fff653f
Updating bulk upload behavior and unit tests
kellrott Sep 11, 2019
13b748c
Cleaning linting issues
kellrott Sep 11, 2019
0b358ef
Updating badger link
kellrott Sep 11, 2019
755bb93
Fixing lint issue
kellrott Sep 11, 2019
86a7bd5
Fixing flake issue
kellrott Sep 11, 2019
ff79aa3
Fixing offline KV bulk command line loader to use new bulk loading API
kellrott Sep 12, 2019
cb30d36
Adding some error logging
kellrott Sep 13, 2019
6a6b38f
Adding logging edge insert rates
kellrott Sep 13, 2019
1a05b77
more error handling for BulkAdd
adamstruck Sep 17, 2019
d0440f1
cmd/kvload: cleanup
adamstruck Sep 24, 2019
02c2fb7
pass logrus logger to badger
adamstruck Sep 24, 2019
c353877
cleanup badger opts
adamstruck Sep 24, 2019
3bc6496
kvgraph: fix error handling in BulkAdd
adamstruck Sep 24, 2019
891e204
wip
adamstruck Sep 26, 2019
43b0361
create grip logger package, allows passing configured logger to subpr…
adamstruck Oct 1, 2019
0ff4dab
set logger to debug for kvload
adamstruck Oct 1, 2019
390b6b1
add a log sub method
adamstruck Oct 1, 2019
b037525
log: fix log level in methods
adamstruck Oct 1, 2019
f958057
updated file stream methods
adamstruck Nov 22, 2019
944711e
increase buffer size used in StreamFile
adamstruck Nov 22, 2019
d0cac8b
debug wip
adamstruck Nov 22, 2019
c1a64fc
Merge remote-tracking branch 'origin/master' into kvindex-bulkwrite-as
adamstruck Dec 3, 2019
773dd0e
Merge branch 'kvindex-bulkwrite-as' of github.com:bmeg/grip into kvin…
adamstruck Dec 3, 2019
0e6805e
fix broken tests
adamstruck Dec 4, 2019
27765ed
server: fix BulkAdd
adamstruck Dec 5, 2019
b0f2008
added global --pprof flag to enable profiling
adamstruck Dec 6, 2019
6815731
addressing PR comments
adamstruck Dec 6, 2019
c412c8d
lint
adamstruck Dec 6, 2019
4cc7482
use badger v2.0.0
adamstruck Dec 9, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updating bulk upload behavior and unit tests
  • Loading branch information
@kellrott
kellrott committed Sep 11, 2019
commit fff653fd31e3fcf214b7819cdd1edf81ca19b08a
31 changes: 30 additions & 1 deletion conformance/tests/ot_bulk.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@ def test_bulkload(O):
bulk.addEdge("6", "3", "created", {"weight": 0.2})
bulk.addEdge("4", "5", "created", {"weight": 1.0})

bulk.execute()
err = bulk.execute()
if err["error_count"] != 0:
print(err)
errors.append("Bulk insertion error")

res = O.query().V().count().execute()[0]
if res["count"] != 6:
Expand All @@ -34,3 +37,29 @@ def test_bulkload(O):
(res["count"], 6))

return errors

def test_bulkload_validate(O):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be worth adding in a unit test that actually tests the batching? IE, lots of vertices or elements with large amounts of attached data?

errors = []

bulk = O.bulkAdd()

bulk.addVertex("1", "Person", {"name": "marko", "age": "29"})
bulk.addVertex("2", "Person", {"name": "vadas", "age": "27"})
bulk.addVertex("3", "Software", {"name": "lop", "lang": "java"})
bulk.addVertex("4", "Person", {"name": "josh", "age": "32"})
bulk.addVertex("5", "Software", {"name": "ripple", "lang": "java"})
bulk.addVertex("6", "Person", {"name": "peter", "age": "35"})

bulk.addEdge("1", None, "created", {"weight": 0.4})
bulk.addEdge("1", "2", "knows", {"weight": 0.5})
bulk.addEdge("1", "4", "knows", {"weight": 1.0})
bulk.addEdge("4", "3", "created", {"weight": 0.4})
bulk.addEdge("6", "3", "created", {"weight": 0.2})
bulk.addEdge("4", "5", None, {"weight": 1.0})

err = bulk.execute()

if err["error_count"] == 0:
errors.append("Validation error not detected")
print(err)
return errors
6 changes: 4 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@ require (
github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9 // indirect
github.com/Shopify/sarama v1.22.1
github.com/bmeg/golib v0.0.0-20170626075926-82a1e1d7a0b2
github.com/bmeg/protoc-gen-grcp-rest-direct v0.0.0-20190228222353-4d40e8b9d305 // indirect
github.com/boltdb/bolt v1.3.1
github.com/ckaznocha/protoc-gen-lint v0.2.1 // indirect
github.com/davecgh/go-spew v1.1.1
github.com/dgraph-io/badger v2.0.0-rc2+incompatible
github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2 // indirect
Expand All @@ -17,12 +19,12 @@ require (
github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8
github.com/go-sourcemap/sourcemap v2.1.2+incompatible // indirect
github.com/go-sql-driver/mysql v1.4.1
github.com/golang/protobuf v1.3.1
github.com/golang/protobuf v1.3.2
github.com/golangci/golangci-lint v1.16.0 // indirect
github.com/graphql-go/graphql v0.7.8
github.com/graphql-go/handler v0.2.3
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0
github.com/grpc-ecosystem/grpc-gateway v1.8.5
github.com/grpc-ecosystem/grpc-gateway v1.11.1
github.com/hashicorp/go-multierror v1.0.0
github.com/imdario/mergo v0.3.7
github.com/jmoiron/sqlx v1.2.0
Expand Down
9 changes: 9 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,12 @@ github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMx
github.com/StackExchange/wmi v0.0.0-20180116203802-5d049714c4a6/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg=
github.com/bmeg/golib v0.0.0-20170626075926-82a1e1d7a0b2 h1:pR4J0Eq9bLu2kRq3Xh3iGDaiOHELIKJLPBhB5K8yTJg=
github.com/bmeg/golib v0.0.0-20170626075926-82a1e1d7a0b2/go.mod h1:UYeNUXxiubPvUu5x/Sm/lIIbdhtkFOAdR0P0NTIQrPc=
github.com/bmeg/protoc-gen-grcp-rest-direct v0.0.0-20190228222353-4d40e8b9d305 h1:jFKpPaxW0TZKlh7u4fYtdzoX+JZV/czsi2j6d8Bh21A=
github.com/bmeg/protoc-gen-grcp-rest-direct v0.0.0-20190228222353-4d40e8b9d305/go.mod h1:1S3ijn0o+w7RmSdfX3EQttlrF9csSziK8dpeh0nP4LU=
github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4=
github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps=
github.com/ckaznocha/protoc-gen-lint v0.2.1 h1:wP+SgbHat4ovpPQayCroxK/1pXtnBR4HIo9G+2gTnHU=
github.com/ckaznocha/protoc-gen-lint v0.2.1/go.mod h1:EveTCMo4KBPAmWqVxMXUDrI/iV6v93ydJyZVdEYyFIg=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
Expand Down Expand Up @@ -58,11 +62,14 @@ github.com/go-toolsmith/strparse v0.0.0-20180903215201-830b6daa1241/go.mod h1:YI
github.com/go-toolsmith/typep v0.0.0-20181030061450-d63dc7650676/go.mod h1:JSQCQMUPdRlMZFswiq3TGpNp1GMktqkR2Ns5AIQkATU=
github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
Expand Down Expand Up @@ -92,6 +99,8 @@ github.com/grpc-ecosystem/go-grpc-middleware v1.0.0 h1:Iju5GlWwrvL6UBg4zJJt3btmo
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
github.com/grpc-ecosystem/grpc-gateway v1.8.5 h1:2+KSC78XiO6Qy0hIjfc1OD9H+hsaJdJlb8Kqsd41CTE=
github.com/grpc-ecosystem/grpc-gateway v1.8.5/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/grpc-ecosystem/grpc-gateway v1.11.1 h1:/dBYI+n4xIL+Y9SKXQrjlKTmJJDwCSlNLRwZ5nBhIek=
github.com/grpc-ecosystem/grpc-gateway v1.11.1/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.0.0 h1:iVjPR7a6H0tWELX5NxNe7bYopibicUzc7uPribsnS6o=
Expand Down
8 changes: 4 additions & 4 deletions gripql/gripql.pb.dgw.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading