Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KV BulkWrite #223

Merged
merged 45 commits into from
Dec 9, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
6b5e2af
Changing KV index code so that it doesn't do a 'set(get+1)' increment…
kellrott Jul 13, 2019
0b934bf
Adding bulk write interface to KV interface. It only does something d…
kellrott Jul 16, 2019
db0f614
Fixing bug in the aggrigation processor.
kellrott Jul 16, 2019
639abcf
Removing RocksDB from driver list and testing
kellrott Jul 16, 2019
920e365
Adding rate counter to KVLoading logging
kellrott Jul 19, 2019
55b75f6
Adding a BulkAdd method to all of the drivers. Reworking the server A…
kellrott Jul 19, 2019
3079c11
Merge remote-tracking branch 'origin/master' into kvindex-bulkwrite
kellrott Jul 19, 2019
7b062a6
Making kvgraph bulk add use the bulk transaction
kellrott Jul 19, 2019
b6e00c2
Removing redundant code
kellrott Jul 19, 2019
8fa93a8
Fixing lint issue
kellrott Jul 19, 2019
b7a8598
Removing redundant code
kellrott Jul 22, 2019
cd0ed62
Having internal bulkAdd call errors get copied if if they occur
kellrott Jul 23, 2019
c9b3e09
Addressing issues raised in PR
kellrott Sep 11, 2019
e5241d0
Merge remote-tracking branch 'origin/master' into kvindex-bulkwrite
kellrott Sep 11, 2019
c49018b
Adding some tests related to StreamBatch method
kellrott Sep 11, 2019
02cb246
Adding graph name validation in SteamBatch
kellrott Sep 11, 2019
fff653f
Updating bulk upload behavior and unit tests
kellrott Sep 11, 2019
13b748c
Cleaning linting issues
kellrott Sep 11, 2019
0b358ef
Updating badger link
kellrott Sep 11, 2019
755bb93
Fixing lint issue
kellrott Sep 11, 2019
86a7bd5
Fixing flake issue
kellrott Sep 11, 2019
ff79aa3
Fixing offline KV bulk command line loader to use new bulk loading API
kellrott Sep 12, 2019
cb30d36
Adding some error logging
kellrott Sep 13, 2019
6a6b38f
Adding logging edge insert rates
kellrott Sep 13, 2019
1a05b77
more error handling for BulkAdd
adamstruck Sep 17, 2019
d0440f1
cmd/kvload: cleanup
adamstruck Sep 24, 2019
02c2fb7
pass logrus logger to badger
adamstruck Sep 24, 2019
c353877
cleanup badger opts
adamstruck Sep 24, 2019
3bc6496
kvgraph: fix error handling in BulkAdd
adamstruck Sep 24, 2019
891e204
wip
adamstruck Sep 26, 2019
43b0361
create grip logger package, allows passing configured logger to subpr…
adamstruck Oct 1, 2019
0ff4dab
set logger to debug for kvload
adamstruck Oct 1, 2019
390b6b1
add a log sub method
adamstruck Oct 1, 2019
b037525
log: fix log level in methods
adamstruck Oct 1, 2019
f958057
updated file stream methods
adamstruck Nov 22, 2019
944711e
increase buffer size used in StreamFile
adamstruck Nov 22, 2019
d0cac8b
debug wip
adamstruck Nov 22, 2019
c1a64fc
Merge remote-tracking branch 'origin/master' into kvindex-bulkwrite-as
adamstruck Dec 3, 2019
773dd0e
Merge branch 'kvindex-bulkwrite-as' of github.com:bmeg/grip into kvin…
adamstruck Dec 3, 2019
0e6805e
fix broken tests
adamstruck Dec 4, 2019
27765ed
server: fix BulkAdd
adamstruck Dec 5, 2019
b0f2008
added global --pprof flag to enable profiling
adamstruck Dec 6, 2019
6815731
addressing PR comments
adamstruck Dec 6, 2019
c412c8d
lint
adamstruck Dec 6, 2019
4cc7482
use badger v2.0.0
adamstruck Dec 9, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
create grip logger package, allows passing configured logger to subpr…
…ocesses
  • Loading branch information
adamstruck committed Oct 1, 2019
commit 43b03614be664380e146bdcbff368d9364176bae
4 changes: 2 additions & 2 deletions cmd/kvload/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ import (
"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/kvgraph"
"github.com/bmeg/grip/kvi"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/util"
"github.com/paulbellamy/ratecounter"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)

Expand Down Expand Up @@ -106,7 +106,7 @@ var Cmd = &cobra.Command{

edgeCounter := ratecounter.NewRateCounter(10 * time.Second)
for _, edgeFile := range edgeFileArray {
log.Printf("Loading %s", edgeFile)
log.Infof("Loading %s", edgeFile)
count := 0
for e := range util.StreamEdgesFromFile(edgeFile) {
graphChan <- &gripql.GraphElement{Graph: graph, Edge: e}
Expand Down
2 changes: 1 addition & 1 deletion cmd/load/example/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import (
"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/gripql/example"

"github.com/bmeg/grip/log"
"github.com/bmeg/grip/util/rpc"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)

Expand Down
4 changes: 2 additions & 2 deletions cmd/load/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ import (

"github.com/bmeg/grip/cmd/load/example"
"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/util"
"github.com/bmeg/grip/util/rpc"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)

Expand Down Expand Up @@ -61,7 +61,7 @@ var Cmd = &cobra.Command{
wait := make(chan bool)
go func() {
if err := conn.BulkAdd(elemChan); err != nil {
log.Printf("bulk add error: %v", err)
log.Errorf("bulk add error: %v", err)
}
wait <- false
}()
Expand Down
2 changes: 1 addition & 1 deletion cmd/mongoload/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ import (
"strings"

"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/mongo"
"github.com/bmeg/grip/util"
"github.com/bmeg/grip/util/rpc"
"github.com/globalsign/mgo"
"github.com/globalsign/mgo/bson"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)

Expand Down
2 changes: 1 addition & 1 deletion cmd/rdf/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ import (
"strings"

"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/util/rpc"
"github.com/golang/protobuf/jsonpb"
"github.com/knakk/rdf"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)

Expand Down
4 changes: 2 additions & 2 deletions cmd/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ import (
_ "github.com/bmeg/grip/kvi/badgerdb" // import so badger will register itself
_ "github.com/bmeg/grip/kvi/boltdb" // import so bolt will register itself
_ "github.com/bmeg/grip/kvi/leveldb" // import so level will register itself
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/mongo"
"github.com/bmeg/grip/psql"
"github.com/bmeg/grip/server"
_ "github.com/go-sql-driver/mysql" //import so mysql will register as a sql driver
"github.com/imdario/mergo"
_ "github.com/lib/pq" // import so postgres will register as a sql driver
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)

Expand All @@ -34,7 +34,7 @@ var schemaFile string
// This opens a database and starts an API server.
// This blocks indefinitely.
func Run(conf *config.Config, schemas map[string]*gripql.Graph) error {
config.ConfigureLogger(conf.Logger)
log.ConfigureLogger(conf.Logger)
log.WithFields(log.Fields{"Config": conf}).Info("Starting Server")

var db gdbi.GraphDB
Expand Down
2 changes: 1 addition & 1 deletion cmd/stream/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ import (

"github.com/Shopify/sarama"
"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/util/rpc"
"github.com/golang/protobuf/jsonpb"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)

Expand Down
5 changes: 3 additions & 2 deletions config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (

"github.com/bmeg/grip/elastic"
esql "github.com/bmeg/grip/existing-sql"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/mongo"
"github.com/bmeg/grip/psql"
"github.com/bmeg/grip/server"
Expand All @@ -35,7 +36,7 @@ type Config struct {
MongoDB mongo.Config
PSQL psql.Config
ExistingSQL esql.Config
Logger Logger
Logger log.Logger
}

// DefaultConfig returns an instance of the default configuration for Grip.
Expand Down Expand Up @@ -65,7 +66,7 @@ func DefaultConfig() *Config {
c.Elasticsearch.DBName = "gripdb"
c.Elasticsearch.BatchSize = 1000

c.Logger = DefaultLoggerConfig()
c.Logger = log.DefaultLoggerConfig()
return c
}

Expand Down
2 changes: 1 addition & 1 deletion elastic/graph.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ import (
"github.com/bmeg/grip/engine/core"
"github.com/bmeg/grip/gdbi"
"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/timestamp"
"github.com/bmeg/grip/util"
"github.com/golang/protobuf/jsonpb"
log "github.com/sirupsen/logrus"
"golang.org/x/sync/errgroup"
elastic "gopkg.in/olivere/elastic.v5"
)
Expand Down
2 changes: 1 addition & 1 deletion elastic/graphdb.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ import (

"github.com/bmeg/grip/gdbi"
"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/timestamp"
log "github.com/sirupsen/logrus"
elastic "gopkg.in/olivere/elastic.v5"
)

Expand Down
2 changes: 1 addition & 1 deletion elastic/index.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ import (

"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/jsonpath"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/protoutil"
log "github.com/sirupsen/logrus"
elastic "gopkg.in/olivere/elastic.v5"
)

Expand Down
2 changes: 1 addition & 1 deletion elastic/util.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ package elastic
import (
"context"

log "github.com/sirupsen/logrus"
"github.com/bmeg/grip/log"
elastic "gopkg.in/olivere/elastic.v5"
)

Expand Down
2 changes: 1 addition & 1 deletion engine/core/processors.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ import (
"github.com/bmeg/grip/jsonpath"
"github.com/bmeg/grip/kvi"
"github.com/bmeg/grip/kvindex"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/protoutil"
structpb "github.com/golang/protobuf/ptypes/struct"
log "github.com/sirupsen/logrus"
"github.com/spenczar/tdigest"
"github.com/spf13/cast"
"golang.org/x/sync/errgroup"
Expand Down
2 changes: 1 addition & 1 deletion engine/pipes.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ import (

"github.com/bmeg/grip/gdbi"
"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/protoutil"
log "github.com/sirupsen/logrus"
)

// Start begins processing a query pipeline
Expand Down
4 changes: 2 additions & 2 deletions existing-sql/graph.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ import (
"github.com/bmeg/grip/engine/core"
"github.com/bmeg/grip/gdbi"
"github.com/bmeg/grip/gripql"
"github.com/bmeg/grip/log"
"github.com/bmeg/grip/timestamp"
"github.com/jmoiron/sqlx"
log "github.com/sirupsen/logrus"
)

// Graph is the interface to a single graph
Expand Down Expand Up @@ -194,7 +194,7 @@ func (g *Graph) VertexLabelScan(ctx context.Context, label string) chan string {
data := make(map[string]interface{})
if err := rows.MapScan(data); err != nil {
log.WithFields(log.Fields{"error": err}).Error("VertexLabelScan: MapScan")
log.Println("VertexLabelScan failed:", err)
log.Errorln("VertexLabelScan failed:", err)
return
}
v := rowDataToVertex(v, data, types, false)
Expand Down
Loading