Support for streaming in Neo4j 2.0?

[bunkat]

Do you have plans to support the new streaming interface for the REST API?

http://docs.neo4j.org/chunked/snapshot/rest-api-streaming.html

[jmcvetta]

No current plans, but it would be nice to have. Any thoughts on the best way to implement?

If you have a client able to fund development, I currently do have bandwidth to work on it. But it's not a personal priority at the moment.

[bunkat]

Looks like there isn't much magic to it.

req, err := http.NewRequest("POST", url, bytes.NewReader(params))
req.Header.Add("content-type", "application/json")
req.Header.Add("accept", "application/json")
req.Header.Add("X-Stream", "true") // this is the new header to add

resp, err := db.client.Do(req)
defer resp.Body.Close()

body, err := ioutil.ReadAll(resp.Body)

Basically this sets the encoding to chunked which the Go http client already handles. In my limited testing, the CPU usage was much lower and returning 75,000 nodes saw about a 25% speedup. Returning a small amount of nodes is a tiny bit slower though.

Since I only use the Cypher interface, I wrote a basic client using net/http directly to test out the streaming stuff. I did a quick benchmark and using Neoism is about 300% slower for the larger queries I was running (25% due to not streaming) and about 35% slower on small queries. I didn't look too much into it but I'm guessing it must be the extra json marshaling.

[freeeve]

It would probably help if the JSON parser were "streaming" capable also, but I think the results actually need to be in a different format to be able to take real advantage of that, maybe. (the ability to marshall each data record as they come in, and call a callback, or something)

[bunkat]

Agree, obviously ioutil.ReadAll is waiting for the last chunk of data to be received before continuing which then gets parsed. To actually get end to end streaming I think you would need to write a Neo4j extension to send the data over the wire in a different format (such that each chunk represents a valid JSON document). However, even with the limitations, chunking still definitely helps since the data is not getting cued up on both the server and client.

[freeeve]

Making a cypher extension that does that actually doesn't seem like a hard problem... hmm. brb.

[jmcvetta]

Most likely it is in fact mostly the JSON parsing that is making neoism slower. The way neoism handles unmarshaling is not particularly efficient - definitely room for improvment. However for insert queries you can skip the JSON handling entirely by leaving CypherQuery.Result nil.

[jmcvetta]

Currently there is no way to tell restclient to use a default set of headers. So adding streaming support would require a bunch of not-too-pretty copypasta. Alternatively, I've been thinking about a partial rewrite of restclient (probably under a less generic name) to make it more closely resemble Python's requests library, which does allow setting default headers.

[jmcvetta]

New underlying HTTP client napping allows streaming support to be set globally.

h := http.Header{}
h.Add("X-Stream", "true")
db, _ := neoism.Connect("http://localhost:7474/db/data")
db.Session.Header = &h

In Neo4j 2.0.0-M03 this causes some behavior changes in the transactional endpoint, inducing test failure. Not sure how this will effect 2.0.0-M05. Will investigate that once #26 is resolved.

[freeeve]

M03 may have a streaming bug--there was one in 1.9.2 as well. Check to see if it is fixed in M05 before troubleshooting, I think.

Also you should add:

h.Add("User-Agent", "neoism")

[jmcvetta]

Added user agent by default in #29. So now we want to add the streaming header to the existing Session.Header rather than creating a new header.

db, _ := neoism.Connect("http://localhost:7474/db/data")
db.Session.Header.Add("X-Stream", "true")

[jmcvetta]

Batched Cypher operations return a different data format when streaming is set: http://docs.neo4j.org/chunked/2.0.0-M05/rest-api-batch-ops.html#rest-api-execute-multiple-operations-in-batch-streaming

It is awkward to handle two different result formats. Is there a use case where it is desirable to execute batch operations without streaming?

[freeeve]

I would say there's no use case for not streaming. I'd be curious to know how much faster streaming is.

[bunkat]

I did a few quick tests before and found that streaming is slightly (1 - 2% on my configuration) slower if you are returning a very small number of results. However, the speedup when returning larger sets and overall reduced CPU is always worth it in my opinion so I've turned it on for all of my queries.