1.7: issues with create edge or create class on distributed configuration #2389

tanji · 2014-05-26T23:57:26Z

Hello,

on latest 1.7, CREATE EDGE on Graph schema causes a timeout and several warnings. Even though creation succeeds in the end it will timeout first (10 seconds).

Server version: OrientDB Server v1.7-SNAPSHOT (build UNKNOWN@r; 2014-05-26 13:56:37-0400)
Ubuntu 14.04 64-bit, with OpenJDK 7
Default Hazelcast configuration, two nodes

Simple test case following.

orientdb {test}> create class Person extends V

Class created successfully. Total classes in database now: 13

orientdb {test}> create class Loves extends E

Class created successfully. Total classes in database now: 14

orientdb {test}> create vertex Person set name = 'xiaodu'

Created vertex 'Person#11:0{name:xiaodu} v1' in 0.029000 sec(s).

orientdb {test}> create vertex Person set name = 'xiaohu'

Created vertex 'Person#11:1{name:xiaohu} v1' in 0.026000 sec(s).

orientdb {test}> create edge Loves from #11:0 to #11:1 set created = sysdate()

[orientdb1] error on reading distributed request: command_sql(create edge Loves from #11:0 to #11:1 set created = sysdate())
Cannot dispatch response to the thread queue orientdb1
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.onMessage(OHazelcastDistributedDatabase.java:500)
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase$1.run(OHazelcastDistributedDatabase.java:243)
-> java.lang.Thread.run(Thread.java:744)
java.lang.NullPointerException
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.onMessage(OHazelcastDistributedDatabase.java:500)
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase$1.run(OHazelcastDistributedDatabase.java:243)
-> java.lang.Thread.run(Thread.java:744)
null
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.onMessage(OHazelcastDistributedDatabase.java:500)
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase$1.run(OHazelcastDistributedDatabase.java:243)
-> java.lang.Thread.run(Thread.java:744)
2014-05-26 23:50:36:124 WARN [orientdb1] timeout (10000ms) on waiting for synchronous responses from nodes=[orientdb2, orientdb1] responsesSoFar=[] request=id=26 from=orientdb1 task=command_sql(create edge Loves from #11:0 to #11:1 set created = sysdate()) [OHazelcastDistributedDatabase]
2014-05-26 23:50:36:127 WARN [orientdb1] detected 2 node(s) in timeout or in conflict and quorum (2) has not been reached, rolling back changes for request: id=26 from=orientdb1 task=command_sql(create edge Loves from #11:0 to #11:1 set created = sysdate()) [ODistributedResponseManager]Cannot route COMMAND operation to the distributed node
Error on sending distributed request against database 'test.[]' to nodes [orientdb2, orientdb1]
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:192)
-> com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.sendRequest(OHazelcastPlugin.java:361)
-> com.orientechnologies.orient.server.distributed.ODistributedStorage.command(ODistributedStorage.java:182)
-> com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:59)
-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1151)
-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:342)
-> com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:169)
-> com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45)
Quorum 2 not reached for request=id=26 from=orientdb1 task=command_sql(create edge Loves from #11:0 to #11:1 set created = sysdate()). No server in conflict. Received: {orientdb2=waiting-for-response, orientdb1=waiting-for-response}
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:192)
-> com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.sendRequest(OHazelcastPlugin.java:361)
-> com.orientechnologies.orient.server.distributed.ODistributedStorage.command(ODistributedStorage.java:182)
-> com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:59)
-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1151)
-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:342)
-> com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:169)
-> com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45)
Error: com.orientechnologies.orient.server.distributed.ODistributedException: Quorum 2 not reached for request=id=26 from=orientdb1 task=command_sql(create edge Loves from #11:0 to #11:1 set created = sysdate()). No server in conflict. Received: {orientdb2=waiting-for-response, orientdb1=waiting-for-response}

On server 2:

[orientdb2]<-[orientdb1] error on reading distributed request: command_sql(create edge Loves from #11:0 to #11:1 set created = sysdate())
Cannot dispatch response to the thread queue orientdb1
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.onMessage(OHazelcastDistributedDatabase.java:500)
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase$1.run(OHazelcastDistributedDatabase.java:243)
-> java.lang.Thread.run(Thread.java:744)
java.lang.NullPointerException
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.onMessage(OHazelcastDistributedDatabase.java:500)
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase$1.run(OHazelcastDistributedDatabase.java:243)
-> java.lang.Thread.run(Thread.java:744)
null
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.onMessage(OHazelcastDistributedDatabase.java:500)
-> com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase$1.run(OHazelcastDistributedDatabase.java:243)

tanji · 2014-05-28T09:53:00Z

Hello, can we have an approximate ETA for the fix of this error? We plan to roll out an application based on OrientDB in production in about one week but this error makes OrientDB Graph Database unusable in distributed mode since there's a huge timeout at each named Edge creation.

Thanks in advance.

lvca · 2014-05-28T09:59:02Z

The line you report the NPE is:

com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.onMessage(OHazelcastDistributedDatabase.java:500)

But I see this line, where it's impossible iRequest is null:

throw new ODistributedException("Cannot dispatch response to the thread queue " + iRequest.getSenderNodeName(), e);

So, please can you repeat the test with last 1.7.1-SNAPSHOT?

tanji · 2014-05-28T10:26:45Z

Hi Luca,

where can I find 1.7.1-SNAPSHOT?
I have been unable to find it on sonatype.

Thanks

enisher · 2014-05-28T11:44:48Z

Hi Tanji,

Sorry, we have not reconfigured a build server to new branching model yet.
It is not pushed to sonatype yet.

To get 1.7.1-SNAPSHOT you can get a 1.7.1 branch with git and execute `ant
installg'

Best regards,
Artem Orobets

Orient Technologiesthe Company behind OrientDB*

2014-05-28 13:26 GMT+03:00 tanji notifications@github.com:

Hi Luca,

where can I find 1.7.1-SNAPSHOT?
I have been unable to find it on sonatype.

Thanks

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2389#issuecomment-44388293
.

lvca · 2014-05-28T11:45:54Z

Waiting for Jenkins I've deployed it manually, so now it's available.

tanji · 2014-05-28T12:46:23Z

Thanks guys. Anyway I've tested with the latest 1.7.1 branch and the error is similar. I don't know why the exception gives out a NPE itself, anyway the problem seems located elsewhere. This is quite a blocker because distributed mode is totally broken when using graph schemas.

Note that creating edges without properties (aka lightweight edges) doesn't break the server.

lvca · 2014-05-28T13:42:35Z

Without the stack trace we can't help you. Please can you provide it?

tanji · 2014-05-28T14:21:44Z

I think the issue is easy enough to reproduce on your own. Just start 2 servers in distributed mode, create edge will property and you will get the error.

I have supplied the full stack trace output above in the initial issue.
Let me know if there's more that I can do.

tanji · 2014-05-30T08:26:23Z

Hello, have you been able to reproduce the issue?
Thanks in advance

lvca · 2014-07-22T17:40:52Z

@tanji We fixed similar issues. Please can you retry with 1.7.7-SNAPSHOT?

lvca · 2014-07-23T10:49:40Z

I'm closing it, in case the error exists with 1.7.7, please comment this to reopen.

tanji · 2014-07-23T19:29:54Z

@lvca There's no more NPE, but the edge is not correctly created: it is not referenced correctly in the Vertex.

----+-----+--------+------------+-----------
#   |@RID |name    |out_distance|in_distance
----+-----+--------+------------+-----------
0   |#16:0|milan   |null        |null       
1   |#16:1|new york|null        |null       
----+-----+--------+------------+-----------

Should be

----+-----+--------+------------+-----------
#   |@RID |name    |out_distance|in_distance
----+-----+--------+------------+-----------
0   |#12:0|milan   |#13:0       |null       
1   |#12:1|new york|null        |#13:0      
----+-----+--------+------------+-----------

Please reopen the issue.

srgg · 2014-07-24T03:22:24Z

@tanji @lvca
here is the same issue #2582

tanji changed the title ~~latest 1.7-SNAPSHOT: issues with create edge or create class on distributed configuration~~ 1.7: issues with create edge or create class on distributed configuration May 27, 2014

enisher added the 1 - Next label May 27, 2014

enisher added this to the 1.7.1 milestone May 27, 2014

enisher added bug labels May 27, 2014

lvca self-assigned this May 28, 2014

lvca modified the milestones: 1.7.2, 1.7.1, 1.7.3 Jun 4, 2014

andrii0lomakin modified the milestones: 1.7.4, 1.7.3 Jun 12, 2014

lvca modified the milestones: 1.7.5, 1.7.4 Jun 23, 2014

lvca modified the milestones: 1.7.6, 1.7.5, 1.7.7 Jul 10, 2014

tanji mentioned this issue Jul 21, 2014

Distributed 1.7 : create edge with property fails with NPE #2292

Closed

lvca closed this as completed Jul 23, 2014

lvca added 6 - Done and removed 2 - Sprint labels Jul 23, 2014

lvca reopened this Jul 23, 2014

lvca modified the milestones: 1.7.8, 1.7.7 Jul 23, 2014

andrii0lomakin closed this as completed Jul 31, 2014

srgg mentioned this issue Aug 6, 2014

(Client-51) Can't properly create not lightweight edge on a distributed cluster of ODB v1.7.7-SNAPSHOT #2582

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1.7: issues with create edge or create class on distributed configuration #2389

1.7: issues with create edge or create class on distributed configuration #2389

tanji commented May 26, 2014

tanji commented May 28, 2014

lvca commented May 28, 2014

tanji commented May 28, 2014

enisher commented May 28, 2014

lvca commented May 28, 2014

tanji commented May 28, 2014

lvca commented May 28, 2014

tanji commented May 28, 2014

tanji commented May 30, 2014

lvca commented Jul 22, 2014

lvca commented Jul 23, 2014

tanji commented Jul 23, 2014

srgg commented Jul 24, 2014

1.7: issues with create edge or create class on distributed configuration #2389

1.7: issues with create edge or create class on distributed configuration #2389

Comments

tanji commented May 26, 2014

tanji commented May 28, 2014

lvca commented May 28, 2014

tanji commented May 28, 2014

enisher commented May 28, 2014

lvca commented May 28, 2014

tanji commented May 28, 2014

lvca commented May 28, 2014

tanji commented May 28, 2014

tanji commented May 30, 2014

lvca commented Jul 22, 2014

lvca commented Jul 23, 2014

tanji commented Jul 23, 2014

srgg commented Jul 24, 2014