Infinite Time-Waiting Threads in OrientDB Client while Adding Edges #1636
Assignees
Milestone
Comments
ghost
assigned 
|
it's so rare receiving a very detailed issue like your, thanks! :-) |
|
Please could you try the same against a "local" database to see if it's the remote protocol the problem? |
|
I tested the code using a local connection this time not only I got a deadlock reported in JConsole, but also some new exceptions.
I changed the code as follows: private void attempt() throws Exception {
Graph g = new OrientGraph("local:/home/edalorzo/fixme", "admin", "admin");
try {
try {
Set<Vertex> found = Sets.newHashSet(g.getVertices("@class", "Sku"));
for(Vertex skuVertex : found) {
for (String attribute : this.attributes) {
Vertex attrVertex = g.getVertex(attribute);
g.addEdge("class:HashValue", skuVertex, attrVertex, "HasValue");
System.out.printf("Added Edge from %s to %s in thread %s%n",
skuVertex,
attrVertex,
Thread.currentThread().getName());
}
}
} catch (Exception propagate) {
if(g instanceof TransactionalGraph) {
((TransactionalGraph) g).rollback();
}
throw propagate;
}
} finally {
g.shutdown();
}
} |
|
Also tried the same scenario with OrientGraphNoTx: Graph g = new OrientGraphNoTx("local:/home/edalorzo/SandBox/databases/fixme", "admin", "admin");With the unfortunate same results. My tests report the following in the output: And my code gets blocked there. It never reaches the end of execution. Actually this problem is pretty serious. It is preventing us from moving to production. The only workaround I have found is to synchronize all modifications to the database, which is very counter effective in terms of performance. Are there any leads on the cause of this problem or any other workaround that I could use while this gets solved? |
|
@Laa does the new index has improved also the locking management? How can we prevent such situation? |
|
Hi all, May be deadlock is related to ridset (I can see only one thread on On Tue, Sep 10, 2013 at 3:14 PM, Luca Garulli notifications@github.comwrote:
Best regards, Orient Technologies |
|
@Laa The screenshot shows one of the final tests that I did with OrientDBNoTx and local connections, However, above you can find a full thread dump where another type deadlock using remote connections happens. The connection never recovers from this. Not sure if this is the same problem. |
|
Today I took a moment to test this again with OrientGraphNoTx. I have been trying this for hours and I have not been able to reproduce the case. The problem seem to occur only while using OrientGraph. public class OrientDBDeadlockTest {
private static final int THREAD_POOL_SIZE = 30;
private static class AddAttributes implements Callable<Object> {
private final String skuId;
private final String[] attributeIds;
public AddAttributes(String skuId, String[] attributeIds) {
this.skuId = skuId; //skuId names (i.e. "SKU-001")
this.attributeIds = attributeIds; //object ids (i.e. #11:0)
}
@Override
public Object call() {
for (int i = 0; i < 10; i++) {
try {
attempt();
return null;
} catch (OTransactionException | ONeedRetryException ignore) {
System.out.println("Retyring transaction: " + ignore.getMessage());
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
System.out.println("Exhausted number of attempts");
return null;
}
private void attempt() throws Exception {
OrientGraphNoTx g = new OrientGraphNoTx("remote:localhost/fixme", "admin", "admin");
try {
try {
Vertex skuVertex = getSkuVertex(skuId, g);
for (String attribute : this.attributeIds) {
Vertex attrVertex = g.getVertex(attribute);
g.addEdge("class:HashValue", skuVertex, attrVertex, "HasValue");
System.out.printf("Added Edge from %s to %s in thread %s%n",
skuVertex,
attrVertex,
Thread.currentThread().getName());
}
} catch (Exception propagate) {
g.rollback();
throw propagate;
}
} finally {
g.shutdown();
}
}
private Vertex getSkuVertex(String sku, Graph g) {
for(Vertex v : g.getVertices("Sku.sku", sku)){
return v;
}
throw new AssertionError("Sku vertex for "+ sku +" should have been found");
}
}
public static void main(String[] args) throws Exception {
String[] skus = new String[20];
for(int i=0; i < 20; i++) {
skus[i] = String.format("SKU-%03d", i+1);
}
String[] attributes = new String[16];
for(int i=0; i <= 15; i++) {
attributes[i] = String.format("#11:%d", i);
}
OGlobalConfiguration.dumpConfiguration(System.out);
Collection<Callable<Object>> tasks = new ArrayList<>();
for (String sku : skus) {
tasks.add(new AddAttributes(sku, attributes));
}
ExecutorService service = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
try {
List<Future<Object>> results = service.invokeAll(tasks);
for (Future<Object> result : results) {
result.get();
}
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
System.out.println("Done!");
System.exit(0); //this line is never reached.
}
} |
|
While using OrientGraph the last thing reported in the console before going into infinite time waiting is this
So, I guess the retrying of the transaction is the problem. This happens despite the fact that I have mvcc=false in both the server and the client. |
|
This is pretty weird, specially since v0 is equals to v0! Let me try it... |
|
Tried. I found a minor problem in the exception just fixed. The code seems to run well now but I'm not sure that fixed this problem. Could you retry it please? My output: |
|
Don't disable MVCC unless you want to have a corrupted db! |
|
Thanks @lvca I retested the scneario using OrientDB 1.6.0 SNAPSHOT and it looks like the problem has been corrected. This are great news, we deployed in production last week and this bug was giving me a hard time. I did not know that disabling mvcc could corrupt the database. If this is the case, the flag should not even exist. Also, disabling mvcc is one of the recommended ways to deal with concurrency exceptions mentioned in the wiki under Troubleshooting Java. Perhaps this recommendation should be updated to avoid people disable mvcc as a mechanism to deal with this problem. |
|
You're right I've removed that line in Troubleshooting Java. I know very old applications that are running in that mode because they didn't manage correctly the concurrency. I agree with you that flag should be turned-off. |
|
@lvca hi ,I have a problem,could you give me some advise? I set the db.pool.min and db.pool.max in the orientdb-server-config.xml , |
|
@NLPScott what's your problem and what are you trying to accomplish? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Overview
My OrientDB client freezes in a very simple concurrent scenario Evidence shows several of my threads get into waiting state. They are waiting on one or more OrientDB threads that went into timed-waiting state and never got out of it.
Following the evidence and the description on how to reproduce the problem.
System Information
Downloads
Preconditions
For experimantation purposes I also I tried the settings below in certain occasions to see if I would get different results:
However, I got the same results of thread-locking problems.
The Failing Scenario
The failing scenario consists simply in using a set of threads to add all attributes to a given SKU. Therefore the code will simply add edges to the database.
This is what my threads do. It looks for an SKU vertex, and then iterates over a collection of attribute IDs, which I look up and then add an edge from the Sku vertex to the attribute vertex.
This is my main method. This code freezes invariably every time I run it:
Initial Diagnosis of the Problem
I ran this test several times and analyzed what was going on using both JConsole and JVisual VM.
Invariably, it seems that several threads get into TIME_WAITING state and never come out of it:
In fact, I found several other threads in this state. They never get out of it. The rest of the treads get blocked by these threads:
The text was updated successfully, but these errors were encountered: