How to solve Neo4jError: read ECONNRESET

[rjgmail88]

Since we started using this driver we are getting following error frequently.

{ Neo4jError: read ECONNRESET
    at new Neo4jError (D:\home\site\wwwroot\node_modules\neo4j-driver\lib\v1\error.js:76:132)
    at newError (D:\home\site\wwwroot\node_modules\neo4j-driver\lib\v1\error.js:66:10)
    at NodeChannel._handleConnectionError (D:\home\site\wwwroot\node_modules\neo4j-driver\lib\v1\internal\ch-node.js:330:41)
    at emitOne (events.js:121:20)
    at TLSSocket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9) code: 'ServiceUnavailable', name: 'Neo4jError' }

Is there any way to fix this ? We are using connectionPoolSize = 0 while creating driver. Also using nodejs promise way run the neo4j queries and doing session.close() every time.

[lutovich]

Hi @rjgmail88,

Such errors happen when either network equipment or the database closes the connection.

I think retrying the query/transaction is a good thing to do if errors are caused by the network equipment/stack. You could use transaction functions API which can retry with exponential backoff on such errors. More info and usage patterns can be found in the developer manual.

Database might close connection because of unrecoverable errors (out of memory, protocol violation, etc). Could you please attach the database logs neo4j.log and debug.log to this issue so we can take a look?

[lutovich]

@rjgmail88 couple more questions:

• what version of the neo4j are you using?
• why do you set the connection pool size to zero?

[rjgmail88]

@lutovich , sorry for the delayed response. I will be attaching log files soon.

1. neo4j-driver version 1.5.0

2.Neo4j
Version: | 3.3.5
Edition: | Community

3. We couldn't decide on right number for pool size. We have a chat bot which is using neo4j to fetch some answers user is asking. This bot has been actively used by our Cerner associates (around 10K users for now. and eventually will be 18K+ ) . We thought setting it 0 will be helpful so we don't need to encounter scenario where pool will exhaust.

[ganeshchippada]

I attached the logs. We are getting the below errors while creating connections.

2018-06-19 14:43:06.665+0000 ERROR [o.n.b.t.SocketTransportHandler] Fatal error occurred when handling a client connection: [id: 0x0b223092, L:/10.0.5.4:7687 - R:/13.84.153.1:58608] syscall:read(..) failed: Connection reset by peer
io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)
2018-06-19 14:43:09.675+0000 ERROR [o.n.b.t.SocketTransportHandler] Fatal error occurred when handling a client connection: [id: 0xca3201d8, L:/10.0.5.4:7687 - R:/13.84.153.1:58616] syscall:read(..) failed: Connection reset by peer
io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)
2018-06-19 14:43:15.676+0000 ERROR [o.n.b.t.SocketTransportHandler] Fatal error occurred when handling a client connection: [id: 0x376ecc4b, L:/10.0.5.4:7687 - R:/13.84.153.1:58600] syscall:read(..) failed: Connection reset by peer
io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)
2018-06-19 14:45:18.970+0000 ERROR [o.n.b.t.SocketTransportHandler] Fatal error occurred when handling a client connection: [id: 0x7989f0e3, L:/10.0.5.4:7687 - R:/13.84.153.1:58626] syscall:read(..) failed: Connection reset by peer
io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)
debug.log

[rjgmail88]

@lutovich , log files have been posted. Please let us know if you can help.

{ Neo4jError: read ECONNRESET
at new Neo4jError (D:\home\site\wwwroot\node_modules\neo4j-driver\lib\v1\error.js:76:132)
at newError (D:\home\site\wwwroot\node_modules\neo4j-driver\lib\v1\error.js:66:10)
at NodeChannel._handleConnectionError (D:\home\site\wwwroot\node_modules\neo4j-driver\lib\v1\internal\ch-node.js:330:41)
at emitOne (events.js:121:20)
at TLSSocket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickDomainCallback (internal/process/next_tick.js:218:9) code: 'ServiceUnavailable', name: 'Neo4jError' }

form above stack trace I'm confused if its Service unavailable error mentioned bellow.

https://neo4j.com/docs/developer-manual/current/drivers/client-applications/#driver-configuration-max-retry-time

[rjgmail88]

From my local instance of nodejs app I am seeing different stack trace for same scenario.

Error: read **ETIMEDOUT**
CernerCampusHelper.js:21
code:"**ServiceUnavailable**"
message:"read ETIMEDOUT"
name:"Neo4jError"
stack:"Neo4jError: read ETIMEDOUT\n    at new Neo4jError .......

[lutovich]

@rjgmail88 @ganeshchippada debug.log shows that it is definitely not the database closing all those connections. There are many "Connection reset by peer" errors in the log which most likely means that something in-between driver and database has closed the connection. Such connections appear to be broken for both driver and database. Does there exist a load balancer or a proxy server in your network?

Provided Neo4jError: read ECONNRESET with ServiceUnavailable status code is the error described in the developer manual. Please note that mentioned retires only happen when transaction functions are used.

Could you please also share the code that creates the driver and executes queries? It would be valuable to see which API is used for query execution (Session#run() or Session#beginTransaction() or Session#read[write]Transaction()). Do you maintain a single driver per application?

[rjgmail88]

Hi @lutovich ,

Does there exist a load balancer or a proxy server in your network?

Ans:- No. We are using public Neo4j VM IP with Bolt url.

Here is how we consume API's

setup.js holds code to create and expose driver object. getGraphDriver()

const neo4jD = require('neo4j-driver').v1;

const driver = neo4jD.driver(bolt://00.000.000.00:1212, neo4jD.auth.basic(USER,PASS),
        {
          connectionPoolSize: 0
        });

 getGraphDriver: function () {
      return driver;
     }

consume of driver is in index.js file in getCampuData() where we run queries using session. We use promise way mentioned in the document.

var neo4jConnection = require('../../setup.js');
var graphDBDriver = new neo4jConnection.getGraphDriver();

   getCampuData: function (campusName) {
        let session = graphDBDriver.session();
        let deferred = Q.defer();
        let query = function () {
            let returnResults = [];
            session
                .run('MATCH (c:Campus) WHERE c.name =~ {campusName} return c',
                    { campusName: '(?i).*' + campusName + '.*' })
                .then(function (result) {
                    result.records.forEach(function (record) {
                        if (record && record.length >= 1) {
                            if (record.get("c").properties) {
                                returnResults.push(record.get("c").properties.address);
                            } else {
                                returnResults
                            }
                        }
                    });
                    session.close();
                    return deferred.resolve(returnResults);
                    
                })
                .catch(function (error) {
                    session.close();
                    console.log(" Neo4j error from getCampuData: " + error);
                    return deferred.reject(error);
                });
        }
        query();
        return deferred.promise;
    }

One more think I would like to mention. I setup connection pool size to 10 yesterday and I was able to reproduce the error in local as well as from azure cloud app. So it might not re related to infrastructure.

[rjgmail88]

@lutovich,

As mentioned in the doc here

const session = driver.session();
const writeTxPromise = session.writeTransaction(tx => tx.run('CREATE (a:Person {name: $name})', {'name': personName}));

writeTxPromise.then(result => {
  session.close();

  if (result) {
    console.log('Person created');
  }
});

I am still able to see Neo4jError: read ECONNRESET and retry isn't working with Transaction functions.

[rjgmail88]

Is there any way to find if ReTry happened ?

[lutovich]

Hi @rjgmail88,

Your code with Session#run() looks correct to me. It does not execute retries, only transaction functions Session#readTransaction() and Session#writeTransaction() perform retries.

Right now there is no built-in way to log retries. We will add it to 1.7 version of the driver. You could insert logging of retries in the function itself:

const session = driver.session();
let retryCount = 0;
const writeTxPromise = session.writeTransaction(tx => {
  if (retryCount === 0) {
    console.log('Executing query for the first time');
  } else {
    console.log('Retrying query: ' + retryCount);
  }
  retryCount++;
  return tx.run('CREATE (a:Person {name: $name})', {'name': personName});
});

writeTxPromise.then(result => {
  session.close();

  if (result) {
    console.log('Person created');
  }
});

By default driver will rety for 30 seconds. This time is configurable via maxTransactionRetryTime config setting.

You can experience these errors even locally right? Could you please provide a reproducer application that I can run locally?

[rjgmail88]

Yes, I can reproduce locally as well. I have refactored my code to use transaction api's. I will test for couple days and see how it goes. If that fixes the issue we should be good.

[rjgmail88]

We are good on this. Thank you.