Skip to content

Breaking change: cluster connection behavior when between workers #1239

Description

@scottnonnenberg

Already reported this on node 0.12, thought I should report it here as well.

On OSX, I've noticed a big difference between the way that connections are dealt with by a master process when there are no workers ready to take care of that incoming connection. In node 0.10.36 (and before), the connection would be held open, and a worker that hadn't been started when that request was made would have the chance to handle it. In all versions of iojs (and node 0.12.0), incoming connections when between workers are outright refused.

At the very least, this should be documented.

Example code and output on both node 0.10.36 and iojs 1.6.1 follows:

var cluster = require('cluster');
var http = require('http');
var supertest = require('supertest');
var PORT = 3000;

// cluster.schedulingPolicy = cluster.SCHED_NONE;

if (!cluster.isMaster) {
  http.createServer(function (req, res) {
    if (req.url === '/error') {
      setTimeout(function() {
        throw new Error('something went wrong!');
      }, 500);
    }
    else {
      res.writeHead(200, {'Content-Type': 'text/plain'});
      res.end('Hello World\n');
    }
  }).listen(PORT);

  console.log('Worker %s running at port %s', cluster.worker.id, PORT);
}
else {
  var count = 0;
  var request = supertest('http://localhost:' + PORT);

  var hitWorker = function(count) {
    console.log('%s: Worker listening! Hitting it...', count);

    request
      .get('/error')
      .expect(200, function(err, res) {
        console.log('%s: Worker taken down, now making second request', count);

        request
          .get('/')
          .expect('Hello World\n')
          .expect(200, function(err, res) {
            console.log('%s: Second request complete. Error:', count, err);
          });
      });
  };

  cluster.on('disconnect', function() {
    count +=1;
    if (count < 2) {
      cluster.fork();
    }
  });

  cluster.on('listening', function() {
    hitWorker(count);
  });

  // start just one worker
  cluster.fork();

  var interval = setInterval(function() {
    console.log('...');
  }, 1000);
  interval.unref();
}

output

iojs 1.6.1 (scheduling policy does not make a difference):

Worker 1 running at port 3000
0: Worker listening! Hitting it...
/Users/scottnonnenberg/Development/thehelp/cluster/test.js:12
        throw new Error('something went wrong!');
              ^
Error: something went wrong!
    at null._onTimeout (/Users/scottnonnenberg/Development/thehelp/cluster/test.js:12:15)
    at Timer.listOnTimeout (timers.js:88:15)
0: Worker taken down, now making second request
0: Second request complete. Error: { [Error: connect ECONNREFUSED 127.0.0.1:3000]
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 3000 }
Worker 2 running at port 3000
1: Worker listening! Hitting it...
...
/Users/scottnonnenberg/Development/thehelp/cluster/test.js:12
        throw new Error('something went wrong!');
              ^
Error: something went wrong!
    at null._onTimeout (/Users/scottnonnenberg/Development/thehelp/cluster/test.js:12:15)
    at Timer.listOnTimeout (timers.js:88:15)
1: Worker taken down, now making second request
1: Second request complete. Error: { [Error: connect ECONNREFUSED 127.0.0.1:3000]
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 3000 }
...

node 0.10.36:

Worker 1 running at port 3000
0: Worker listening! Hitting it...

/Users/scottnonnenberg/Development/thehelp/cluster/test.js:13
        throw new Error('something went wrong!');
              ^
Error: something went wrong!
    at null._onTimeout (/test.js:13:15)
    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
0: Worker taken down, now making second request
Worker 2 running at port 3000
1: Worker listening! Hitting it...
0: Second request complete. Error: null
...

/Users/scottnonnenberg/Development/thehelp/cluster/test.js:13
        throw new Error('something went wrong!');
              ^
Error: something went wrong!
    at null._onTimeout (/test.js:13:15)
    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
1: Worker taken down, now making second request
...
...
...
...
^C

This version hangs, because third worker not started, and master keeps connection open. Note also that '0: second request complete' actually comes after '1: worker listening!'. This is because that initial second request actually ends up hitting the second worker.

Activity

  1. added
    clusterIssues and PRs related to the cluster subsystem.
    on Mar 23, 2015
  2. benjamingr commented on Mar 25, 2015

    @benjamingr
    Member
  3. rpaterson commented on May 22, 2015

    @rpaterson

    The OP is being too charitable calling this a "Breaking change". IMO this is a serious regression that is preventing us from upgrading from v0.10 to v0.12. One of the main reasons for using a cluster is to provide a high availability service - in v0.12 that is impossible because if all the workers die the clients will see "connection refused".

  4. Fishrock123 commented on Aug 27, 2015

    @Fishrock123
    Contributor

    This commit is the origin of the changed behavior, investigating.

  5. Fishrock123 commented on Aug 27, 2015

    @Fishrock123
    Contributor

    Hmm, does

    • Handles in the master process are now closed when the last worker
      that holds a reference to them quits. Previously, they were only
      closed at cluster shutdown.

    Sound like this issue?

  6. Fishrock123 commented on Aug 27, 2015

    @Fishrock123
    Contributor

    cc @bnoordhuis since he wrote that commit.

  7. bnoordhuis commented on Aug 27, 2015

    @bnoordhuis
    Member

    A documentation issue, perhaps. The change itself is intentional.

    One of the main reasons for using a cluster is to provide a high availability service

    That's not a design goal of the cluster module and never has been.

  8. added
    docIssues and PRs related to Node.js documentation.
    on Aug 27, 2015
  9. self-assigned this
    on Aug 28, 2015
  10. added this to the 4.0.0 milestone on Aug 28, 2015
  11. Fishrock123 commented on Aug 29, 2015

    @Fishrock123
    Contributor

    A docs fix has been landed in 4c5fc3b, thanks for reporting this!

  12. parshap commented on Sep 3, 2015

    @parshap

    @bnoordhuis: Would it be possible to implement the previous behavior in userland?

    I'm thinking something along the lines of having the master process create and bind the socket, but never call accept() on it. Basically I want a require("net").createServer().listen() that never accepts connections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

clusterIssues and PRs related to the cluster subsystem.docIssues and PRs related to Node.js documentation.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions