CPU Usage to 100%

[motleycrew]

I am using phantom-pool which interns uses generic-pool.

After running few hundered urls, phantomJs CPU usage shoots up to 100% and instances are not killed. Is anybody else seeing this? I have to manually restart the pool. Is there a better way to fix this?

Is there a way to kill old processes let say after 1 minute which have stopped responding?

My current configuration bellow:

self.pool = phantomPool.default({
max: 5, // default
min: 1, // default
// how long a resource can stay idle in pool before being removed
idleTimeoutMillis: 30000, // default.
// maximum number of times an individual resource can be reused before being destroyed; set to 0 to disable
maxUses: 10, // default
// function to validate an instance prior to use; see https://github.com/coopernurse/node-pool#createpool
validator: () => Promise.resolve(true), // defaults to always resolving true
// validate resource before borrowing; required for maxUses andvalidatortestOnBorrow: true, // default // For all opts, see opts at https://github.com/coopernurse/node-pool#createpool phantomArgs: [phantomArgs, { logLevel: 'error', }], // arguments passed to phantomjs-node directly, default is[]`. For all opts, see https://github.com/amir20/phantomjs-node#phantom-object-api

[t3hmrman]

I think I might have found something that might explain this -- you need to make sure your create promise does not fail.

I just spent some time debugging this issue, and it looks like node-pool will retry create in a tight loop, so if your pool does something expensive (which is why you're pooling in the first place), it will cause memory to run away.

You need to set a handler for factoryCreateError -- if you don't, the create just gets retried endlessly. I added some code to stop the pool:

        this.pool.on("factoryCreateError", err => {
            this.poolErrors.push(err);
            // Stop the pool if resource allocation fails too many times
            if (this.poolErrors.length > this.maxPoolErrors) {
                this.emit("error", new Errors.ResourceAllocationFailureLimitExceeded());
                this.disconnect(); // stops the pool and does some other stuff
            }
        });

[alolis]

It seems that others (including myself) have this issue if you search for "infinite" in Issues

[ImHype]

I think it is a good default behavior which can solve this problem.

this._pool.on('factoryCreateError', err => {
       this._pool._waitingClientsQueue.dequeue().reject(err);
});

[mattfysh]

this._pool.on('factoryCreateError', err => {
       this._pool._waitingClientsQueue.dequeue().reject(err);
});

This doesn't seem safe as there's no guarantee the error came from the first waiting client in the queue.

[RogerKang]

this._pool.on('factoryCreateError', err => {
       this._pool._waitingClientsQueue.dequeue().reject(err);
});

This doesn't seem safe as there's no guarantee the error came from the first waiting client in the queue.

In my understanding, _waitingClientsQueue should be a list waiting for new clients which being creating, when an error is thrown in creating, give this error as rejection in any item in this _waitingClientsQueue should be ok to notice that this waiting will not succeed due to this create error. And naturally, pick the first one from the queue to reject will be the common way.