Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Something went wrong. cron reached maximum iterations #478

Closed
mtjn opened this issue Mar 17, 2020 · 24 comments
Closed

Error: Something went wrong. cron reached maximum iterations #478

mtjn opened this issue Mar 17, 2020 · 24 comments

Comments

@mtjn
Copy link

mtjn commented Mar 17, 2020

Time Zone: "" - Cron String: 0 0,30 * * * * - UTC offset: +00:00 - current Date: Mon Mar 16 2020 16:31:21 GMT+0000

cron@1.8.2
@types/cron@1.7.2

stacktrace:

at CronTime._getNextDateFrom (/usr/src/app/node_modules/cron/lib/cron.js:235:12)
at CronTime.sendAt (/usr/src/app/node_modules/cron/lib/cron.js:156:17)
at CronTime.getTimeout (/usr/src/app/node_modules/cron/lib/cron.js:175:29)
at CronJob.start (/usr/src/app/node_modules/cron/lib/cron.js:613:31)
at Timeout.callbackWrapper [as _onTimeout] (/usr/src/app/node_modules/cron/lib/cron.js:665:29)
at listOnTimeout (internal/timers.js:549:17)
at processTimers (internal/timers.js:492:7)

index.ts:

import cron from "cron";

...

cron.job("0 0,30 * * * *", () => task(arg), undefined, true);
@ghost
Copy link

ghost commented Mar 28, 2020

yes, it reappeared, though solved in 1.7 one year ago

cron@1.8.2

Error: Something went wrong. cron reached maximum iterations. 						
Please open an  issue (https://github.com/kelektiv/node-cron/issues/new) and provide the following string 	
					
Time Zone: "" - Cron String: 39 0,15,30,45 * * * * - UTC offset: +00:00 - current Date: Sat Mar 28 2020 13:30:45 GMT+0000     

at CronTime._getNextDateFrom (/app/node_modules/cron/lib/cron.js:235:12)     
at CronTime.sendAt (/app/node_modules/cron/lib/cron.js:156:17)     
at CronTime.getTimeout (/app/node_modules/cron/lib/cron.js:175:29)     at CronJob.start (/app/node_modules/cron/lib/cron.js:613:31)     
at Timeout.callbackWrapper [as _onTimeout] (/app/node_modules/cron/lib/cron.js:665:29)     
at listOnTimeout (internal/timers.js:531:17)     
at processTimers (internal/timers.js:475:7)

@mikelam14
Copy link

Same. I am also using cron@1.8.2.

@mikelam14
Copy link

I realized the error would show whenever my CPU utilization is too high (>9x%). That might cause some delay of code and affect something.

@ghost
Copy link

ghost commented Mar 30, 2020

agree, happens when load goes high

@GitUser0001
Copy link

Same for me, using @1.8.2

@leonardfactory
Copy link

Happens to me too, Time Zone: "" - Cron String: 0,15,30,45 * * * * * - UTC offset: +00:00 - current Date: Sat Apr 11 2020 08:21:55 GMT+0000

@bs32g1038
Copy link

Happens to me too, Time Zone: "" - Cron String: 0 0,10,20,30,40,50 * * * * - UTC offset: +00:00 - current Date: Thu Apr 16 2020 18:32:09 GMT+0000

@rusekr
Copy link

rusekr commented Apr 17, 2020

Yes, got this error too after some time after upgrade to 1.8. Before it version 1.7 worked perfectly over more one year on different installations.
Got on high load.

@rusekr
Copy link

rusekr commented Apr 17, 2020

Thinking (but not tested) it related to this commit:
e1b392f
Can anyone test without it?

Reverted back to 1.7.2 for now.

@leonardfactory
Copy link

leonardfactory commented Apr 17, 2020

Maybe with setTimeout previous implementation, even if CPU stalled, setTimeout stalled too so didn't update the findingTimeout variable? Not sure about this, just my 2 cents

Calling setTimeout will add a message to the queue after the time passed as second argument. If there is no other message in the queue, the message is processed right away; however, if there are messages, the setTimeout message will have to wait for other messages to be processed. For that reason the second argument indicates a minimum time and not a guaranteed time.

@phips28
Copy link

phips28 commented Apr 19, 2020

I have the same problem, any fix for that?

@rusekr
Copy link

rusekr commented Apr 19, 2020

I have the same problem, any fix for that?

Revert to 1.7.2 for now. For me 1.7.2 was bugless.

@miguelarios
Copy link

How does one upgrade from 1.5? On docker hub for NodeRed/node-red docker images i can’t see anything tagged 1.7

@knolleary
Copy link

@mar005 this issue is related to the cron module and the discussions around version numbers related to the cron module, not the version of any application that is using the cron module internally.

@miguelarios
Copy link

Ah ok thank you for clarifying. How would I go about finding or applying this workaround then? Apologies for not knowing.

knolleary added a commit to node-red/node-red that referenced this issue Apr 24, 2020
@ncb000gt
Copy link
Member

I'm curious about this because it is obviously affecting a bunch of projects. If it's truly due to cpu load, then can that cpu load be reduced/offloaded, or the jobs placed on a different machine? Would you rather the job run even if the task execution is a little fuzzy? For instance, would it be better to have an option to say runWhenPossible that "does its best" even if it doesn't run exactly when specified (in the case of cpu load) or would it be better to have this strictness in there? What about a warning instead that outputs but doesn't kill the process?

@rusekr what makes you suspect the commit you pointed to? The biggest change there is that rather than having an additional timer running, we just grab the time before the while loop and then just check against the current time. It's possible that in the prior cases, when cpu load was high, the timeout was affected and caused that timeout to delay as well. This makes the execution a little more strict.

@knolleary cool project. :)

@rusekr
Copy link

rusekr commented Apr 27, 2020

@ncb000gt From my perspective - used this module in production high load systems since 2016 and not got critical (uncatchable?) error even once and this module worked like a charm. No need to catch this errors and handle this.
If it slowed down on critical loads more then 5 secs it was minor. It wasn't bothered me. No critical cron task requires this precision. Nor task can`t be critical if it runs as frequent as 5 secs or less. Use event system and buses if you need such precision/speed.

Now after not major and not breaking update of this module my projects start to crash and you suggest me update hardware on all my (future) productions to what specs? Place additional monitoring, restart handlers? Patch on all projects such type of minor error that can be a just warning (where to place the handlers?)? It's rediculous.

For commit - you answered yourself - it was only commit from 1.7.2 to current version that changed related to error behaviour. Therefore was my suggestion of cause.

@rusekr
Copy link

rusekr commented Apr 27, 2020

And not mention causes - it for now is bug. If it can't be reverted or fixed suggesting 2.0.0 version of component with breaking changes and example of catching and handling such error.

@ghost
Copy link

ghost commented May 4, 2020

Happening to me as well:

v1.8.2

/var/www/node_modules/cron/lib/cron.js:235
					throw new Error(
					^

Error: Something went wrong. cron reached maximum iterations.
						Please open an  issue (https://github.com/kelektiv/node-cron/issues/new) and provide the following string
						Time Zone: "" - Cron String: 0 * * * * * - UTC offset: +00:00 - current Date: Mon May 04 2020 01:12:09 GMT+0000
    at CronTime._getNextDateFrom (/var/www/node_modules/cron/lib/cron.js:235:12)
    at CronTime.sendAt (/var/www/node_modules/cron/lib/cron.js:156:17)
    at CronTime.getTimeout (/var/www/node_modules/cron/lib/cron.js:175:29)
    at CronJob.start (/var/www/node_modules/cron/lib/cron.js:613:31)
    at Timeout.callbackWrapper [as _onTimeout] (/var/www/node_modules/cron/lib/cron.js:665:29)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! sparkly-api@1.0.0 start: `node .`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the sparkly-api@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-05-04T01_12_14_983Z-debug.log

[Edit]
I have 4 docker containers running on the server, each for different project but same structure and same crons. Only one of the docker crashes, others are running fine. Can this be a reason for crash?

ncb000gt added a commit that referenced this issue May 27, 2020
… hard error isn't required.

Signed-off-by: Nick Campbell <nicholas.j.campbell@gmail.com>
@ncb000gt
Copy link
Member

I pushed up a PR with some changes to allow a "soft warning". Can you please test and report back?

@rusekr
Copy link

rusekr commented May 27, 2020

Great PR - it converts bug to feature, but if default for softTimeout is false - it is breaking change (with main commit caused this issue). And after upgrade to newer versions people must add this config in all cronjobs.

Meanwhile - even with softTimeout turned on this warning may occur in between server syncing time with ntp or changin from/to dst. Fot this js now have: https://developer.mozilla.org/en-US/docs/Web/API/Performance/now and some examples https://developers.google.com/web/updates/2012/08/When-milliseconds-are-not-enough-performance-now
Page on nodejs https://nodejs.org/api/perf_hooks.html#perf_hooks_performance_now

May be use this instead?

@ncb000gt
Copy link
Member

@rusekr

Yea. I hear you re: still occurring.

This pr is to hopefully allow the developer to make the decision at to whether ms precision is necessary or not rather than throwing an exception.

I can take care of the warning showing at all later. Thanks for the docs.

As for the first point, re: default of false. I think you're right. I'll default it to true.

ncb000gt added a commit that referenced this issue May 28, 2020
… hard error isn't required.

Signed-off-by: Nick Campbell <nicholas.j.campbell@gmail.com>
ncb000gt added a commit that referenced this issue May 28, 2020
… hard error isn't required.

Signed-off-by: Nick Campbell <nicholas.j.campbell@gmail.com>
ncb000gt added a commit that referenced this issue Jun 2, 2020
… hard error isn't required.

Signed-off-by: Nick Campbell <nicholas.j.campbell@gmail.com>
@P4sca1

This comment was marked as spam.

@intcreator
Copy link
Collaborator

closing in favor of #467

@intcreator intcreator closed this as not planned Won't fix, can't repro, duplicate, stale Feb 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests