Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception in seyren logs for null data points. #430

Open
gargnisha1605 opened this issue Feb 10, 2017 · 6 comments
Open

Exception in seyren logs for null data points. #430

gargnisha1605 opened this issue Feb 10, 2017 · 6 comments

Comments

@gargnisha1605
Copy link

Following are the logs formed:
Can anyone suggest how to resolve this?

I have used Meter.mark(); to push to graphite

[seyren.check-scheduler-2] WARN c.s.c.service.schedule.CheckRunner - No value present for Fail
09/02/2017 16:44:53.516 [seyren.check-scheduler-0] WARN c.s.c.s.c.GraphiteTargetChecker - {"target":"REQUEST_FAILED","datapoints":[[null,1486638240],[null,1486638300],[null,1486638360],[null,1486638420],[null,1486638480],[null,1486638540],[null,1486638600],[null,1486638660],[null,1486638720],[null,1486638780]]}
[seyren.check-scheduler-0] WARN c.s.c.s.c.GraphiteTargetChecker - REQUEST FAILURE failed to read from Graphite
com.seyren.core.exception.InvalidGraphiteValueException: Could not find a valid datapoint for target: "REQUEST_FAILED"
at com.seyren.core.service.checker.GraphiteTargetChecker.getLatestValue(GraphiteTargetChecker.java:83) ~[seyren-core-1.0.1-SNAPSHOT.jar:na]
at com.seyren.core.service.checker.GraphiteTargetChecker.check(GraphiteTargetChecker.java:54) ~[seyren-core-1.0.1-SNAPSHOT.jar:na]
at com.seyren.core.service.schedule.CheckRunner.run(CheckRunner.java:65) ~[seyren-core-1.0.1-SNAPSHOT.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_30]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_30]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_30]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) ~[na:1.6.0_30]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) ~[na:1.6.0_30]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_30]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_30]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]
[seyren.check-scheduler-0] WARN c.s.c.service.schedule.CheckRunner - No value present for REQUEST_FAILED

@TyBrown
Copy link

TyBrown commented Feb 10, 2017

It looks like Seyren is unable to find the metric in your Graphite tree named "REQUEST_FAILED". Double check that you are entering the complete metric name in the "Target" field on the Check in the Seyren GUI.

Perhaps share with us a screenshot of your Seyren Check and an example Graph from the Graphite web application so we can see what you're working with?

Cheers!

@gargnisha1605
Copy link
Author

gargnisha1605 commented Feb 10, 2017

image
Pls find the graphite screenshot, there is no datapoint against it in graphite.

And target for seyren : alias(sumSeries(derivative(prod...*.REQUEST_FAILED.count)),'REQUEST_FAILED')

@TyBrown
Copy link

TyBrown commented Feb 10, 2017

If your'e not seeing any data points in the Graph in Graphite, it could mean a few things.

  1. Check that your smallest storage schema matches how frequently you are sending data to Graphite.
  2. Check that the metrics are actually arriving in Graphite.

You can try applying the function transformNull(metric, 0) to the metric to test and see if perhaps your data is arriving with gaps.

Basically though, you need to make sure that you have a graph that displays properly in Graphite web so that you can convert it to a Seyren check, otherwise exceptions like you posted will be what you get.

@gargnisha1605
Copy link
Author

My storage schemas :
[carbon]
pattern = ^carbon.
retentions = 60:90d

[metric_1min_for_3m]
pattern = .*
retentions = 1m:45d,60m:90d

How can I check that metric is actually arriving in graphite?

Also I am getting some other exceptions as well, after I deleted the checks which were in unknown state in seyren alert dashboard. Please find the stack trace:
10/02/2017 12:48:22.137 [http-bio-8080-exec-13] WARN o.j.resteasy.core.ExceptionHandler - failed to execute
javax.ws.rs.NotFoundException: Could not find resource for full path: http://30.0.9.122:8080/api/metrics/total
at org.jboss.resteasy.core.registry.SegmentNode.match(SegmentNode.java:112) ~[resteasy-jaxrs-3.0.6.Final.jar:na]
at org.jboss.resteasy.core.registry.RootNode.match(RootNode.java:43) ~[resteasy-jaxrs-3.0.6.Final.jar:na]
at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:48) ~[resteasy-jaxrs-3.0.6.Final.jar:na]
at org.jboss.resteasy.core.ResourceMethodRegistry.getResourceInvoker(ResourceMethodRegistry.java:444) ~[resteasy-jaxrs-3.0.6.Final.jar:na]
at org.jboss.resteasy.core.SynchronousDispatcher.getInvoker(SynchronousDispatcher.java:234) ~[resteasy-jaxrs-3.0.6.Final.jar:na]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:171) ~[resteasy-jaxrs-3.0.6.Final.jar:na]
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) ~[resteasy-jaxrs-3.0.6.Final.jar:na]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) ~[resteasy-jaxrs-3.0.6.Final.jar:na]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) ~[resteasy-jaxrs-3.0.6.Final.jar:na]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) ~[seyren-1.1.0.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) ~[seyren-1.1.0.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) ~[seyren-1.1.0.jar:na]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) ~[seyren-1.1.0.jar:na]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) ~[seyren-1.1.0.jar:na]

@TyBrown
Copy link

TyBrown commented Feb 10, 2017

I'm not sure about that 2nd exception.

As for checking if the metrics are arriving... you can look at /opt/graphite/storage/log/carbon-cache/*/creates.log to see if the metrics got created. You should also be able to see the metric whisper files at /opt/graphite/storage/whisper, unless you specified a different path in your carbon.conf file.

@gargnisha1605
Copy link
Author

gargnisha1605 commented Feb 11, 2017

I have checked in the location and only m1_rate.wsp is present. There is no count.wsp file. But graphs are visible in Grafana. So how can it be possible.

Also after deleting those alerts, this problem must have resolved, right? But I am not getting alerts for other metrics as well even after WARN level has reached.
Also if one metric is throwing exception, so alerts will not be generated for other metrics as well?

Also, until count.wsp of the metric gets created, I should not create an alert to avoid this Exception? Or is there any way, to stop these alerts from pushing to seyren, instead of deleting those checks and once again creating them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants