[Synthetics] Improve synthetics alerting #186585

shahzad31 · 2024-06-21T04:50:13Z

Summary

Improve synthetics alerting !!

We have improved synthetics alerting to allow users to create custom rules, users will be able to limit rules scope by filtering monitors by Monitor name, tags, type or location. This will allow users to send alerts to different teams for different monitors by defining respective action connectors in each rule.

User will be able to create custom synthetics status alert by defining two kind of criteria

First kind of threshold is Monitor is down over last consecutive checks with threshold from x locations. This will allow users to let’s say define a threshold, Monitor abc is down 3 times from the last 5 checks from 1 location. Users can also change the location threshold to specify from how many locations the threshold should be met.

The other threshold specifies a time window for checks, example being Monitor is down 3 times in the last 10 minutes from 2 locations.

By default if only 1 location threshold is selected, users can choose either to get an alert per location or get a summary message for all locations. If the user chooses multiple locations threshold then a summary message is always sent.

Monitor is down over last consective checks with threshold

with Locations threshold

user can also select location threshold to say only when down from x locations

Over time with checks threshold just like uptime custom status alert

shahzad31 · 2024-06-21T04:50:24Z

/ci

obltmachine · 2024-06-21T04:50:26Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

shahzad31 · 2024-06-21T06:51:41Z

/ci

shahzad31 · 2024-06-21T12:07:54Z

/ci

shahzad31 · 2024-07-01T07:29:58Z

/ci

…-synth-alerting

shahzad31 · 2024-07-18T16:06:15Z

/ci

…fix'

shahzad31 · 2024-07-18T17:21:17Z

/ci

…-synth-alerting

…/kibana into improve-synth-alerting

shahzad31 · 2024-07-19T11:44:27Z

/ci

elasticmachine · 2024-07-19T12:39:33Z

💔 Build Failed

Buildkite Build
Commit: d180f77

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #28 / Synthetics API Tests GetMonitorsOverview accepts search queries
[job] [logs] FTR Configs #28 / Synthetics API Tests GetMonitorsOverview accepts search queries

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`synthetics`	914	1079	+165

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/observability-synthetics-test-data`	-	4	+4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`synthetics`	854.8KB	1.1MB	⚠️ +247.7KB
`triggersActionsUi`	1.7MB	1.7MB	+119.0B
total			+247.8KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/observability-synthetics-test-data`	-	1	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`synthetics`	19.4KB	21.7KB	+2.3KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/observability-synthetics-test-data`	-	4	+4

async chunk count

id	before	after	diff
`synthetics`	8	9	+1

ESLint disabled line counts

id	before	after	diff
`synthetics`	84	83	-1

Total ESLint disabled count

id	before	after	diff
`synthetics`	91	90	-1

History

💔 Build #222448 failed 424bc3d
💔 Build #222419 failed 7c19f87

…-synth-alerting

justinkambic

This is looking a lot better and pretty close IMO.

In terms of smoke testing, I:

ran TLS rules that should not fail
ran TLS rules that should alert
ran MS rules that check location
ran MS rules that check project
ran MS rules that check monitor name
ran MS rules that check tags
changed the value of the failure threshold
changed the number of locations required to be down

I did see this error logged in my Kibana

Deprecation warning: value provided is not in a recognized RFC2822 or ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions. Non RFC2822/ISO date formats are discouraged. Please refer to http://momentjs.com/guides/#/warnings/js-date/ for more info.
Arguments: 
[0] _isAMomentObject: true, _isUTC: false, _useUTC: false, _l: undefined, _i: now-2m, _f: undefined, _strict: undefined, _locale: [object Object]
Error: 
    at Function.createFromInputFallback (kibana/node_modules/moment/moment.js:324:25)
    at configFromString (kibana/node_modules/moment/moment.js:2613:19)
    at configFromInput (kibana/node_modules/moment/moment.js:3056:13)
    at prepareConfig (kibana/node_modules/moment/moment.js:3039:13)
    at createFromConfig (kibana/node_modules/moment/moment.js:3006:44)
    at createLocalOrUTC (kibana/node_modules/moment/moment.js:3100:16)
    at createLocal (kibana/node_modules/moment/moment.js:3104:16)
    at cloneWithOffset (kibana/node_modules/moment/moment.js:3353:20)
    at Moment.diff (kibana/node_modules/moment/moment.js:3917:16)
    at StatusRuleExecutor.getRange (status_rule_executor.ts:223:60)
    at StatusRuleExecutor.getDownChecks (status_rule_executor.ts:136:24)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.executor (monitor_status_rule.ts:77:60)
    at rule_type_runner.ts:212:28
    at TaskRunnerTimer.runWithTimer (task_runner_timer.ts:50:20)
    at RuleTypeRunner.run (rule_type_runner.ts:187:57)
    at TaskRunner.runRule (task_runner.ts:355:9)
    at TaskRunner.run (task_runner.ts:645:9)
    at TaskManagerRunner.run (task_runner.ts:389:22)

...tion/synthetics/public/apps/synthetics/components/alerts/common/field_popover_expression.tsx

justinkambic · 2024-09-30T16:25:59Z

...ility_solution/synthetics/public/apps/synthetics/components/alerts/common/field_selector.tsx

+        isClearable
+        fullWidth
+        isLoading={isLoading}
+        onChange={(selected: Array<EuiComboBoxOptionOption<string>>) => {


It'd be nice to have this handler tested.

...ution/synthetics/public/apps/synthetics/components/alerts/common/for_the_last_expression.tsx

justinkambic · 2024-09-30T16:33:54Z

...y_solution/synthetics/public/apps/synthetics/components/alerts/common/popover_expression.tsx

+import React, { useState, ReactNode } from 'react';
+import { EuiExpression, EuiPopover } from '@elastic/eui';
+
+type ExpressionColor = 'subdued' | 'primary' | 'success' | 'accent' | 'warning' | 'danger';


...ins/observability_solution/synthetics/public/apps/synthetics/components/alerts/query_bar.tsx

…-synth-alerting

…/kibana into improve-synth-alerting

justinkambic

LGTM!!

kibana-ci · 2024-10-01T16:12:21Z

💚 Build Succeeded

Buildkite Build
Commit: c9fa0da
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-186585-c9fa0da1c8f1

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`synthetics`	1019	1154	+135

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/observability-synthetics-test-data`	-	4	+4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`synthetics`	964.5KB	1.2MB	⚠️ +237.5KB
`triggersActionsUi`	1.6MB	1.6MB	+115.0B
total			+237.6KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/observability-synthetics-test-data`	-	1	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`synthetics`	36.9KB	37.1KB	+221.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/observability-synthetics-test-data`	-	4	+4

ESLint disabled line counts

id	before	after	diff
`@kbn/synthetics-e2e`	19	18	-1
`@kbn/test-suites-xpack`	721	722	+1
`synthetics`	49	52	+3
total			+3

Total ESLint disabled count

id	before	after	diff
`@kbn/synthetics-e2e`	20	19	-1
`@kbn/test-suites-xpack`	745	746	+1
`synthetics`	55	58	+3
total			+3

History

💛 Build #238291 was flaky b89059c
💚 Build #238129 succeeded 6fec0e2
💔 Build #238058 failed 27c2816
💔 Build #237898 failed 2b4f971
💔 Build #237731 failed cf339ed

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2024-10-01T16:48:57Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11129829761

kibanamachine · 2024-10-01T16:53:06Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [Synthetics] Remove dead code (#193335)

Manual backport

To create the backport manually run:

node scripts/backport --pr 186585

Questions ?

Please refer to the Backport tool documentation

## Summary Fixes elastic#175298 Improve synthetics alerting !! User will be able to create custom synthetics status alert by defining three kind of criteria ### Monitor is down over last consective checks with threshold <img width="639" alt="image" src="https://github.com/elastic/kibana/assets/3505601/390da238-f7f2-4eb0-9606-3279b3199fdf"> ### From Locations threshold Will be considered down only when from defined number of locations <img width="618" alt="image" src="https://github.com/elastic/kibana/assets/3505601/24741a10-0880-4247-9048-8ce03df25bf5"> ### Over time with checks threshold just like uptime custom status alert <img width="631" alt="image" src="https://github.com/elastic/kibana/assets/3505601/64e1c808-8d4b-4dd0-b794-eb7f4e5d1e6b"> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Dominique Clarke <dominique.clarke@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Maryam Saeidi <maryam.saeidi@elastic.co> Co-authored-by: Justin Kambic <jk@elastic.co>

shahzad31 · 2024-10-02T09:50:22Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

## Summary Fixes elastic#175298 Improve synthetics alerting !! User will be able to create custom synthetics status alert by defining three kind of criteria ### Monitor is down over last consective checks with threshold <img width="639" alt="image" src="https://github.com/elastic/kibana/assets/3505601/390da238-f7f2-4eb0-9606-3279b3199fdf"> ### From Locations threshold Will be considered down only when from defined number of locations <img width="618" alt="image" src="https://github.com/elastic/kibana/assets/3505601/24741a10-0880-4247-9048-8ce03df25bf5"> ### Over time with checks threshold just like uptime custom status alert <img width="631" alt="image" src="https://github.com/elastic/kibana/assets/3505601/64e1c808-8d4b-4dd0-b794-eb7f4e5d1e6b"> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Dominique Clarke <dominique.clarke@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Maryam Saeidi <maryam.saeidi@elastic.co> Co-authored-by: Justin Kambic <jk@elastic.co> (cherry picked from commit 82d0b00) # Conflicts: # .github/CODEOWNERS # x-pack/plugins/observability_solution/synthetics/public/apps/synthetics/components/alerts/query_bar.tsx

# Backport This will backport the following commits from `main` to `8.x`: - [[Synthetics] Improve synthetics alerting (#186585)](#186585)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)

shahzad31 force-pushed the improve-synth-alerting branch from bd7d959 to 49a2a8f Compare June 27, 2024 14:17

shahzad31 added 3 commits July 15, 2024 17:37

Merge branch 'main' of https://github.com/elastic/kibana into improve…

48327cc

…-synth-alerting

Merge branch 'main' of https://github.com/elastic/kibana into improve…

fe81cb8

…-synth-alerting

wip

1c45b11

dominiqueclarke self-requested a review July 16, 2024 13:18

shahzad31 added 9 commits July 16, 2024 22:14

Merge branch 'main' of https://github.com/elastic/kibana into improve…

a4e81e6

…-synth-alerting

Merge branch 'main' of https://github.com/elastic/kibana into improve…

6ff1481

…-synth-alerting

Merge branch 'main' of https://github.com/elastic/kibana into improve…

dc3616d

…-synth-alerting

wip

aedb99e

Merge branch 'main' of https://github.com/elastic/kibana into improve…

83c9688

…-synth-alerting

wip

6b1fc33

Merge branch 'main' of https://github.com/elastic/kibana into improve…

ea80bbb

…-synth-alerting

update tests

d7abefd

update test

7c19f87

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

424bc3d

…fix'

shahzad31 added 6 commits July 18, 2024 19:21

Merge branch 'main' of https://github.com/elastic/kibana into improve…

44379fe

…-synth-alerting

update

12f90f8

added alerts page

325d00c

Merge branch 'main' of https://github.com/elastic/kibana into improve…

00a787c

…-synth-alerting

update test

17d4c14

Merge branch 'improve-synth-alerting' of https://github.com/shahzad31…

d180f77

…/kibana into improve-synth-alerting

shahzad31 added 7 commits September 27, 2024 16:23

Merge branch 'main' of https://github.com/elastic/kibana into improve…

46d4184

…-synth-alerting

update api test

cf339ed

update

8bbdb96

Merge branch 'main' of https://github.com/elastic/kibana into improve…

2b4f971

…-synth-alerting

Merge branch 'main' of https://github.com/elastic/kibana into improve…

e3a74e7

…-synth-alerting

jest

27c2816

Merge branch 'main' into improve-synth-alerting

6fec0e2

justinkambic reviewed Sep 30, 2024

View reviewed changes

shahzad31 and others added 5 commits October 1, 2024 10:06

Merge branch 'main' of https://github.com/elastic/kibana into improve…

76dc10e

…-synth-alerting

Merge branch 'main' of https://github.com/elastic/kibana into improve…

f5bfc2f

…-synth-alerting

More PR feedback

d715c31

Merge branch 'improve-synth-alerting' of https://github.com/shahzad31…

b89059c

…/kibana into improve-synth-alerting

PR feedback.

c9fa0da

justinkambic approved these changes Oct 1, 2024

View reviewed changes

shahzad31 merged commit 82d0b00 into elastic:main Oct 1, 2024
42 checks passed

shahzad31 deleted the improve-synth-alerting branch October 1, 2024 16:48

kibanamachine added the v9.0.0 label Oct 1, 2024

shahzad31 mentioned this pull request Oct 2, 2024

[8.x] [Synthetics] Improve synthetics alerting (#186585) #194671

Merged

kibanamachine mentioned this pull request Oct 2, 2024

[fix] type issue from race condition on merge #194676

Merged

This was referenced Oct 2, 2024

[Request]: Synthetics custom alerts elastic/observability-docs#4337

Open

[Synthetics] Support n-location check for monitor rules #153571

Closed

kibanamachine added the v8.16.0 label Oct 2, 2024

colleenmcginnis mentioned this pull request Oct 3, 2024

[synthetics] Update monitor status rule docs elastic/observability-docs#4345

Open

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Synthetics] Improve synthetics alerting #186585

[Synthetics] Improve synthetics alerting #186585

shahzad31 commented Jun 21, 2024 •

edited

Loading

shahzad31 commented Jun 21, 2024

obltmachine commented Jun 21, 2024

shahzad31 commented Jun 21, 2024

shahzad31 commented Jun 21, 2024

shahzad31 commented Jul 1, 2024

shahzad31 commented Jul 18, 2024

shahzad31 commented Jul 18, 2024

shahzad31 commented Jul 19, 2024

elasticmachine commented Jul 19, 2024 •

edited

Loading

API count

async chunk count

ESLint disabled line counts

Total ESLint disabled count

justinkambic left a comment

justinkambic Sep 30, 2024

justinkambic Sep 30, 2024

justinkambic left a comment

kibana-ci commented Oct 1, 2024 •

edited

Loading

API count

ESLint disabled line counts

Total ESLint disabled count

kibanamachine commented Oct 1, 2024

kibanamachine commented Oct 1, 2024

shahzad31 commented Oct 2, 2024

[Synthetics] Improve synthetics alerting #186585

[Synthetics] Improve synthetics alerting #186585

Conversation

shahzad31 commented Jun 21, 2024 • edited Loading

Summary

Monitor is down over last consective checks with threshold

with Locations threshold

Over time with checks threshold just like uptime custom status alert

shahzad31 commented Jun 21, 2024

obltmachine commented Jun 21, 2024

🤖 GitHub comments

shahzad31 commented Jun 21, 2024

shahzad31 commented Jun 21, 2024

shahzad31 commented Jul 1, 2024

shahzad31 commented Jul 18, 2024

shahzad31 commented Jul 18, 2024

shahzad31 commented Jul 19, 2024

elasticmachine commented Jul 19, 2024 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

async chunk count

ESLint disabled line counts

Total ESLint disabled count

History

justinkambic left a comment

Choose a reason for hiding this comment

justinkambic Sep 30, 2024

Choose a reason for hiding this comment

justinkambic Sep 30, 2024

Choose a reason for hiding this comment

justinkambic left a comment

Choose a reason for hiding this comment

kibana-ci commented Oct 1, 2024 • edited Loading

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

kibanamachine commented Oct 1, 2024

kibanamachine commented Oct 1, 2024

💔 All backports failed

Manual backport

Questions ?

shahzad31 commented Oct 2, 2024

💚 All backports created successfully

Questions ?

shahzad31 commented Jun 21, 2024 •

edited

Loading

elasticmachine commented Jul 19, 2024 •

edited

Loading

kibana-ci commented Oct 1, 2024 •

edited

Loading