Skip to content

Add Alertmanager Integration Tests and Static File Backend #1686

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
27c77c7
refactor alertmanager storage and add static file alert store
jtlisi Sep 18, 2019
858d1ec
add integration test for alertmananger
jtlisi Sep 19, 2019
6d3405b
revert client
jtlisi Jan 14, 2020
056bc51
delete unneeded files
jtlisi Jan 14, 2020
5497b46
refactor to match rebase
jtlisi Jan 14, 2020
71b73ed
fix struct import
jtlisi Jan 14, 2020
0b04e5b
fix interface change
jtlisi Jan 14, 2020
c4935b2
remove unused function
jtlisi Jan 14, 2020
ffcc546
remove unused config
jtlisi Jan 15, 2020
0ac7460
instantiate map
jtlisi Jan 15, 2020
4409f94
address linting issues around file/local nomenclature
jtlisi Jan 24, 2020
caf9db7
ensure user alertmanagers are deleted
jtlisi Jan 28, 2020
f2e7846
rename addNewConfigs to syncConfigs
jtlisi Jan 28, 2020
5315819
rename server to store
jtlisi Jan 28, 2020
513c1ad
fix comments to use store language
jtlisi Jan 28, 2020
7a0ddb0
remove unused file
jtlisi Jan 28, 2020
26de24d
refactor logic when setting map
jtlisi Jan 28, 2020
1c036ee
add unit tests for syncing alertmanager configs
jtlisi Jan 29, 2020
2086938
goformat test file
jtlisi Jan 29, 2020
c6e8c3a
fix mild race condition
jtlisi Jan 29, 2020
2932615
update docs
jtlisi Jan 29, 2020
4eaefac
remove mutex and stop linearly
jtlisi Jan 29, 2020
33dc8c5
revert change to dispatcher
jtlisi Jan 29, 2020
fae9c12
revert alermanager to original file and remove delete test case
jtlisi Jan 29, 2020
d461ee6
fix alertmanager test
jtlisi Jan 29, 2020
765581b
refactor per PR comments
jtlisi Feb 11, 2020
bf4b4d2
add Go based integration test
jtlisi Feb 12, 2020
cbd8c18
rebase alertmanager integration test
jtlisi Feb 12, 2020
d96c619
check errors
jtlisi Feb 12, 2020
ef86964
update vendored deps
jtlisi Feb 12, 2020
5002d5b
revert alertmanager upgrade
jtlisi Feb 12, 2020
4c1214a
fix goimports lint
jtlisi Feb 12, 2020
aeca3b5
revert debugging vendor change
jtlisi Feb 12, 2020
e7eb89f
update docs
jtlisi Feb 12, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## master / unreleased

* [CHANGE] Config file changed to remove top level `config_store` field in favor of a nested `alertmanager.store.configdb` field. #1686
* [CHANGE] Removed unnecessary `frontend.cache-split-interval` in favor of `querier.split-queries-by-interval` both to reduce configuration complexity and guarantee alignment of these two configs. Starting from now, `-querier.cache-results` may only be enabled in conjunction with `-querier.split-queries-by-interval` (previously the cache interval default was `24h` so if you want to preserve the same behaviour you should set `-querier.split-queries-by-interval=24h`). #2040
* [CHANGE] Removed remaining support for using denormalised tokens in the ring. If you're still running ingesters with denormalised tokens (Cortex 0.4 or earlier, with `-ingester.normalise-tokens=false`), such ingesters will now be completely invisible to distributors and need to be either switched to Cortex 0.6.0 or later, or be configured to use normalised tokens. #2034
* [CHANGE] Moved `--store.min-chunk-age` to the Querier config as `--querier.query-store-after`, allowing the store to be skipped during query time if the metrics wouldn't be found. The YAML config option `ingestermaxquerylookback` has been renamed to `query_ingesters_within` to match its CLI flag. #1893
Expand Down
21 changes: 16 additions & 5 deletions docs/configuration/config-file-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,11 +86,6 @@ Supported contents and default values of the config file:
# and used by the 'configs' service to expose APIs to manage them.
[configdb: <configdb_config>]

# The configstore_config configures the config database storing rules and
# alerts, and is used by the Cortex alertmanager.
# The CLI flags prefix for this block config is: alertmanager
[config_store: <configstore_config>]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an entry to CHANGELOG.md explaning all changes (this is a config breaking change).


# The alertmanager_config configures the Cortex alertmanager.
[alertmanager: <alertmanager_config>]

Expand Down Expand Up @@ -821,6 +816,22 @@ externalurl:
# Root of URL to generate if config is http://internal.monitor
# CLI flag: -alertmanager.configs.auto-webhook-root
[autowebhookroot: <string> | default = ""]

store:
# Type of backend to use to store alertmanager configs. Supported values are:
# "configdb", "local".
# CLI flag: -alertmanager.storage.type
[type: <string> | default = "configdb"]

# The configstore_config configures the config database storing rules and
# alerts, and is used by the Cortex alertmanager.
# The CLI flags prefix for this block config is: alertmanager
[configdb: <configstore_config>]

local:
# Path at which alertmanager configurations are stored.
# CLI flag: -alertmanager.storage.local.path
[path: <string> | default = ""]
```

## `table_manager_config`
Expand Down
8 changes: 0 additions & 8 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,6 @@ github.com/NYTimes/gziphandler v1.1.1 h1:ZUDjpQae29j0ryrS0u/B8HZfJBtBQHjqw2rQ2cq
github.com/NYTimes/gziphandler v1.1.1/go.mod h1:n/CVRwUEOgIxrgPvAQhUUr9oeUtvrhMomdKFjzJNB0c=
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5/go.mod h1:lmUJ/7eu/Q8D7ML55dXQrVaamCz2vxCfdQBasLZfHKk=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/OneOfOne/xxhash v1.2.5 h1:zl/OfRA6nftbBK9qTohYBJ5xvw6C/oNKizR7cZGl3cI=
github.com/OneOfOne/xxhash v1.2.5/go.mod h1:eZbhyaAYD41SGSSsnmcpxVoRiQ/MPUTjUdIIOT9Um7Q=
github.com/OneOfOne/xxhash v1.2.6 h1:U68crOE3y3MPttCMQGywZOLrTeF5HHJ3/vDBCJn9/bA=
github.com/OneOfOne/xxhash v1.2.6/go.mod h1:eZbhyaAYD41SGSSsnmcpxVoRiQ/MPUTjUdIIOT9Um7Q=
Expand Down Expand Up @@ -99,7 +98,6 @@ github.com/apache/thrift v0.12.0 h1:pODnxUFNcjP9UTLZGTdeh+j16A8lJbRvD3rOtrk/7bs=
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878 h1:EFSB7Zo9Eg91v7MJPVsifUysc/wPdN+NOnVe6bWbdBM=
github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878/go.mod h1:3AMJUQhVx52RsWOnlkpikZr01T/yAVN2gn0861vByNg=
github.com/armon/go-metrics v0.3.0 h1:B7AQgHi8QSEi4uHu7Sbsga+IJDU+CENgjxoo81vDUqU=
github.com/armon/go-metrics v0.3.0/go.mod h1:zXjbSimjXTd7vOpY8B0/2LpvNvDoXBuplAD+gJD3GYs=
Expand Down Expand Up @@ -375,7 +373,6 @@ github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/gnostic v0.0.0-20170426233943-68f4ded48ba9/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=
github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=
github.com/googleapis/gnostic v0.3.0 h1:CcQijm0XKekKjP/YCz28LXVSpgguuB+nCxaSjCe09y0=
github.com/googleapis/gnostic v0.3.0/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=
github.com/googleapis/gnostic v0.3.1 h1:WeAefnSUHlBb0iJKwxFDZdbfGwkd7xRNuV+IpXMJhYk=
github.com/googleapis/gnostic v0.3.1/go.mod h1:on+2t9HRStVgn95RSsFWFz+6Q0Snyqv1awfrALZdbtU=
Expand Down Expand Up @@ -454,7 +451,6 @@ github.com/hashicorp/memberlist v0.1.4/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2p
github.com/hashicorp/memberlist v0.1.5 h1:AYBsgJOW9gab/toO5tEB8lWetVgDKZycqkebJ8xxpqM=
github.com/hashicorp/memberlist v0.1.5/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I=
github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc=
github.com/hashicorp/serf v0.8.3 h1:MWYcmct5EtKz0efYooPcL0yNkem+7kWxqXDi/UIh+8k=
github.com/hashicorp/serf v0.8.3/go.mod h1:UpNcs7fFbpKIyZaUuSW6EPiH+eZC7OuyFD+wc1oal+k=
github.com/hashicorp/serf v0.8.5 h1:ZynDUIQiA8usmRgPdGPHFdPnb1wgGI9tK3mO9hcAJjc=
github.com/hashicorp/serf v0.8.5/go.mod h1:UpNcs7fFbpKIyZaUuSW6EPiH+eZC7OuyFD+wc1oal+k=
Expand All @@ -474,7 +470,6 @@ github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht
github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901/go.mod h1:Z86h9688Y0wesXCyonoVr47MasHilkuLMqGhRZ4Hpak=
github.com/jonboulle/clockwork v0.1.0 h1:VKV+ZcuP6l3yW9doeqz6ziZGgcynBVQO+obU0+0hcPo=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7 h1:K//n/AqR5HjG3qxbrBCL4vJPW0MVFSs9CPK1OOJdRME=
github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7/go.mod h1:2iMrUgbbvHEiQClaW2NsSzMyGHqN+rDFqY705q49KG0=
github.com/jpillora/backoff v1.0.0 h1:uvFg412JmmHBHw7iwprIxkPMI+sGQ4kzOWsMeHnm2EA=
github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4=
Expand Down Expand Up @@ -652,7 +647,6 @@ github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXP
github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM=
github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod h1:p2iRAGwDERtqlqzRXnrOVns+ignqQo//hLXqYxZYVNs=
github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
github.com/prometheus/client_golang v1.1.0 h1:BQ53HtBmfOitExawJ6LokA4x8ov/z0SYYb0+HxJfRI8=
github.com/prometheus/client_golang v1.1.0/go.mod h1:I1FGZT9+L76gKKOs5djB6ezCbFQP1xR9D75/vuwEF3g=
github.com/prometheus/client_golang v1.2.0/go.mod h1:XMU6Z2MjaRKVu/dC1qupJI9SiNkDYzz3xecMgSW/F+U=
github.com/prometheus/client_golang v1.2.1 h1:JnMpQc6ppsNgw9QPAGF6Dod479itz7lvlsMzzNayLOI=
Expand Down Expand Up @@ -1096,12 +1090,10 @@ k8s.io/klog v1.0.0 h1:Pt+yjF5aB1xDSVbau4VsWe+dQNzA0qv1LlXdC2dF6Q8=
k8s.io/klog v1.0.0/go.mod h1:4Bi6QPql/J/LkTDqv7R/cd3hPo4k2DG6Ptcz060Ez5I=
k8s.io/kube-openapi v0.0.0-20190228160746-b3a7cee44a30/go.mod h1:BXM9ceUBTj2QnfH2MK1odQs778ajze1RxcmP6S8RVVc=
k8s.io/kube-openapi v0.0.0-20190709113604-33be087ad058/go.mod h1:nfDlWeOsu3pUf4yWGL+ERqohP4YsZcBJXWMK+gkzOA4=
k8s.io/kube-openapi v0.0.0-20190722073852-5e22f3d471e6 h1:s9IxTKe9GwDH0S/WaX62nFYr0or32DsTWex9AileL7U=
k8s.io/kube-openapi v0.0.0-20190722073852-5e22f3d471e6/go.mod h1:RZvgC8MSN6DjiMV6oIfEE9pDL9CYXokkfaCKZeHm3nc=
k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a h1:UcxjrRMyNx/i/y8G7kPvLyy7rfbeuf1PYyBf973pgyU=
k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a/go.mod h1:1TqjTSzOxsLGIKfj0lK8EeCP7K1iUG65v09OM0/WG5E=
k8s.io/utils v0.0.0-20190221042446-c2654d5206da/go.mod h1:8k8uAuAQ0rXslZKaEWd0c3oVhZz7sSzSiPnVZayjIX0=
k8s.io/utils v0.0.0-20190809000727-6c36bc71fc4a h1:uy5HAgt4Ha5rEMbhZA+aM1j2cq5LmR6LQ71EYC2sVH4=
k8s.io/utils v0.0.0-20190809000727-6c36bc71fc4a/go.mod h1:sZAwmy6armz5eXlNoLmJcl4F1QuKu7sr+mFQ0byX7Ew=
k8s.io/utils v0.0.0-20191114200735-6ca3b61696b6 h1:p0Ai3qVtkbCG/Af26dBmU0E1W58NID3hSSh7cMyylpM=
k8s.io/utils v0.0.0-20191114200735-6ca3b61696b6/go.mod h1:sZAwmy6armz5eXlNoLmJcl4F1QuKu7sr+mFQ0byX7Ew=
Expand Down
41 changes: 41 additions & 0 deletions integration/alertmanager_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
package main

import (
"context"
"testing"

"github.com/stretchr/testify/require"

"github.com/cortexproject/cortex/integration/framework"
)

func TestAlertmanager(t *testing.T) {
s, err := framework.NewScenario()
require.NoError(t, err)
defer s.Shutdown()

AlertmanagerConfigs := map[string]string{
"-alertmanager.storage.local.path": "/integration/alertmanager_test_fixtures/",
"-alertmanager.storage.type": "local",
"-alertmanager.web.external-url": "http://localhost/api/prom",
}

// Start Cortex components
require.NoError(t, s.StartAlertmanager("alertmanager", AlertmanagerConfigs, ""))
require.NoError(t, s.Service("alertmanager").WaitMetric(80, "cortex_alertmanager_configs", 1))

c, err := framework.NewClient("", "", s.Endpoint("alertmanager", 80), "user-1")
require.NoError(t, err)

status, err := c.GetAlertmanagerStatus(context.Background())
require.NoError(t, err)

// Ensure the returned status config matches alertmanager_test_fixtures/user-1.yaml
require.NotNil(t, status)
require.Equal(t, "example_receiver", status.ConfigJSON.Route.Receiver)
require.Len(t, status.ConfigJSON.Route.GroupByStr, 1)
require.Equal(t, "example_groupby", status.ConfigJSON.Route.GroupByStr[0])
require.Len(t, status.ConfigJSON.Receivers, 1)
require.Equal(t, "example_receiver", status.ConfigJSON.Receivers[0].Name)
require.NoError(t, s.StopService("alertmanager"))
}
5 changes: 5 additions & 0 deletions integration/alertmanager_test_fixtures/user-1.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
route:
receiver: "example_receiver"
group_by: ["example_groupby"]
receivers:
- name: "example_receiver"
4 changes: 2 additions & 2 deletions integration/backward_compatibility_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ func TestBackwardCompatibilityWithChunksStorage(t *testing.T) {
now := time.Now()
series, expectedVector := generateSeries("series_1", now)

c, err := framework.NewClient(s.Endpoint("distributor", 80), "", "user-1")
c, err := framework.NewClient(s.Endpoint("distributor", 80), "", "", "user-1")
require.NoError(t, err)

res, err := c.Push(series)
Expand All @@ -69,7 +69,7 @@ func TestBackwardCompatibilityWithChunksStorage(t *testing.T) {
require.NoError(t, s.Service("querier").WaitMetric(80, "cortex_ring_tokens_total", 512))

// Query the series
c, err := framework.NewClient(s.Endpoint("distributor", 80), s.Endpoint("querier", 80), "user-1")
c, err := framework.NewClient(s.Endpoint("distributor", 80), s.Endpoint("querier", 80), "", "user-1")
require.NoError(t, err)

result, err := c.Query("series_1", now)
Expand Down
21 changes: 20 additions & 1 deletion integration/framework/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (

"github.com/gogo/protobuf/proto"
"github.com/golang/snappy"
alertClient "github.com/prometheus/alertmanager/client"
promapi "github.com/prometheus/client_golang/api"
promv1 "github.com/prometheus/client_golang/api/prometheus/v1"
"github.com/prometheus/common/model"
Expand All @@ -17,6 +18,7 @@ import (

// Client is a client used to interact with Cortex in integration tests
type Client struct {
alertmanagerClient promapi.Client
distributorAddress string
timeout time.Duration
httpClient *http.Client
Expand All @@ -25,7 +27,7 @@ type Client struct {
}

// NewClient makes a new Cortex client
func NewClient(distributorAddress string, querierAddress string, orgID string) (*Client, error) {
func NewClient(distributorAddress string, querierAddress string, alertmanagerAddress string, orgID string) (*Client, error) {
// Create querier API client
querierAPIClient, err := promapi.NewClient(promapi.Config{
Address: "http://" + querierAddress + "/api/prom",
Expand All @@ -43,6 +45,17 @@ func NewClient(distributorAddress string, querierAddress string, orgID string) (
orgID: orgID,
}

if alertmanagerAddress != "" {
alertmanagerAPIClient, err := promapi.NewClient(promapi.Config{
Address: "http://" + alertmanagerAddress + "/api/prom",
RoundTripper: &addOrgIDRoundTripper{orgID: orgID, next: http.DefaultTransport},
})
if err != nil {
return nil, err
}
c.alertmanagerClient = alertmanagerAPIClient
}

return c, nil
}

Expand Down Expand Up @@ -95,3 +108,9 @@ func (r *addOrgIDRoundTripper) RoundTrip(req *http.Request) (*http.Response, err

return r.next.RoundTrip(req)
}

// GetAlertmanagerStatus gets the status of an alertmanager instance
func (c *Client) GetAlertmanagerStatus(ctx context.Context) (*alertClient.ServerStatus, error) {
statusAPI := alertClient.NewStatusAPI(c.alertmanagerClient)
return statusAPI.Get(ctx)
}
20 changes: 20 additions & 0 deletions integration/framework/scenario.go
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,26 @@ func (s *Scenario) StartTableManager(name string, flags map[string]string, image
))
}

func (s *Scenario) StartAlertmanager(name string, flags map[string]string, image string) error {
if image == "" {
image = GetDefaultCortexImage()
}

return s.StartService(NewService(
name,
image,
NetworkName,
[]int{80},
nil,
NewCommandWithoutEntrypoint("cortex", BuildArgs(MergeFlags(map[string]string{
"-target": "alertmanager",
"-log.level": "warn",
}, flags))...),
// The table-manager doesn't expose a readiness probe, so we just check if the / returns 404
NewReadinessProbe(80, "/", 404),
))
}

func (s *Scenario) StopService(name string) error {
service := s.Service(name)
if service == nil {
Expand Down
2 changes: 1 addition & 1 deletion integration/ingester_flush_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ func TestIngesterFlushWithChunksStorage(t *testing.T) {
require.NoError(t, s.Service("distributor").WaitMetric(80, "cortex_ring_tokens_total", 512))
require.NoError(t, s.Service("querier").WaitMetric(80, "cortex_ring_tokens_total", 512))

c, err := framework.NewClient(s.Endpoint("distributor", 80), s.Endpoint("querier", 80), "user-1")
c, err := framework.NewClient(s.Endpoint("distributor", 80), s.Endpoint("querier", 80), "", "user-1")
require.NoError(t, err)

// Push some series to Cortex
Expand Down
2 changes: 1 addition & 1 deletion integration/ingester_hand_over_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ func runIngesterHandOverTest(t *testing.T, flags map[string]string, setup func(t
require.NoError(t, s.Service("distributor").WaitMetric(80, "cortex_ring_tokens_total", 512))
require.NoError(t, s.Service("querier").WaitMetric(80, "cortex_ring_tokens_total", 512))

c, err := framework.NewClient(s.Endpoint("distributor", 80), s.Endpoint("querier", 80), "user-1")
c, err := framework.NewClient(s.Endpoint("distributor", 80), s.Endpoint("querier", 80), "", "user-1")
require.NoError(t, err)

// Push some series to Cortex
Expand Down
11 changes: 9 additions & 2 deletions pkg/alertmanager/alertmanager.go
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,15 @@ func (am *Alertmanager) ApplyConfig(userID string, conf *config.Config) error {

am.api.Update(conf, func(_ model.LabelSet) {})

am.inhibitor.Stop()
am.dispatcher.Stop()
// Ensure dispatcher is set before being called
if am.dispatcher != nil {
am.dispatcher.Stop()
}

// Ensure inhibitor is set before being called
if am.inhibitor != nil {
am.inhibitor.Stop()
}

am.inhibitor = inhibit.NewInhibitor(am.alerts, conf.InhibitRules, am.marker, log.With(am.logger, "component", "inhibitor"))

Expand Down
Loading