Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: Remove unnecessary configuration reload from ContentPathReloader and improve its tests #6496

Merged
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
- [#6467](https://github.com/thanos-io/thanos/pull/6467) Mixin (Receive): add alert for tenant reaching head series limit.

### Fixed
- [#6496](https://github.com/thanos-io/thanos/pull/6496): *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests.
- [#6456](https://github.com/thanos-io/thanos/pull/6456) Store: fix crash when computing set matches from regex pattern
- [#6427](https://github.com/thanos-io/thanos/pull/6427) Receive: increasing log level for failed uploads to error
- [#6172](https://github.com/thanos-io/thanos/pull/6172) query-frontend: return JSON formatted errors for invalid PromQL expression in the split by interval middleware.
Expand Down
1 change: 1 addition & 0 deletions pkg/extkingpin/path_content_reloader.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ func PathContentReloader(ctx context.Context, fileContent fileContent, logger lo
reloadFunc()
level.Debug(logger).Log("msg", "configuration reloaded after debouncing")
})
reloadTimer.Stop()
}
defer watcher.Close()
for {
Expand Down
10 changes: 9 additions & 1 deletion pkg/extkingpin/path_content_reloader_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,15 @@ func TestPathContentReloader(t *testing.T) {
testutil.Ok(t, pathContent.Rewrite([]byte("test modified again")))
},
},
wantReloads: 1,
wantReloads: 2,
},
{
name: "Chmod doesn't trigger reload",
args: args{
runSteps: func(t *testing.T, testFile string, pathContent *staticPathContent) {
testutil.Ok(t, os.Chmod(testFile, 0777))
testutil.Ok(t, os.Chmod(testFile, 0666))
testutil.Ok(t, os.Chmod(testFile, 0777))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this trigger 2 reloads now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one triggers zero reload... see line 71 below.

The one above it triggers 2 reloads, which is truly what we expect, because the file is rewritten twice. Previously it could pass due to the code being much faster than the async fs watcher notification for the second rewrite of the file.

},
},
wantReloads: 0,
Expand All @@ -79,7 +81,9 @@ func TestPathContentReloader(t *testing.T) {
},
}
for _, tt := range tests {
tt := tt
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
testFile := path.Join(t.TempDir(), "test")
testutil.Ok(t, os.WriteFile(testFile, []byte("test"), 0666))
pathContent, err := NewStaticPathContent(testFile)
Expand All @@ -98,6 +102,10 @@ func TestPathContentReloader(t *testing.T) {
testutil.Ok(t, err)

tt.args.runSteps(t, testFile, pathContent)
if tt.wantReloads == 0 {
// Give things a little time to sync. The fs watcher events are heavily async and could be delayed.
time.Sleep(1 * time.Second)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Crossposting from c6623f8#r120681785. Please check testutil.Equals(t, tt.wantReloads, reloadCount) in a loop instead or something. This time.Sleep will probably lead to flakiness.

Copy link
Contributor Author

@douglascamata douglascamata Jul 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@GiedriusS I have some other follow ups to this part of the code and will include this suggested change for the next PR. 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this loop will have a timeout though, as I don't want a test failure to end up with an infinite loop or rely on test.timeout when running tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 you can use runutil.Repeat with a context to achieve that

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@GiedriusS I actually have to keep the sleep there for one reason: a real scenario might have a delayed event from the fs and the test might pass if the assertion runs quickly before the event arrives... but would fail, as expected, if it had waited slightly more.

I'm adding the repeat there though for all the other scenarios.

}
wg.Wait()
testutil.Equals(t, tt.wantReloads, reloadCount)
})
Expand Down