Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.*: Fix Azure MSI token refresh #4611

Merged
merged 1 commit into from
Sep 1, 2021

Conversation

phillebaba
Copy link
Contributor

@phillebaba phillebaba commented Aug 30, 2021

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

The changes made in #4607 was missing a token refresh as I expected that to be done by the client. This was not the case and after leaving things running over the weekend the token eventually expired and everything stopped working. This change adds a token refresh function which renews the token before it expires.

Verification

Have tested a build of this branch in an AKS cluster, and it works. I will leave it running for another day just to make sure that the token refresh actually works.

@phillebaba phillebaba changed the title .*: Fix MSI token refresh .*: Fix Azure MSI token refresh Aug 30, 2021
@phillebaba phillebaba force-pushed the fix/msi-token-refresh branch 2 times, most recently from 6c9665b to b1fb8d4 Compare August 30, 2021 09:55
bwplotka
bwplotka previously approved these changes Aug 30, 2021
Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, love it. Left some suggestions. Thanks! 💪🏽

pkg/objstore/azure/helpers.go Outdated Show resolved Hide resolved
pkg/objstore/azure/helpers.go Show resolved Hide resolved
var tokenRefresher blob.TokenRefresher = func(credential blob.TokenCredential) time.Duration {
err := azureServicePrincipalToken.Refresh()
if err != nil {
// Retry later as the error can be related to API throttling
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would 100% add metric or error log to indicate that we hit this and why (err)

pkg/objstore/azure/helpers.go Outdated Show resolved Hide resolved
pkg/objstore/azure/helpers.go Outdated Show resolved Hide resolved
azureServicePrincipalToken, err := msiConfig.ServicePrincipalToken()
if err != nil {
return nil, err
}

// Get a new token.
var tokenRefresher blob.TokenRefresher = func(credential blob.TokenCredential) time.Duration {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about defining this function outside? (normally) ;p

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cant really do that as it need access to the spt.

Signed-off-by: Philip Laine <philip.laine@xenit.se>
@phillebaba
Copy link
Contributor Author

@bwplotka hopefully this solves all comments and issues. I will leave a build of this running for at least 12 hours to make sure that the token refresh works. The only thing I am not super happy about is the token refresh failure but it might be something that we need to re visit in the future instead.

Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM

pkg/objstore/azure/helpers.go Show resolved Hide resolved
@bwplotka bwplotka merged commit 5bda2d4 into thanos-io:main Sep 1, 2021
@bwplotka
Copy link
Member

bwplotka commented Sep 1, 2021

Nice, thanks. Can we add issue to the library we use, so they are aware?

@phillebaba
Copy link
Contributor Author

Created an issue in the library, linking it here to make it easier to track.

Azure/azure-storage-blob-go#303

someshkoli pushed a commit to someshkoli/thanos that referenced this pull request Nov 7, 2021
Signed-off-by: Philip Laine <philip.laine@xenit.se>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants