Failure to get snapshot information when there's a repository with incorrect credentials

[mattkime]

Elasticsearch Version

8.18.1

Installed Plugins

No response

Java Version

bundled

OS Version

cloud

Problem Description

When there's a snapshot repo with incorrect credentials it can block loading the list of snapshots.

Steps to Reproduce

• add keystore values for client access/secret keys (invalid ones for example)
• update repo to use the client with the invalid credentials
• reload secure settings (ie: POST _nodes/reload_secure_settings)
• GET /_snapshot/*/* - should 500 error

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

This behaviour is by design and I'm not really seeing what else we could do in this case given all the other constraints on this API's behaviour. In particular, it offers a sorted and paginated view of the snapshots in all the requested repositories, plus summary statistics, so if even one of them isn't available there's no way we can know what's missing and the results will be (possibly-dangerouly) incorrect.

[mattkime]

This behaviour is by design

I guess I don't understand why a resilient api wasn't designed.

so if even one of them isn't available there's no way we can know what's missing and the results will be (possibly-dangerouly) incorrect

Doesn't strike me as different from unavailable shards. Return the results and report what was missing.

[DaveCTurner]

I guess I don't understand why a resilient api wasn't designed.

As mentioned above: pagination. I'm guessing this UI isn't really doing GET /_snapshot/*/* since this would attempt to retrieve all snapshots, possibly thousands or more. Instead I'd expect it to be using the ?size= and ?offset= query params to retrieve the snapshots a page at a time instead. If so, how can we know where the current page even starts if one of the repositories is unavailable? If the unavailability is flaky or temporary then the results will jump around in ways that will suggest that extant snapshots have gone missing.

Moreover the response already indicates what's missing in the message. If it's a repository-level failure and you want to know the contents of the remaining repositories then you can skip the broken repository with e.g. GET /_snapshot/*,-brokenrepo/.... However from the screenshots in the linked issue it looks like the UI won't let the user adjust the repository list if this API returns an error:

That needs fixing, we must expose the repository list so that users can choose to skip the faulty repository.

[sophiec20]

Not keen on a UX which requires users to pick which snapshot repo to exclude. Let's put that idea to one side.

The atomic API behaviour makes sense from one angle, but less so for the practical realities for larger/complex customers with multiple repos who are just trying to manage their estates. Leniency is a preferred experience (along with surfacing relevant errors). e.g. For searching indices with a wildcard, we don't 500 the response if one index is closed or unauthorised. And I think returning partial search results is default.

Considering some options... Perhaps the most lenient option (and imho most usable) would list the snapshots we can and surface an error to say one repo was unauthorized. It sounds like today's APIs would require us to get the results from each repo and then merge and sort results client-side which doesn't sound like a great idea and negates the point of a wildcard. Alternatively we could show a better error perhaps.

Can I see an example of the API response under these circumstances? (the UI shows the elasticsearch errors its given)
What does the Repositories tab look like? Is it showing the problem repo?

[DaveCTurner]

To check I understand, you're saying you think it's ok to hide the UI components that would let the user adjust the request parameters and retry in the event of an error? I must be missing something because that seems pretty bad to me. There are possible errors other than just "one repo has incorrect credentials". I imagine we can find a way to handle that one particular case, but the general point about hiding this UI seems more fundamental to me.

[mattkime]

Is it possible to create a better error message? Its pretty obtuse.

Yes, we're using pagination.