[Fleet] Add workflow for requesting and downloading agent diagnostics from Fleet UI

[kpollich]

Blocked by https://github.com/elastic/security-team/issues/4661

Background

A common supportability concern with Fleet/Agent is the collection and investigation of diagnostics. Elastic Agent exposes the elastic-agent diagnostics collect command, which outputs a .zip containing various diagnostics information that's crucial for debugging purposes.

We'd like to expose these diagnostics files in Fleet UI to improve debug-ability and reduce support overhead when requesting these diagnostics.

There are a few components at play here

• Fleet Server's "file upload" API that will allow agents to upload their diagnostics files (or any other arbitrary files) to Kibana. Tracked in https://github.com/elastic/security-team/issues/4661 and slotted for delivery during 8.6.
• A new UPLOAD_DIAGNOSTICS (name not final) action type for initiating the collection -> upload of Agent diagnostics
• A Fleet UI workflow for requesting diagnostics, being notified of upload completion, and viewing previously requested diagnostics (this issue)

Implementation

• API requirements
  • Support creating the REQUEST_DIAGNOSTICS action type in the existing POST /api/fleet/agents/<id>/action API
  • Add a new API for listing the files an agent has uploaded: GET /api/fleet/agents/<id>/uploads
  • Use the Kibana File Storage HTTP API for downloading: GET /api/files/files/<id>/blob[/<filename>]
• Agent Details Page - /agents/:id
  • Add a new action to the "actions" dropdown: "Request diagnostics .zip"
  • Clicking "Request diagnostics" creates a new REQUEST_DIAGANOSTICS action and navigates the user to a new "Diagnostics" tab on the agent details page
  • Diagnostics tab
    • A list of diagnostics files for the current agent is displayed as a table with two columns: File and Date
      • Each file name is prefixed with a "download" icon, and clicking the file name downloads the file using the browser's standard file download process
      • Any pending uploads (e.g action is created, but not yet complete) are displayed with a filename of "Generating diagnostics" and a date corresponding to the action's creation
      • Any errored uploads (e.g. action reports an error status) are displayed with a an error icon and a derived timestamp filename based on the action's creation date
    • A "Request diagnostics .zip" button exists to initiate (create action) a new diagnostics upload. The table dynamically updates to include this new request.
• Agent Listing Page - /agents
  • Add a new "Request diagnostics .zip" action to each agent's action menu that creates the action record -> navigates the user to the Diagnostics tab for the selected agent as above
  • Requesting diagnostics for multiple agents at once should be possible through the bulk actions menu when multiple agents are selected
• Global functionality
  • When a diagnostics upload is completed, the user should be notified via a toast message. This can probably be implemented by polling for the action's status for a set amount of time after diagnostics are requested.

Demo

• Prepare recorded demo

Designs

Overview:

Show individual screens

Agent details screen:

Diagnostics tab:

Agent listing page

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

Requesting diagnostics for multiple agents at once should be possible through the bulk actions menu when multiple agents are selected

I've added this requirement based on an internal conversation w/ support where this was identified as a big win. @juliaElastic or @michel-laterman do you foresee any issues with "queuing up" diagnostics requests for multiple agents via .fleet-actions? The way I imagine this working is that we'll create an action record for each agent ID with type REQUEST_DIAGNOSTICS and we should process them eventually. Any concerns with that approach?

[michel-laterman]

I don't see an issue with this.
But just to clarify, an action document has the agents: [] attribute, so we can give the same action to multiple agents.

[kpollich]

an action document has the agents: [] attribute, so we can give the same action to multiple agents.

Even better. Thanks for clarifying! This should make the process of requesting diagnostics from many agents simpler.

[juliaElastic]

Yes, this can work the same way as other bulk actions.

[juliaElastic]

@kpollich I think the bulk action of Request Diagnostics is more complex than we assumed:

For bulk selection, it doesn't sound logical to navigate to the Agent details page of one agent.
Alternatively there could be a new screen to list the diagnostics of bulk actions or show the file for each agent details that was included in the bulk action. In the latter case it wouldn't be easy to find all the files that were generated as part of one bulk action.
Related question, is the backend implementation planned to produce one zip per agent for bulk action?

There is also a question on the size of files the bulk action would produce, are there any concerns of uploading hundreds of Mb per each agent? This could quickly reach a very big file size if actioned on large agent selections.

[kpollich]

For bulk selection, it doesn't sound logical to navigate to the Agent details page of one agent.

I agree with this. We'll probably want to do something else after a bulk "request diagnostics" action is created.

Alternatively there could be a new screen to list the diagnostics of bulk actions or show the file for each agent details that was included in the bulk action. In the latter case it wouldn't be easy to find all the files that were generated as part of one bulk action.

Could we open the "agent activity" flyout after this bulk action is created to show the pending action w/ some detail about its status? Maybe we can display an expandable section where each agent is listed w/ a link to its /diagnostics tab?

Related question, is the backend implementation planned to produce one zip per agent for bulk action?

Yes there'd be one zip per agent.

There is also a question on the size of files the bulk action would produce, are there any concerns of uploading hundreds of Mb per each agent? This could quickly reach a very big file size if actioned on large agent selections.

This is true, and users should be aware that diagnostics incur storage costs + storage costs incur monetary costs on cloud. We may want to document this in a callout on the diagnostics tab. Something like Diagnostics files are stored in Elasticsearch, and as such can incur storage costs. Fleet will automatically remove old diagnostics files after 30 days.

[juliaElastic]

Could we open the "agent activity" flyout after this bulk action is created to show the pending action w/ some detail about its status? Maybe we can display an expandable section where each agent is listed w/ a link to its /diagnostics tab?

Good idea, though showing a link for each agent would only work for a limited number of agents.
When we implement the View agents functionality of the flyout, we would have a way to navigate to agent details from the actioned agent list as well.

This is true, and users should be aware that diagnostics incur storage costs + storage costs incur monetary costs on cloud. We may want to document this in a callout on the diagnostics tab. Something like Diagnostics files are stored in Elasticsearch, and as such can incur storage costs. Fleet will automatically remove old diagnostics files after 30 days.

Yes, we can do that. Even a confirmation window could be added with a warning message.

[kpollich]

Good idea, though showing a link for each agent would only work for a limited number of agents.

When we implement the #141206 functionality of the flyout, we would have a way to navigate to agent details from the actioned agent list as well.

Good points here. For bulk actions, let's just opt not to open the activity flyout once the action is created then. Eventually, we'll enhance the flyout with some more granular info about each agent for which an action was created. For now though, I think just showing the status of the "request bulk diagnostics" operation is good enough. The user will likely dismiss the flyout and visit the agents individually to access diagnostics afterwards.

[kpollich]

I'm realizing we don't have expiry captured anywhere here. We'll want to create a new index for the uploaded files to be stored in, and create an ILM policy during Fleet setup to manage it.

One thing I'm not clear on is whether we need to add to https://github.com/elastic/elasticsearch/tree/main/x-pack/plugin/core/src/main/resources (see .fleet-actions example) for this index, or if the file upload functionality will expose some kind of API for creating these "upload destination" indices. @pzl maybe you could answer this or point us in the right direction based on your work over on https://github.com/elastic/security-team/issues/4661?

[juliaElastic]

@kpollich We discussed on the call today that there will be different indices needed for fleet and endpoint security (and potentially other integrations in the future).
I think it makes sense to create and manage the fleet index in fleet setup code, including the ILM policy.