[Change Proposal] Define requirements for agentless support

[jsoriano]

There are integrations that could be deployed on fully managed agents. These integrations are going to be called "agentless". We need to define the mechanisms required for packages that include such integrations.

The main proposal would be to have a supports_agentless flag that packages that support serverless should set to true.

There are some open questions:

• In what manifest to set the flag. Package, policy (for multi-tile) or data stream level?
• Requirements to support agentless:
  • Data should be collected from publicly reachable APIs.
• Are there integrations that require agentless?
• Are there variables that we need to mark as non-configurable for agentless? Variables available only on agentless?
• What is the impact in Kibana UX?
• Testing: Is there any way to validate that a package or data stream actually supports agentless?

Document to facilitate the discussion on what needs to be filtered and where in the Fleet/Integrations UI: https://docs.google.com/document/d/1ccAswS94hr6qNnzQ_HKsmygQch2wgSQYMkMoQHo21Ds/edit?usp=sharing

cc: @kpollich

Implementation

• Add a new deployment_modes property to the policy template schema that contains an object with default and agentless properties. Each property in this object should have a single nested enabled boolean property for now, but should be extensible to support additional configuration in the future.
• Add a new hide_in_deployment_modes property to var declarations at all manifest levels that allows integration maintainers to opt certain variables in or out of being displayed in agentless mode. This would be an enum array, supporting default and agentless strings.

Example policy template declaration:

format_version: 3.0.0
name: aws
title: AWS
version: 2.13.1
...
policy_templates:
  - name: billing
    title: AWS Billing
    description: Collect billing metrics with Elastic Agent
    deployment_modes: # <---
      default:
        enabled: false
      agentless:
        enabled: true
    data_streams:
      - billing
    ...

Example variable declaration:

streams:
  - input: logfile
    vars:
      - name: paths
        type: text
        title: Paths
        multi: true
        required: true
        show_user: true
        default:
          - /var/log/my-package/*.log
      - name: agentless_only
        type: text
        title: Agentless only variable
        multi: false
        required: false
        show_user: true
        hide_in_deployment_modes: # <---
          - default
     - name: hidden_in_agentless 
        type: text
        title: Hidden in agentless variable
        multi: false
        required: false
        show_user: true
        hide_in_deployment_modes: # <---
          - agentless

[nimarezainia]

We need to provide integration developers with a way to explicitly "hide" sections of their integrations settings.

for a pull integration as an example, when the integration is hosted by Elastic, integration developer may want to allow the user to change the polling interval. Certainly they would need the user to provide the credentials. However they may need to restrict changes to namespace or datastream.

[nimarezainia]

@eyalkraft @olegsu @tinnytintin10 I wanted to explore whether it would be prudent for the package to convey other information that we should cater for as part of this work. Right now it's pretty much just a boolean (managed or not).

From CSPM perspective do you see a need to carry other information? for example details about the container sizing? etc

[eyalkraft]

Thanks @nimarezainia !

From CSPM perspective do you see a need to carry other information? for example details about the container sizing? etc

You're on-spot with your question - with our future plans for agentless + agent k8s runtime, I think it would be valuable for us to be able to configure per-integration package resource requirements (like container sizing), and possibly even things like persistent volumes for keeping state, horizontal scaling options, deployment method (for example, maybe CSPM can be a K8s Job that runs every x hours rather than a deployment).
Maybe we should have an agentless json field, currently only containing the supported boolean, but extendable down the road to meet further extensions.

@andrewkroh @aleksmaus might have additional input.

Some thoughts on the questions from @jsoriano:

In what manifest to set the flag. Package, policy (for multi-tile) or data stream level?

I think it can't be on the package since for example within our single cloud_security_posture integration package we have both the CSPM policy template that will support agentless, and CNVM that won't (for now).
Also not on the data stream since we have a single "findings" data stream used both by KSPM (not going to be agentless) and CSPM. Maybe on the inputs themselves... not sure.

Are there integrations that require agentless?
Are there variables that we need to mark as non-configurable for agentless? Variables available only on agentless?
What is the impact in Kibana UX?

Integrations that only work on agentless? We don't have any in mind right now but it's a good question.
All these questions make me think wether we should mark a policy template / input as "supports agentless", and start figuring out how me make every field enabled/disabled, or just have a separate "agentless" policy template/input with the proper variables configured just like any other policy template. I think we want to avoid having separate tiles for the agent-based/agentless versions of the same integration, but from the technical perspective it seems simpler to have them separate.

@maxcold

[aleksmaus]

Maybe we should have an agentless json field, currently only containing the supported boolean, but extendable down the road to meet further extensions.

@andrewkroh @aleksmaus might have additional input.

Making agentless as it's own sub config object for integrations spec might make sense for future expandability.

As far as I understand, some of the k8s runtime constraints such as minimum cpu/memory requirements would be specific to the actual beat containers, and might be tied to the beat specific build.
Some other runtime constrains such as cpu/memory requirements and limits, "auto-scaling", etc, would depend on amount work related to the particular integration+beat combination, possibly reconfigurable by the integration user.

Potentially there are three pieces of configuration that might need to come together:

1. Integration configuration for "agentless" (possibly some defaults can be defined there)
2. The beat spec "agentless" configuration, the beat runtime requirements/limits/constraints .
3. The "agentless" runtime/behavour configuration for a specific integration+beat+workload combination

Thoughts?

[kpollich]

Hi all, I've started catching up here and I want to add a bit of my thinking so far.

So far it seems like we've identified a few requirements here

• Integrations need a way to indicate that they support agentless deployment at the policy template level
• Integrations likely need the ability to change various pieces of configuration (variables? mappings? etc) based on whether the integration is running in agentless mode
• There should be some top-level config for agentless where things like resource limits can be configured

A few things I'd like to add:

• We will likely have additional deployment modes in the future, e.g. Helm chart
• We should think of this like a "capabilities" and "preferences" relationship. e.g. if an integration expresses that it can be deployed in agentless mode, and the environment to which it is installed supports agentless deployment, it will run in agentless mode

Perhaps the "capabilities" implementation is a future thought, though - I think what's in front of us to build today is essentially a mapping of deployment modes to UIs. Imagine this as a "tabs" component where each tab's title is a deployment mode, and the tab's contents are the particular instructions for that deployment mode. Our current default/agent deployment mode uses a Fleet policy for deployment, agentless will use a "managed" policy, and a Helm chart or K8's native deployment mode will have different instructions. Let's build this mapping first and foremost, and think about more nuanced behaviors like capabilities once the mapping is fleshed out.

One thing I'm not 100% sure on is whether agentless integrations can only be deployed in agentless environments. For example, if an environment does not support agentless, should agentless policy templates even show up?

[cmacknz]

We will likely have additional deployment modes in the future, e.g. Helm chart

I think this is key. Some thoughts on how to model this:

Today Fleet manages agent policies, which are sent directly to individual agents and describe 100% of the information needed to run the installed integrations. The agent is already installed and deployed, and we only need to configure the agent.

With Agentless, instead of targeting individual agents we are now targeting the agent controller first. The controller needs additional information beyond just the agent policy for how to deploy an agent to run an integration.

This is very similar to a Helm chart or Kubernetes YAML, where in addition to an agent policy we must tell the Kubernetes API how we would like it to run the agent.

All this to say I don't think we should try to jam all of the agentless configuration into the existing agent policy.

We should consider introducing something like a deployment policy, or deployment type, and have that be a separate piece of configuration that contains or is associated with an agent policy. The way the agent policy itself is generated may depend on the deployment type it is associated with. The agent can know and report which deployment policy it is running under to allow us determine capabilities as described by Kyle above.

This would allow us to add additional deployment types in the future, with their own constraints and capabilities.

[kpollich]

I like the idea of creating a new deployment policy data model to capture these concerns. I'm in agreement that jamming deployment concerns into the existing agent policy model is not a good fit.

Some quick scratchpad diagrams about how things are structured today, and what we could move to:

---

In this what we'd be building would be something like:

• Package spec support to indicate packages support the agentless deployment mode (all packages will support agent or default as what we have today by default)
• Package spec support for what configuration to include in Fleet's generated deployment policy when a given package is installed, e.g. resource limits
• Fleet support for creating deployment policy objects populated based on the deployment configuration provided alongside each integration installed on a given agent policy
• Fleet support for rendering different UI's, onboarding experiences, etc based on the deployment policy types supported by a given integration. e.g. in ESS where agentless is supported, integrations that support agentless will render the agentless onboarding experience and generate agentless deployment policies by default - maybe with no other option
• Probably a lot of other stuff

[cmacknz]

Probably a lot of other stuff

In the case of agentless, the deployment policy configuration itself would have to be made available to the agentless controller along with the agent policy.

[eyalkraft]

Small note about agentless-controller and it's role as we imagine it currently:

agentless-controller is multi-tenanted, and takes care of spinning up single-tenanted elastic agents, enrolled to fleet, on the serverless platform (MKI).
We don't plan on having it aware or concerned with what the customer does with this agent.
It doesn't know anything about agent-policy, and I don't think it should know anything about deployment policy.

Our current plan for how deploy integrations/beats in separate containers/pods/deployments is to (roughly) follow the plan for agent k8s runtime, which for us basically says - the agent has k8s API access, and it creates deployments rather than executes binaries.

To conclude, In this approach it's the agent that cares about the deployment-policy, not the agentless-controller.

[olegsu]

To protect the agent from running undesired integrations we use the capabilities feature and we define allowed inputs and outputs. Today its statically configured for all the agents. To properly support agentless integration this approach also needs to be revisited.

[kpollich]

Hello all, I'm revisiting our technical definition around agentless support in Fleet + Integrations, and I wanted to bump this issue with a few things I'm thinking of.

For one: the deployment policy model is a good long term direction here, but I'm concerned about the level of effort required to enable it. I think modeling deployments mode as a distinct data model in Fleet is the right solution for the eventual state we're seeking here, but I think we should move forward with some more short term solutions that more immediately patch some holes in our current implementation of agentless support, namely supporting more than one integration.

To this end, I am proposing we move forward on two things in the package spec:

1. Add an agentless property to the package-level manifest.yml file that can contain configuration related to agentless. Immediately, this will only support a single nested property: enabled: true

e.g.

format_version: "3.0.2"
name: nginx
title: My package
version: "1.0.0"
description: Lorem ipsum.
type: integration
categories:
  - web
conditions:
  kibana:
    version: "^8.12.0"
  elastic:
    subscription: basic
agentless: # <---
  enabled: true

2. Add a new hide_in_deployment_modes property to var declarations at all manifest levels. This will allow integration maintainers to opt certain variables out of being displayed in agentless mode, or introduce agentless-only variables. This would be an enum array, supporting default and agentless strings.

e.g.

streams:
  - input: logfile
    vars:
      - name: paths
        type: text
        title: Paths
        multi: true
        required: true
        show_user: true
        default:
          - /var/log/my-package/*.log
      - name: agentless_only
        type: text
        title: Agentless only variable
        multi: false
        required: false
        show_user: true
        hide_in_deployment_modes: # <---
          - default
     - name: hidden_in_agentless 
        type: text
        title: Hidden in agentless variable
        multi: false
        required: false
        show_user: true
        hide_in_deployment_modes: # <---
          - agentless

With both of these changes, I'm thinking towards the future. I don't want to limit our technical options if we move towards the deployment policy model proposed above, so I'm proposing these as an extensible foundation.

Curious to see if folks have some thoughts on the specifics here, I'm certainly open to suggestions or alternatives.

[eyalkraft]

++ @kpollich
Looks like a great first step that would be enough to begin with.

I think that soon enough we would want a standard way of providing our users with a clouformation/ARM template/gcloud shell way of installing/authenticating the integration with their cloud account, but as you propose this could be a later extension.

[jsoriano]

@kpollich plan about having a general toggle, and another one per variable sounds good to me.

Only thing I would recommend is to add the agentless enablement flag at the policy_templates level, as mentioned in other comments. Packages such as AWS have many policy templates and we may want to enable agentless only in some of them, at least during the migration period.

format_version: 3.0.0
name: aws
title: AWS
version: 2.13.1
...
policy_templates:
  - name: billing
    title: AWS Billing
    description: Collect billing metrics with Elastic Agent
    agentless:
      enabled: true
    data_streams:
      - billing
    ...

Maybe we could also consider to have a section about deployment modes, to follow the naming used in hide_in_deployment_modes, and to allow to have policies available only on agentless:

format_version: 3.0.0
name: aws
title: AWS
version: 2.13.1
...
policy_templates:
  - name: billing
    title: AWS Billing
    description: Collect billing metrics with Elastic Agent
    deployment_modes:
      default:
        enabled: false
      agentless:
        enabled: true
    data_streams:
      - billing
    ...

[olegsu]

Thanks @kpollich
I think if the deployment_mode.defaults is clear compared to deployment_mode.agentless, wdyt about deploymet_mode.self_hosted or something similar?

[kpollich]

I slightly prefer deployment_mode.default because it encapsulates the various different deployment models we have today, which include self-hosted agents run by users as well as managed agents like the one that runs on the Integration Server in ESS. Rather than having the default value of this setting be something opinionated about how an agent running this integration is deployed (e.g. self-hosted), I think default conveys that the integration "doesn't care" how it is deployed.

[jlind23]

@kpollich is this one ready to be worked on now? I do see it is still under a Needs tech definition status.

[kpollich]

Whoops yes this is ready to go. Changed the status.

[jlind23]

Removing Nima and you from the assignee.
@jsoriano @mrodm feel free to pick this up whenever you have some spare cycle.