Migrate documentation for the component framework (#23429)

.github/CODEOWNERS

@@ -202,6 +202,11 @@
 /docs/dev/checks/                       @DataDog/documentation @DataDog/agent-metrics-logs
 /docs/cloud-workload-security/          @DataDog/documentation @DataDog/agent-security
 
+/docs/public/                                                @DataDog/agent-platform
+/docs/public/architecture/components/                        @DataDog/agent-shared-components
+/docs/public/guidelines/components/                          @DataDog/agent-shared-components
+/docs/public/how-to/components/                              @DataDog/agent-shared-components
+
 /google-marketplace/                    @DataDog/container-ecosystems
 
 # These files are owned by all teams, but assigning them to @DataDog/agent-all causes a lot of spam

docs/public/.snippets/links.txt

@@ -1 +1,2 @@
+[agent-components-listing]: https://github.com/DataDog/datadog-agent/blob/main/comp/README.md
 [agent-troubleshooting]: https://docs.datadoghq.com/agent/troubleshooting/

docs/public/architecture/components/fx.md

@@ -0,0 +1,194 @@
+# Overview of Fx
+
+The Agent uses [Fx](https://uber-go.github.io/fx) as its application framework. While the linked Fx documentation is thorough, it can be a bit difficult to get started with. This document describes how Fx is used within the Agent in a more approachable style.
+
+## What Is It?
+
+Fx's core functionality is to create instances of required types "automatically," also known as [dependency injection](https://en.wikipedia.org/wiki/Dependency_injection). Within the agent, these instances are components, so Fx connects components to one another. Fx creates a single instance of each component, on demand.
+
+This means that each component declares a few things about itself to Fx, including the other components it depends on. An "app" then declares the components it contains to Fx, and instructs Fx to start up the whole assembly.
+
+## Providing and Requiring
+
+Fx connects components using types. Within the Agent, these are typically interfaces named `Component`. For example, `scrubber.Component` might be an interface defining functionality for scrubbing passwords from data structures:
+
+=== ":octicons-file-code-16: scrubber/component.go"
+    ```go
+    type Component interface {
+        ScrubString(string) string
+    }
+    ```
+
+Fx needs to know how to *provide* an instance of this type when needed, and there are a few ways:
+
+* [`fx.Provide(NewScrubber)`](https://pkg.go.dev/go.uber.org/fx#Provide) where `NewScrubber` is a constructor that returns a `scrubber.Component`. This indicates that if and when a `scrubber.Component` is required, Fx should call `NewScrubber`. It will call `NewScrubber` only once, using the same value everywhere it is required.
+* [`fx.Supply(scrubber)`](https://pkg.go.dev/go.uber.org/fx#Supply) where `scrubber` implements the `scrubber.Component` interface. When another component requires a `scrubber.Component`, this is the instance it will get.
+
+The first form is much more common, as most components have constructors that do interesting things at runtime. A constructor can return multiple arguments, in which case the constructor is called if _any_ of those argument types are required. Constructors can also return `error` as the final return type. Fx will treat an error as fatal to app startup.
+
+Fx also needs to know when an instance is *required*, and this is where the magic happens. In specific circumstances, it uses reflection to examine the argument list of functions, and creates instances of each argument's type. Those circumstances are:
+
+* Constructors used with `fx.Provide`. Imagine `NewScrubber` depends on the config module to configure secret matchers:
+      ```go
+      func NewScrubber(config config.Component) Component {
+          return &scrubber{
+              matchers: makeMatchersFromConfig(config),
+          }
+      }
+      ```
+* Functions passed to [`fx.Invoke`](https://pkg.go.dev/go.uber.org/fx#Invoke):
+    ```go
+    fx.Invoke(func(sc scrubber.Component) {
+        fmt.Printf("scrubbed: %s", sc.ScrubString(somevalue))
+    })
+    ```
+    Like constructors, Invoked functions can take multiple arguments, and can optionally return an error. Invoked functions are called automatically when an app is created.
+* Pointers passed to [`fx.Populate`](https://pkg.go.dev/go.uber.org/fx#Populate).
+   ```go
+   var sc scrubber.Component
+   // ...
+   fx.Populate(&sc)
+   ```
+   Populate is useful in tests to fill an existing variable with a provided value. It's equivalent to `fx.Invoke(func(tmp scrubber.Component) { *sc = tmp })`.
+
+    Functions can take multple arguments of different types, requiring all of them.
+
+## Apps and Options
+
+You may have noticed that all of the `fx` methods defined so far return an `fx.Option`. They don't actually do anything on their own. Instead, Fx uses the [functional options pattern](https://commandcenter.blogspot.com/2014/01/self-referential-functions-and-design.html) from Rob Pike. The idea is that a function takes a variable number of options, each of which has a different effect on the result.
+
+In Fx's case, the function taking the options is [`fx.New`](https://pkg.go.dev/go.uber.org/fx#New), which creates a new [`fx.App`](https://pkg.go.dev/go.uber.org/fx#New). It's within the context of an app that requirements are met, constructors are called, and so on.
+
+Tying the example above together, a very simple app might look like this:
+
+```go
+someValue = "my password is hunter2"
+app := fx.New(
+    fx.Provide(scrubber.NewScrubber),
+    fx.Invoke(func(sc scrubber.Component) {
+        fmt.Printf("scrubbed: %s", sc.ScrubString(somevalue))
+    }))
+app.Run()
+// Output: scrubbed: my password is *******
+```
+
+For anything more complex, it's not practical to call `fx.Provide` for every component in a single source file. Fx has two abstraction mechanisms that allow combining lots of options into one app:
+
+* [`fx.Options`](https://pkg.go.dev/go.uber.org/fx#Options) simply bundles several Option values into a single Option that can be placed in a variable. As the example in the Fx documentation shows, this is useful to gather the options related to a single Go package, which might include un-exported items, into a single value typically named `Module`.
+* [`fx.Module`](https://pkg.go.dev/go.uber.org/fx#Module) is very similar, with two additional features. First, it requires a module name which is used in some Fx logging and can help with debugging. Second, it creates a scope for the effects of [`fx.Decorate`](https://pkg.go.dev/go.uber.org/fx#Decorate) and [`fx.Replace`](https://pkg.go.dev/go.uber.org/fx#Replace). The second feature is not used in the Agent.
+
+So a slightly more complex version of the example might be:
+
+=== ":octicons-file-code-16: scrubber/component.go"
+    ```go
+    func Module() fxutil.Module {
+        return fx.Module("scrubber",
+        fx.Provide(newScrubber))    // now newScrubber need not be exported
+    }
+    ```
+
+=== ":octicons-file-code-16: main.go"
+    ```go
+    someValue = "my password is hunter2"
+    app := fx.New(
+        scrubber.Module(),
+        fx.Invoke(func(sc scrubber.Component) {
+            fmt.Printf("scrubbed: %s", sc.ScrubString(somevalue))
+        }))
+    app.Run()
+    // Output: scrubbed: my password is *******
+    ```
+
+## Lifecycle
+
+Fx provides an [`fx.Lifecycle`](https://pkg.go.dev/go.uber.org/fx#Lifecycle) component that allows hooking into application start-up and shut-down. Use it in your component's constructor like this:
+
+```go
+func newScrubber(lc fx.Lifecycle) Component {
+    sc := &scrubber{..}
+    lc.Append(fx.Hook{OnStart: sc.start, OnStop: sc.stop})
+    return sc
+}
+
+func (sc *scrubber) start(ctx context.Context) error { .. }
+func (sc *scrubber) stop(ctx context.Context) error { .. }
+```
+
+This separates the application's lifecycle into a few distinct phases:
+
+* Initialization - calling constructors to satisfy requirements, and calling invoked functions that require them.
+* Startup - calling components' OnStart hooks (in the same order the components were initialized)
+* Runtime - steady state
+* Shutdown - calling components' OnStop hooks (reverse of the startup order)
+
+## Ins and Outs
+
+Fx provides some convenience types to help build constructors that require or provide lots of types: [`fx.In`](https://pkg.go.dev/go.uber.org/fx#In) and [`fx.Out`](https://pkg.go.dev/go.uber.org/fx#Out). Both types are embedded in structs, which can then be used as argument and return types for constructors, respectively. By convention, these are named `dependencies` and `provides` in Agent code:
+
+```go
+type dependencies struct {
+    fx.In
+
+    Config config.Component
+    Log log.Component
+    Status status.Component
+)
+
+type provides struct {
+    fx.Out
+
+    Component
+    // ... (we'll see why this is useful below)
+}
+
+func newScrubber(deps dependencies) (provides, error) { // can return an fx.Out struct and other types, such as error
+    // ..
+    return provides {
+        Component: scrubber,
+        // ..
+    }, nil
+}
+```
+
+In and Out provide a nice way to summarize and document requirements and provided types, and also allow annotations via Go struct tags. Note that annotations are also possible with [`fx.Annotate`](https://pkg.go.dev/go.uber.org/fx#Annotate), but it is much less readable and its use is discouraged.
+
+### Value Groups
+
+[Value groups](https://pkg.go.dev/go.uber.org/fx#hdr-Value_Groups) make it easier to produce and consume many values of the same type. A component can add any type into groups which can be consumed by other components.
+
+For example:
+
+Here, two components add a `server.Endpoint` type to the `server` group (note the `group` label in the `fx.Out` struct).
+
+=== ":octicons-file-code-16: todolist/todolist.go"
+    ```go
+    type provides struct {
+        fx.Out
+        Component
+        Endpoint server.Endpoint `group:"server"`
+    }
+    ```
+
+=== ":octicons-file-code-16: users/users.go"
+    ```go
+    type provides struct {
+        fx.Out
+        Component
+        Endpoint server.Endpoint `group:"server"`
+    }
+    ```
+
+Here, a component requests all the types added to the `server` group. This takes the form of a slice received at
+instantiation (note once again the `group` label but in `fx.In` struct).
+
+=== ":octicons-file-code-16: server/server.go"
+    ```go
+    type dependencies struct {
+        fx.In
+        Endpoints []Endpoint `group:"server"`
+    }
+    ```
+
+# Day-to-Day Usage
+
+Day-to-day, the Agent's use of Fx is fairly formulaic. Following the [component guidelines](../../guidelines/components/purpose.md), or just copying from other components, should be enough to make things work without a deep understanding of Fx's functionality.

docs/public/architecture/components/overview.md

@@ -0,0 +1,50 @@
+# Overview of Components
+
+The Agent is structured as a collection of components working together. Depending on how the binary is built, and how it is invoked, different components may be instantiated. The behavior of the components depends on the Agent configuration.
+
+Components are structured in a dependency graph. For example, the `comp/logs/agent` component depends on the `comp/core/config` component to access Agent configuration. At startup, a few top-level components are requested, and [Fx](fx.md) automatically instantiates all of the required components.
+
+## What is a Component?
+
+Any well-defined portion of the codebase, with a clearly documented API surface, _can_ be a component. As an aid to thinking about this question, consider four "levels" where it might apply:
+
+1. Meta: large-scale parts of the Agent that use many other components. Example: DogStatsD or Logs-Agent.
+2. Service: something that can be used at several locations (for example by different applications). Example: Forwarder.
+3. Internal: something that is used to implement a service or meta component, but doesn't make sense outside the component. Examples: DogStatsD's TimeSampler, or a workloadmeta Listener.
+4. Implementation: a type that is used to implement internal components. Example: Forwarder's DiskUsageLimit.
+
+In general, meta and service-level functionality should always be implemented as components. Implementation-level functionality should not. Internal functionality is left to the descretion of the implementing team: it's fine for a meta or service component to be implemented as one large, complex component, if that makes the most sense for the team.
+
+## Bundles
+
+There is a large and growing number of components, and listing those components out repeatedly could grow tiresome and cause bugs. Component bundles provide a way to manipulate multiple components, usually at the meta or service level, as a single unit. For example, while Logs-Agent is internally composed of many components, those can be addressed as a unit with `comp/logs.Bundle`.
+
+Bundles also provide a way to provide parameters to components at instantiation. Parameters can control the behavior of components within a bundle at a coarse scale, such as whether the logs-agent should start or not.
+
+## Apps and Binaries
+
+The build infrastructure builds several agent binaries from the agent source. Some are purpose-specific, such as the serverless agent or dogstatsd, while others such as the core agent support many kinds of functionality. Each build is made from a subset of the same universe of components. For example, the components comprising the DogStatsD build are precisely the same components implementing the DogStatsD functionality in the core agent.
+
+Most binaries support subcommands, such as `agent run` or `agent status`. Each of these also uses a subset of the components available in the binary, and perhaps some different bundle parameters. For example, `agent status` does not need the logs-agent bundle (`comp/logs.Bundle`), and does not need to start core-bundle services like component health monitoring.
+
+These subcommands are implemented as Fx _apps_. An app specifies, the set of components that can be instantiated, their parameters, and the top-level components that should be requested.
+
+There are utility functions available in `pkg/util/fxutil` to eliminate some common boilerplate in creating and running apps.
+
+### Build-Time and Runtime Dependencies
+
+Let's consider sets and subsets of components. Each of the following sets is a subset of the previous set:
+
+1. All implemented components (everything in [this document][agent-components-listing])
+1. All components in a binary (everything directly or indirectly referenced by a binary's `main()`) -- the _build-time dependencies_
+1. All components available in an app (everything provided by a bundle in the app's `fx.New` call)
+1. All components instantiated in an app (all explicitly required components and their transitive dependencies) -- the _runtime dependencies_
+1. All components started in an app (all instantiated components, except those disabled by their parameters)
+
+The build-time dependencies determine the binary size. For example, omitting container-related components from a binary dramatically reduces binary size by not requiring kubernetes and docker API libraries.
+
+The runtime dependencies determine, in part, the process memory consumption. This is a small effect because many components will use only a few Kb if they are not actually doing any work. For example, if the trace-agent's trace-writer component is instantiated, but writes no traces, the peformance impact is trivial.
+
+The started components determine CPU usage and consumption of other resources. A component polling a data source unnecessarily is wasteful of CPU resources. But perhaps more critically for correct behavior, a component started when it is not needed may open network ports or engage other resources unnecessarily. For example, `agent status` should not open a listening port for DogStatsD traffic.
+
+It's important to note that the size of the third set in the list above, "all components available", has no performance effect. As long as the components would be included in the binary anyway, it does no harm to make them available in the app.