Skip to content

Commit c40c406

Browse files
maycmleemaycmlee
authored
[DOCS-7981] Add Datadog Agent source to log volume control (#23226)
* add dd agent source * remove images
1 parent 421ddb5 commit c40c406

File tree

6 files changed

+174
-12
lines changed

6 files changed

+174
-12
lines changed

‎config/_default/menus/main.en.yaml

+8-3
Original file line numberDiff line numberDiff line change
@@ -3965,21 +3965,26 @@ menu:
39653965
parent: observability_pipelines
39663966
identifier: observability_pipelines_log_volume_control
39673967
weight: 2
3968+
- name: Datadog Agent
3969+
url: observability_pipelines/log_volume_control/datadog_agent/
3970+
parent: observability_pipelines_log_volume_control
3971+
identifier: observability_pipelines_log_volume_control_datadog_agent
3972+
weight: 2001
39683973
- name: Splunk HTTP Event Collector
39693974
url: observability_pipelines/log_volume_control/splunk_hec/
39703975
parent: observability_pipelines_log_volume_control
39713976
identifier: observability_pipelines_log_volume_control_splunk_hec
3972-
weight: 2001
3977+
weight: 2002
39733978
- name: Splunk Forwarders (TCP)
39743979
url: observability_pipelines/log_volume_control/splunk_tcp/
39753980
parent: observability_pipelines_log_volume_control
39763981
identifier: observability_pipelines_log_volume_control_splunk_tcp
3977-
weight: 2002
3982+
weight: 2003
39783983
- name: Sumo Logic Hosted Collector
39793984
url: observability_pipelines/log_volume_control/sumo_logic_hosted_collector/
39803985
parent: observability_pipelines_log_volume_control
39813986
identifier: observability_pipelines_log_volume_control_sumo_logic_hosted_collector
3982-
weight: 2003
3987+
weight: 2004
39833988
- name: Dual Ship Logs
39843989
url: observability_pipelines/dual_ship_logs/
39853990
parent: observability_pipelines

‎content/en/observability_pipelines/log_volume_control/_index.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,14 @@ As your infrastructure and applications grow, so does your log volume and the co
1414
- **Dedupe**: Drop duplicate copies of your logs, for example, due to retries because of network issues.
1515
- **Remap**: Add, drop, or rename a field in your logs.
1616

17-
{{< img src="observability_pipelines/use_cases/log_volume_control.png" alt="The log sources, processors, and destinations available for the split logs use case" width="100%" >}}
18-
1917
Select a log source to get started:
2018

19+
- [Datadog Agent][4]
2120
- [Splunk HTTP Event Collector (HEC)][1]
2221
- [Splunk Heavy and Universal Forwarders (TCP)][2]
2322
- [Sumo Logic Hosted Collector][3]
2423

2524
[1]: /observability_pipelines/log_volume_control/splunk_hec
2625
[2]: /observability_pipelines/log_volume_control/splunk_tcp
27-
[3]: /observability_pipelines/log_volume_control/sumo_logic_hosted_collector
26+
[3]: /observability_pipelines/log_volume_control/sumo_logic_hosted_collector
27+
[4]: /observability_pipelines/log_volume_control/datadog_agent

‎content/en/observability_pipelines/log_volume_control/datadog_agent.md

+163
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
---
2+
title: Log Volume Control for the Datadog Agent
3+
kind: document
4+
disable_toc: false
5+
---
6+
7+
## Overview
8+
9+
Set up the Observability Pipelines Worker with the Datadog Agent source so that you route only useful logs to your destinations.
10+
11+
This document walks you through the following steps:
12+
1. The [prerequisites](#prerequisites) needed to set up Observability Pipelines
13+
1. [Setting up Observability Pipelines](#set-up-observability-pipelines)
14+
1. [Connecting the Datadog Agent to the Observability Pipelines Worker](#connect-the-datadog-agent-to-the-observability-pipelines-worker)
15+
16+
## Prerequisites
17+
18+
{{% observability_pipelines/prerequisites/datadog_agent %}}
19+
20+
{{< tabs >}}
21+
{{% tab "Splunk HEC" %}}
22+
23+
{{% observability_pipelines/prerequisites/splunk_hec %}}
24+
25+
{{% /tab %}}
26+
{{% tab "Sumo Logic" %}}
27+
28+
{{% observability_pipelines/prerequisites/sumo_logic %}}
29+
30+
{{% /tab %}}
31+
{{< /tabs >}}
32+
33+
## Set up Observability Pipelines
34+
35+
1. Navigate to [Observability Pipelines][1].
36+
1. Select the **Log Volume Control** template to create a new pipeline.
37+
1. Select **Datadog Agent** as the source.
38+
39+
### Set up the source
40+
41+
{{% observability_pipelines/source_settings/datadog_agent %}}
42+
43+
### Set up the destinations
44+
45+
Enter the following information based on your selected logs destination.
46+
47+
{{< tabs >}}
48+
{{% tab "Datadog" %}}
49+
50+
{{% observability_pipelines/destination_settings/datadog %}}
51+
52+
{{% /tab %}}
53+
{{% tab "Splunk HEC" %}}
54+
55+
{{% observability_pipelines/destination_settings/splunk_hec %}}
56+
57+
{{% /tab %}}
58+
{{% tab "Sumo Logic" %}}
59+
60+
{{% observability_pipelines/destination_settings/sumo_logic %}}
61+
62+
{{% /tab %}}
63+
{{< /tabs >}}
64+
65+
### Set up processors
66+
67+
{{% observability_pipelines/processors/intro %}}
68+
69+
{{% observability_pipelines/processors/filter_syntax %}}
70+
71+
{{% observability_pipelines/processors/add_processors %}}
72+
73+
{{< tabs >}}
74+
{{% tab "Filter" %}}
75+
76+
{{% observability_pipelines/processors/filter %}}
77+
78+
{{% /tab %}}
79+
{{% tab "Sample" %}}
80+
81+
{{% observability_pipelines/processors/sample %}}
82+
83+
{{% /tab %}}
84+
{{% tab "Quota" %}}
85+
86+
{{% observability_pipelines/processors/quota %}}
87+
88+
{{% /tab %}}
89+
{{% tab "Dedupe" %}}
90+
91+
{{% observability_pipelines/processors/dedupe %}}
92+
93+
{{% /tab %}}
94+
{{% tab "Edit fields" %}}
95+
96+
{{% observability_pipelines/processors/remap %}}
97+
98+
{{% /tab %}}
99+
{{< /tabs >}}
100+
101+
### Install the Observability Pipelines Worker
102+
1. Select your platform in the **Choose your installation platform** dropdown menu.
103+
1. Enter the Datadog Agent address. This is the address and port where your Datadog Agent is sending its logging data. The Observability Pipelines Worker listens to this address for incoming logs.
104+
1. Provide the environment variables for each of your selected destinations.
105+
{{< tabs >}}
106+
{{% tab "Datadog" %}}
107+
108+
{{% observability_pipelines/destination_env_vars/datadog %}}
109+
110+
{{% /tab %}}
111+
{{% tab "Splunk HEC" %}}
112+
113+
{{% observability_pipelines/destination_env_vars/splunk_hec %}}
114+
115+
{{% /tab %}}
116+
{{% tab "Sumo Logic" %}}
117+
118+
{{% observability_pipelines/destination_env_vars/sumo_logic %}}
119+
120+
{{% /tab %}}
121+
{{< /tabs >}}
122+
1. Follow the instructions for your environment to install the Worker.
123+
{{< tabs >}}
124+
{{% tab "Docker" %}}
125+
126+
{{% observability_pipelines/install_worker/docker %}}
127+
128+
{{% /tab %}}
129+
{{% tab "Amazon EKS" %}}
130+
131+
{{% observability_pipelines/install_worker/amazon_eks %}}
132+
133+
{{% /tab %}}
134+
{{% tab "Azure AKS" %}}
135+
136+
{{% observability_pipelines/install_worker/azure_aks %}}
137+
138+
{{% /tab %}}
139+
{{% tab "Google GKE" %}}
140+
141+
{{% observability_pipelines/install_worker/google_gke %}}
142+
143+
{{% /tab %}}
144+
{{% tab "Linux (APT)" %}}
145+
146+
{{% observability_pipelines/install_worker/linux_apt %}}
147+
148+
{{% /tab %}}
149+
{{% tab "Linux (RPM)" %}}
150+
151+
{{% observability_pipelines/install_worker/linux_rpm %}}
152+
153+
{{% /tab %}}
154+
{{% tab "CloudFormation" %}}
155+
156+
{{% observability_pipelines/install_worker/cloudformation %}}
157+
158+
{{% /tab %}}
159+
{{< /tabs >}}
160+
161+
{{% observability_pipelines/log_source_configuration/datadog_agent %}}
162+
163+
[1]: https://app.datadoghq.com/observability-pipelines

‎content/en/observability_pipelines/log_volume_control/splunk_hec.md

-2
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,6 @@ This document walks you through the following steps:
1313
1. [Setting up Observability Pipelines](#set-up-observability-pipelines)
1414
1. [Sending logs to the Worker over Splunk HEC](#send-logs-to-the-observability-pipelines-worker-over-splunk-hec)
1515

16-
{{< img src="observability_pipelines/use_cases/log_volume_control.png" alt="The log sources, processors, and destinations available for the split logs use case" width="100%" >}}
17-
1816
## Prerequisites
1917

2018
{{% observability_pipelines/prerequisites/splunk_hec %}}

‎content/en/observability_pipelines/log_volume_control/splunk_tcp.md

-2
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,6 @@ This document walks you through the following steps to set up the Observability
1212
1. [Setting up Observability Pipelines](#set-up-observability-pipelines)
1313
1. [Connecting Splunk Forwarder to the Observability Pipelines Worker](#connect-splunk-forwarder-to-the-observability-pipelines-worker)
1414

15-
{{< img src="observability_pipelines/use_cases/log_volume_control.png" alt="The log sources, processors, and destinations available for the split logs use case" width="100%" >}}
16-
1715
## Prerequisites
1816

1917
{{% observability_pipelines/prerequisites/splunk_tcp %}}

‎content/en/observability_pipelines/log_volume_control/sumo_logic_hosted_collector.md

-2
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,6 @@ This document walks you through the following steps to set up the Observability
1212
1. [Setting up Observability Pipelines](#set-up-observability-pipelines)
1313
1. [Sending logs to the Observability Pipelines Worker over Sumo Logic HTTP Source](#send-logs-to-the-observability-pipelines-worker-over-sumo-logic-http-source)
1414

15-
{{< img src="observability_pipelines/use_cases/log_volume_control.png" alt="The log sources, processors, and destinations available for the split logs use case" width="100%" >}}
16-
1715
## Prerequisites
1816

1917
{{% observability_pipelines/prerequisites/sumo_logic %}}

0 commit comments

Comments
 (0)